1. Introduction
John is an American Chief Executive Officer (CEO) of a fintech company that employs machine learning (ML) to conduct credit assessments and expand access to loans for people in low- and middle-income countries (LMICs). He knows that women receive fewer loans and loans at lower amounts compared to men in the markets where his company operates. He is dedicated toward his company’s social impact mission of enhancing financial inclusion, while also identifying as a businessman in a tough industry. He remains frustrated with what he describes as “social justice warriors,” whom he perceives as insisting that lenders distribute loans equally and take responsibility for remedying economic injustice. When asking about fairness in our interview, particularly after discussing gender differences in lending outcomes, I seemed to have struck a nerve. Though he finds it interesting to consider fairness, having studied philosophy in his graduate education, it stirred irritation, and he reflected that he’s found himself turning away from the topic. His experiences and emotions expose the complexity of ethical decisions that underlie choices in artificial intelligence (AI) model design and management. John’s company now reaches millions of people every year. As his company’s algorithmic lending tool, along with similar smartphone-based applications, scales across the Global South, critical questions arise about how the use of ML in credit assessment may both enhance financial inclusion and reinforce or mitigate gender biases in access to finance.
This paper – drawing on a theoretical framework based in feminist Science and Technology Studies (STS) – examines the different ways fintechs define fairness, with gender serving as a consistent thread through which fairness approaches are examined and their consequences traced. The contribution of this paper is threefold. First, it sheds light on how fintechs’ definitions and considerations of fairness are value-laden and shape decisions in the development and management of algorithmic lending tools, while leaving gender inequities in access to finance unaddressed. Second, it builds on and pushes beyond theoretical debates about the role and limitations of AI fairness in algorithmic accountability by using empirical evidence to critically demonstrate and examine how fairness is operationalized in the real-world context of fintechs that are leveraging “AI for Good.” Third, it reveals the normative decisions made by developers and managers of the technologies that inevitably intersect with tensions, contradictions and power dynamics embedded in these efforts. Alternative approaches to fairness in ML offer a path forward.
This paper begins with an overview of why fairness is a critical topic to examine, how different disciplines understand it and how fairness is operationalized in ML technologies, in order to ground the paper and its findings. After explaining my methods, I present how fintech leaders perceive fairness in the context of algorithmic lending. This examination considers fairness from both a process perspective (e.g., what constitutes a fair approach in designing ML tools) and an outcome perspective (e.g., what is deemed a fair result from the application of ML in credit assessments). I conclude with a discussion on my findings and propose feminist alternatives to fairness in ML.
2. Background
This background section situates the study within existing literature on ML and fairness, with a focus on credit scoring and lending. It outlines the rise of ML as a tool for expanding financial inclusion and its promises, before exploring fairness in ML systems and critiques of popular approaches.
2.1 ML unlocks new opportunities for financial inclusion
Financial market imperfections and lack of access to finance are seen as critical reasons for persistent income inequality and reduced economic growth (Beck et al. Reference Beck, Demirguc-Kunt and Honohan2009). Tackling financial market imperfections and enhancing financial inclusion can accelerate economic growth, as well as reduce income inequality and poverty, while access to finance for firms can promote entrepreneurship and innovation, resulting in firm growth and broader economic gains (Beck et al. Reference Beck, Demirguc-Kunt and Honohan2009). There are 1.4 billion adults considered “unbanked” and outside of traditional financial folds (World Bank 2021).
Despite large recent gains in financial access, persistent divides remain along axes of gender, socioeconomic status, education and more. Women’s account ownership is 6 percentage points lower than men’s in LMICs (World Bank 2021). Linked to persistent gender discrimination and limiting gender norms, women – and those with lower socioeconomic status – are more likely to lack identification, lack a mobile phone, live further from formal financial services and need more support to use financial accounts (World Bank 2021). Women also face issues accessing formal finance due to a lack of collateral, a lack of credit history and gender discrimination from loan officers (Demirguc-Kunt et al. Reference Demirguc-Kunt, Klapper and Singer2013).
Mobile phones are increasingly key in advancing opportunities for digital payments, savings and borrowing – a trend catalyzed by COVID-19. Worldwide financial account ownership has reached 76 percent of people (71% in LMICs), with mobile phones playing a key role in fueling growth in sub-Saharan Africa and for women, particularly (World Bank 2021). Mobile phones and alternative data found on them – when combined with ML – open new opportunities to facilitate loans to those otherwise unable to access them. ML-based alternative lending tools collect mobile phone data and utilize ML to provide credit scores and subsequent access to loans to those left out of traditional banking folds in LMICs. These smartphone apps ask for permission to view data on a user’s smartphone and collect real-time data (Björkegren et al. Reference Björkegren, Blumenstock, Folajimi-Senjobi, Mauro and Nair2022). The application can have access to various data stored on the device (Óskarsdóttir et al. Reference Óskarsdóttir, Bravo, Sarraute, Baesens and Vanthienen2018). The ML model assesses such data to make predictions about one’s creditworthiness and facilitate access to loans, which tend to be small with high interest rates and short repayment windows.
Use of and investment in fintech for alternative lending is skyrocketing globally alongside excitement over its economic and social impact potential. The alternative financing market was valued at $US10.82 billion in 2022 (Grand View Research 2023). The global AI in credit scoring market is set to experience significant growth with a projected compound annual growth rate of 25.9 percent from 2024 to 2031 (InsightAce Analytic 2024). India and Kenya are two of the most common markets for these apps. In Kenya, 77 percent of borrowers have taken only digital loans, and in 2018, over 90 percent of loans taken were digital (MicroSave 2019). Meanwhile, multilateral organizations, NGOs and government agencies are eager to support and fund these types of “AI for good” systems. Despite their popularity, research on impacts is lacking.
2.2 Why is it important to understand fairness and what does it mean?
AI and ML tools are increasingly governing more aspects of our lives – in finance as well as healthcare, education and more – raising urgent ethical questions about how these systems should be designed and governed (Taddeo and Floridi Reference Taddeo and Floridi2018). These systems make predictions (e.g., if one is creditworthy) while optimizing for specific objectives (e.g., maximizing a certain definition of accuracy and/or profit). Fairness is not inherent in ML but must be deliberately defined and encoded into the system. When fairness is not explicitly prioritized, ML systems are left to optimize freely for the goal they are given, identifying and reinforcing patterns in data without questioning whether those patterns reflect structural inequalities. This can lead to outcomes that systematically disadvantage certain groups, such as women, while still appearing neutral or technically sound.
Across domains, there are pervasive fairness concerns and recognition that it is important to ensure models are fair (Barocas and Selbst Reference Barocas and Selbst2016; Pessach and Shmueli Reference Pessach and Shmueli2022). But what does fairness mean? Fairness is commonly defined as the “quality or state of being fair, especially fair or impartial treatment” (Merriam-Webster 2025). It is a ubiquitous, yet contested, term in AI and ML research and practice. After all, the “state of being fair” or impartial remains unclear and nuanced.
In ML applications like lending, fairness takes on especially high stakes. Developers must make normative choices: should a lending model aim to maximize profit, increase the number of approved loans or ensure equitable access across groups? These choices are not merely technical, and they directly invoke questions of fairness with ensuing impacts on how the algorithm learns and operates. The choice of fairness definitions and approaches has immense social impact consequences in lending (Liu et al. Reference Liu, Dean, Rolf, Simchowitz and Hardt2018). Misclassification costs individuals dearly by denying loans that could have been repaid, resulting in missed opportunities for the individual to enhance their social or economic position; or receiving a loan one cannot repay may worsen one’s financial situation (Kozodoi et al. Reference Kozodoi, Jacob and Lessmann2022).
Fairness is not a universally agreed-upon concept and has different meanings in various domains. Legally, fairness relates to antidiscrimination and protecting individuals or groups based on protected characteristics. In the social sciences, it is understood through the lens of power dynamics, institutional arrangements and structural inequality. In quantitative fields (e.g., computer science), fairness is framed as a mathematical optimization problem, using formal criteria such as equal error rates, representation or allocation to assess outputs. Philosophical perspectives often tie fairness to broader principles of justice and equity (Mulligan et al. Reference Mulligan, Kroll, Kohli and Wong2019).
Fairness can also mean different things in different contexts to different people. In his book, Voices in the Code, David Robinson explores a Kidney Allocation System, which uses an algorithm to prioritize kidney transplant recipients. The system incorporated perspectives from a variety of stakeholders – including patients, surgeons, data scientists, public officials and more – who debated and defined what should be considered fair. Moral choices and ensuing decisions around fairness were both necessary and consequential for the design and operation of the algorithm (Robinson Reference Robinson2022).
2.3 How is fairness in ML operationalized and measured?
In ML practice, fairness is often approached as a quantitative optimization problem, focusing on constructing an “optimal ML model subject to fairness constraints” (Mulligan et al. Reference Mulligan, Kroll, Kohli and Wong2019). These constraints introduce formal rules into the model training process, requiring the algorithm not only to perform well overall but also to satisfy specific fairness criteria. In domains like credit scoring, models predict whether an applicant is creditworthy and the likelihood of default (e.g., through a score). These scores are typically converted into binary decisions (e.g., whether to approve or deny a loan). Model performance is, therefore, commonly evaluated using a confusion matrix, which breaks down predictions into true positives, false positives, true negatives and false negatives (Mirpourian et al. Reference Mirpourian, Jonathan and Kelly2022). This structure allows developers to calculate fairness metrics (e.g., false-negative rates) across demographic groups. These metrics form the basis for applying fairness constraints. For example, a model may be trained to minimize disparities in false-negative rates or ensure equal true-positive rates between groups (Mirpourian et al. Reference Mirpourian, Jonathan and Kelly2022).
Before fairness constraints can be implemented, developers must decide what fairness means in a given context and how it should be measured. Conceptual frameworks provide high-level guidance on what fairness means in practice, often by specifying how to balance outcomes across groups. In finance and credit scoring, three common conceptual frameworks are often discussed (Kozodoi et al. Reference Kozodoi, Jacob and Lessmann2022): independence ensures equal loan acceptance rates across groups but may increase false positives and defaults in disadvantaged groups (Hardt et al. Reference Hardt, Price and Srebro2016); separation equalizes error rates (i.e., false negatives) but can still lead to unequal approval rates that reinforce systemic inequalities (Kozodoi et al. Reference Kozodoi, Jacob and Lessmann2022); and sufficiency equalizes true-positive rates across groups but may allow for disparities in other errors (Liu et al. Reference Liu, Dean, Rolf, Simchowitz and Hardt2018). These frameworks can then be assessed using statistical metrics like demographic parity, equalized odds or treatment equality. Some organizations also consider outcome-level measures, such as equal loss ratios (Kelly and Mirpourian Reference Kelly and Mirpourian2021). These conceptual frameworks also influence how fairness is pursued throughout the ML pipeline.
Fairness can be pursued at different stages of the ML pipeline: pre-processing (e.g., balancing datasets or reweighting to address group imbalances), in-processing (e.g., modifying ML algorithms during training by adding fairness constraints) and post-processing (e.g., enforcing fairness criteria by applying calibration, reweighting schemes or decision threshold adjustments) (Caton and Haas Reference Caton and Haas2024; Jui and Rivas Reference Jui and Rivas2024; Mehrabi et al. Reference Mehrabi, Morstatter, Saxena, Lerman and Galstyan2019; Pessach and Shmueli Reference Pessach and Shmueli2022). Taken together, fairness in ML-based credit scoring can involve aligning conceptual frameworks with specific statistical metrics and implementing approaches at different stages of the ML pipeline.
2.4 Critiques of common fairness approaches in ML
Quantitative approaches to operationalizing fairness are limited. First, while conceptual frameworks and statistical measures can serve as useful diagnostic tools, they are narrow in what they capture and can obscure deeper questions about justice, harm and context. For example, focusing on independence (focusing on equal loan approval rates across groups) can miss disproportionate harms in denials, such as higher false-rejection rates for marginalized groups. Second, statistical criteria approaches can conflict with each other, making it impossible from a technical perspective to satisfy all at once (Corbett-Davies et al. Reference Corbett-Davies, Gaebler, Nilforoshan, Shroff and Goel2023). When base rates differ across demographic groups – as they commonly do – it is mathematically impossible for a model to simultaneously satisfy all fairness metrics, such as calibration and equal error rates (Kleinberg et al. Reference Kleinberg, Mullainathan and Raghavan2016). Furthermore, adding different fairness constraints can mean tradeoffs with the overall performance accuracy of the model (Caton and Haas Reference Caton and Haas2024; Haas Reference Haas2019). Developers, therefore, are forced to choose which fairness goals to prioritize, and within that, what kind of accuracy will be pursued by the algorithm.
While fairness frameworks and metrics offer valuable starting points, they frequently obscure critical normative questions: Whose outcomes matter most? What kinds of errors are considered tolerable? And who decides what fairness means in practice? These metrics embed assumptions about acceptable disparities and trade-offs. They can also fail to engage with the structural conditions that shape disparities in the first place, thereby not capturing dimensions of fairness that disciplines such as law, ethics and the social sciences emphasize. Fairness framed through technical accuracy only offers a partial explanation for algorithmic harms, while taking on normative weights in domains like credit scoring (Burrell Reference Burrell2024).
A focus on quantitative approaches can also result in unintended consequences. Fairness approaches in ML practice tend to justify solutions that employ surveillance and classification, while masking “broken social systems” that create a “patina of legitimacy” while perpetuating inequity (Pasquale Reference Pasquale2019). In addition, technical fairness approaches distract from institutional structures that produce inequity, making it harder to “challenge upstream decision-making about whether automated decision-making can or should be implemented” (Burrell Reference Burrell2024). A prominent example of these tensions is the COMPAS algorithm, used in US courts to assess a defendant’s risk of reoffending. ProPublica found that Black defendants were nearly twice as likely to be incorrectly labeled as high risk for reoffending, compared to similar white defendants (highlighting racial disparities in error rates, invoking the conceptual framework of separation) (Angwin et al. Reference Angwin, Larson, Mattu and Kirchner2016). COMPAS developers defended the tool based on sufficiency, citing equal overall predictive accuracy across groups (Angwin and Larson Reference Angwin and Larson2016). This example highlights how different fairness frameworks and statistical definitions can lead to contradictory conclusions about fairness – and how developers can use technical reasoning to sidestep deeper questions of justice and equity.
This study fills an important gap. While a growing body of scholarship examines technical definitions and critiques of fairness, with some studies exploring fairness considerations implemented and their issues in private (Holstein et al. Reference Holstein, Vaughan, Daumé, Dudik and Wallach2019) and public domains (Veale et al. Reference Veale, Van Kleek and Binns2018), there is limited research examining how fairness is understood by those building or deploying these systems, particularly in financial domains. Understanding how fairness is interpreted, including the beliefs that shape those interpretations, is critical to move beyond principlist approaches that often rely on abstract, universal notions like fairness that carry unexamined presumptions and superficial solutions to moral problems (Clouser and Gert Reference Clouser and Gert1990). This study employs elite interviewing and “studying up” (Nader Reference Nader1972), a theoretical orientation that allows for a reorientation “upward,” increasingly recognized as critical in studies of algorithmic fairness and power (Barabas et al. Reference Barabas, Doyle, Rubinovitz and Dinakar2020). This qualitative design allows an examination of underlying logics, priorities and approaches that illuminate the “ground truth” around root causes of how data and AI models are assembled (Marda and Narayan Reference Marda and Narayan2021). Fairness is a particularly compelling point of inquiry, as it exposes moral assumptions and value systems that become embedded in technological systems.
3. Methods
The findings in this paper are drawn from 25 semi-structured interviews conducted with corporate leaders, data scientists and investors at fintech companies developing and managing ML-based alternative lending apps in LMICs. I focus on fintechs that leverage apps and use alternative data to inform credit assessment in LMICs, given their roots with Silicon Valley, their mission-focused approaches on financial inclusion, their increasing prominence globally (including in “AI for Good” spaces) and the lack of research on them. I utilized purposive sampling to identify and recruit the study sample, which followed a landscape review of ML-based alternative lending companies. To recruit, I sent emails or messages through LinkedIn with the interview request and attended events, where target interviewees were speaking or attending. I also utilized snowball sampling. Over 160 participants were invited for interviews, and 25 interviews were conducted between August 2023 and February 2024. The majority of interviewees were in leadership and management positions outside of the data science team (48%), followed by people occupying leadership roles in the data science arm of the organization (20%), followed by investors (16%), and then other data scientist team members (8%) and partner researchers (8%). The majority of participants were based in the United States (60%), followed by India and other Asian countries (28%), Europe (8%) and Kenya (4%). Each interview lasted approximately 1 hour, allowing for in-depth discussions with participants. This paper builds on a subsection of the interviews on how fintechs define and operationalize fairness in the context of ML-based credit assessment and loan disbursement. Interviews were transcribed using a speech-to-text transcription service. One interviewee did not consent to their interview being recorded or to quotes being used.
In analyzing data, I utilized an inductive approach using a grounded theory method. This allowed for themes to emerge and nuance to be captured in how fairness is understood and operationalized. I drew on reflexive thematic analysis, subsequently reviewing and refining themes to identify patterns and core elements in the data (Braun and Clarke Reference Braun and Clarke2006). NVivo software facilitated the coding process. Drawing on reflexive thematic analysis, after familiarization with the data round, I developed an initial generation of codes related to AI fairness. I developed and reviewed themes from the codes before subsequently refining my codebook and writing.
Drawing on a theoretical framework that integrates insights from feminist STS, this research highlights how technology development is not neutral but rather shaped by individual choice and power dynamics (Kuhn Reference Kuhn1970; MacKenzie and Wajcman Reference MacKenzie and Wajcman1999; West Reference West2019). Specifically, I use Donna Haraway’s situated knowledges and Ruha Benjamin’s concept of do-gooderness to analyze how perceptions of fairness influence the ways ML-based credit assessment in LMICs is produced and their implications for inclusion and inequality. Haraway’s situated knowledges posit that all knowledge is produced from specific, socially and historically situated standpoints (Haraway Reference Haraway1988). This lens allows me to examine epistemological positions on fairness and ensuing design choices. I incorporate Social Shaping of Technology (SST) to analyze how organizational and industry-wide structural pressures (e.g., profitability imperatives) influence fintechs and their employees in constructing fairness perspectives and approaches (MacKenzie and Wajcman Reference MacKenzie and Wajcman1999). This is complemented by Benjamin’s concept of do-gooderness, which critiques the framing of technologies as solutions to social problems without addressing their deeper, often hidden, implications. This lens allows me to further critique and analyze complexities and pitfalls in how fintechs define fairness (Benjamin Reference Benjamin2019).
4. Findings
Fairness can be defined in different ways in the context of ML-based credit assessment. None of the fintechs interviewed desire to be unfair or unfairly discriminatory, and all (except one) insisted they are fair. However, what fairness means is not always clear, and there is no single definition or approach to achieving fairness across gender or other demographic groups. Lucy, an American woman who conducts research and works as a funder of fintechs in this space, shared: “When we talk to digital lenders …. They say yes, absolutely it’s fair. And then they lack an understanding or an ability to articulate what their definition of fairness is that they’re measuring against.” Similarly, Sean, the Chief Technology Officer of a company headquartered in California with operations in Africa and Asia, spoke candidly about their own attempts at grappling with the complicated question of fairness:
The truth is, I don’t have an answer …. Would I love to be able to say this thing is totally fair and this is why and this is how and it’s all good? Obviously I don’t have. Why? Well, first we need to find out what fairness is. So you have a metric, right? Or you could do something more qualitative and ask people and whatnot, that’s a very expensive process. So I would rather have a quantitative one where I have a formula. Maybe that’s just my background. But I would like to say, no, it’s fair, because this number here is 7.3 and that’s fair. But that’s not so easy to get.
Sean highlighted how there is a subjectivity to considering whether an output is fair in algorithmic-facilitated lending. Similarly, Aditi, based in Washington D.C., who works at an organization funding fintechs operating in LMICs, noted that “fairness has different implications in different cultures.” This recognition that fairness can mean different things in different contexts adds to the uncertainty for fintechs.
While fintech actors express uncertainty about what fairness means and there is a lack of clear alignment in how they define or measure it, these ambiguities are revealing of how fairness becomes a site of negotiation and differentiation in algorithmic-lending practices. Two main ways of thinking about fairness become visible: one focused on the outcomes that models produce, and another on the processes through which outcomes are reached. The following sections explore these orientations and their implications, before turning to the role of investors and market incentives in shaping how fairness is articulated and operationalized.
4.1 Process: fairness is achieved by …
Most interviewees are not considering fairness in the design and implementation of their ML models, or note that they are achieving fairness through unawareness. From a gender perspective, this means that if gender is not considered by the model, it is fair from a gender perspective. This reasoning reflects a logic of leaving it to the machine, whereby approaches to being fair mean not having humans get in the way of the (objective) technology. In doing so, actors position the machine as a neutral arbiter.
In some cases, interviewees noted that they did not see fairness as a priority, nor did they consider how to operationalize it. Gary, based in the United States (US) and working for a US-headquartered company operating in Asian markets, responded when asked about fairness: “By and large, we are relying on our credit risk engine and what our machine learning actually presents to us.” On the surface, this appears as if they lack a fairness approach; however, it is in line with achieving fairness by being unaware (i.e., fairness through unawareness), which is aligned with the belief in the objectivity of ML and AI. Dante, a white man working for a US-headquartered fintech with operations globally, put it bluntly:
The short answer is that we don’t [consider fairness]. Because we don’t bring bias into the model. We don’t know the gender, we don’t know the ethnicity. I smile at your question because it’s kind of a US kind of question …. Believe me, nobody cares, outside of the US or perhaps the UK, if the model is fair or not.
This response highlights how perceptions of fairness can often be linked to social justice – and that algorithmic-facilitated lending is not the appropriate place to advance social justice. Rather, algorithmic-facilitated lending is seen as apolitical and objective.
Others more explicitly attached fairness to approaches of being “blind” to gender. While not focused on social justice, these perspectives support the belief that algorithmic-facilitated lending is objective and can cut through unfair human-caused bias and discrimination. The overarching sense is that ML makes lending more fair and responsible by avoiding bias that humans have through using machines that detect patterns and make decisions based on a set of rules. Therefore, the act of using ML is what makes the tool fair. From this perspective, fairness is achieved not necessarily through a careful design of outcomes or constraints but through the very act of delegating decisions to the machine.
Beyond being gender “blind,” several interviewees noted an approach in line with fairness through unawareness, in which the company randomly issues loans in new markets. This method allows for the machine to identify variables empirically associated with repayment, though some discretion remains in selecting features for the final model. Here, too, confidence in the machine’s objectivity is relied upon to produce the best outcomes.
Feminist scholars have critiqued this “view from nowhere” (the idea that knowledge or technology can be detached from social position or perspective) as an illusion, and one that hides how technical systems are built from particular social and cultural standpoints (Haraway Reference Haraway1988). Claims of fairness through unawareness similarly depend on erasing the situated nature of various choices embedded in model design. This belief in neutrality can be seen as a perspective in itself, one that reflects situated knowledges by revealing the specific assumptions about objectivity and neutrality that shape how fairness is understood and enacted (Haraway Reference Haraway1988). Furthermore, the idea that machines, when left alone, will be fair ignores the ways that data are rife with biases. Data reflect limiting stereotypes and norms about people, while datasets can under-represent certain groups, leading to lower performance (Criado-Perez Reference Criado-Perez2021; D’Ignazio and Klein Reference D’Ignazio and Klein2023). It also ignores how features and proxies used in models carry gendered norms and reflect structural inequality (D’Ignazio and Klein Reference D’Ignazio and Klein2023). From the perspective of fairness being achieved through the act of delegating decisions to the algorithm itself, fairness becomes framed as a property of a method and as an automatic outcome of using ML.
Some interviewees noted that the tool is fair because it is accessible to people, especially marginalized groups. As Rakesh, the US-based Founder and CEO of a fintech operating in Asian countries, put it, “The way we look at fairness is that this is a system that is open to anyone …. This is a free to use app by anyone who wants to access bank loans. There are no further limitations on this beyond just having their mobile phone [and] being able to download an app.” Here, the tool is fair because it is available for anyone to hypothetically be able to use, while owning a mobile phone and being able to download apps is seen as basic and easy. This reflects a fairness claim rooted in how the system is designed and accessed but does not necessarily include who actually benefits from the tool or whether outcomes are equitable. This was echoed by Suraj, a US-based senior leader of a fintech with global operations, who said: “I don’t want to call it fair, but not because it’s unfair, but I just think that doesn’t apply. Like it’s an ML based model, so you can scale to everyone.” Here, fairness as a concept doesn’t apply as the tool is accessible to people and can reach scale. This kind of claim reflects a moral assurance that making technology broadly available is inherently good, while overlooking how structural inequalities shape who can meaningfully benefit from the tool.
4.2 Outcome: fairness is achieved when …?
Interviewees also discussed “fair” outcomes, whereby fairness is judged based on what the model achieves versus how it gets there. This outcome-based view of fairness puts an emphasis on the effects of the model in the real-world, such as increasing financial access to people who are underserved. In this framing, fairness is validated by the result.
4.2.1 Fairness is achieved when securing social impact
Various interviewees see fairness as being achieved simply by doing good. In this view, tools are fair because they are advancing financial inclusion for people who would otherwise be financially excluded. Fairness is seen as inherent in the social mission of enabling access to financial systems and in the belief that ML tools are better than the status quo. Suraj asserted: “If this product is focused only on one segment, like every loan you sell is a fair loan. You know what I’m saying? …. Every loan you make is very impactful because you’re giving loans to people who are pulling hand carts or electricians, day laborers. Right?” Various interviewees echoed this sentiment, noting that financial inclusion is the goal of these tools, and they are fair because they are enhancing financial inclusion and reaching underbanked people. Jack, a US-based employee at an organization that funds fintechs in this space and conducts research, noted, “If financial inclusion is the goal, it’s fair because it’s reaching people who are underbanked.” Similarly, some interviews discussed fairness as being better than the status quo. This is a common sentiment of being fair because it is better than the status quo and avoids some of the historical biases in finance that have inhibited women from accessing credit.
Other interviewees take this further and see fairness not just as doing good but doing as much good or securing as much benefit as possible (i.e., benefit maximizing). Inclusion is prized: More people accessing loans is seen as the most socially impactful approach and therefore the fairest. This approach prioritizes inclusion over considerations of equity. It does not consider the number of women versus men who benefit, rather people overall. Some fintechs track these numbers to understand their reach but don’t have particular goals related to loan distribution across gender and other demographics.
Fairness as benefit maximizing or doing good echoes a consequentialist logic, whereby the fairness of a system is judged by its outcomes (e.g., how many people receive loans) rather than how decisions are made. This logic often appears alongside procedural notions of fairness (e.g., fairness through unawareness). The blend reinforces the belief that if a model applies the same rules to everyone and helps people, it must be fair. Beyond reflecting a situated knowledge, this logic can help conceal the structural inequalities it sustains, legitimizing uneven outcomes through appeals to objectivity and the moral language of social good. This illustrates an example of Benjamin’s do-gooderness, where good intentions get conflated with good outcomes without distributional critique.
No interviewees discussed fairness in terms of parity – whereby both men and women have equal probability of getting a good credit score and access to a loan. In fact, this is generally looked at negatively by interviewees who do not see it as the role of fintechs to correct for societal inequalities. Megan, a California-based leader at a US-headquartered fintech with operations across several LMICs, explained that tracking approvals by gender is
a very black and white way to look at it …. You can’t say that [if an] approval rate of women versus men is higher in India that our models are negatively biased against men …. You can’t really just look at all approval rates and [say] they have to be the same.
From a lender and business perspective, this approach is logical, as loans are tailored based on various factors, many of which are shaped by structural inequalities.
4.2.2 Fairness is achieved when the tool is accurate
Several interviewees had a more when fairness is achieved. They discussed that fairness is achieved by the ML tool when it accurately predicts creditworthiness across different protected groups. John, the California-based CEO with training in ML, noted that he thinks of bias from the technical definition in regard to ML. In this case, if you’re consistently wrong in the same direction, that would indicate bias, such as being consistently wrong about women. But if they are correct about people (even if they are more often giving loans to men than women), it would not be unfairly biased. He continued that women in the countries where they work have less money, on average, than men. So one would want to give them a lower credit score, and this lower credit score would be accurate in regard to whether one should get a loan – even though it reflects inequality. Relatedly, giving out loans to people who could not repay them can have unintended consequences for that individual, so it is important to ensure the person can repay the loan. John noted that this is the reason why lending is not a good way to correct for social inequality.
From a lender’s perspective, some degree of bias is seen as inevitable, and the primary goal becomes delivering accurate credit decisions – offering the “right” amount of credit to individuals likely to repay. While this might seem like a straightforward objective, complications arise: Accuracy depends not only on what the model is designed to predict but also on the lending terms that shape repayment outcomes in the first place. This raises deeper questions: what should be measured to determine if a tool is accurate, and what outcome should the model be optimized for – minimizing defaults, maximizing profit or expanding access? These questions reveal that even the pursuit of accuracy involves value-laden decisions about goals, trade-offs and priorities. A fintech might define accuracy in terms of correctly predicting default, on-time payment or customer profitability. Each of these reflects different situated knowledges and institutional logics. In this sense, accuracy operates as both a technical and moral category, produced through specific standpoints and organizational pressures.
These value-laden decisions were visible in Megan’s description of how her team compares credit margins and optimizes across risk groups:
I don’t try to optimize that much against female and male …. The main metric I look at is credit margin. What is my credit margin across customer risk segments? …. If you’re receiving more in revenue given a group’s level of risk, that is unfair. Either you have to find a way to adjust the interest rate down of that protected class or increase the interest rate of your other groups to get to that uniform loss ratio.
The type of approach that Megan presented considers the revenue that customers provide and aligns financial margins across different risk groups, regardless of demographics. Here, late payment may not be considered a negative outcome, as long as the loan is ultimately repaid and the financial return is acceptable. This represents a “post-processing” approach (i.e., adjusting model outputs to ensure a fairness criterion) and reflects a quantitative, revenue-centered notion of fairness rooted in the lender’s business objectives. An equalized (uniform) loss ratio extends accuracy beyond simple repayment prediction to account for the timing and profitability of that repayment and is used in lending and insurance (Schreiber Reference Schreiber2019), as well as by other fintechs (Kelly and Mirpourian Reference Kelly and Mirpourian2021). Indeed, Megan shared that this approach was taken after careful consideration and reading industry reports on the topic. Other interviewees also discussed tracking and balancing loss ratios. However, equalized loss ratio departs from fairness frameworks and metrics in the AI ethics literature, which focus on conceptual frameworks like separation and metrics such as demographic parity or equalized odds. Here, fairness is framed from a risk management and financial lens: no group should systematically generate more profit relative to its risk. There is still a choice in terms of what is grouped and measured (e.g., demographic groups, risk groups) and how outcomes are interpreted.
Importantly, this approach doesn’t aim to eliminate differences in approval outcomes; rather, it accepts outcome differences as long as resulting margins or loss ratios are uniform. While appealing in industry contexts, this logic raises normative concerns. It may systematically favor groups that generate more money, even if they are less reliable. Also, equal loss ratios can obscure differences in access: a model may equalize loss ratios while still denying disproportionately more loans to historically marginalized groups or offering them worse terms.
Notably, Megan was the only interviewee who explicitly described using a fairness metric and statistical approach (i.e., equalized loss ratios across risk groups). Others either did not mention fairness measurement or give any indication that they conducted audits or evaluations. The lack of standardized testing and transparency across most interviewees reflects a limited maturity around how fairness is considered and operationalized in the ML-based lending space in LMICs. Despite widespread perceptions that their models are fair, actual implementation and measurement are largely ad hoc or absent.
This gap between rhetoric and practice sits in contrast to the academic literature on ML, which offers a proliferation of conceptual frameworks, fairness metrics and strategies for intervention across different stages of model development. Many fintechs either aren’t yet engaging with these fairness frameworks and metrics or are doing so in fragmented, inconsistent ways. The result is a significant gap between perceived and demonstrated fairness, whereby fintech actors operate under a belief that their models are fair, without systematically testing across demographic groups. Fairness claims are not disingenuous; rather, they reflect a common belief that fairness has already been achieved or is inherently embedded through using “objective” ML and/or pursuit of social impact. This belief exemplifies Benjamin’s notion of do-gooderness: the conflation of good intentions with good outcomes, which forecloses critical reflection on how “for good” technologies can reproduce structural inequities.
Leadership plays a crucial role in shaping whether and how these ethical questions are prioritized. Returning to Megan, her organization was the only one to establish an Ethics Committee, which reflects recognition at the leadership level of the ethical tensions inherent in developing such technologies. By creating organizational structures to examine ethics and questions of fairness, they legitimized it as a meaningful and actionable concern rather than an abstract ideal.
4.3 The role of investors and market incentives in shaping fairness
Beyond organizational leaders, investors play a critical role in shaping priorities and informing the approaches of fintechs. Several interviewees who are employees of fintechs discussed pressure from investors as a reason they had to update their approach or focus on easier-to-reach customer segments. Hannah, who is a US-based researcher at an organization that supports and invests in fintechs, shared:
[Fintechs] would say, well you know we have to keep in mind the incentives we have from our investors. We need to hit certain growth targets. We have certain KPIs that are part of our covenants or agreements with investors …. We would need more resources to build out more checks for fairness or even to reach more excluded customers to build models for them.
Many of the fintechs providing ML-based credit assessment tools are startups or small businesses operating in a tough industry. As a result of various macro-level factors, including high interest rates globally after the COVID-19 pandemic, venture capital fell drastically, leading to a “fintech winter.” In 2023, venture capital in fintechs decreased 42 percent (S&P Global 2024). In many cases, fintechs are trying to keep the doors open and, therefore, pursuing profitability. Gary, the US-based President of a fintech operating in Asian markets, explained:
We are a startup … and many of the players out there that you’re speaking to are startups …. Ultimately we need to manage for profitability …. It’s really managing your loss rate, especially with cost of funds going up …. I think everybody that you’re speaking to generally is loss making as a business.
Limited resources (time, money, people) can impede more robust responsibility approaches. At the same time, business pressures push fintechs toward efficiency and profitability, often shaping how their models are developed and evaluated. Several interviewees noted that the models are predicting creditworthiness while optimizing for lifetime value, a common metric used in lending, which estimates revenue over time minus acquisition and servicing costs.
Kumar, working at an Indian fintech company operating across the country, shared tensions between priorities of making money and achieving fairness for different groups, including women. He said,
[The fintech is] trying to make money so you can’t, you know, push fairness when you look at income bands …. I don’t think the models are that fair …. Maybe housewives would probably be in a less fair situation to get a higher ticket size …. You’re not taking gender [into account, but in an] employment setting, [they are] probably homemakers.
This honest assessment applied understandings of parity in considering fairness, while exposing how priority on profits impacts fairness in practice.
5. Discussion
There are different understandings and maturity levels of fairness, with many fintechs lacking a formal or structured approach to incorporating it into their models. Fairness is commonly invoked from the perspective of process (i.e., what is a fair approach when using ML for credit assessment) and/or as an outcome (i.e., what is a fair outcome from using ML for credit assessment). These framings set the stage for the discussion that follows, which traces five dynamics that build on one another. The discussion begins with fairness as largely undeveloped or implicitly assumed, before considering how fairness was at times equated with accuracy. It then examines how institutional incentives structure these understandings and practices and highlights the normative tensions these dynamics reveal. It concludes by outlining alternative paths forward for fairness in ML.
5.1 Fairness by declaration, not by deliberate design or evaluation
The rhetorical framings of fairness identified, whether through unawareness or through doing good, share a common feature: they position fairness as a passive state versus something that needs to be actively pursued. Fairness becomes both a declaration and a moral stance, invoked to signal good intent rather than to guide design or evaluation. This rhetorical dynamic mirrors Sara Ahmed’s concept of nonperformativity, whereby the declaration of fairness without executing corresponding action illustrates how institutional language can make fintech tools appear equitable, while actively maintaining the status quo of inequity (Ahmed Reference Ahmed2005). This mechanism effectively facilitates Benjamin’s (Reference Benjamin2019) concept of do-gooderness, in which technologies are framed as solutions to social problems while ignoring deeper, hidden implications, often through the use of narrow definitions and operationalization of fairness. Benjamin argues that algorithmic racism arises from this moral cover, which overlooks power hierarchies under simple, mathematical framings of fairness. In algorithmic lending in LMICs, declarations of fairness grounded in claims of doing good masks unintended consequences. It becomes a blindfold, even to the developers and managers of the fintechs themselves. The result is an alluring – yet selective – fiction about the potential of “AI for Good” to lift marginalized communities, while leaving gendered and structural inequities unexamined.
These moral logics are sustained by epistemic ones. Fairness is treated as inherently embedded and achieved in the use of “objective” ML and the pursuit of social impact. Drawing on Haraway’s concept of situated knowledges, these assumptions can be understood as epistemological positions that reflect specific standpoints, yet present fairness as a neutral standard. By framing fairness as something already accomplished, they render evaluation, formal metrics or auditing unnecessary. As a result, fairness functions more as an implicit belief, rather than something to explicitly consider in model design or for which to test. Fairness becomes subjective and unaccountable: something that doesn’t need to be measured, because it is a presumed outcome of ML’s objectivity and one’s good intentions. This faith in the neutrality of ML locks in the logic of do-gooderness: moral virtue and technical objectivity reinforce one another, creating a sense that deeper scrutiny is unnecessary.
Compounding these logics are other barriers that limit the uptake of fairness practices. Following the SST framework to examine industry-wide influences, the sheer number of fairness definitions, metrics and techniques (without clear consensus or regulatory guidance) may be part of the problem, contributing to confusion, inaction or ad hoc adoption. Even among those who recognize that considering fairness is important, they often lack resources or incentives to build robust approaches, operating under tight constraints reflecting broader structural pressures in the fintech sector. In addition, they operate in various markets, which tend to have immature AI regulatory landscapes. These conditions reinforce treating fairness as an assumed quality versus something to rigorously design for and pursue. The result was that systematic approaches to fairness remained the exception rather than the norm.
5.2 Fairness as accuracy and situated priorities
Framing fairness as accuracy aligns with dominant ML practices where fairness is often treated as a mathematical optimization problem, despite there being different ways to consider fairness in social science and philosophical disciplines (Mulligan et al. Reference Mulligan, Kroll, Kohli and Wong2019). Furthermore, creditworthiness assessments can become self-fulfilling prophecies, complicating the idea of accuracy. For instance, if someone is deemed a riskier borrower, their loan offer and repayment conditions are adjusted (e.g., higher interest rates, shorter repayment periods). If they default, the algorithm could be seen as accurate for determining them to be a riskier borrower. But with better loan terms, they might have successfully repaid, thereby “contributing to or creating the situation they claim merely to predict” (Citron and Pasquale Reference Citron and Pasquale2014).
Within economic and industry discourse, bias is not necessarily framed as harmful but as rational differentiation, folded into the definition of accuracy itself and inherently part of the lending process (Lauer Reference Lauer2017). Researchers have shown that biases against female loan applicants can be seen as rational responses from lenders, yet they also perpetuate gender segregation in lending and further entrench economic differences (Buvinic and Gokhroo Reference Buvinic and Gokhroo2023). Interviewees reflected this perspective of gender differences as rational; after all, being fair means predicting creditworthiness accurately, not rectifying societal imbalances. This illustrates how fairness framed through accuracy can overlook social science definitions, which emphasize fairness through a lens of justice and addressing systemic inequalities. Equating fairness with accuracy in ML may appear objective, but it only offers a partial explanation for algorithmic harms and can obscure broader social impacts (Burrell Reference Burrell2024).
The varied ways that fintechs define and pursue accuracy reveal that it, too, is a subjective and value-laden concept shaped by specific standpoints, organizational pressures and industry-wide perspectives. This intersection forms what I term situated priorities. Situated priorities is an original theoretical concept that synthesizes and extends SST and situated knowledges, bridging a gap in existing theories. Frameworks like SST focus on broader social and structural pressures, while situated knowledges emphasize epistemological positioning; yet neither fully examines how individual actors negotiate and prioritize specific goals within those structures, which then become embedded in the technology. Situated priorities include several key propositions: (1) priorities are situated, constructed through a combination of epistemological positioning and institutional logics; (2) priorities are dynamic and negotiated within decision-making and operational processes; and (3) priorities shape different technological outcomes by defining what matters most and embedding those priorities in technology-related decisions. Situated priorities offer a theoretical lens for understanding how actors’ decisions are shaped by what they consider important or valuable – priorities that are formed through their institutional contexts and situated perspectives. It calls for examination of how principles and values are defined, navigated and translated into particular goals and implemented in practice over time. This reveals how the same principles can be actively shaped through organizational dynamics and practical constraints, leading to diverse technological outcomes. Situated priorities extend beyond fintech lending. This is evident in the development of foundation models, where the priorities of a handful of largely Western, for-profit actors become embedded in tools and infrastructure that organizations worldwide draw on, shaping technological outcomes (Puutio and Lin Reference Puutio, Lin and Hacker2025).
5.3 Fairness is shaped by institutional incentives and priorities
While fairness is often discussed as a technical or ethical question, in practice it is deeply shaped by institutional incentives and market logics. Technology is not only constituted through human agency but through the strong influences of organizations and institutions, resulting in outcomes of technology that are a result of this complex interflow (Fountain Reference Fountain2001). Several interviewees described models being optimized for business objectives like lifetime value, reflecting profit priorities. Fairness constraints were not mentioned during model development. While several interviewees discussed balancing loss ratios, one organization described its post-processing equalized loss ratio approach.
5.3.1 Fairness through equalized loss ratios and profit priorities
Quantification serves as a moral veil, obscuring the consequences even from those doing the quantifying without the right guardrails. In his examination of kidney allocation algorithms, Robinson discussed quantification as “a moral anesthetic,” in which numbers seem dispassionate and impartial. Numbers and algorithms help “find a way through the moral conflict” (Robinson Reference Robinson2022). In this case, I argue that quantification is not necessarily serving as a moral anesthetic, as much as a moral veil for certain decisions. This moral veil happens somewhat unknowingly, as developers haven’t fully grasped the implications and consequences themselves. This is partly a consequence of the misguided and widespread belief that algorithms are technical – as opposed to sociotechnical – systems.
Profit as a priority is not a new concept in credit scoring, nor in digital financial inclusion. Several studies highlight how digital financial inclusion interventions in LMICs – developed by foreign fintech firms and funded by shareholders largely from the Global North – commodify and monetize poor people under the guise of financial inclusion, without addressing underlying inequalities and economic issues that persist (Akolgo Reference Akolgo2023; Gabor and Brooks Reference Gabor and Brooks2017; Jain and Gabor Reference Jain and Gabor2020; Langley and Leyshon Reference Langley and Leyshon2022). When fairness is shaped by profit imperatives, its gendered consequences are important to consider, especially in contexts where structural inequalities already limit women’s access to finance.
5.3.1 Gender implications of fairness under a profit orientation
In contexts where women are better repayers but take smaller loans (a pattern noted by the vast majority of interviewees), approaches like lifetime value and equal loss ratio may fail to reward women’s reliability. Larger loan sizes generate higher revenue, which can offset the costs of late payments and defaults. Smaller loans, on the other hand, may incur higher transaction and administrative costs per dollar lent (Blanco-Oliver et al. Reference Blanco-Oliver, Irimia-Diéguez and Vázquez-Cueto2023). If fintechs optimize for lifetime value or apply equal loss ratios without appropriate safeguards, they risk replicating gender bias by penalizing women for generating less revenue. Men could be overextended if they are allocated too much credit at higher risk thresholds, increasing the likelihood of overborrowing or falling into debt traps.
This would not be the first documented case of ML tools in lending exhibiting a bias due to optimizing for profit: Research on a peer-to-peer lending platform in China found that the introduction of ML to inform interest rates resulted in higher interest rates for women. This was not due to higher predicted risk or lower creditworthiness, but because women had lower price sensitivity and the platform could, therefore, better optimize revenue by offering them loans at higher interest rates (Chu et al. Reference Chu, Sun, Zhang and Zhao2023).
At a higher level, if models optimized for profit are learning that men are more profitable, they may learn to downgrade women altogether, even under gender “blind” algorithms, by inferring that certain proxies for gender are less profitable. This pattern of a certain identity being unintentionally and systematically downgraded has occurred in other algorithms; for example, an Amazon hiring algorithm that systematically downgraded women’s resumes, even without explicit gender indicators (Dastin Reference Dastin2018).
Importantly, as actors begin to recognize that their own perspectives and situated priorities shape technological outcomes, a reflexive moment emerges. This is reflected in the development of the Ethics Committee at the fintech where Megan works. This is also reflected by Sean, who acknowledged his own uncertainty and positionality in questions of fairness. These reflexive moments open space to question the neutrality of AI systems and surface tough ethical questions.
5.4 Businesses, not “social justice warriors”
Fairness cannot be separated from the priorities of those who build, manage and fund these systems – in this case, within for-profit and social impact-oriented startups. The perception of fairness as doing as much good for as many people, or benefit maximizing, is tied to business priorities and goals of market expansion, while also linked to priorities for social impact scale found in the development industry. This focus on inclusion justifies gender differences in lending. Prioritizing inclusion over equity can result in algorithmic decisions that reinforce power hierarchies and mask structural injustice. Meanwhile, measuring and optimizing models against fairness perspectives that are grounded in profit priorities can justify discriminatory practices that uphold – or even worsen – existing gender economic inequalities.
These findings underscore how ML-based credit assessment algorithms are normative, reflecting situated priorities. In a study of credit scores in insurance pricing, Barbara Kiviat notes, “Algorithmic prediction is imbued with normative viewpoints – they are viewpoints that suit the goals of the corporation” (Kiviat Reference Kiviat2019). Corporate profit priorities are embedded in algorithms, even while drawing legitimacy from claims of objectivity. Such systems can treat profit-driven success as truth (Sadowski Reference Sadowski2025). Data-driven scoring systems for risk classification are not new in the financial sector and include scientific and technical methods to justify their interests and actions (Fourcade and Healy Reference Fourcade and Healy2024). This dynamic reflects the concept of do-gooderness, illustrating how ethical narratives are co-opted by institutional logics focused on scalability, efficiency and market expansion. In this framing, inclusive and “fair” technology becomes a mirage for reputational legitimacy, even as it potentially projects existing inequalities into the future. Without transparency of models and regulation, “fair” ML systems can entrench feedback loops that reinforce an unjust status quo.
Still, as John pointed out, giving loans to people who cannot or will not repay them (which may result in contexts where “social justice warriors” insist on giving loans equally to men and women or to those who are more marginalized) also has unintended consequences and results in immense harm. Fairness does not necessarily mean parity or the provision of equal loans to men and women. Research has exposed how giving more loans to vulnerable populations (often at higher interest rates) reflects a sort of “reverse redlining,” whereby people can be targeted for loans under worse conditions with consequences such as overindebtedness and debt traps (Garcia et al. Reference Garcia, Garcia and Rigobon2024). This illustrates the tricky moral and operational balance that fintechs must walk: enhancing inclusion and financial sustainability in a space where both inaction and overcorrection can produce unintended consequences.
Ultimately, fairness in ML is not just a technical challenge to be optimized; it is a sociotechnical problem tied to institutional priorities, value judgments and power. In for-profit domains such as lending, economic values and priorities bubble to the top. In the absence of regulatory guardrails and accountability mechanisms, fairness risks being defined in ways that serve organizational interests over social equity. Addressing fairness requires grappling with these realities – and being transparent about how fairness is defined, who it serves and why certain choices are made, while recognizing the limitations of for-profit actors to self-regulate.
Fintechs can take various actions, spurred by reflective moments when actors recognize the influence of their perspectives and situated priorities. First, they should clearly define their fairness approach and be transparent about tradeoffs. Second, transparency can be improved through tools like model cards, which detail fairness frameworks, metrics used and model-stage interventions. Third, internal and external audits are crucial for identifying disparities across demographic groups. Critical consideration must be taken to ensure that women are not disproportionately denied loans or offered different terms, incorporating an intersectional perspective. Collaborating with social scientists can help surface blind spots and guide responsible design. More broadly, fintechs must reflect on how institutional priorities shape gender equity efforts. Future research could explore how non-profit institutions approach ML-based lending differently. Regulation also plays a key role.
5.5 Alternative paths forward for fairness in ML
The empirical findings inform an alternative approach to fairness in ML. I present a two-dimensional feminist reframing of fairness that combines process and outcome. From a process perspective, this includes centering the voices and agency of those most impacted in defining what fairness entails for a particular system and context. From an outcome perspective, fairness is assessed through an equity lens that asks whether systems address or reproduce historical patterns of exclusion.
5.5.1 Fairness as process: centering community voice
As knowledge is situated and standpoints matter in the production of meaning (Haraway Reference Haraway1988; Harding Reference Harding1995), process is not peripheral but foundational to questions of fairness. Technologies are deeply value-laden tools; therefore, there is an opportunity to embed the values of broader audiences and vulnerable populations whose lives AI technologies impact. More specifically, my findings call for redistributing epistemic authority by co-constructing fairness approaches through lived experiences versus top-down metrics derived from priorities of external actors. This orientation resonates with feminist work on epistemic injustice, which calls attention to the silencing and devaluation of marginalized knowers (Fricker Reference Fricker2007). It also resonates with hooks’ insistence that transformative justice requires shifting knowledge production from the margin to the center (hooks Reference hooks1984). This can be implemented through meaningful community deliberation, co-creation and governance practices that grapple directly with meanings of fairness and considerations of power in technology. These approaches can require meaningful time and resources, which can push against business pressures to move quickly and may surface community perspectives of fairness that sit in tension with profit priorities. Nonprofits and civil society organizations can be well-positioned not only to support these processes but also to develop algorithmic models that center community voice. Prior community-informed fairness and governance approaches for algorithmic systems (including nonprofit-led models with broad stakeholder participation) show this is possible, while also illustrating the complexity (Robinson Reference Robinson2022). Such approaches may surface, honor and innovate for a plurality of fairness perspectives.
While community-informed approaches foreground local agency and situated knowledge, they can also generate tensions with outcome-oriented goals. For instance, a community might define fairness pragmatically as maximizing the number of people who receive loans, even if most recipients are men. From this perspective, the tool appears fair because it benefits the collective, yet it risks reproducing gendered exclusion. A feminist reframing holds these aims (process and outcome) in productive tension: centering the voices of those most impacted ensures fairness definitions are contextually grounded, while positioning equity as the evaluative horizon provides an anchor that pushes fairness efforts to be accountable to structural oppression and broader questions of justice.
5.5.2 Fairness as outcome: reframing toward equity
At a broader level, fairness in ML has limited utility and becomes somewhat meaningless given its ability to mean everything and nothing, capable of justifying almost any design choice. Reframing fairness through the lens of equity extends feminist accounts that link redistribution and recognition (Fraser Reference Fraser1997) and incorporates an emphasis on structural and institutional processes of oppression (Young Reference Young1990). This framing also centers intersectionality (Crenshaw et al. Reference Crenshaw, Gotanda, Peller and Thomas1996) as key to understanding how algorithmic systems reproduce overlapping forms of exclusion. An equity lens rejects the moral cover of “inclusion” and “social impact,” asking not just whether outcomes are balanced, but whether systems account for and address the conditions that produce disparity in the first place, avoiding Benjamin’s do-gooderness pitfall. This reframing, which connects to recent literature refocusing algorithmic fairness to justice (Fazelpour et al. Reference Fazelpour, Lipton and Danks2022; Kontiainen et al. Reference Kontiainen, Koulu and Sankari2022), also aligns with data feminist approaches that operationalize equity as a design principle, emphasizing power, purpose and the transformation of unjust social relations (D’Ignazio and Klein Reference D’Ignazio and Klein2023).
Focusing on equity as the desired outcome accepts that results need not be identical, but that groups have equal access to opportunity. In the context of algorithmic lending, this requires examining and removing proxies, features and practices that, while correlated with repayment, may perpetuate patterns of discrimination by encoding structural inequalities rather than genuine indicators of creditworthiness at the size and purpose of the loan. For instance, while formal employment may reasonably be causally related to repayment for a mortgage, it can be a poor proxy for a $20 microloan, where such correlations reflect gender norms and economic inequality more than true repayment capacity. Equity becomes the standard for evaluation, asking: Do AI systems counter structural exclusion and historical patterns of discrimination? This includes identifying the features most predictive of loan access and determining whether they encode historical and current discrimination (e.g., employment status). It also means ensuring that profit priorities do not disadvantage certain groups due to perceived lower profitability.
Equity in algorithmic lending does not mean disregarding financial viability. It recognizes repayment disparities may reflect structural barriers versus individual deficiencies. Therefore, an equity approach extends beyond the model itself to include broader commitments by fintechs, government and development actors that enhance equitable access and participation. For example, related to the tool, this could include interface options that provide financial literacy education, savings support tools in the app or offline, more flexible repayment schedules and helping more people access the tool, including through offering access on basic phones or supporting efforts to enhance digital inclusion. It could also include partnerships with nonprofits or cooperatives that build people’s financial resilience and address broader issues of economic insecurity for women and other marginalized populations. These strategies would not be one-size-fits-all but rather respond to the particular contexts and power asymmetries within and between different communities. Realizing them will require resources and dedicated commitment from actors to addressing the root causes of structural economic inequality. This reflects an ethic of care, whereby institutions support people’s participation and well-being within unequal social and economic structures (J. Tronto Reference Tronto2020; J. C. Tronto Reference Tronto2010). By centering an ethic of care, efforts confront and reshape historical and social conditions that produce inequality through technical and non-technical solutions and partnerships.
6. Conclusion
By unpacking how fintechs understand and measure fairness, this paper shows how deeper gender and other social inequalities are often sidestepped, and their reproduction legitimized, through ML tools. Fintechs tend to invoke fairness either as a matter of process (“fair” algorithmic design) or as an outcome (“fair” credit assessment), but in both cases, fairness is largely declared rather than interrogated. Process-oriented accounts often rely on “fairness through unawareness,” revealing a deep trust in the objectivity of ML technologies. Outcome-oriented accounts frequently equate fairness with doing good or benefit maximizing, whereby fairness is implicitly assumed under social impact intentions. Some fintechs equate achieving fairness with being accurate, yet the act of defining and measuring accuracy is shaped by situated priorities, revealing how epistemic positioning and institutional logics structure technological outcomes.
Ultimately, fairness in ML is neither neutral nor purely technical, but value-laden and shaped by institutional priorities of growth, scalability and profitability. Profit priorities are not inherently wrong in algorithmic lending, but models risk deepening existing disparities without actors confronting gendered power dynamics and structural inequities embedded in access to apps, data and algorithms. These dynamics unfold in contexts with limited regulation to guide companies and under opaque models, presenting immense risks, including shaping perceptions and realities of who is deemed creditworthy not only by ability and willingness to repay but also by projected profitability. Fintechs can take several actions, from being transparent about their fairness approach and its tradeoffs to conducting and sharing audits, collaborating with social scientists and critically reflecting on tensions with institutional priorities.
Further, these findings call for a feminist reframing of fairness in ML that reorients the field from narrow metrics toward epistemic justice and an ethic of care. By centering community voice in processes of fairness and setting equity as a normative horizon, this perspective broadens our imagination of what AI “for good” can be, inviting new possibilities for collective flourishing and more just futures.
Acknowledgements
The author would like to thank Prof. Xiaolan Fu, Prof. Ekaterina Hertog and Prof. Masooda Bano for their guidance and feedback on earlier versions of this work. This research builds on a doctoral dissertation completed at the University of Oxford.
Funding statement
Funding from the Horowitz Foundation for Social Policy supported this research.
Competing interests
The author declares none.
Dr. Genevieve Smith is a postdoctoral research fellow at Stanford University, founder of the Responsible AI Initiative at the UC Berkeley AI Research Lab, and on professional faculty at UC Berkeley. Dr. Smith recieved her doctorate from the University of Oxford. She is a research affiliate at the Minderoo Centre for Technology & Democracy at Cambridge University and the Technology & Management Centre for Development at University of Oxford. Prior, Dr. Sm served as the Co-Director of the UC Berkeley AI Policy Hub and the Responsible AI Fellow at the U.S. Agency for International Development. Her research has been published at various journals and published in proceedings of leading conferences including the International Conference on Machine Learning (ICML), the ACM Conference on Fairness, Accountability & Transparency (FAccT), and the Academy of Management. It has also been shared in Nature, Wall Street Journal, Forbes, the Economist, and more.
Open access
