Introduction
Public benefits fraud is common in the U.S. across services such as social security (Social Security Administration 2016) and unemployment insurance (Committee of Ways and Means, 2002). However, combatting public benefits fraud is difficult. First, those who commit public benefits fraud are often from precarious or vulnerable populations, making punitive policies controversial. Financial penalties can potentially escalate financial hardship among these populations, which can then increase the likelihood of further fraudulent activity (Baron, Reference Baron2007; Felson et al., Reference Felson, Osgood, Horney and Wiernik2012). Second, investigations and audits are both labor intensive and expensive, and it is unlikely that agencies can successfully recover the amounts lost to an impoverished population; costs of detection, investigation, and recovery efforts may even cost more than the fraudulent act itself (Gustafson, Reference Gustafson2011). Third, there is limited evidence that deterrence efforts are effective (Regev-Messalem, Reference Regev-Messalem2013).
Given the costs of welfare fraud are at an all-time high (estimated at $89 billion annually, see Office of Inspector General (2021)), as well as the difficulties in enforcing the ethical use of welfare services, one potential promising avenue from behavioral economics is the use of ‘nudges’ to try to guide individuals using these services towards more ethical behavior (Thaler and Sunstein, Reference Thaler and Sunstein2009; Grimmelikhuijsen et al., Reference Grimmelikhuijsen, Jilke, Olsen and Tummers2017; John, Reference John2018; Tummers, Reference Tummers2019). Governmental agencies across the globe are increasingly using ‘nudging’ interventions (as opposed to changing economic incentives) which are typically more cost effective for trying to achieve policy goals (Benartzi et al., Reference Benartzi, Beshears, Milkman, Sunstein, Thaler, Shankar, Tucker-Ray, Congdon and Galing2017). For instance, an intervention might leverage the use of defaults to automatically enroll individuals where it leads to socially desirable outcomes (Johnson and Goldstein, Reference Johnson and Goldstein2003). However, interventions are not always effective in changing average behavior (Hauser et al., Reference Hauser, Gino and Norton2018). To address the question for whom an intervention works, we turn to algorithm prediction as a tool to target the intervention at those in whom behavior change is desired and most needed (Athey, Reference Athey2017; Lee et al., Reference Lee, Inceoglu, Hauser and Greene2022).
In this paper, we propose and demonstrate that combining behavioral interventions with algorithmic prediction can help improve the usefulness of such interventions, especially in the public sector (De Vries et al., Reference De Vries, Bekkers and Tummers2016; Tummers, Reference Tummers2020). The goal of this study is to provide a direct comparison of algorithmically targeted to untargeted nudges. We illustrate this approach in the context of the public sector—a US state agency—to which claimants return on a weekly basis to claim unemployment benefits on their platform. To be eligible for unemployment benefits, claimants need to disclose (truthfully) whether and how much they earned income in the previous week. We find that the interventions causally increase disclosure, relative to a control group, but only among claimants who are predicted by an algorithm to behave unethically in that week. Correspondingly, these claimants who are both in the high-risk group report higher earnings when they are in the treatment group relative to being in the control group. In exploratory analyses, we investigate the variation of the effectiveness of different messages, finding consistent effects for messages that include information on social norms, impact on others, and audits and verification, while penalty messages yield some but less robust effects. Aside from encouraging more truthful disclosures, higher disclosed earnings in the treatment group have positive financial consequences for the state agency (both directly through disclosures and indirectly through saved administrative costs of recovering owed money).
Our work contributes to three streams of literature. First, an emerging body of research studies uses behavioral (or ‘nudging’) interventions to reduce unethical behavior. Nudges are interventions that ‘alter people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives’ (Thaler and Sunstein, Reference Thaler and Sunstein2009, p. 6). Ample extant literature documents that such interventions can change behavior in the public and private sector (Thaler and Sunstein, Reference Thaler and Sunstein2009; Benartzi et al., Reference Benartzi, Beshears, Milkman, Sunstein, Thaler, Shankar, Tucker-Ray, Congdon and Galing2017). In some cases, nudges have helped to achieve policy outcomes efficiently and cheaply, such as increasing energy savings (Allcott, Reference Allcott2011; Allcott and Rogers, Reference Allcott and Rogers2014; Jachimowicz et al., Reference Jachimowicz, Hauser, O’Brien, Sherman and Galinsky2018), savings (Karlan et al., Reference Karlan, McConnell, Mullainathan and Zinman2016; Kessler et al., Reference Kessler, Milkman and Zhang2019), vaccination uptake (Milkman et al., Reference Milkman, Beshears, Choi, Laibson and Madrian2011), diversity in hiring (Arslan et al., Reference Arslan, Chang, Chilazi, Bohnet and Hauser2024) and giving to charity (Barasz et al., Reference Barasz, John, Keenan and Norton2017). At the same time, nudge-based interventions are not always effective (Sunstein, Reference Sunstein2017; Hauser et al., Reference Hauser, Gino and Norton2018).Footnote 1 Studies have found null results for similar nudges in different contexts (e.g. Kettle et al., Reference Kettle, Hernandez, Sanders, Hauser and Ruda2017, Bird et al., Reference Bird, Castleman, Denning, Goodman, Lamberton and Rosinger2021) and sometimes nudges have even led to opposite effects of those predicted by researchers and policy experts (Beshears et al., Reference Beshears, Choi, Laibson, Madrian and Milkman2015; Robinson et al., Reference Robinson, Gallus, Lee and Rogers2019), prompting new theories of ways that nudging can be improved (e.g. Nudge +, see Banerjee and John, Reference Banerjee and John2021). Even for relatively well-studied interventions, such as penalties threats and deterrence messages (Ariel, Reference Ariel2012; De Neve et al., Reference De Neve, Imbert, Spinnewijn, Tsankova and Luts2021) or social norms (Larkin et al., Reference Larkin, Sanders, Andresen and Algate2019), results can be mixed. Consequently, relatively little is known about the conditions under which interventions are most likely to succeed. In the current study, we examine one potential avenue for increasing the effectiveness of interventions to achieve policy outcomes: the use of predictive algorithms to identify subgroups most likely to benefit from receiving an intervention.
Second, we draw on the fast-growing literature on algorithmic targeting, while also advancing this literature by adding causal investigations in a real-world field setting (Hofman et al., Reference Hofman2021; Lee et al., Reference Lee, Inceoglu, Hauser and Greene2022). Predictive algorithms are already commonly used to analyze user and consumer behavior (Webb et al., Reference Webb, Pazzani and Billsus2001) and, in the context of unethical behavior, these algorithms have mostly been employed for the detection of consumer credit card fraud and firm-level financial statement fraud (Ngai et al., Reference Ngai, Hu, Wong, Chen and Sun2011; West et al., Reference West, Bhattacharya, Islam, Tian, Jing and Srivatsa2015). They also have the potential to be used to increase social welfare through optimal policy design (Balaguer et al., Reference Balaguer, Koster, Summerfield and Tacchetti2022; Koster et al., Reference Koster, Balaguer, Tacchetti, Weinstein, Zhu, Hauser, Williams, Campbell-Gillingham, Thacker, Botvinick and Summerfield2022, Reference Koster, Pîslar, Tacchetti, Balaguer, Liu, Elie, Hauser, Tuyls, Botvinick and Summerfield2024). The idea that such prediction algorithms can be used to identify people who might benefit from an intervention has recently received increasing attention (Athey and Imbens, Reference Athey and Imbens2016; Xiao et al., Reference Xiao, Hauser, Kirkwood, Li, Jones and Higgins2024), in particular in marketing. Matz et al. (Reference Matz, Kosinski, Nave and Stillwell2017) use Facebook ‘Likes’ to infer ‘Big Five’ personality types, increasing response to targeted advertisements, whilst Matz et al. (Reference Matz, Gladstone and Farrokhnia2023) demonstrate a similar targeting approach for savings behaviors. Ghose et al. (Reference Ghose, Li and Liu2019) show that fine-grained location data from cell phones can be used to target advertisements at nearby shoppers, leading to greater purchasing, while Pe’er et al. (Reference Pe’er, Egelman, Harbach, Malkin, Mathur and Frik2019) demonstrate that customizing nudges to suit a person’s decision-making style can improve their effectiveness. Hauser et al. (Reference Hauser, Urban, Liberali and Braun2009) demonstrate that a website that automatically ‘morphs’ its design to suit different cognitive styles leads to more sales. In sum, research demonstrates algorithms are promising to selectively target individuals to increase sales.
Our final contribution relates to the literature on behavior change by focusing on the use of targeting as a means to achieve the most scope for behavior change. In our context, we demonstrate that our treatments only have an effect among claimants who are predicted to behave unethically. This should be unsurprising: those claimants had something to disclose in the first place. However, all too often, nudges in the field are applied to the whole population, but doing so might underestimate the actual treatment effect among the relevant population. This point was raised in early field experiments by Slemrod et al. (Reference Slemrod, Blumenthal and Christian2001) and Blumenthal et al. (Reference Blumenthal, Christian, Slemrod and Smith2001) who both observe heterogeneous treatment effects among populations who had more (versus less) of an opportunity to evade taxes. Relative to these early papers which focused on heterogeneous treatment effects by demographic and tax group characteristics post hoc, our study offers a more generalizable, a prior targeting approach with machine learning (Lee et al., Reference Lee, Inceoglu, Hauser and Greene2022): by targeting claimants algorithmically, we do not need to impose any structure or assumptions on which variables are most likely to be predictive of unethical behavior; we can take advantage of non-linearities and interactions of variables that might be predictive; and we introduce an approach by which claimants can be targeted in real time and, therefore, the interventions can be employed efficiently for only the relevant subgroup. This combination of algorithmic targeting and causal investigations under one umbrella falls into a relatively understudied area of computational social science (Hofman et al., Reference Hofman2021). Our paper shows a first proof-of-concept of combining algorithmic targeting with a causal study of the effectiveness of ‘nudging’ messages in the public sector in the field, which we hope will be a blueprint for future studies and for policy-makers.
Methods
Study context. The field context for our study is the U.S. public sector: in collaboration with an unemployment office in a U.S. state, we conducted a large-scale field experiment to attempt to reduce dishonesty in unemployment claims. This context involves a population of unemployed workers who claim state unemployment benefits, an understudied population. This population is likely without significant wealth, creating a predicament for them in filing accurate unemployment claims, which can reduce their income when filing honestly. Yet, an individual who is caught filing a fraudulent claim also incurs financial penalties and potential exclusion from the system as well as legal trouble, placing an already vulnerable population in even greater financial and employment difficulty. Furthermore, the state has an incentive to minimize fraudulent claims prospectively as it reduces government spending on monitoring and recouping fraudulent claims.
The experiment was conducted on the government’s online platform with all submissions over a three-week period in August–September of 2016, for a total of 9,833 unique claimants and 22,457 submissions. This type of unobtrusive procedural field experiment (Harrison and List, Reference Harrison and List2004; Hauser et al., Reference Hauser, Linos and Rogers2017) can help to reduce potential concerns regarding demand effects and self-selection. Claimants were randomized into control (where they received no message) and treatment, which is comprises one of several ‘nudging’ messages (Table 1), based on the extant literature (see Supplementary Information (SI) Section 1) and to which claimants were randomly assigned within the treatment condition.
Examples of intervention messages used in the field experiment. See Table S1 in the appendix for the full list of messages
Table 1 Long description
The table lists five categories of intervention messages used in a field experiment, with an example message for each category and representative prior literature sources. Social Norm messages state that nearly all people in a claimant’s county report earnings accurately and prompt accurate reporting for the reporting period. Impact Others messages warn that misreporting may affect other unemployed people in the state and ask the claimant to report earnings for the reporting period. Audits and Verification messages, shown as two variations, emphasize that employment and earnings are verified and that overpaid benefits can be recovered if earnings are reported inaccurately. Penalties messages, shown as five variations, warn that fraud requires repayment of all benefits plus substantial penalties. Reminder messages instruct claimants to report earnings for the reporting period even if they have not yet been paid. Citations indicate that each message type aligns with established approaches in prior research, but the table provides examples rather than the full set of messages or any outcome results.
Algorithmic targeting. We employed an off-the-shelf machine learning algorithm that calculates each week a Risk Assessment Result (RAR) value (as a continuous value from 1 to 100) for each claimant who submitted an unemployment claim to the state government. The RAR is a measure of likelihood of behaving fraudulently in any given week, based on the predictive algorithm. Claimants are assigned a value by the algorithm from 1 to 100, with higher RAR values indicating a greater likelihood of behaving fraudulently. The RAR takes into account various historic and real-time behavioral variables (e.g. type and date of submission, behaviors on the platform, last employment and industry, etc.). A new RAR value is calculated each week, and can thus vary within a claimant over time (whereas the randomized assignment to treatment or control group does not change over time).
The algorithm returns a RAR value that is calculated each week and varies from 1 to 100 for each claimant who submitted an unemployment claim to the state government. We later created ‘RAR bins’ for ease of presentation, but our results hold regardless of whether the RAR value is categorized into bins or on a continuous scale. The RAR value is the variable we use in the field experiment to conduct analyses of the causal treatment effect of the messages in each RAR bin, where high RAR values imply that the algorithm predicts that the claimant is more likely to behave unethically at that time.
Further details on the algorithmic training procedure can be found in SI Section 2. Note that, due to the sensitive nature of the data and the research question in collaboration with a public sector field partner, several details of the algorithm are confidential and proprietary. Specifically, the algorithm and data used in the training stage are confidential due to potential concerns by the field partner that revealing these details could lead to ‘gaming the system.’ However, we can share several procedural steps in how we arrived at the final algorithm (see SI Section 2, Figure S1 and Table S5). We also emphasize that the algorithm used here is neither the primary goal, nor of key interest, of this research; we relied on an off-the-shelf, carefully calibrated machine learning algorithm that uses sensitive data and variables that we are not allowed to disclose at this point. Future research may choose to apply a similar algorithmic targeting approach by choosing the appropriate statistical algorithm based on their research or policy question and the data at hand, and by calibrating it for the specific context.
Behavioral interventions. Table 1 summarizes the messages used in the experiment. These messages can be classified as ‘nudges’ because, while they may include information about the presence or severity of a sanction, the government’s ability to take action against a fraudulent claimant, or simply information about the behavior of others (i.e. social norms), none of these messages materially change the underlying incentive structure. In all conditions (control or treatment) in our field context, claimants are required by law to truthfully disclose information as requested by the state agency and failing to do so can result in fines, penalties, and other sanctions. The nudge therefore does not change the ‘economic incentives’ (Thaler and Sunstein, Reference Thaler and Sunstein2009, p. 6) and preserves the decision-makers’ ultimate choice but it may change the likelihood that claimants make a disclosure if they have earnings to disclose.
Benefits claim setup and process. When a claimant loses their job and is eligible for unemployment insurance, they set up an online account with the government agency responsible for distributing unemployment benefits. (A fraction of claimants (approx. 9% in our data) do not set up an online account, either because they do not have access to a computer or do not feel comfortable navigating the system online. They can instead have an in-person or phone conversation with a government official each week to certify their eligibility. Because they do not use the online platform, they are not part of our experiment.) Once the account is set up, they need to start an ‘initial claims’ process. The randomization took place for each claimant at the time when they started the initial claim. Once the initial claim is approved by the government agency, claimants need to return to the online portal on a weekly basis to apply for unemployment benefits, through a process known as ‘weekly certification.’ Claimants are eligible if they do not hold a regular job and have not earned more (through, for example, occasional jobs) than the amount of the unemployment benefits in the week prior to completing the certification. If a claimant has earned money in the previous week, the amount of unemployment benefits they are eligible for this week is reduced accordingly.Footnote 2
Treatment delivery. For claimants in the treatment group, the treatment is incorporated into the weekly eligibility certification process (see Figure S2 in the SI for a screenshot of an illustrative message in the treatment). Claimants in the treatment are exposed to a popup message on the same page during the eligibility certification process, after they clicked the submit button of the form where our dependent variable was collected. Specifically, in the treatment group, the popup was shown after participants had selected ‘no earnings’ and submitted the page. This meant that behavioral information collected on this page could also be fed back into the algorithm, which calculated the RAR value as the page was submitted based on both historic and real-time data; showing the popup as the page loads would have been another possibility but would not have incorporated the most recent behavioral patterns into the RAR calculation, which was an important consideration to the field partner.
Once randomized, claimants in the treatment group see the same intervention every week when they came back to complete the eligibility certification process (and they see no message if they are in the control group). This means that claimants can see the same message multiple times for a few weeks in a row. The number of times a claimant in the treatment group is exposed to the treatment is a function of the number of times they return to claim unemployment benefits. Conversely, if they were assigned to the control group, they do not see any message at any point, no matter how often they return to the online platform to claim unemployment benefits.
Dependent variable in the field experiment. The dependent variable, which we refer to as ‘disclosure,’ is whether claimants answered ‘yes’ versus ‘no’ in response to the question ‘Did you work during the reporting period listed above?’ during the weekly certification process. (The ‘reporting period’ refers to two dates, spanning the beginning to the end of the previous week, at the top of the page.) Claimants who answered ‘yes’ to this question are asked to indicate the amount earned. Of course, selecting ‘no income to report’ during the weekly certification process does not automatically equate to fraud. Only those who did work, but who deliberately do not disclose their earnings, are behaving unethically. If the claimant reported earnings, the amount is deducted from the weekly payment. Not disclosing earned income could be considered fraud, illegal, and punishable by state law, and the state has multiple mechanisms for determining actual earnings. For example, employers report wages to the state and new employment relationships are included in a database. (There is a lag between when the earnings are reported by the claimant and employer.) Overpayments must be paid back in full. If, in addition, the intention behind the misreporting is deemed fraudulent by the state it is considered fraud and additional penalties apply. However, in the absence of an audit, there is no way to know whether a particular individual behaved unethically when they reported no earnings. While full audit would be the most accurate and preferential outcome variable, they are only conducted in few cases and the available audit data is too small for the duration of this study.
Therefore, while the ‘disclosure’ outcome does not necessarily measure fraudulent behavior on an individual level, it is a useful proxy of fraudulent behavior when aggregated on a higher level (i.e. condition level): That is, with a randomized experimental design, we can observe whether the treatment increased reporting relative to the control group. Since claimants are randomly assigned to conditions and the only difference between them is the treatment, any difference in disclosure rates is the result of the intervention. This condition-level proxy measure of fraudulent behavior in a field experimental design mirrors the commonly used condition-level measures of dishonest in laboratory settings (e.g. die-roll task, see Fischbacher and Föllmi-Heusi, Reference Fischbacher and Föllmi-Heusi2013, or matrix task, see Gino et al., Reference Gino, Ayal and Ariely2009).
Statistical analysis in the field experiment. Results are estimated using a linear probability model (LPM) predicting the likelihood of disclosure with robust standard errors clustered at the claimant level. Our results are robust to using variation in econometric specifications (e.g. logistic regressions). To help with interpretation and ease of reading of the coefficients in the regression tables, the disclosure outcome variable is represented as a percentage (0–100) rather than a fraction (0–1).
In all our analyses, we take a conservative approach by adjusting for multiple comparisons. We apply the False Discovery Rate (FDR) procedure to reduce the likelihood of detecting false-positives. Introduced by Benjamini and Hochberg (Reference Benjamini and Hochberg1995), the FDR is one of the most widely used procedures to adjust p-values when multiple tests are being performed. The FDR provides more statistical power than Bonferroni-type corrections while ensuring that p-values (also sometimes called q-values after application of the FDR) are adjusted on the basis of a cut-off value. Modern implementations of FDR algorithms automatically determine the cut-off based on the empirical distribution of p-values for the given context. We use the ‘p.adjust’ function in the ‘stats’ core package in R. All our regressions that involve multiple comparison groups (e.g. different messages or multiple RAR bins) have been adjusted using this FDR procedure.
This research was approved by the ethics board at Harvard University (IRB16-0813).
Results
Untargeted average treatment effect. We begin by looking at the average treatment effect (i.e. without the use of algorithmic targeting). We start with this analysis because it represents the status quo of the kind of empirical analysis one would conduct when analyzing such a field experiment in the absence of any algorithmic targeting. This analysis reveals what the treatment effect is for an intervention across the board for everyone, not taking into consideration the targeting approach we have outlined above and which we analyze below.
Using a LPM, we regress the disclosure rate by the treatment status, testing whether the treatment (relative to control) increases the average likelihood of disclosure across all claimants. While the coefficient on treatment is positive, there is no significant main effect of the intervention relative to the control group (see Table 2 Column 1: b = 0.490, SE = 0.336, p = 0.145).
Differential impact of intervention on the likelihood of disclosure by RAR. Unit of analysis is claimants’ weekly submissions. Column 1 is the main effect without algorithmic targeting by RAR. Column 2 studies the effect of algorithmic targeting and finds an interaction effect between treatment and RAR (as a continuous variable). Column 3 corroborates this finding by showing that the treatment effect is concentrated in the 5th RAR bin (with RAR values between 81 and 100). Standard errors are clustered at the claimant level. P-values are adjusted for multiple comparisons using FDR.
Table 2 Long description
Regression results estimate how an intervention relates to the likelihood of disclosure using weekly claimant submissions, reported across three model specifications. In model 1, the baseline level is about 4.44 and the treatment change from baseline is small and not statistically reliable. In model 2, baseline is about 4.73; the treatment term is strongly negative, while the interaction between treatment and RAR is strongly positive, indicating the treatment effect becomes more positive as RAR increases. The standalone RAR term in model 2 is near zero and not statistically reliable. In model 3, baseline is about 4.52; treatment and the RAR bin indicators are not statistically reliable on their own, but the treatment interaction for the highest RAR bin is positive and statistically reliable, suggesting the treatment effect is concentrated among the highest RAR group. All models use 22,457 observations and have very low explained variance, so effects should be interpreted as small in overall predictive power.
Note:
* p< 0.05; **p < 0.01; ***p < 0.001.
This result, thus far, suggests that the average behavior in the treatment group is not significantly different from the average behavior in the control group in the absence of algorithmic targeting. We note that, if this study were conducted in this ‘traditional’ fashion (i.e. with randomly assigning nudge messages across all claimants but without application of a targeting algorithm), a policy-maker would likely conclude at this stage that the interventions had failed (or, at least, that there is a lack of evidence for the treatment being effective). This is, however, where we turn to the use of algorithmic targeting to illustrate the added value of using such an approach.
Algorithmic-targeted treatment effects. We next consider the role of the algorithmic targeting approach based on the real-time generated RAR variable. The analytical approach is as follows: we take advantage of the random assignment of claimants to control or treatment groups, and then interact the treatment assignment with the RAR variable. The RAR variable captures the extent to which a claimant is predicted to behave fraudulently (with higher values indicating a greater likelihood of behaving fraudulently).
Note that, in the absence of the control group, the use of the RAR variable would simply be targeting nudges for everyone and would not allow for a causal estimation of the treatment and targeting approach; however, because a control group is present (i.e. a group of claimants across all RAR values, for whom no nudge is shown), we can compare claimants with a similar RAR values between the control and treatment groups, in order to identify the causal effect of the nudges relative to the control for those specific claimants.Footnote 3
Using LPM with both treatment assignment and the RAR value as independent variables, we find that there exists a significant interaction between the treatment and RAR (Table 2 Column 2; b = 0.056, SE = 0.015, p < 0.001).Footnote 4 As Figure 1 illustrates, the treatment effect is positive and significant only for claimants with the highest RAR values, suggesting that the treatment increased disclosure among claimants that were predicted to behave unethically.
Intervention only increases disclosure among claimants with high RAR values. Relative to the control group (red bars), the treatment (blue) significantly increases the likelihood of disclosure among claimants in the highest RAR bin but not among claimants in lower RAR bin. Error bars are standard errors from the mean (clustered at the claimant level).
Figure 1 Long description
A grouped bar graph with error bars. Legend title Condition with entries Control and Treatment. Horizontal axis label RAR Bin with tick labels 1, 2, 3, 4, 5. Vertical axis label Likelihood of Disclosure left parenthesis percent right parenthesis with tick labels 0, 2, 4, 6, 8. RAR Bin 1: Control bar about 4.5. Treatment bar about 3.9. RAR Bin 2: Control bar about 4.4. Treatment bar about 4.0. RAR Bin 3: Control bar about 5.1. Treatment bar about 3.4. RAR Bin 4: Control bar about 4.0. Treatment bar about 3.9. RAR Bin 5: Control bar about 3.6. Treatment bar about 7.1. Error bars are drawn above each bar with upper and lower caps.
By way of example of how the algorithm moderates the effect of the treatment, consider the disclosure rates in the control and treatment groups in lowest and the highest RAR bins (Figure 1). In the lowest RAR bin (with RAR values 0-19), the control and treatment groups have comparable disclosure rates of 4.52% and 3.91%, respectively. In contrast, disclosure rates of the control and treatment groups in the highest RAR bin (with RAR values 80–100) are notably different: relative to the control group (3.52%), the treatment doubles disclosure rates (7.07%) among ‘high-risk’ claimants. Since, by design, all claimants are part of our randomized field experiment, this difference in disclosure rates in the top bin can be attributed to the causal effect of the treatment.
Robustness analyses. We conduct several robustness checks on our data. First, in our main specification, we used a LPM to estimate the effect of the treatment on disclosure rates. However, since the dependent variable is binary, a logistic regression is another common analytical approach in the literature. We initially chose LPM over a logistic regression approach for our main analysis since logistic models can be problematic for estimates of causal treatment effects, especially when interaction terms are involved (Ai and Norton, Reference Ai and Norton2003), making linear models a safer choice (Gomila, Reference Gomila2020). However, as we show in Table S2, our results are robust to an alternative choice of statistical model.
Next, to understand where along the RAR spectrum this treatment effect is concentrated, we repeat the same analysis using a discretization of the RAR spectrum into five equally-sized ‘bins’ (i.e. 0–19, 20–39, 40–59, 60–79, and 80–100). We find that the treatment effect is driven exclusively by an increase in disclosure rates among the treatment group in the highest RAR bin (b = 4.158, SE = 1.346, FDR-adjusted p = 0.010). (This categorization is also used in Figure 1).
In addition, we further explore a finer discretization by creating 10 equally spaced bins along the RAR spectrum. As shown in Table S4 in the appendix, these results demonstrate that the treatment effect is especially pronounced in the highest RAR bin (i.e., RAR values 90–100) where the disclosure rate in the treatment group is 2.5 times larger than in the control group (8.52% vs. 3.37%, respectively).
Exploratory analysis of individual messages. In this section, we examine the variation in treatment messages on the likelihood of disclosure. We acknowledge that this is an exploratory analysis and as such, our findings in this section should be interpreted with caution. To avoid spurious findings, we take two steps. First, we adjust for multiple comparisons in all our regressions using the FDR procedure described above. Second, we conduct the same analyses as above where we look at the interaction between the treatment messages and RAR.
We consider two ways that RAR could be operationalized, first as a continuous variable and, second, RAR discretized into five bins. The second analysis is more conservative than the first because, by virtue of having multiple RAR bins, we also adjust for more multiple comparisons. We only conclude that a message is reliably effective for high-RAR claimants if both analyses agree. That is, we require that both the interaction between the message and the continuous RAR as well as the interaction between the message and discretized highest-RAR bin are significant after FDR-adjusting for multiple comparisons, for us to conclude that the algorithm-moderated treatment effect is real and robust. We believe that this procedure ensures that false positives are minimized.
The results of the two complementary analyses can be summarized as follows: the Social Norm, Impact Others, Audits & Verification and Penalties message are significant only for claimants with the highest RAR values in both analyses (FDR-adjusted ps < 0.05 in Column 5 in Table 3, which uses discrete RAR bins; see also Table S4, which uses the continuous RAR variable). In contrast, the Reminder message is not significant in either analyses. In sum, after adjusting for multiple comparisons in both analyses, our results suggest that the Social Norm, Impact Others, Audits & Verification and Penalties messages significantly increase disclosure rates among high-RAR claimants, whereas the Reminder message does not. (For further analyses on minor variations in message, see SI Section 2.)
Exploratory analyses of differential impact of messages by discretized RAR. Linear probability model predicting disclosure by type of message interacted with RAR bin. For ease of reading, the regression table is presented such that the regression coefficients of the interactions between each RAR bin and each message are shown in the columns and rows, respectively. Specifically, the coefficients in Column 1 in the table are the baseline effects in RAR bin 1 while Columns 2–6 show the interaction coefficients between the message and the corresponding RAR bin. Standard errors are clustered at the claimant level. P-values are adjusted for multiple comparisons using FDR.
Table 3 Long description
The table reports linear probability model coefficients for likelihood of disclosure by message type across five discretized RAR bins, with standard errors in parentheses. Column 1 gives baseline effects in RAR bin 1, while columns 2 through 5 show how each message’s effect differs in RAR bins 2 through 5. In RAR bin 1, the control baseline is strongly positive at 4.520 with high statistical significance, while message coefficients are small and not statistically significant. In RAR bins 2 to 4, most message effects remain mixed in sign and are not statistically significant. In RAR bin 5, every message shows a positive coefficient, and four are statistically significant: Social norm 5.915, Impact others 8.499, Audits and verification 7.040, and Penalties 4.132; Reminder is positive at 2.955 but not significant. Overall, the pattern suggests messages have their clearest positive association with disclosure among claimants in the highest RAR bin, while effects are weaker or inconsistent at lower RAR levels. Interpret results cautiously because coefficients are adjusted for multiple comparisons and uncertainty is reflected in clustered standard errors.
Note:
* p < 0.05; **p < 0.01; ***p < 0.001.
Discussion
While nudges have been shown to be practical and cost-effective in many domains (Benartzi et al., Reference Benartzi, Beshears, Milkman, Sunstein, Thaler, Shankar, Tucker-Ray, Congdon and Galing2017), they have not consistently replicated across different settings (Hummel and Maedche, Reference Hummel and Maedche2019). Here, we proposed that nudges can be made more effective with algorithmic targeting. Providing experimental evidence of nudge interventions that worked in the field helps strengthen the external validity of interventions, which is both theoretically important and policy relevant (Hauser et al., Reference Hauser, Linos and Rogers2017). We created an algorithm to detect likely unethical behavior, which then enabled us to create and direct targeted interventions: The interventions that significantly increased disclosure among high-risk claimants included impact on others, social norms, and audits and verification messages, consistent with recent studies in other domains (e.g. Kleven et al., Reference Kleven, Knudsen, Kreiner, Pedersen and Saez2011; Hallsworth et al., Reference Hallsworth, List, Metcalfe and Vlaev2017; Bott et al., Reference Bott, Cappelen, Eø and Tungodden2019).
By targeting interventions at claimants with the most opportunity to evade, our results may also shed light on some contradicting findings in the past literature, where one group of scholars may have found that these interventions work while others have not found an effect. For example, in our sample of claimants of social benefits, if we had only considered the impact of the interventions on the average claimant, one might have assumed that the treatment had not worked; however, it would be wrong to assume that the treatment should work on claimants who have already made an honest decision of whether earnings needed to be disclosed (which is the vast majority of claimants). We therefore a priori employed an algorithm to identify which claimants would be most likely to behave fraudulently and only predicted a treatment effect to occur for this relevant population. Future researchers may benefit from targeting more precisely claimants who they believe are able to change their behavior, rather than applying an intervention to all available claimants.
Our results had real-world, financial implications for the public sector organization. While the treatment did not produce an average significant effect on disclosure and disclosed earnings, it did show significant increases in disclosure rates for high-risk claimants. Based on back-of-the-envelope estimates, we can approximate the additional earnings that the government was able to collect by exposing the highest RAR claimants to the treatment. During the three-week experimental period, the treatment was observed 5,440 times by claimants in the highest RAR bin, leading to approximately $23,011 in earnings disclosed by claimants, which would otherwise not have been collected by the government. Extrapolating from these numbers, the treatment could generate up to $400,000 per year in additional revenue that claimants would otherwise not disclose voluntarily and would be lost to the government. In addition, further savings for the public administration would also result from not having to investigate and recover any earnings from claimants who have not truthfully disclosed such earnings. However, this rough back-of-the-envelope calculation does not consider potential general equilibrium effects, including potential habituation to the treatments or learning effects, which could affect the response to the treatment and remains an important area of future research. Furthermore, both our research design and the cost savings implications are contextual and may vary depending on the specific behavior to be nudged, the algorithm’s ability to predict such behavior, and the success of the interventions used. To generalize to other contexts, we encourage future researchers to apply similar approaches that go beyond prediction alone and aim for behavior change (Athey, Reference Athey2017), while also rigorously evaluating the policy outcomes and cost savings of such interventions (H.M. Treasury, Reference Treasury2007).
Finally, our paper also offers proof of concept and practical contributions surrounding the effectiveness of algorithms to detect risk levels of unethical behavior over time. Algorithmic targeting is fast becoming ubiquitous, from social media to savings to health (Benartzi et al., Reference Benartzi, Beshears, Milkman, Sunstein, Thaler, Shankar, Tucker-Ray, Congdon and Galing2017; Matz et al., Reference Matz, Kosinski, Nave and Stillwell2017, Nahum-Shani et al., Reference Nahum-Shani, Smith, Spring, Collins, Witkiewitz, Tewari and Murphy2018), but the same algorithms have not been used widely in identifying unethical behaviors. However, future applications can be imagined: identifying illegal transactions, employee mistreatment, or even domestic abuse. By identifying potentially unethical subgroups using an algorithm and applying interventions to those subgroups only may provide a practical and non-invasive means to change, or even pre-empt, unethical behavior. We believe the greatest use of this application is where nudges are costly to organizations or cumbersome for users, or where the majority of benefits are concentrated in a small subset of individuals (Einav et al., Reference Einav, Finkelstein, Mullainathan and Obermeyer2018). Recent advances with generative AI technology, such as large language models, further add to the possibility of greater personalization of persuasive messages (Matz et al., Reference Matz, Teeny, Vaid, Peters, Harari and Cerf2024) or other uses of creative outputs co-created with AI (Doshi and Hauser, Reference Doshi and Hauser2024).
However, it is critical to note that there are important ethical considerations at play in organizations’—and especially the public sector’s—use of machine learning to detect risk of unethical behavior. On the one hand, nudges have been compared in their use to manipulation to achieve certain behavioral change (e.g., see Wilkinson, Reference Wilkinson2013). Artificial intelligence attracts similar worries: The in-depth monitoring, the use of dynamic artificial intelligence systems, and attempts to subtly influence behavior by organizations could all be manipulative, autonomy-reducing, and invasive. On the other hand, nudging can improve welfare and, by not constraining choice, this approach emphasizes the importance of preserving freedom of choice in attempts to influence behavior (Sunstein, Reference Sunstein2015). Combining nudges with algorithms can potentially resolve this argument: while the algorithm in our study identified individuals with elevated risk of unethical behavior, it was not interpreted by the government as unethical behavior itself (and no action was taken), leaving the decision and behavior up to the individual. Importantly, the way in which algorithms are described to citizens can affect their acceptance, which is a particularly important consideration for their applications in the public sector (Sunstein and Reisch, Reference Sunstein and Reisch2023).
It is important to acknowledge that both the use of machine learning and nudges could potentially be used in nefarious ways, such as identifying individuals for the purposes of discrimination (Wang and Kosinski, Reference Wang and Kosinski2018). While responsible use of these techniques has the power to decrease the negative effects of human biases, temptations, and assumptions, private and public sector organizations must educate themselves on the possibilities for negative side effects or misguided use. Furthermore, regulators need to step up to oversee and manage what is currently an unregulated and potentially problematic tool that can be used on vulnerable populations. As organizations and governments embrace the use of machine learning and algorithms, a set of standards and rules ought to be developed, maintained and observed, ensuring transparency of the algorithms used and individuals’ privacy respected (Horvitz and Mulligan, Reference Horvitz and Mulligan2015; Athey, Reference Athey2017).
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/bpp.2024.50.
Acknowledgements
We are grateful to the New Mexico Department of Workforce Solutions (DWS) and Deloitte for their collaboration and support. In particular, we wish to thank Sue Anne Athens (DWS), Celina Bussey (formerly DWS), Joy Forehand (formerly DWS), Jim Guszcza (formerly Deloitte), and Scott Malm (Deloitte) for their instrumental support in making this study succeed. For the purpose of open access, the authors have applied a ‘Creative Commons Attribution’ (CC BY) licence to any Author Accepted Manuscript version arising.
Open access
