The effect of incentives on motivated numeracy amidst COVID-19

Abstract How does political ideology affect the processing of information incongruent with one’s worldview? The disagreement in prior research about this question lies in how one’s ideology interacts with cognitive ability to shape motivated numeracy or the tendency to misinterpret data to confirm one’s prior beliefs. Our study conceptually replicates and extends previous research on motivated numeracy by testing whether monetary incentives for accuracy lessen motivated reasoning when high- and low-numeracy partisans interpret data about mask mandates and COVID-19 cases. This research leverages the ongoing COVID-19 crisis, as Americans are polarized along party lines regarding an appropriate government response to the pandemic.


Introduction
How does political ideology affect the processing of data incongruent with one's worldview?When individuals face uncongenial information, they tend to interpret it inaccurately, engaging in motivated reasoning (Bénabou and Tirole, 2006;Cohen, 2003;Kunda, 1990); this phenomenon is also known as "the congeniality effect/bias" (Khanna and Sood, 2018).Motivated numeracy, a type of motivated reasoning, arises when individuals interpret uncongenial data.Recent studies find that the congeniality bias is a politically neutral phenomenon, as both conservatives and liberals experience discomfort when confronting worldview conflict, and both enjoy consuming congenial information (Brandt et al., 2014(Brandt et al., , 2019)). 1  This study conceptually replicates and extends Kahan et al. (2017) by testing whether monetary incentives for accuracy lessen the congeniality bias when highand low-numeracy partisans interpret data about mask mandates and COVID-19 cases.This research leverages the ongoing COVID-19 crisis, as Americans are polarized along party lines regarding an appropriate government response to the pandemic.For example, states that voted Republican in 2016 are 40 percentage points less likely to adopt mask-wearing mandates (Makridis and Rothwell, 2020).

Motivation
Do incentives for accuracy reduce motivated reasoning among numerate partisans?
Scholars disagree about how worldview conflict interacts with scientific literacy to shape motivated numeracy.On the one hand, subjects highest in numeracy appear to exhibit more bias when interpreting uncongenial data (Kahan et al., 2017), consistent with findings that cognitive ability does not impede in-group bias (Stanovich et al., 2013) and that receiving new evidence exacerbates biases in experts (Baekgaard et al., 2017).
On the other hand, Mérola and Hitt (2016) demonstrate that more scientifically literate subjects are more persuaded by data sponsored by an opposing party, suggesting that higher numeracy may provide immunity against the congeniality bias. 2 This is consistent with experimental research on persuasion (Redlawsk et al., 2010) and studies on citizens' updating from observing economic reality (Parker-Stephen, 2013), which suggest voters may update from uncongenial information.This latter research is also indirectly reinforced by two sets of findings around incentives for accuracy and "motivated responding," which indicate that the congeniality effect may have been overstated.First, the congeniality bias drops if individuals are motivated to answer accurately by monetary payments or appeals to accuracy (Bullock et al., 2015;Prior et al., 2015).Partisans' biased reasoning appears to be driven (in part) by the lack of stimulus for accuracy.These studies do not, however, test whether incentives lessen motivated numeracy, 3 instead focusing on bias in recalling facts or interpreting verbal information.This research, by contrast, tests whether incentives reduce motivated interpretation of data, a critical question to answer if we are to understand how individuals form evidence-based opinions.
Second, other studies differentiate "motivated learning" from "motivated responding" (Bisgaard, 2019;Khanna and Sood, 2018), where the former describes absorbing congenial facts more readily than uncongenial information and the latter giving incorrect but congenial answers even after correctly processing the worldview-conflicting information. 4Our study speaks to the difference in these concepts as follows.To measure motivated responding, Khanna and Sood (2018) 5announce incentives for accuracy after respondents were shown numerical tasks to examine motivated responding.Our research announces incentives before respondents are shown in table.This design allows us to answer to what extent incentives reduce motivated learning when interpreting uncongenial data.
In summary, evidence indicates that incentives for accuracy should lessen the congeniality bias.We now examine how incentives may interact with cognition and ideology to shape motivated numeracy.

Hypotheses
Replication of prior results is key to understanding contradictory findings.Replications may be direct (i.e., testing whether the original findings are true under the same conditions, using as-similar-as-possible design and analysis) or conceptual (i.e., retesting the underlying ideas to specify the conditions under which the same results would be found, potentially using different design or analysis) (Chambers, 2017).Our study conceptually replicates and extends Kahan et al. (2017) by modifying treatments from and adding incentives for accuracy.

Conceptual replication
Prior research replicated Kahan et al.'s (2017) findings directly and conceptually.First, Kahan and Peters (2017) directly retested their own results in a new sample and found the same results.Second, Khanna and Sood (2018) found partially consistent results in a conceptual replication; they did not, however, revisit the initial question of how numeracy interacts with ideology.Third, Baker et al.'s (2020) conceptual replication finds that motivated numeracy depends only on partisanship, not on scientific literacy; this result contradicts both Kahan et al.'s (2017) and Mérola and Hitt's (2016) contrasting findings. 6hese mixed findings emphasize the need for conceptual replication that will revisit the questions of (1) whether accuracy is higher when interpreting congenial data, and (2) whether this effect is amplified among numerate individuals.
Hypothesis 1: Among unincentivized respondents, the rate of correct data interpretation increases as the data become more congenial to one's ideological beliefs.(The congeniality bias exists.) Hypothesis 2: Among unincentivized respondents, the congeniality bias increases with one's numeracy.

Extension
How do incentives for accurate learning influence motivated numeracy?Prior research reveals that rewarding accuracy reduces the congeniality bias (Bullock et al., 2015;Prior et al., 2015), when respondents are asked to recall facts or interpret passages.
But do incentives reduce motivated processing of numerical data rather than verbal information?Khanna and Sood (2018) employed incentives for accuracy when solving numerical tasks and found mixed results: monetary rewards reduced the congeniality bias among some but not all partisans.Given the limitations of Khanna and Sood's (2018) sample,7 it is difficult to determine how incentives shaped motivated numeracy in partisans vs. moderates and in high-vs.lownumeracy subjects.We thus test the impact of incentives for accuracy in a sample with sufficient variation in numeracy and ideology.
Additionally, Khanna and Sood (2018) focus on motivated responding as opposed to learning, announcing rewards for accuracy after respondents were given the numerical task.In our extension of Kahan et al. (2017), we focus on the impact of incentives on motivated learning from numerical data; therefore, we announce incentives for accuracy before respondents are shown the task.
Our expectations for introducing an incentives condition are twofold.First, if incentives lessen motivated numeracy, then slightly higher stakes of giving a correct answer may alleviate motivated learning from data.Second, theoretically, incentives should increase the accuracy of high-numeracy individuals in both congenial and uncongenial conditions.This means that with incentives for accuracy, the congeniality bias, that is, the difference between correct answers in congenial and uncongenial treatment conditions for a given ideology, should not increase with numeracy.
Hypothesis 3: Relative to unincentivized respondents, those incentivized will exhibit greater accuracy in all conditions.
Hypothesis 4: The congeniality bias among incentivized respondents increases at a lower rate with one's numeracy, compared to the rate of bias increase among unincentivized respondents.

Registration
The research design was peer-reviewed in this journal from August 2020-January 2021, and the embargoed OSF pre-registration (https://osf.io/935xb)was released in August 2021.No explorative tests were done on the data.

Deviations from pre-registration
The post-analysis peer-review process uncovered two mistakes in the pre-registered regression equations, and these deviations from the pre-registered design do not substantively alter the results.Table 1 lists said changes, section "Deviations from pre-registration" of the appendix also includes the original analysis that follows the pre-registered design without deviations.

Case
From March 2020-August 2021, the U.S. (4.2% of the world population) experienced 18% of COVID-19 cases and 14% of COVID-19-related deaths (Ritchie et al. 2021).Despite the evidence that masks curb the spread of the virus (Ingraham, 2020), in 2020, then President Trump's and his allies' dismissive statements about masks politicized masking further (Miller et al., 2020), such that Republicans reported most reluctance to wear masks (Pew Research Center, 2020).Out of nineteen "much more Republican" states based on the 2016 election margin, seventeen did not introduce mask mandates (Markowitz, 2020).
This survey was fielded in May 2021, when mask mandates remained politicized.Republican-led states opted to lift mask mandates earlier, notwithstanding low vaccination rates in those states.In May, eleven Republican governors lifted mask mandates, while House Speaker Pelosi indicated that the chamber would keep its mandate despite new CDC guidelines, sparking a GOP backlash (Elfrink, 2021).In May-August 2021, eight states (all with Republican governors) prohibited local governments from issuing mask mandates, while ten states (all with Democratic governors) required masks in schools (Vestal 2021).

Experimental conditions
The study randomly assigned participants to either the no-incentives or incentives treatment.Within each group, respondents were randomly assigned to one of two data tasks shown in Figure 1.Participants were asked to judge, based on the table, whether cities that had a mask mandate were more likely to experience an increase/ decrease in COVID-19 cases.The description of the tables clarified that the numbers represent cases since the mandate was implemented.The correct answers for conditions 1 and 2 are mutually opposite.Cell values remained the same across conditions, which differed only in their column headings.This covariance-detection task represents a difficult numeracy problem because one needs to consider ratios (Baker et al., 2020).
Participants' numeracy was captured by six questions, randomized to appear either before or after the contingency table; the analysis section discusses order effects.

Dependent variable
Correct i equals 1 when a respondent correctly answered the contingency table question, 0 otherwise.

Independent variables
Numeracy i is the number of correct answers the participant i gives to the six questions aimed at measuring numeracy (Lipkus et al., 2001;Peters et al., 2007;Schwartz et al., 1997). 8Items on disease/infection were removed considering their relevance to the treatment.
Congenial i is a continuous measure of attitude-consistent message that captures the degree of congeniality of the contingency table's data with one's ideology.This variable is constructed using, first, the continuous measure of ideology (Conservative i ) and, second, two binary indicators of the condition to which a respondent was assigned (Covid_decreases i or Covid_increases i ).Using Cronbach's α, two 7-point Likert scales of party affiliation and ideology form an aggregate scale, Conservative i , where negative values indicate liberal Democrats and positive values indicate conservative Republicans (α = 0.795 which is similar to the α value of 0.83 in Kahan et al. 2017).Conservative i is identical to the measure of ideology in Kahan et al. (2017) For a conservative respondent (Conservative i >0), the Congenial i variable is positive in the Covid_increases i condition and negative in the Covid_decreases i condition, while for a liberal (Conservative i <0), Congenial i is positive in Covid_decreases i and negative in Covid_increases i condition.The absolute magnitude of Congenial i increases with the strength of one's ideology.
Incentive i is a binary indicator of whether a participant is assigned to that condition.

Controls
Indicators of age, ethnicity, gender, education, and voting behavior were collected to ensure that treatment groups are balanced on observable covariates.

Statistical model
A linear probability model estimates the parameters of equations ( 1)-(4).Equations (1) and ( 2) are estimated on unincentivized observations to test hypotheses 1 and 2; equations (3) and (4) test hypotheses 3 and 4, utilizing the full sample.The appendix also includes the logistic regression models that estimate these equations.

Sample
No pilot data were collected.Qualtrics fielded the survey.Per the pre-registered power analysis, the required sample size is 3,000 with 1/3 of observations unincentivized and 2/3incentivized; Qualtrics collected 1,016 (512 and 504 in each contingency table condition) unincentivized responses and 2,034 incentivized (992 and 1,042 in each table condition), a total of 3,050 responses.

Simulations for power calculations
The power was calculated using Monte Carlo (MC) simulations with 1,000 repetitions; the appendix details all assumptions.

Data inclusion criteria
Qualtrics assembled a nonprobability online sample of U.S. residents of 18 years old.To avoid common biases in online samples (Kennedy et al., 2016;Rivers, 2016), we calculated nested quotas.Qualtrics was unable to fulfill all planned nested quotas; the appendix describes the differences between planned and collected quotas.Nevertheless, the resultant sample largely avoids underrepresenting nonvoters and less educated individuals (see the appendix).

Data exclusion criteria based on quality checks
After the treatment, respondents answered the factual manipulation check question (Kane and Barabas, 2019); those who failed it were removed from the sample.Qualtrics removed additional low-quality responses, as the appendix describes.

Termination of data collection
Qualtrics terminated data collection after 3,050 quality responses were reached, conducting quality checks after 10%, 75%, and 100% of the data were collected.

Balance on covariates
Per the pre-registered power calculations, respondents were randomly assigned to either a no-incentive or incentive treatment with a probability of 1/3 and 2/3, respectively.Respondents were also assigned randomly to conditions Covid Increases vs. Covid Decreases with a probability of 1/2.While assignment was random, some respondents were removed after failing the manipulation check and/or other quality controls outlined in the "Data exclusion [ : : : ]" section.To verify that passing the exclusion criteria did not correlate with any observable attributes, we obtained differences-in-means between each pair of treatments (see the appendix).None of the differences-in-means is statistically discernible from 0 at the 5% level; the treatment groups are, therefore, balanced on observable attributes.

Multivariate analyses
The impact of congeniality on accuracy among unincentivized respondents (Hypothesis 1) To test hypothesis 1, models 1-2 of Table 1 estimate equation ( 1), using a linear probability model on unincentivized responses; model 2 also adds the controls.
Hypothesis 1 states that, among unincentivized respondents, accuracy is higher when interpreting more congenial data.The coefficient estimate on Congenial i implies that one-unit increase in Congenial i (ranges from −1.88 to 1.88, mean = 0, SD = 1) generates a 4.4-4.8percentage point (pp) increase in accuracy (statistically discernible from 0).Section "Inferiority tests" of the appendix shows that we cannot conclude that a meaningful effect of Congenial i on Correct i is absent.We thus reject the first null of no effect.
The impact of congeniality and numeracy on accuracy among unincentivized respondents (Hypothesis 2) Hypothesis 2 posits that, among unincentivized respondents, the congeniality bias increases with one's numeracy.The coefficient estimate for the multiplicative interaction term Numeracy×Congenial is positive (only marginally significant in model 3), indicating that increases in numeracy improve accuracy even as congeniality increases.Following Kahan et al. (2017), we also include the squared Numeracy term; the coefficient estimate for the interaction Numeracy 2 ×Congenial is not statistically different from zero.We use predicted probability densities (Figure 2) and differences in the predicted congeniality bias for various numeracy levels (Table 3) to further understand if Numeracy i has a range of values for which Congenial i generates a statistically meaningful impact among unincentivized respondents.
Section "Inferiority tests" of the appendix shows that we cannot conclude that a meaningful effect of the interaction terms on accuracy is absent.
The impact of incentives on accuracy (Hypothesis 3) Hypothesis 3 expects that incentivized participants are more likely to correctly interpret the contingency table compared to those unincentivized.Models 5 and 6 of The Effect of Incentives on Motivated Numeracy Amidst COVID-19 Table 2 test this expectation, estimating parameters in equation ( 3), using all observations.The coefficient estimates on Incentive i in models 3 and 4 are substantively and statistically negligible; we thus fail to reject the null hypothesis of monetary incentives having no impact on accuracy.
The impact of incentives, numeracy, and congeniality on accuracy (Hypothesis 4) Hypothesis 4 posits that the congeniality bias among incentivized respondents increases at a lower rate with one's numeracy, compared to the rate of the congeniality bias increase among unincentivized respondents.To test hypothesis 4, linear probability models 7 and 8 of Table 2 estimate parameters in equation ( 4).The interaction of numeracy and congeniality is positive, while the interaction of the squared numeracy term and congeniality is negative; however, neither interaction term reaches statistical significance at the 0.05 level.Both three-way interaction coefficients have statistically and substantively negligible effects on correct answers.While this suggests that the increase in the congeniality bias with numeracy is not different between incentivized and unincentivized respondents, we use visualizations to understand if there are ranges of values of Congenial i and Numeracy i for which Incentive i has a statistically meaningful impact.Note: Density distributions derived via MC simulation from logistic regression that estimates equation 4 (output is shown in the appendix), when Congenial is set at -1 SD (i.e., respondents facing data contradicting their beliefs), at mean (i.e., ideological moderates on the ideology-party affiliation spectrum), and 1 SD (i.e., data are consistent with beliefs), and numeracy set at -1SD (1 out of six correct questions) for "low numeracy" and 1SD (4.35 out of six correct questions) for "high numeracy."Numeracy is centered at "0" for ease of interpretation./-1SD Numeracy value is /-1.654, and numeracy squared term is 2.736.

Visualization of the interaction effects
We supplement the linear probability model results with logistic regression results (in the appendix) to estimate the out-of-sample predicted probabilities of correct answer (based on MC simulations).Figure 2 graphs these probability densities, visualizing how incentives, numeracy, and congeniality interact to shape motivated numeracy. 9igure 2 demonstrates that the congeniality bias increases with numeracy.The predicted accuracy rates for respondents facing uncongenial, neutral, and congenial data constitute (1) 41%, 43%, and 45%, that is, the congeniality bias equals 4pp for the less numerate unincentivized respondents (top left), (2) 34%, 41%, and 49%, that is, the congeniality bias is 15pp for the more numerate unincentivized (top right), (3) 40%,40%, and 39%, that is, the bias equals -1pp for the less numerate incentivized (bottom left), and (4) 40%, 45%, and 49%, that is, the bias constitutes 9pp for the more numerate incentivized (bottom right).
Table 3 uses differences-in-means to test whether the differences in the congeniality bias between less and more numerate individuals are statistically distinct.To test hypothesis 2, consider respondents in the no-incentive condition first.Highly numerate respondents exhibit greater congeniality bias than less numerate subjects by 12pp, 18pp, and 23pp for respondents with numeracy of −/1SD, −/1.5SD, and /−2SD above/ below the mean, however these differences are not statistically distinct at 0.05 level.Our data, therefore, indicate that numeracy does not have an impact on the congeniality bias of unincentivized respondents shown by Kahan et al. (2017); we thus fail to reject the second null hypothesis of no effect of numeracy on congeniality bias among unincentivized respondents.Table 4 shows that incentives shrink the gap in motivated numeracy from −4pp to 1pp (5pp reduction) among less numerate (/−1SD) and from 15pp to 9pp (6pp reduction) among more numerate individuals.Similar reductions in the congeniality bias are observed at other levels of numeracy.Hypothesis 4 expected that incentives would generate larger reductions in the bias among more numerate individuals; however, the changes we observe are not statistically distinct between less and more numerate individuals; we fail to reject the fourth null hypothesis of no effect.

Order effects
In summary, this paper tested four hypotheses.With exception of hypothesis 1, we fail to reject the nulls of no effect.
Section "Order effects" of the appendix reruns all analyses on split samples by order (this robustness check was pre-registered).Split sample analysis reveals that one-unit increase in congeniality increases accuracy by 7-8pp when numeracy questions appear after treatment.The congeniality, however, has no effect on accuracy when numeracy questions precede treatment, implying that our rejection of the first null hypothesis in the pooled sample is driven by the subsample that received numeracy questions after treatment.
Additionally, split sample analysis reveals thatconsistent with the pooled sampleincentives generate no reductions in the congeniality bias among more numerate respondents when numeracy questions follow treatment.However, in the subsample "numeracy before treatment" we observe that incentives increase the congeniality bias among the more numerate individualscontrary to the expectation in hypothesis 4.
The results from split samples do not alter our conclusions regarding hypotheses 2 and 3.The Effect of Incentives on Motivated Numeracy Amidst COVID-19

Figure 2 .
Figure 2. Predicted Probabilities of Correctly Interpreting the Data.

Table 1 .
Deviations from Pre-registration . Second, conservatives received uncongenial data in Covid_decreases i and liberalsin Covid_increases i .The Congenial i variable follows the formula:

Table 3 .
The Impact of Incentives, Numeracy, and Congeniality on Accuracy (All Participants)

Table 4 .
Differences in the Predicted Congeniality Bias Between Less and More Numerate Individuals at Various Levels of Numeracy