1 Introduction
Decisions are frequently influenced by numerical information. Sometimes this information is provided as a descriptive summary (e.g., the stated return from an investment or the risk of side effects from a medication), while in other situations it is gleaned from prior experience (e.g., memory of previous returns from stock or the amount of time a particular route home took). Interestingly, risky decisions based on experience or descriptive summaries do not always align (Reference Barron and ErevBarron & Erev, 2003; see for review, Reference Rakow, Newell and ZougkouRakow & Newell, 2010). As a means of understanding this divergence, there has been particular interest in the way information is sought out (see for review, Erev et al., 2010; Reference Mehlhorn, Newell, Todd, Lee, Morgan, Braithwaite and GonzalezMehlhorn et al., 2015). Research suggests that the amount of information collected, and to some extent the way it is used, is influenced by cognitive and numeric abilities (Reference Ashby and RakowAshby & Rakow, 2014; Reference LejarragaLejarraga, 2010; Reference Rakow, Newell and ZougkouRakow, Newell & Zougkou, 2010), emotional states (Frey, Hertwig & Reiskamp, 2014), age (Reference Frey, Mata and HertwigFrey, Mata & Hertwig, 2015), search strategy (Reference Hills and HertwigHills & Hertwig, 2010), and perspective (Reference Pachur and ScheibehennePachur & Scheibehenne, 2012). The current work extends these findings by providing an analysis of the role numeric ability (numeracy) plays in search (exploration), and its influence on preference stability (i.e., the extent to which the same choices are made) across descriptive and experiential formats. Specifically, I test whether individuals with greater numeric ability seek out more experience before making a decision and show greater choice consistency across decisions made based on descriptive information or experience.
Numeracy tends to be the single strongest predictor of general decision making skill, including the ability to understand and evaluate risk (i.e., risk literacy; Cokely et al, in press, 2012). Statistical numeracy in particular (i.e., the ability to solve problems involving probabilistic information), has been shown to be predictive of performance in decisions based on descriptive information (decisions-from-description; Peters et al., 2006, 2012). For example, individuals with higher numeric ability are less influenced by frame (e.g., when outcomes are framed as gains or losses; Reference Peters and LevinPeters & Levin, 2008), make more normatively consistent decisions (Reference Cokely and KelleyCokely & Kelley, 2009), perform better on the job (Reference Burks, Carpenter, Goette and RustichiniBurks, Carpenter, Goette & Rustichini, 2009), and are generally more adept at using numerical information when making decisions (Reference Cokely, Galesic, Schulz, Ghazal and Garcia-RetameroCokely et al., 2012; Reference Garcia-Retamero and CokelyGarcia-Retamero & Cokely, 2017; Reference Pachur and GalesicPachur & Galesic, 2013; Reference PetersPeters et al., 2012; Petrova et al., in press; Reference Reyna, Nelson, Han and DieckmannReyna, Nelson, Han & Dieckmann, 2009). In addition, there is evidence that numeracy influences how probabilities (Reference Patalano, Saltiel, Machlin and BarthPatalano et al., 2015; Reference Traczyk and FulawkaTraczyk & Fulawka, 2016) and outcomes (Reference Schley and PetersSchley & Peters, 2014) are treated, with numerate individuals appearing to treat them more objectively (i.e., treating 90% more like 90% rather than 90% ± some bias). Numeracy has also been found to be related to increases in the amount of information sought out in decisions-from-samples paradigms (where decision makers simulate playing available options before making a consequential choice; Reference LejarragaLejarraga, 2010) and choosing options with higher expected value (Reference Frey, Mata and HertwigEV; Reference Jasper, Bhattacharya, Levin, Jones and BossardJasper, Bhattacharya, Levin, Jones & Bossard, 2013). Given that individuals with greater statistical numeracy make more normatively consistent decisions when options are described and experienced, one might suspect that they would also show greater preference consistency across descriptive and experiential formats.
This paper tests the hypothesis that individuals with higher statistical numeracy (referred to as numeracy from here on) generally support skilled decision making by searching for more information (Reference Cokely and KelleyCokely & Kelley, 2009; Reference Jasper, Bhattacharya and CorserJasper, Bhattacharya & Corser, 2017; Reference LejarragaLejarraga, 2010), and as a result will tend to express greater consistency in choices (e.g., revealed preferences) across decisions made from description and experience. In addition, I test whether numeracy influences the rate of alternation (i.e., switching from one option to another on consecutive samples) during information search, influencing choice. Specifically, Hills and Hertwig (2010) reported that individuals who alternated less made more EV maximizing choices (i.e., choosing the option with the highest EV). Given that numerate individuals have been found to make choices that align with an EV maximization strategy, one might predict that they also alternate between options less during search.
Second, I examine whether numeracy increases sampling from riskier options (i.e., options containing outcomes that occur with smaller probabilities). Such a bias in sampling is of interest because greater sampling from riskier options is required if one wishes to see all possible outcomes they may contain.Footnote 1 Furthermore, previous research indicates that one of the largest contributors to the divergence between risky choices made from experience and description (when rare events are involved) is the result of limited sampling from riskier options (Reference Glöckner, Hilbig, Henninger and FiedlerGlöckner, Hilbig, Henninger & Fiedler., 2016; Reference Hadar and FoxHadar & Fox, 2009; Reference Hau, Pleskac, Kiefer and HertwigHau, Pleskac, Kiefer & Hertwig, 2008; Reference Rakow, Demes and NewellRakow, Demes & Newell, 2008). Therefore, in order for more numerate individuals’ choices to align across decision formats as I predict, they should draw more representative samples from the riskier option.
Lastly, given that previous studies have reported that numerate individuals are more likely to choose higher EV options in both descriptive and experiential formats in higher stakes gambles with larger EV ratios (e.g., where one option returns twice the amount of another on average; Cokely & Kelly, 2009; Reference Jasper, Bhattacharya, Levin, Jones and BossardJasper et al., 2013; Reference Peters, Västfjäll, Slovic, Mertz, Mazzocco and DickertPeters et al., 2006), I examine whether such effects are found when stakes and EV ratios are smaller. To test these predictions, I present participants with the same choices in decisions-from-description and decisions-from-samples (Hertwig, Barron, Weber & Erev, 2005) and assess their numeracy (Reference Cokely, Galesic, Schulz, Ghazal and Garcia-RetameroCokely et al., 2012).
2 Study 1
2.1 Participants
One-hundred and eighty participants (M age = 34.37; 41% male) were recruited from Amazon Mechanical Turk and completed the study in full: The study was run in a session along with several unrelated studies. Participants reported relatively diverse educational backgrounds: Approximately 12% of participants reported having a high-school education or GED, 28% some college, 12% a 2-year degree, 35% a 4-year degree, 9% a masters, 2% a professional degree, and 1% a doctoral degree. As an attention check participants had to click an invisible box instead of “continue” on an instruction page. Participants who failed this attention check were not allowed to take part in the study. Participants received $0.80 as well as a bonuses contingent on their choices during the task (see below). The experimental session lasted approximately 30 minutes and participants earned $3.00 on average.
2.2 Materials and Procedure
After providing informed consent and answering demographic questions (e.g., age, gender, education, etc.) participants were presented with the adaptive version of the Berlin Numeracy Test (Reference Cokely, Galesic, Schulz, Ghazal and Garcia-RetameroCokely et al., 2012)Footnote 2: This test has undergone extensive validation in large diverse and representative samples, and correlates robustly with other (longer) numerical ability, reasoning, and decision making skill inventories (e.g., Cokely et al., in press). The Berlin Numeracy Test provides an efficient means of sorting people into quartiles (normed against college educated adults from industrialized countries). In the task, participants are presented with narrative math problems that become harder (require greater numerical ability/reasoning) to solve as they are answered correctly (two to three questions are asked). Scores range from one to four with higher scores indicating greater statistical numeracy.
After taking part in unrelated tasks, participants were told that they would be making choices across several different pairs of options, and that the points (pts) they earned from these choices would be converted to a bonus payment (40 pts = $0.01). Participants were not shown the outcomes of their choices but they were added to their bonus earnings. They then made consequential choices between the fully described (see Figure 1 for a screenshot of a trial) pairs of options in Table 1 (presented in random order for each participant). Pairs consisted of a safer (less outcome variability) option paying out two modest amounts with nearly equivalent probabilities and a riskier (more outcome variability) option paying out a larger amount about a quarter of the time and a smaller amount more frequently: In half of the pairs the riskier option provided a higher EV than the safer option.
Screenshot of a decisions-from-description trial. Participants clicked on the option they wanted to play and the resulting outcome (not shown to participants) was added to their bonus earnings.
Intermixed with the target pairs were 11 pairs of options drawn from Reference Holt and LauryHolt and Laury (2002). These pairs were constructed to assess individual differences in risk aversion, but were employed in the current study to serve as distractors and to screen out inattentive participants (see Table A1 in the Appendix). Specifically, in one pair the safer option paid out 200 pts with certainty while the riskier option paid out 385 pts with certainty, while in another pair the safer option paid out 160 pts with certainty while the riskier option paid out 10 pts with certainty. Participants who chose the option of lower value (e.g., choosing the 10 pts over the 160 pts) in these pairs are excluded from all analyses since they were either not paying attention to the task or had not understood the task.
Following a brief personality questionnaire used as a filler taskFootnote 3 (Reference Gosling, Rentfrow and SwannGosling, Rentfrow & Swan, 2003), participants selected between the same pairs of options in a decisions-from-samples paradigm (Hertwig et al., 2005); pairs were presented in random order for each participant. In this task participants were told that they would be presented with two options and that they would need to sample from the options (i.e., simulate playing them) in order to see what outcomes they provided before making one consequential choice (see Figure 2). Participants were free to sample from each option zero to 100 times: The upper bound of a 100 was known to participants and rarely reached. As with the choices in the decisions-from-description task the outcomes of the consequential choices in the decisions-from-samples task were not told to participants but were added to their earnings. After making their last consequential choice participants were told of their total earnings across the two tasks and thanked for their time.
Example of the course of a decisions-from-samples trial. Participants first explored (sampled from) the options by clicking on them to simulate plays (Panel 1), seeing the outcome of each sample for half a second (Panel 2). When ready participants indicated they wished to make their one consequential choice (Panel 3) and then selected the option they wanted to play once with consequence (Panel 4). This sequence was repeated until all option pairs were encountered.
2.3 Exclusions
Observations were excluded for two reasons that were established before the study was conducted. First, trials where a participant did not sample from each option at least once (658 observations across 76 participants – four participants never sampled) were removed, as were their corresponding choices in the decisions-from-description format. These exclusions are made since samples of size zero indicate a complete lack of engagement in the task. Sample sizes of zero are included in the analyses examining the number of samples drawn.Footnote 4 Next, 26 participants who chose either (or both) of the dominated options (e.g., choosing 200 pts over 385 pts) in the Holt-Laury pairs were excluded (one participant chose an inferior option and never sampled). Thus, the final analysis includes 2,775 observations from 151 participants.
2.4 Results
2.4.1 Numeracy
Participants were fairly distributed across possible numeracy scores: Numeracy scores of one (low numeracy: 36 participants, 39 including three who never sampled in any trial), two (49 participants), three (24 participants), and four (high numeracy; 42 participants) were observed.
2.4.2 Sample size
Overall, individuals sampled a relatively small proportion of the available information (M = 12.49; CI95%[11.87, 13.12]) as is common in the decisions-from-samples paradigm (see for example studies conducted in the lab with higher pay and larger outcomes, Reference Ashby and RakowAshby & Rakow, 2014; Reference Hertwig and PleskacHertwig & Pleskac, 2010). The number of samples drawn across the two options ranged from zero to 200 (95% range [0, 31]; see the left panel of Figure 3). To test whether numeracy was related to the amount of information sought out the average number of samples drawnFootnote 5 was regressed on numeracy. Replicating previous findings (Reference LejarragaLejarraga, 2010), numeracy was positively related to the number of samples drawn, b = 2.09, t(152) = 1.99, p = .048, R 2 = .03 (see the right panel of Figure 3 and Table A2 in the Appendix). Removing observations where each option was not sampled from at least once reduced this relationship to non-significance, p = .17. Thus, it appears that the increase in sampling that is related to numeracy in the current study is primarily driven by less numerate individuals who did not sample from each option at least once before making a decision.
The histogram in the left panel displays the distribution of samples drawn across both options in each trial. The scatterplot in the right panel displays the relationship between participants’ average number of samples and their numeracy score; dashed line represents the identity line.
2.4.3 Exploration Strategy
Alternation Rate
To test whether numeracy influenced the rate of alternation (selecting different options on consecutive samples) I regressed the average rate of alternation on numeracy. As predicted, the rate of alternation was negatively related to numeracy, b = –.07, t(149) = –3.05, p = .003, R 2 = .06 (Figure 4 and Table A2 in the Appendix).
Scatterplot showing the relationship between participants’ average alternation rate and their numeracy score. Dashed line represents the identity line.
Sampling biases
To test whether numeracy increased the proportion of samples drawn from riskier options I created a variable indexing the proportion of total samples drawn from the riskier option for each individual and pair of options:
I then regressed the average proportion on numeracy and found a significant positive relationship, b = .02, t(149) = 3.49, p = .001, R 2 = .08 (Figure 5 and Table A2 in the Appendix). As predicted, more numerate individuals explored riskier options to a greater extent than less variable options.Footnote 6
Scatter plot displaying the relationship between the average proportion of samples a participant drew from the riskier option and their numeracy score. Dashed line represents the identity line.
2.4.4 Choice consistency
To test the prediction that the choices of those with greater numeric ability would align across decision formats to a greater extent I generated a variable indicating the average rate of choice alignment across the described and experiential formats (i.e., picking the riskier option in a pair in both the decisions-from-description and decisions-from-samples tasks). I then regressed this variable on numeracy.Footnote 7 As predicted, the relationship between numeric ability and the rate of choice consistency was significant, with more numerate individuals showing greater consistency across formats, b = .03, t(149) = 1.99, p = .048, R 2 = .03 (see the left panel of Figure 6 and Table A2 in the Appendix).
Scatter plots showing participants’ average rate of choice alignment across the decisions-from-description and decisions-from-samples formats and their numeracy score in Study 1 (free sampling, left panel) and Study 2 (fixed sampling, right panel). Dashed lines represent identity lines.
2.4.5 EV Maximization
In order to test whether individuals with greater numeracy selected higher EV options to a greater extent, two variables were constructed: The average rate at which a participant selected the option with a higher objective EV in the decisions-from-description format, and the average rate at which they selected the option with a higher experienced EV (i.e., the mean of the outcomes sampled from) in the decisions-from-samples format. Neither the relationship between numeracy and the rate of selecting the higher EV option in decisions-from-description (upper left panel Figure 7 and Table A2 in the Appendix), p = .85, nor in decisions-from-samples (upper right panel Figure 7 and Table A2 in the Appendix), p = .76, were significant.Footnote 8
Scatter plots showing the relationship between participants’ average rate of selecting options with higher expected value (EV) and their numeric ability in Study 1 (Free Sampling — upper panels) and Study 2 (Fixed Sampling — lower panels). Plotted separately for choices in the decisions-from-description (left panels) and decisions-from-samples (right panels) paradigms. Dashed lines represent the identity line.
3 Study 2
One reason numerate individuals’ preferences were more consistent across formats may be that their exploration strategies differed (i.e., increased sampling, alternating less, and drawing a larger proportion of samples from more variable options).Footnote 9 For example, sampling more and drawing a greater proportion of samples from variable options should lead to objective and experienced option EVs aligning to a great extent, increasing the likelihood that the same option would be preferred across the two formats. In order to test whether the mechanism behind this increased choice consistency for numerate individuals stems from a more skillful exploration strategy I conducted a follow up study where sampling was the same for all participants. To the extent increased preference consistency in the previous study directly resulted from numeracy’s impact on exploration, the relationship between consistency and numeracy may be reduced or eliminated when sampling is equated.
3.1 Participants
One-hundred and forty-four new participants (M age = 37.78; 50% male) were recruited from Amazon Mechanical Turk and completed the study in full. Approximately 8% of participants reported having a high-school education or GED, 33% some college, 8% a 2-year degree, 33% a 4-year degree, 14% a masters, 2% a professional degree, and 1% a doctoral degree. As in the previous study an attention check was included and those who failed were not allowed to take part in the study. Participants received $0.75 as well as a bonuses contingent on their decisions as in the previous study. The experimental session, which contained other unrelated studies, lasted approximately 30 minutes and participants earned roughly $3.00.
3.2 Changes to method and materials
Three changes were made: First, participants encountered only five pairs of options, to reduce repetition (see column S2 in Table 1). Second, instead of sampling freely from the available options, participants were shown 100 samples from each option. Specifically, either the safer or riskier option (counter balanced across participants) was drawn with the outcome of each draw being shown for 300 milliseconds. Importantly, the outcomes drawn from each option aligned perfectly with the outcomes’ objective probabilities, eliminating any potential sampling error. For instance, for the first riskier option in Table 1 25 draws returned an outcome of 126 pts while 75 draws returned an outcome of 8 pts. Outcomes were dispersed randomly across draws. Lastly, the Holt-Laury pairs from Study 1 were not included.
3.3 Results
3.3.1 Choice Alignment
As in the previous study the average rate of choice alignment was regressed on numeracy. Counter to the previous study there was no robust relationship between the rate of choice alignment and numeracy (right panel Figure 6 and Table A2 in the Appendix), b = .03, t(142) = 1.29, p = .19, R 2 = .01.Footnote 10
3.3.2 EV Maximization
As in the previous study the relationship between the rate of selecting the higher EV option and numeracy was examined. As before the relationship was not significant for decisions-from-description (bottom left panel Figure 7 and Table A2 in the Appendix), b = –.01, t(142) = –.41, p = .68, R 2 < .01, nor for decisions-from-samples (bottom right panel Figure 7 and Table A2 in the Appendix), b = .01, t(142) = .47, p = .64, R 2 < .01.
4 Discussion
The current studies examined whether statistical numeric ability (numeracy) influenced pre-decision search and preference consistency across described (decisions-from-description) and experiential formats (decisions-from-samples). As predicted, numerate individuals drew larger samples before making their final consequential choices. In addition, they sampled from options with higher variability more (i.e., riskier options) and alternated between options less during search. More numerate individuals also made choices that were more consistent across formats. Study 2 employed fixed sampling and failed to find greater choice consistency among participants with higher numeracy scores. Thus, the current studies provide novel evidence indicating that numeric ability influences exploration strategies, which in turn leads to greater preference consistency across decision formats. Nevertheless, decision strategies do not appear to be based on EV maximization strategies (see below). These findings further demonstrate that numeracy is an important individual difference underpinning decisions under conditions of risk and uncertainty, and provide additional insight into decision strategies that skilled decision makers employ (see discussion of Skilled Decision Theory; Cokely et al., in press; Garcia-Retamero & Cokely, in press).
One area where individual differences in numeracy could prove particularly informative is in the description-experience gap (i.e., the propensity to overweight rare events in descriptive formats while underweighting them in experiential formats; Reference Barron and ErevBarron & Erev, 2003; Hertwig et al., 2014). Given that numerate individuals’ choices aligned across formats in the current studies it would be interesting to see if this alignment is also found when choices are between safer (certain) and very risky options (i.e., options with extremely rare outcomes). Given that numerate individuals sampled more in general, and drew a greater proportion of their samples from riskier options, one might expect them to be more consistent between the two formats. This should be the case if limited sampling is indeed the primary driver behind the description-experience gap (Reference Hadar and FoxHadar & Fox, 2009; Reference Glöckner, Hilbig, Henninger and FiedlerGlöckner et al., 2016; Reference Rakow, Demes and NewellRakow et al., 2008).
More numerate participants did not show a greater propensity to select options with higher expected value (EV). In addition, there was no evidence that more numerate individuals “played the odds” more through increased risk seeking — which may be a defensible strategy when stakes are low and differences between accepting risk or playing it safe are minimal. This indicates that the benefits of numeracy in terms of making “better” choices is not apparent in all circumstances (Reference Peters, Västfjäll, Slovic, Mertz, Mazzocco and DickertPeters et al., 2006). Moreover, the current results suggest that more numerate individuals might actually fare worse than their less numerate cohorts if one assumes there are “costs” of investing more time and effort while obtaining little monetary benefit in return (Reference LejarragaLejarraga, 2010). However, to the extent that preference consistency is valuable, then these additional search costs might be viewed as reasonable (e.g., necessary in order to achieve well-considered preferences). As such, the current studies suggest there may be many opportunities for future research to map the boundary conditions that determine when and why numeric ability benefits or harms decision making under risk, broadly defined.
From a pragmatic standpoint, the current studies suggest potential ways in which choices made by those of lower numeracy might be made more consistent with their higher numeracy peers. Specifically, a growing body of work, including the results of present study, suggests that less numerate individuals explore their options less in general (Reference Garcia-Retamero and CokelyGarcia-Retamero & Cokely, 2017; Reference Hess, Visschers, Siegrist and KellerHess, Visschers, Siegrist & Keller, 2011; Reference Okan, Galesic and Garcia-RetameroOkan, Galesic & Garcia-Retamero, 2015). In addition, the current study highlights a novel effect whereby numerate individuals allot a greater proportion of their samples to riskier (more variable) options. As noted previously, increased sampling from options, particularly those with greater outcome variability, leads to estimates of worth that are more likely to align with their EVs. Importantly, Study 2 found that insuring all participants encountered the same amount of information diminished the effect of numeracy on choice alignment. Thus, one way to reduce differences between the decisions made by those with lower and higher numeracy is to ensure equivalency in the information encountered and encoded into memory (e.g., through transparent visual aids; Garica-Retamero & Cokely, 2013, 2017). As such, future research should continue to investigate ways that less numerate individuals can be nudged toward more efficient and deliberative information search, so that they too might become more independent and skilled decision makers (Reference Cokely, Galesic, Schulz, Ghazal and Garcia-RetameroCokely et al., 2012; in press; Baron, 1985, 2000).
Appendix
Note. Values calculated after all exclusion criteria applied and averaged within subject.
Note. |r| ≥ .18 for p<.05; |r| ≥ .14 for p<.10.
Open access
