1 Introduction
On your drive in to work you run into traffic. Do you “exploit” the route you have taken many times, a familiar option that is likely to make you 30 minutes late, or do you “explore” a road that you have not tried before, an unfamiliar option the outcome of which is unknown? Exploring could be bad if the road is long or you encounter an unanticipated obstacle; but if the road is faster, you could make up some lost time. Either way the information you gain from exploring could be useful the next time your commute is slow.
Such “explore-exploit” decisions are ubiquitous in daily life, from the mundane moments, such as picking out a TV show, to major life choices, such as selecting a spouse. Theoretically, computing an optimal solution to the explore-exploit dilemma is impossible in all but the simplest settings (Reference GittinsGittins, 1979), and there is a great deal of interest in how humans and animals solve the dilemma in practice (Reference Meyer and ShiMeyer & Shi, 1995; Reference Banks, Olson and PorterBanks, Olson & Porter, 1997; Reference Aston-Jones and CohenAston-Jones & Cohen 2005; Reference Daw, O’Doherty, Dayan, Seymour and DolanDaw, O’Doherty, Dayan, Seymour & Dolan, 2006; Reference Cohen, McClure and YuCohen, McClure & Yu, 2007; Reference Frank, Doll, Oas-Terpstra and MorenoFrank, Doll, Oas-Terpstra & Moreno, 2009; Reference Steyvers, Lee and WagenmakersSteyvers, Lee & Wagenmakers, 2009; Payzan-LeNestour & Bossaerts, 2011; Reference Lee, Zhang, Munro and SteyversLee, Zhang, Munro & Steyvers, 2011; Reference Payzan-LeNestour and BossaertsPayzan-LeNestour & Bossaerts, 2012; Reference Zhang and YuZhang & Yu, 2013; Reference Teodorescu and ErevTeodorescu & Erev 2013).
While the explore-exploit tradeoff has been defined in various ways (Reference NewellMehlhorn et al., 2015), here we focus on two particular types of exploration that we identified in a previous study (Reference Wilson, Geana, White, Ludvig and CohenWilson et al., 2014). In particular, our previous work suggests that humans solve the explore-exploit dilemma with a mixture of two strategies: directed exploration in which exploration is driven by explicit information seeking, and random exploration in which exploration is driven by chance through behavioral variability. A key insight in identifying these strategies was the idea that exploration should change with the number of choices that will be made in the future — the time “horizon”. When the horizon is long, it is often better to start by exploring, as there will be plenty of time in the future to make use of new information. When the horizon is short, it is often better to exploit a more familiar option, since there is less opportunity to learn and put that learning to use. With this horizon manipulation, we were able to identify directed exploration as an increase across horizon in selecting the more uncertain option, and random exploration as an increase across horizon in behavioral variability.
One limitation of our previous work was that, like most work on the explore-exploit dilemma, it focused exclusively on decisions in the domain of gains, where the subjects’ goal was to find the most rewarding option. However, many decisions, such as the traffic-jam example above, involve choices for which the goal is to minimize losses. Logically, decisions about gains and losses should be equivalent, but psychologically many human decisions are strongly influenced by the shift in outcome valence (Reference Daw, O’Doherty, Dayan, Seymour and DolanKahneman & Tversky, 1979).
One well-known asymmetry in how humans respond to reward valence is the reflection effect, whereby subjects are risk averse for gains and risk seeking for losses (Baucells & Villas 2007; Weller, Levn & Tversky, 1979; Reference EllsbergTversky & Kahneman, 1992). Similar effects are also seen in decisions under ambiguity in which subjects are ambiguity averse for gains but ambiguity seeking for losses (Reference Einhorn and HogarthEinhorn & Hogarth, 1986; Reference Kahn and SarinKahn & Sarin, 1988; Reference DiMauro and MaffiolettiDi Mauro & Maffioletti, 1996; Reference KuhnKuhn, 1997; Reference DiMauro and MaffiolettiDi Mauro & Maffioletti, 2002; Reference Ho, Keller and KeltykaHo, Keller & Keltyka, 2002; Reference Abdellaoui, Vossmann and WeberAbdellaoui, Vossmann & Weber, 2005; Reference Du and BudescuDu & Budescu, 2005; Reference Chakravarty and RoyChakravarty & Roy, 2009; Reference Davidovich and YassourDavidovich & Yassour, 2009). Unlike risk, for which the uncertainty arises from known outcome probabilities, ambiguity arises when outcome probabilities are unknown and is thus the main type of uncertainty associated with exploratory options. These findings for increased risk and ambiguity seeking in the losses domain suggest an overall increase in uncertainty seeking for losses. That is, decisions in the domain of losses are biased towards the option with the more variable outcome.
This switch in uncertainty preference between gains and losses suggests that subjects should show a bias toward the uncertain option in the loss domain, but that this bias will not change with the time horizon. This is because behavior driven by uncertainty seeking is not the same as exploration, which is driven by information seeking. Uncertainty seeking values outcome variability itself, irrespective of its potential future use; information seeking favors an uncertain option because it provides more information for the future. Therefore, information seeking scales with time horizon, as information becomes more valuable over a long horizon, but uncertainty seeking does not. Thus, manipulating the time horizon should provide a way to experimentally distinguish uncertainty seeking and exploration.
While Prospect Theory predicts increased uncertainty seeking in the loss domain, other possibilities exist. For example, a simple win-stay/lose-shift heuristic would predict more exploration in the domain of losses and, indeed, such a model was recently found to predict exploratory behavior better than Prospect Theory (Reference Lejarraga and HertwigLejarraga & Hertwig 2016). This finding was also consistent with previous work by the same authors showing that people spend more time searching in the loss domain in a purely information seeking task (Reference Lejarraga, Hertwig and GonzalezLejarraga, Hertwig & Gonzalez 2012). Another possibility, and one that we consider in detail here, is that people have different priors for gains vs. losses — that is they are more optimistic when the glass is half empty than when the glass is half full.
In this study we compared exploratory behavior and uncertainty seeking in the domains of gain and loss. We ran a modified version of our earlier experiment (Reference Wilson, Geana, White, Ludvig and CohenWilson et al., 2014) with the addition of a losses condition. In this task, subjects made a series of explore-exploit decisions between two options, both of which yielded either positive (gains condition) or negative (losses condition) outcomes. As before we quantified directed exploration as an increase across horizon in information seeking, an “information bonus”, and random exploration as an increase across horizon in behavioral variability, or “decision noise”. In addition, we quantified uncertainty seeking as the bias toward the more uncertain option that was independent of horizon.
Consistent with our previous work, we found directed and random exploration in the gains condition. We also found identical directed and random exploration in the losses condition. This suggests that both directed and random exploration strategies are used in the domain of losses in the same way as they are in the domain of gains.
In addition to directed and random exploration, we also found an overall increase in uncertainty seeking in the domain of losses that was not modulated by horizon. This uncertainty seeking is qualitatively consistent with the Prospect Theory prediction of increased uncertainty seeking with losses. However, a quantitative analysis revealed this bias to be inconsistent with predictions of Prospect Theory, but consistent with a Bayesian account in which subjects incorporate a prior that is optimistic in the case of losses and pessimistic in the case of gains.
In general, our findings are consistent with an interpretation in which explore-exploit decisions are driven by three independent processes: baseline uncertainty seeking that is driven by a prior that is optimistic for losses and pessimistic for gains, and directed and random exploration, driven by information seeking and decision noise, respectively, that scale with horizon irrespective of valence.
— Task design. Screenshots of four different games showing the gains condition (A & B) and losses condition (C & D), and the horizon 1 condition (A & C) and horizon 6 condition (C & D). In the gains condition, points are added to subjects’ scores, and in the losses condition, points are subtracted. The height of the bandits represents the game length, either 5 or 10 trials. The first four trials of every game are forced, wherein subjects are instructed which bandit to select. In the [1 3] unequal uncertainty condition illustrated here, subjects are instructed to choose one option once and the other three times. In the [2 2] equal uncertainty condition, not shown, subjects play both options twice. Free-choice trials are cued by a pair of green squares located inside the box of each bandit. Once a subject presses a button to choose a bandit, the lever of that bandit flips down and the number of points for that bandit is displayed, followed by the onset of any remaining free-choice trials.
2 Methods
2.1 Subjects
39 subjects (22 women, mean age 22.5, range 18–42) were recruited from the Princeton campus and surrounding area. Subjects were paid $12 for their time plus a performance bonus of up to $3.
2.2 Gains and losses task
The experiment was a version of the “Horizon Task” from Wilson et al. (2014) that was modified to include two valence conditions (gains or losses) for each subject. The valence conditions were divided into two blocks, with block order counterbalanced across subjects. The Horizon Task is used to measure directed and random exploration by comparing behavior across short and long time horizons.
Briefly, subjects played a series of games, each lasting either 5 or 10 trials, in which they made decisions between two slot machines or “one-armed bandits”, of the sort one might find in a casino (Figure 1). When played, each bandit yielded a probabilistic outcome that either added (in the gains condition) or subtracted (in the losses condition) points ranging from 1 to 99. This outcome was sampled from a Gaussian distribution (rounded to the nearest integer), the standard deviation of which was fixed throughout the experiment at 8 points. The mean of the underlying Gaussian changed from game to game but was fixed within each game. The mean of the Gaussian was different for each bandit and therefore one bandit was always better than the other on average. For each game, the mean of one option was set to a “center mean” of 40 or 60 points for the gains condition, and –60 or –40 for the losses condition; the mean of the other option was set relative to the mean of the first, with a difference in means of –30, –20, –12, –8, –4, 4, 8, 12, 20, or 30 points.
To control the information subjects had about the bandits before they made a choice, the first four trials of every game were “forced trials” in which subjects were instructed which option to select. This instruction was given by a green square in the next empty box of the bandit to be selected. Only after selecting the instructed bandit was the subject able to proceed. Once that bandit was selected, a number was revealed for that bandit, and “XX” was displayed for the other bandit to indicate that it had not been played on that trial. The four forced trials were used to set up one of two uncertainty conditions: an unequal uncertainty (or [1 3]) condition in which one option was played once and the other three times, and an equal uncertainty (or [2 2]) condition in which each option was played twice.
After the four forced trials, subjects were given at least one free choice between the two bandits, indicated by a green square in the next open slot on each bandit. Thereafter, either the game ended or they had 5 additional opportunities to select from the same bandits. Thus, after the forced trials, subjects had a horizon of either 1 (in games with 5 trials) or 6 (in games with 10 trials). This horizon manipulation eliminated the explore-exploit dilemma in horizon 1, while leaving it intact in horizon 6, allowing us to distinguish baseline uncertainty preference and decision noise (observed equally in horizon 1 and horizon 6) from changes in these factors associated with exploration (observed as increases from horizon 1 to horizon 6 in selecting the more uncertain option and in behavioral variability).
Finally, the games were organized into two separate blocks per subject, one involving gains and the other involving losses. The order of these blocks was counterbalanced across subjects. The horizon conditions and uncertainty conditions, as well as center mean, difference in mean, and the side of the ambiguous option (left or right) were all randomly interleaved (separately for each subject) and fully counterbalanced across each gains vs. losses block, such that there were 160 unique games in each block (2 horizon conditions x 2 uncertainty conditions x 2 center means x 10 differences in means x 2 sides). The side of the bandit (left or right) with a center mean of ±40 or ±60 versus the side of the bandit with a mean offset from this value, as well as the order of the forced trials (determining which trial in 1–4 the ambiguous option was revealed) were also randomized but not explicitly counterbalanced.
Instructions for the task were presented onscreen before the experiment began; see the Supplementary Material for the full text of the instructions. They were explicitly told that for a given game, one of the bandits would add more points (gains condition) or subtract fewer points (losses condition) on average, and hence be the better bandit to play. They were also explicitly told that the variability in the outcome from either bandit was fixed across the entire experiment. At the start of each block they were again reminded of the valence condition for that block.
2.3 Exclusion criteria
Subjects who performed at or below chance levels were excluded from further analyses. For each subject we used Bayesian inference to infer the frequency, f high, with which they chose the option with highest mean reward based on the samples they had actually seen. In particular, we computed the posterior distribution over f high given the data, p(f high ∣ α , β), as a beta distribution:
where α was the number of times subjects chose the high mean option and β was the number of times subjects chose the low mean option. We then excluded subjects for whom we inferred f high was less than or equal to 0.5 with greater than 1% probability. That is, subjects were excluded when
This test is equivalent to a Bayes factor comparing the model f high<0.5 to the model f high>0.5, and setting a cut-off of Bayes factor of 0.99 to classify subjects as contaminants. This resulted in the exclusion of 5 subjects, leaving 34 for the remaining analysis. Block order was counterbalanced across these 34 subjects.
2.4 Behavioral analysis
We analyzed behavior on the task in two different ways. First, we used a simple model-free analysis to illustrate our main findings. Second, we used a more sophisticated model-based approach to extract more sensitive estimates of uncertainty bias, information seeking, and decision noise. Both analyses lead to the same conclusions, so readers may skip over the sections on the model-based analysis and results without missing out on the main conclusions. In both analyses, we focused solely on the first free-choice trial because this is the only trial that can be fairly compared between horizon 1 and horizon 6 and because of a subtle confound that occurs between reward and information after the first free choice trial in horizon 6. In particular, because subjects tend to choose high reward options, on average they gain more information about the higher value option than the lower value option. This leads to a confound between the average reward of an option and the amount of information it yields that complicates the analysis in this and other explore-exploit experiments (see Wilson et al., 2014, for a more complete description of this reward-information confound).
2.4.1 Model-free analysis
In the model-free analysis, we focused on the first free choice and defined model-free measures of directed and random exploration in the following way. We measured the fraction of trials in which subjects chose the more informative (and also more uncertain) option in the [1 3] condition, p(high info). Directed exploration was measured as an increase in p(high info) with the time horizon. Decision noise was quantified as the fraction of trials in which subjects chose the low-mean option in the [2 2] condition, p(low mean). Random exploration was measured as an increase in p(low mean) with the time horizon. The intuition behind this measure of decision noise is that the more randomly people respond, the more likely they will be to choose the low-mean option in the [2 2] condition, when it is always best to choose the high-mean option.
— Graphical depiction of the hierarchical Bayesian model. In this plot, each node corresponds to a variable in the model. Shaded nodes correspond to observed variables (e.g. choices) and unshaded nodes correspond to hidden variables (e.g. information bonus or decision noise). Discrete variables are represented as squares and continuous variables as circles. The group variables, which are illustrated as different “plates,” have different values for different games, g, subjects, s, or conditions, n (defined by the valence, uncertainty and horizon). For each game, the observable data (shaded nodes) consisted of a choice, c nsg, the difference in mean between each option, Δ R nsg, and the difference in uncertainty between each option, Δ I nsg. The model estimates posterior distributions of both the single subject-level parameters: the information bonus, A ns, decision noise, σns, and spatial bias, B ns, and the group-level parameters: µnA, σnA, k nσ, λnσ, µnB and σBn.
2.4.2 Model-based analysis
We fit the behavioral data of the first free-choice trial using a logistic model. This model assumes that decisions are made by first computing a noisy value, Q a, for bandit a, and Q b for bandit b, then choosing the option with the highest value. In particular we assume that the value Q a is computed as the weighted sum of four factors: the expected reward from option a, R a, the uncertainty associated with playing it, I a, its spatial location (i.e. the bandit on the left vs. the bandit on the right), s a, and a logistic random noise term, n.
In this equation, A denotes bias towards more informative/uncertain options, which for simplicity we call the information bonus, B is the spatial bias, and σd the standard deviation of the decision noise. I a was chosen such that I a=+1/2 if option a was the more uncertain option (i.e. the option played once in the forced trials) in the [1 3] condition, I a=−1/2 if option a was the less uncertain option (the option played three times in the forced trials) in the [1 3] condition, and I a=0 in the [2 2] condition. s a was set to +1/2 if a was on the left-hand side of the screen and −1/2 if a was on the right.
If we assume that subjects choose the option with highest value and that by convention bandit a is always on the left (for exposition, though not in actuality), then the probability of choosing bandit a is
where Δ R = R b − R a is the difference between the mean observed outcomes from the forced-choice trials for bandits a and b. Δ I = I b − I a is the difference in expected uncertainty from choosing the two options, defined such that Δ I = 0 in the [2 2] equal uncertainty condition, Δ I = +1 in the [1 3] unequal uncertainty condition, when bandit b was more informative than bandit a (i.e. when bandit a had been selected three times and bandit b only once) and Δ I = −1, when bandit a was more uncertain than bandit b. Note that defining Δ I in this way sets A, the information bonus, in units of points.
— Performance plots. (A) Learning curves showing the fraction of responses in which subjects (solid lines) and the optimal model (green asterisks) chose the bandit with the greater underlying generative mean, as a function of free-choice trial number. (B) Performance of all 39 subjects in the gains and losses conditions. Five subjects performing at chance in either condition were excluded (red crosses), while the remaining 34 subjects performed equally well in both the gains and losses conditions.
We used a hierarchical Bayesian approach to fit the parameters of the model simultaneously at both the individual and group levels. A graphical depiction of this hierarchical model is shown in Figure 2, using the notation described in Reference Lee and WagenmakersLee and Wagenmakers (2013). At the individual level, we assumed that each subject, s, could have a separate information bonus, A ns, decision noise, σns, and spatial bias, B ns, in each of the eight conditions, n, defined by the valence, uncertainty, and horizon. At the group level, we assumed that these parameters for each subject and condition were sampled from population-level priors. Thus, the information bonus for each condition was sampled from a Gaussian distribution with mean µnA and standard deviation σnA, i.e.
The decision noise was sampled from a Gamma distribution with shape parameter k nσ and length scale λnσ and the spatial bias was sampled from a Gaussian with mean µnB and standard deviation σnB
The hyperparameters, µnA, σnA, k nσ, λnσ, µnB, and σnB are themselves assumed to come from hyperpriors whose parameters are set such that these hyperprior distributions are very broad and have relatively little influence on the final fits. In particular, the means µnA and µnB are assumed to come from zero-mean Gaussian distributions with standard deviation 1000; the standard deviations, σnA and σnB from Gamma distributions with shape parameter 1 and length scale 0.001; the shape and length-scale parameters k nσ and λnσ come from Exponential distributions with length scale 0.001.
All parameters (A ns, σns, B ns, µnA, σnA, k nσ, λnσ, µnB, and σnB) were fit simultaneously using a Markov Chain Monte Carlo (MCMC) approach to sample from the joint posterior. This was implemented using the JAGS sampler (Reference PlummerPlummer, 2003) via the Matjags interface (Reference SteyversSteyvers, 2011). In all we ran 4 separate Markov Chains with 500 burn-in steps to generate 1000 samples from each chain with a thin rate of 5. Convergence was assessed through visual inspection (see the Supplementary Material for serial plots of samples).
3 Results
3.1 Basic performance is similar for gains and losses
— Model-free measures of directed and random exploration. (A) The fraction of trials in which subjects choose the more uncertain option increases from horizon 1 to horizon 6, indicative of directed exploration. It also increases from gains to losses, but this does not interact with the horizon condition. This is consistent with increased uncertainty seeking in losses, but not a difference in directed exploration between gains and losses. (B) The decision noise (calculated as the fraction of trials in which the low-mean option was chosen) increases from horizon 1 to horizon 6, indicative of random exploration. There is no significant difference in decision noise between gains and losses. Data points are averaged across 34 subjects, with error-bars indicating the standard error of the mean.
We quantified performance as the fraction of times subjects chose the objectively correct option, i.e. the bandit who’s mean was actually highest. As shown in Figure 3A, the group-averaged performance for all subjects (i.e., before eliminating any subjects) was above chance (0.5) in all conditions, with performance improving over the course of horizon 6 games. Subjects were well below optimal performance, but gradually converged toward optimal throughout the course of a horizon 6 game (see the Supplementary Material for an explanation of how we computed optimal behavior).
Out of 39 subjects, the vast majority (34) achieved above chance performance in both valence conditions and performance in one condition was highly correlated with performance in the other (Figure 3B). The five subjects whose performance was at or below chance (as determined by Eq. 2) in one or both conditions were excluded from further analysis.
3.2 Directed and random exploration are equivalent for gains and losses, while uncertainty seeking is increased for losses
To measure the extent of directed exploration, random exploration, and baseline uncertainty seeking we looked at behavior on the first free-choice trial in the Horizon Task in the gains and losses conditions. To give a more complete picture of our data, and to illustrate the robustness of our findings, we performed both a model-free and model-based analysis of the behavior. The model-free analysis is much simpler and is perhaps more intuitive, but comes at the cost of reduced sensitivity and of ignoring variables. The model-based analysis offers more detailed and quantitative insights but at the cost of complexity. Our conclusions are similar for both types of analyses, and some readers may wish to skip the model-based results.
3.3 Model-free analysis
As outlined in the Methods section, we used two model-free metrics to capture directed and random exploration. We computed the fraction of trials in the [1 3] condition in which subjects chose the more uncertain option, p(high info). Because directed exploration involves information seeking, we compared this value across horizons. That is, directed exploration was measured as an increase in p(high info) from horizon 1 to horizon 6, when the information gained from selecting the more uncertain option is useful for future choices. Likewise, because decision noise leads subjects to make “mistakes” we quantified it as the probability of choosing the low mean option in the [2 2] condition, p(low mean). Random exploration was quantified as an increase in p(low mean) from horizon 1 to horizon 6.
A repeated measures ANOVA found a significant increase in p(high info) from horizon 1 to horizon 6 (F(1, 135) = 9.67, p = 0.0039), and from gains to losses (F(1, 135) = 6.00, p = 0.046, one-sided) (Figure 4A). There was no interaction between the valence condition and the horizon condition (F(1, 135) = 0.39, p = 0.54). The increase in p(high info) with horizon is indicative of directed exploration; when subjects are afforded extra trials to explore, they are more likely to choose the more uncertain option in order to gain information about it for future trials. The lack of interaction between valence and horizon indicates that the greater tendency to choose the more uncertain option with losses does not vary across horizons. This horizon-invariant shift towards choosing the uncertain option with losses is consistent with an overall bias toward uncertainty-seeking in the loss domain, but no difference in directed exploration between gains and losses.
For random exploration, a repeated measures ANOVA found a significant increase in p(low mean) from horizon 1 to horizon 6 (F(1, 135) = 56.72, p < 10−8) and no significant difference between gains and losses (F(1, 135) = 1.09, p = 0.30). This increase in decision noise from horizon 1 to horizon 6 is indicative of random exploration as behavioral variability increases when there is opportunity to explore.
3.4 Model-based analysis
To more precisely quantify our findings we used model fitting. We estimated posterior distributions for all parameters in the model for each subject in each condition, using hierarchical Bayesian estimation. For simplicity we focus on the group-level parameters that summarize the group means of the information bonus, µnA, and decision noise, k nσ / λnσ. These are shown in Figure 5, with error-bars indicating 95% credible intervals. The spatial bias was near zero across all conditions and is omitted.
— Model-based measures of directed and random exploration. Parameter fits, averaged across 34 subjects, with error-bars indicating 95% credible intervals across subjects. (A) As with the model-free results, the information bonus is greater in the losses condition than in the gains condition, and greater for horizon 6 than for horizon 1. Decision noise is greater in horizon 6 than in horizon 1 in both uncertainty conditions (B & C), and greater overall in the unequal uncertainty condition (B) than the equal uncertainty condition (C). Decision noise is not significantly different across the gains and losses conditions.
Both the information bonus and the decision noise increased from horizon 1 to horizon 6, in both valence conditions, suggesting that subjects use directed and random exploration for gains and for losses (information bonus and decision noise increased from horizon 1 to horizon 6 in 99.98% and 100% of samples, respectively). Decision noise was not reliably different between gains and losses (greater for losses in 17.8% of samples), and decision noise was greater in the [1 3] unequal uncertainty condition than the [2 2] equal uncertainty condition in 97.4% of samples.
The information bonus was different for gains and losses. In particular there was an overall increase in the bonus in the losses domain (in 98.7% of samples). Thus if we compute the posterior over the difference between the bonus for losses and gains in both horizon conditions (Figure 6) we see approximately the same change between losses and gains in both horizon 1 (losses > gains in 99.8% of samples) and horizon 6 (losses > gains in 88.0% of samples) although the distribution for horizon 6 is broader. The Bayes Factor for our [model versus a model the included an additional information bonus for the losses condition was 0.4997.
There was no interaction between valence and horizon conditions as shown by the fact that the difference in the posteriors for the weight on the more uncertain option in horizon 6 and horizon 1 was identical for both gains and losses (information bonus increased from horizon 1 to horizon 6 in 99.8% of samples from the gains condition, and in 99.6% of samples from the losses condition.).
— Posterior distributions showing the estimated information bonus is greater for losses than for gains. The difference between gains and losses in the posterior distributions of the information bonus shows that the estimated information bonus is greater for losses than for gains. This overall shift in the domain of losses is indicative of uncertainty seeking with losses.
3.5 Uncertainty seeking in the losses condition is consistent with Bayesian Shrinkage, not Prospect Theory
- The interaction between reward and uncertainty according to Prospect Theory (A & B) and a Bayesian Shrinkage hypothesis (C & D). (A) Under conditions of uncertainty about rewards, the average utility, ⟨ U(R) ⟩ (black dots), will deviate below the utility of the actual average, U(⟨ R ⟩) (gray dots) in the domain of gains, and above the utility of the mean in the domain of losses. (B) The difference | ⟨ U(R) ⟩ − U(⟨ R ⟩) | is larger in magnitude closer to zero. As a result, Prospect Theory predicts that for gains, the more uncertain option is more aversive than the less uncertain option for low-mean rewards, and less aversive for high-mean rewards; for losses, the more uncertain option is more favorable for small negative losses, and less favorable for large negative losses. (C) The Bayesian Shrinkage hypothesis postulates that the posterior estimate of reward is biased by a prior that is optimistic for losses and pessimistic for gains. (D) The difference between the mean of the posterior distribution and the mean of the likelihood distribution increases further from zero, and increases when the likelihood distribution is more uncertain. As a result, the Bayesian Shrinkage hypothesis predicts that for gains, the more uncertain option becomes more aversive than the less uncertain option as the mean reward increases; for losses, the more uncertain option becomes more preferable as losses increase.
The results of both the model-free and model-based analyses revealed an overall bias toward the more uncertain option in the domain of losses. This result is consistent with the classic predictions of Prospect Theory, in which humans seek uncertainty in the domain of losses and avoid uncertainty in the domain of gains. However, our results are also consistent with a Bayesian interpretation that we refer to as the “Bayesian Shrinkage” hypothesis. It should be noted that this Bayesian Shrinkage hypothesis is referring metaphorically to how people do inference, which is distinct from the formal Bayesian statistical analyses presented in this paper (Reference KruschkeKruschke 2010; Reference Lee, Zhang, Munro and SteyversLee 2011; Reference LeeLee 2016).
The Bayesian Shrinkage hypothesis assumes that subjects compute the expected value of each option, R, using Bayesian inference with some prior. This prior has the effect of biasing the estimated expected value towards the mean of the prior — it “shrinks” the estimate towards the mean of the prior. Because the effect of the prior is greater on more uncertain options, increased uncertainty seeking in the domain of losses could simply reflect a more optimistic prior, compared to the expectation of the gamble, for losses than for gains.
Intriguingly, despite making similar qualitative predictions about uncertainty seeking with losses, Prospect Theory and Bayesian Shrinkage make different quantitative predictions about the interaction between reward and uncertainty. In the following sections, we describe these predictions in detail and show how our results are consistent with the Bayesian Shrinkage account.
3.6 Prospect Theory and Bayesian Shrinkage make different predictions
— Model-free analysis of reward magnitude effect is consistent with Bayesian Shrinkage hypothesis. In both horizon conditions, for both losses (A) and gains (B) there is a negative association between mean reward and choice of the more uncertain option, p(high info). Error-bars indicate the standard error of the mean across subjects.
Prospect Theory posits that the change in uncertainty preference between gains and losses arises because of a change in the curvature of the utility function from concave for gains to convex for losses. As a result of concavity in the gain domain, the uncertainty in outcome causes the average of the utility, ⟨ U(R) ⟩, to be less than the utility of the average outcome, U(⟨ R ⟩) , and thus uncertain options are under-valued (Figure 7A). Conversely, as a result of convexity in the loss domain, uncertainty in outcome causes the average utility to be greater than the utility of the average, and thus uncertain options are over-valued. In both domains, the size of this uncertainty effect on utility (i.e. | ⟨ U(R) ⟩ − U(⟨ R ⟩) |) decreases with the magnitude of the reward (Figure 7B). This implies a positive relationship between reward and uncertainty seeking for both gains and losses. As we explain next, the Bayesian Shrinkage hypothesis makes the opposite prediction.
Bayesian Shrinkage explains the change in uncertainty preference between gains and losses by appealing to different prior distributions. If the prior for gains is pessimistic and the prior for losses is optimistic, then these priors cause uncertain options to be undervalued for gains and overvalued for losses (Figure 7C). This in turn leads to uncertainty aversion for gains and uncertainty seeking for losses. The size of the Bayesian Shrinkage effect increases as a function of the difference between the mean of the rewards and the mean of the prior (Figure 7D). This implies a negative relationship between reward and uncertainty seeking for both gains and losses.
Thus, Prospect Theory and Bayesian Shrinkage make opposite predictions about the interaction between uncertainty and reward. Specifically, Prospect Theory predicts an increase in uncertainty seeking with reward, while Bayesian Shrinkage predicts a decrease (Figure 7B, D). These predictions are straightforward to test with our data.
3.7 Model-free and model-based analyses favor Bayesian Shrinkage over Prospect Theory
To distinguish between Prospect Theory and Bayesian Shrinkage in a model-free way, we computed p(high info) separately for high and low reward magnitude trials. To do this we took advantage of the fact that one of the bandits always had a mean of magnitude either 40 or 60 points (see Methods), while the mean of the other bandit was set relative to this. We therefore defined high-magnitude trials as those in which the mean of the main bandit was ±60, and low-magnitude trials as those in which the mean of the main bandit was ±40. The null hypothesis asserts that p(high info) will be independent of the reward level. Prospect Theory asserts that p(high info) will be positively correlated with reward level. Thus, for losses, we expect to see less uncertainty seeking in the –60 than –40 games, while for gains, we expect less uncertainty seeking in +40 than +60 games. The Bayesian Shrinkage hypothesis asserts that p(high info) will be negatively correlated with mean reward, i.e. more uncertainty seeking for –60 than –40 in the losses condition and more uncertainty seeking for +40 than +60 in the gains condition.
We computed p(high info) separately for the eight different magnitude x horizon x valence conditions. The pattern of behavior is consistent with the Bayesian Shrinkage hypothesis with a negative relationship between mean reward and p(high info) (Figure 8A, B). In particular a repeated-measures ANOVA revealed a significant interaction between the gains/losses condition, and the high/low-magnitude condition (F(1, 135) = 10.40, p = 0.0029) in the model-free analysis. There was still a main effect of the valence condition (F(1, 135) = 6.0, p = 0.046, one-sided), and of the horizon condition (F(1, 135) = 9.67, p = 0.0039). There was no main effect of the magnitude condition (F(1, 135) = 0.48, p = 0.49) since it goes in opposite directions for gains and losses. For the losses trials alone, there was a main effect of the low/high-magnitude condition (F(1, 135) = 5.92, p = 0.020), and a trend to this main effect for gains trials alone (F(1, 135) = 3.46, p = 0.072).
For the model-based analysis we added a single factor to our model, an interaction term between uncertainty, Δ I, and mean reward, M. This interaction term allows us to determine whether people are more or less likely to choose the more uncertain option as the mean reward increases and thus distinguish between a Prospect Theory and Bayesian Shrinkage account. More specifically, in this updated model we rewrote the choice probabilities (Eq. 4) as
where γ denotes the strength of the interaction term and is predicted to be positive by Prospect Theory and negative by the Bayesian Shrinkage account.
As with the other variables in the model, we assume that γ is sampled from a group-level prior distribution that is potentially different for each subject and each condition. The group-level priors are assumed to be Gaussian with mean µγ and standard deviation σγ. Finally, the hyperparameters, µγ and σγ, are themselves drawn from broad hyperpriors (see Figure S1 for the graphical depiction of this model).
As before, we fit the model using a sampling based approach to compute posterior distributions for all parameters. This clearly shows that the mean of the group-level prior, µγ, is negative in all conditions (Figure 9) with 100% of samples below zero apart from the horizon 6 gains condition where 95.4% of samples are below zero. As discussed above, this negative value is inconsistent with Prospect Theory and provides strong support for the Bayesian Shrinkage hypothesis.
— Distribution of group-level mean of the mean reward scale factor, µγ. In all conditions the µγ is less than zero with high probability, providing strong support for the Bayesian Shrinkage hypothesis.
This model that includes the interaction term was strongly favored over the model without such a term, by 453 points using the deviance information criterion (6,862 for the more sophisticated model versus 7,315 for the more basic model, across 34 subjects). This supports the notion that it is important to account for the interaction between uncertainty and reward.
4 Discussion
In this study we quantified the effects of reward valence (gains vs. losses) on human explore-exploit decision making. Consistent with our previous work (Reference Wilson, Geana, White, Ludvig and CohenWilson et al., 2014), we found evidence that people use two distinct strategies for exploration: directed exploration in which exploration is driven by an increase across horizons in information seeking, and random exploration in which exploration is driven by decision noise that increases with horizon. Moreover, the extent to which subjects used these two strategies was quantitatively comparable in the gains and losses conditions.
In addition we found an overall increase in uncertainty seeking in the domain of losses that initially appeared consistent with the predictions of Prospect Theory (Reference Daw, O’Doherty, Dayan, Seymour and DolanKahneman et al., 1979; Reference Einhorn and HogarthEinhorn et al., 1986; Reference Kahn and SarinKahn et al., 1988; Reference EllsbergTversky et al., 1992; Reference DiMauro and MaffiolettiDi Mauro et al., 1996; Reference KuhnKuhn, 1997; Reference DiMauro and MaffiolettiDi Mauro et al., 2002; Reference Einhorn and HogarthHo et al., 2002; Reference Abdellaoui, Vossmann and WeberAbdellaoui et al., 2005; Reference Du and BudescuDu et al., 2005; Reference Baucells and VillasísBaucells et al., 2009; Reference Chakravarty and RoyChakravarty et al., 2009; Davidovichet al., 2009). However, a more detailed, quantitative analysis revealed an interaction between uncertainty and reward magnitude that was inconsistent with Prospect Theory, but that could be explained with the Bayesian Shrinkage hypothesis in which people have a prior that is optimistic for losses and pessimistic for gains.
An obvious question is why do we see this departure from Prospect Theory? One reason may be that, unlike classical tests of Prospect Theory, the uncertainties in our task are not explicitly described but instead must be inferred from experience. A number of authors have found that decisions based on described uncertainties can be quite different from decisions based on uncertainties that are experienced (e.g. Reference Ludvig and SpetchLudvig & Spetch, 2011; Reference DiMauro and MaffiolettiHertwig et al., 2004; Reference Banks, Olson and PorterBarron & Erev, 2003). However, it is important to note that recent work looking at decisions under ambiguity do not see differences between gains and losses when the decision is made under experience (Reference Dutt, Arló‐Costa, Helzner and GonzalezDutt et al., 2014; Guney & Newell, 2015). Clearly more work will be required to understand why our experiment gives different results.
One result from the description-experience literature that is inline with our findings is the paper of Reference Teodorescu and ErevTeodorescu & Erev (2013) who used an explore-exploit task that combined both gains and losses. In this experiment, the authors found behavior that was consistent with increased uncertainty seeking for losses. However, because losses and gains were always present together — i.e. there were no separate losses-only or gains-only conditions — it was impossible to separate the effects of losses from gains. Our work is also qualitatively consistent with Reference Lejarraga and HertwigLejarraga & Hertwig (2016) who found that exploratory behavior in the loss domain was better explained by a simple win-stay/lose-shift heuristic strategy than by loss aversion. While our task, with continuously valued rewards, is not well suited to studying win-stay/lose-shift behavior, it will be important to reconcile these two findings in future work. Finally, our work is consistent with Reference Yechiam, Zahavi and ArditiYechiam, Zahavi & Arditi (2015) who found increased exploration in the domain of losses. Intriguingly, this latter paper also found a hysteresis effect whereby past exposure to losses increased exploration even when the losses where no longer present. Such a long-term effect of past losses may be driven by the “break-even” effect (Reference Thaler and JohnsonThaler & Johnson, 1990) whereby people who have experienced a loss are more likely to choose an uncertain option if it offers the chance to erase the past loss. While our experiment did not offer any possibility of breaking even in the losses condition, future work should examine this idea more closely.
Another task that combines gains and losses in an explore-exploit setting is the Iowa Gambling Task (IGT) (Reference Bechara, Damásio, Damásio and AndersonBechara et al., 1994). In this task, subjects choose between four decks of cards. Each deck contains both winning and losing cards and the relative amount of gains and losses varies between decks such that two decks are winning on average while the other two are losing. In recent years, a number of authors have modeled behavior in this task in detail (e.g. Wetzels et al., 2010; Reference Worthy, Hawthorne and OttoWorth et al., 2013). While the results of these modeling efforts are not directly comparable with our findings here, it would be interesting to include factors for directed exploration and a Bayesian prior in models of the IGT.
At the neural level, a question of particular interest for future work is how these decisions are processed in the brain. Neuroimaging studies have identified areas of the brain involved in value-based decision making (Reference Rangel, Camerer and MontagueRangel, Camerer & Montague, 2008; Reference Christopoulos, Tobler, Bossaerts, Dolan and SchultzChristopoulos, Tobler, Bossaerts, Dolan & Schultz, 2009; Reference Hare, Camerer and RangelHare, Camerer & Rangel, 2009; Reference Kahnt, Heinzle, Park and HaynesKahnt, Heinzle, Park & Haynes, 2011), the representation of monetary gains versus losses (Reference Breiter, Aharon, Kahneman, Dale and ShizgalBreiter, Aharon, Kahneman, Dale & Shizgal, 2001; Reference Gehring and WilloughbyGehring & Willoughby, 2002; Reference DeMartino, Kumaran, Seymour and DolanDe Martino et al., 2006; Reference Yacubian, Gläscher, Schroeder, Sommer, Braus and BüchelYacubian et al., 2006; Reference Seymour, Daw, Dayan, Singer and DolanSeymour et al., 2007; Reference Weller, Shiv and BecharaWeller et al., 2007; Reference San, Appelbaum, Pearson, Huettel and WoldorffSan Martín et al., 2013), and exploration versus exploitation (Reference Daw, O’Doherty, Dayan, Seymour and DolanDaw et al., 2006; Reference Laureiro-Martínez, Brusoni and ZolloLaureiro-Martínez, Brusoni & Zollo, 2010; Reference Badre, Doll, Long and FrankBadre, Doll, Long & Frank, 2012). In light of our finding that directed exploration, random exploration, and uncertainty seeking are independent, additive terms in the value computation, neuroimaging could reveal whether these brain regions also function in a summative way under explore-exploit decisions.
Finally, it is important to note that, despite being due to uncertainty seeking rather than exploration, subjects did choose the uncertain, and hence more informative option more in the losses condition of our task. While this did not lead to a significant difference in performance in our experiment (Figure 3), it would be interesting to investigate whether this bias for losses is advantageous in some settings, or whether it is simply a suboptimal bias in human decision making.
Open access
