2.1 Main part of the experiment

After playing the risk attitude part described in more detail below, in the main part of the experiment subjects played 7 stages, each with 8 rounds, thus generating $T=56$ observations. At the beginning of each stage the computer randomly chose one of two urns (Urn 1 or Urn 2), with Urn 1 being selected at a known probability of 0.6. Each urn represents a different state of the world. While this prior probability was known and it was known that the urn would remain the same throughout the stage, the chosen urn was not known to subjects. It was known that Urn 1 had seven white balls and three orange balls, and Urn 2 had three white balls and seven orange balls. At the beginning of each of the 8 rounds (round = t), there was a draw from the chosen urn (with replacement) and subjects were told the color of the drawn ball. These were therefore signals that could be used by subjects to update their beliefs.Footnote ⁵ It was made clear to the subjects that the probability an urn was chosen in each of the seven stages was entirely independent of the choices of urns in previous stages. A visual representation of the urns was provided on the computer screens to facilitate understanding (see the computer screens file).

Once they saw the draw for the round, subjects were asked to make a probability guess between 0 and 100%, on how likely it was that the chosen urn was Urn 1. The corresponding variable for analysis is their probability guess expressed as a proportion, denoted g. Once a round was completed, the following round started with a new ball draw, up to the end of the 8th round.

Payment for the main part of the experiment was based on the guess made in a randomly chosen stage and round picked at the end of the experiment. A standard quadratic scoring rule (e.g. Davis and Holt Reference Davis and Holt1993) was used in relation to this round to penalize incorrect answers. The payoff for each subject was equal to 18 GBP minus 18 GBP $\times$ ${\left(g,u,e,s,s,,-,,c,o,r,r,e,c,t,,p,r,o,b,a,b,i,l,i,t,y\right)}^2$ . Therefore, for the randomly chosen stage and round, subjects could earn between 0 and 18 GBP depending on the accuracy of their guesses. It was clarified to subjects that “if the chosen urn was Urn 1, then the correct probability of the chosen urn being Urn 1 is 100%; if the chosen urn was Urn 2, then the correct probability of the chosen Urn being Urn 1 is 0%”. While instructions were generally provided on the computer screen, a table with payoffs for each level of accuracy of the guesses was provided in print, to facilitate understanding (see online appendix 3). Table 8 in “Appendix 5: Robustness and understanding” includes a regression model with maths ability as an explanatory variable, which we measure in the C treatment. This variable is insignificant even in the treatment where potentially it should have mattered the most, which undermines its relevance. Furthermore, subjects could make use of a calculator. Specifically, a ‘calculate consequences’ button gave subjects ready information about the payoffs arising from their guess, depending on which urn was drawn.

2.2 Risk attitude part of the experiment

The risk attitude part was similar to the main part but simpler and therefore genuinely useful as practice. It was modelled after Offerman et al. (Reference Offerman, Sonnemans, Van De Kuilen and Wakker2009) to enable us to infer people’s risk attitude, as detailed in Sect. 3.

It consisted of 10 stages with one round each. In each stage a new urn was drawn (with probabilities 0.05, 0.1, 0.15, 0.2, 0.25, 0.75, 0.8, 0.85, 0.9, 0.95).Footnote ⁶ Subjects were told the prior probability of Urn 1 being chosen but did not receive any further information. In particular, no balls were drawn. The guessing task (single round) was to nominate a probability that Urn 1 was chosen, where subjects could rely on the information available to them and the payment mechanism to identify which probability guess would maximize their expected utility. For subjects who are not risk neutral, the task is non-trivial as they should take account of the payoff structure rather than repeat the announced probabilities. Payment for the risk attitude of the experiment was based on the guess made in a randomly chosen round picked at the end of the experiment. A quadratic scoring rule was applied as in the main part, but this time this was equal to 3 GBP minus 3 GBP $\times$ ${\left(g,u,e,s,s,,-,,c,o,r,r,e,c,t,,p,r,o,b,a,b,i,l,i,t,y\right)}^2$ .Footnote ⁷ Again, it was clarified to subjects that the correct probability was 0 or 1 depending on the urn being chosen, and again a table with payoffs for each level of accuracy of the guesses was provided in print (see online appendix 3) and a calculator was also available.

2.3 Experimental treatments

There were three treatments. The risk attitude parts were identical across all treatments, and the main part of the Baseline treatment (B) was as described.

In the main part (only) of the Complexity treatment (C), the information on the ball drawn from the chosen urn at the beginning of each round was presented as a statement about whether the sum of three numbers (of three digits each) is true or false. If true (e.g., $731+443+927=2101$ ), this meant that a white ball was drawn. If false (e.g., $731+443+927=2121$ ), this meant that an orange ball was drawn.

In the main part (only) of the Inattention treatment (I), subjects were given a non-incentivized alternative counting task which they could do instead of working on the probability. The counting task was a standard one from the real effort experimental literature (see Abeler et al. Reference Abeler, Falk, Goette and Huffman2011, for an example) and consisted in counting the number of 1s in matrices of 0s and 1s. Subjects were told that they could do this exercise for as little or as long as they liked within 60 s for each round, and that we were not asking them in any way to engage in this exercise at all unless they wanted to.Footnote ⁸

As in Caplin et al. (Reference Caplin, Csaba, Leahy and Nov2020) and the key treatments in Sitzia et al. (Reference Sitzia, Zheng and Zizzo2015), we see as important not to incentivize the alternative task. If the alternative task were incentivized—financially or in terms of doing something fun such as browsing the internet—, it would be rational for an agent to split his or her time allocation between tasks, which would trivially imply worse decision making in the guessing task. This would make it difficult to precisely identify what is due to inattention as a psychological mechanism, and could be construed to simply reflect the fact that agents are bad at multitasking (e.g., Buser and Peter Reference Buser and Peter2012). In our setting, instead, agents should just focus on a single task and not be distracted by the alternative task.Footnote ⁹ Undoubtedly, future research could change the incentives associated to the alternative task.

3.1 Model variables and risk attitude correction

In this section, we build a model of subject play in our experiment where the key driver is the way subjects form beliefs. Table 2 lays out the main variables from the experiment and the interrelationships between them, when the event is described in terms of the chosen urn (row 1) and when it is described in terms of the probability of a white ball being drawn (row 2). The two descriptions are equivalent since the subject’s subjective guess of the probability that Urn 1 was chosen generates an implied subjective probability that a white ball is drawn.Footnote ¹⁰ In our modelling of this experimental environment, we sometimes use the former probability—that Urn 1 was chosen—and it will be useful to transform this probability guess using the inverse cumulative Normal distribution (so that the support has the same dimensionality as a classic z-statistic). Alternatively, we sometimes describe agents’ guesses in terms of the probability that a white ball is drawn because the sample proportion from repeated Bernoulli trials has a tractable sampling distribution for hypothesis testing.

Table 2

Model variables

Event	Estimator of event probability	Estimator symbol	P(Event) Subject guesses	Transformed guess	Strength of evidence $z_t$ against earlier choice at m
Urn 1 drawn	Bayes rule	$P_t$	$g_t:$ optimal guess (observed) $g_t^{\ast }:$ true guess (inferred)	$r_t^{\ast }={\Phi }^{-1}\left(g_t^{\ast }\right)$	$z_t=\frac{2({\Phi }^{-1}\left(P_t\right)-{\Phi }^{-1}\left(P_m\right))}{\sqrt{t}}$
White drawn	Prop. white balls out of t	$P_t^w$	Not guessed	Not guessed	$z_t\approx \frac{P_t^w-P_m^w}{\sqrt{0.5^2/t}}$

Time is measured by t, the draw (round) number for the ball draws in each stage. We define the value of t for which subjects last moved their guess (viz. updated their beliefs) to be m (for ‘last move’). Thus, for any sequence of ball draws at time t, the time that has elapsed since the last change in the guess is always $t-m$ .

Along the top row the theoretical estimator for the probability that Urn 1 was drawn is provided by Bayes rule, which we denote by $P_t$ after t ball draws. Many subjects do not use Bayes rule when they are guessing the probability that Urn 1 is chosen, though some guesses are closer to it than others.

As derived in Offerman et al. (Reference Offerman, Sonnemans, Van De Kuilen and Wakker2009), the elicited guess $g_t$ in the fourth column is the result of maximizing expected utility based on a Constant Relative Risk Aversion (CRRA) utility function, $U\{$ Payoff $\}$ :

$\begin{array}{c}U\left\{\text{Payoff}\right\}=\frac{{\text{Payoff}}^{1-\theta }-1}{1-\theta },\end{array}$

and a true guess $g_t^{\ast }$ in $E\left(U,\left\{,\text{Payoff},\right\}\right)=g_t^{\ast }U\{1-{\left(1,-,g_t\right)}^2\}+(1-g_t^{\ast })U\{1-g_t^2\}$ where the payoffs for Urn 1 and Urn 2 are proportional to $1-{\left(1,-,g_t\right)}^2$ and $1-g_t^2$ , according to the quadratic scoring rule, as explained in Sect. 2.Footnote ¹¹ Expected utility is assumed to be maximized with respect to $g_t$ and yields the following relationship between $g_t^{\ast }$ and $g_t$ :

(1) $\begin{array}{c}ln\left(\frac{g_t^{\ast }\left(1,-,g_t\right)}{g_t\left(1,-,g_t^{\ast }\right)}\right)=\theta ln\left(\frac{g_t\left(2,-,g_t\right)}{\left(1,+,g_t\right)\left(1,-,g_t\right)}\right).\end{array}$

In the risk attitude part, the prior probabilities given to the subjects (by way of reminder, 0.05, 0.1, 0.15, 0.2, 0.25, 0.75, 0.8, 0.85, 0.9, 0.95 for 10 separate stages/rounds) are in fact the correct probabilities $P_t$ . We see no reason not to credit subjects with realizing this, and they possess no other information anyway, so we define their true guess to be $g_t^{\ast }=P_t$ . Offerman et al. (Reference Offerman, Sonnemans, Van De Kuilen and Wakker2009) then interpret the deviations of $g_t$ from $g_t^{\ast }$ as being due to the subjects’ risk preferences, and so do we. We use the ten datapoints $\left(g_t^{\ast },,,g_t\right)$ for each subject to estimate $\theta$ in a version of (1) appended with a regression error.Footnote ¹² Armed with a subject-specific value of $\theta$ from the risk attitude part, all the observable $g_t$ values in the main experiment can be transformed to a set of inferred $g_t^{\ast }$ . This transformation is accomplished by exponentiating both sides of (1), and solving for $g_t^{\ast }$ . By taking the inverse cumulative Normal function, ${\Phi }^{-1}$ , of $g_t^{\ast }$ we move it outside the [0, 1] interval and give it the same dimensionality as a z-test statistic, namely $(-\infty ,\infty )$ . The variable in the penultimate column, $r_t^{\ast }={\Phi }^{-1}\left(g_t^{\ast }\right)$ , thus becomes the basis for our econometric analysis.Footnote ¹³

We will explain the last column of Table 2 and why it is useful later.

3.2 Expectation processes

Using the notation of Table 2, we define three processes of expectation formation that will be relevant for our double hurdle model in Sect. 4.

3.2.1 Rational expectations

The rational expectations solution predicts straightforward Bayesian updating. The (conditional) probability the subject is being asked to guess is the rational expectation (RE), which is given by $P_t$ . Calling $P_{\mathrm{initial}}$ the initial prior probability and noting that the number of white balls is $t{P}_t^w$ we can write down $P_t$ in a number of ways:

(2) $\begin{array}{cc}{P}_t& =\left(\frac{{\left(0.7\right)}^{t{P}_t^w}{\left(0.3\right)}^{t-t{P}_t^w}}{{\left(0.7\right)}^{t{P}_t^w}{\left(0.3\right)}^{t-t{P}_t^w}{P}_{\mathrm{initial}}+{\left(0.3\right)}^{t{P}_t^w}{\left(0.7\right)}^{t-t{P}_t^w}\left(1,-,P_{\mathrm{initial}}\right)}\right)P_{\mathrm{initial}}\\ & =\frac{1}{1+\frac{{\left(0.3\right)}^{t{P}_t^w}{\left(0.7\right)}^{t-t{P}_t^w}\left(1,-,P_{\mathrm{initial}}\right)}{{\left(0.7\right)}^{t{P}_t^w}{\left(0.3\right)}^{t-t{P}_t^w}{P}_{\mathrm{initial}}}}.\end{array}$

The second line is a useful simplification (which we use in “Appendix 1: Closeness of two strength-of-evidence measures”) whereas the bracketed fraction in the first line is the probability of obtaining the $t{P}_t^w$ white balls when Urn 1 is drawn versus the total probability of obtaining this number of white balls. We called this the likelihood ratio in Table 1.

3.2.2 Quasi-Bayesian updating

In our version of Quasi-Bayesian updating (QB), agents use Bayesian updating as each new draw is received, but they incorrectly weight the likelihood ratio:

(3) $\begin{array}{c}{P}_t^{\mathrm{QB}}={\left(\frac{{\left(0.7\right)}^{t{P}_t^w}{\left(0.3\right)}^{t-t{P}_t^w}}{{\left(0.7\right)}^{t{P}_t^w}{\left(0.3\right)}^{t-t{P}_t^w}{P}_{\mathrm{initial}}+{\left(0.3\right)}^{t{P}_t^w}{\left(0.7\right)}^{t-t{P}_t^w}\left(1,-,P_{\mathrm{initial}}\right)}\right)}^{\beta }{P}_{\mathrm{initial}}\end{array}$

The parameter $\beta$ may be thought of as the QB parameter: if $\beta =1$ , agents are straightforward Bayesians; if $\beta >1$ they overuse information and under-weight priors; if $0\le \beta <1$ they underuse information and over-weight priors and if $\beta <0$ they respond the wrong way to information—raising the conditional probability when they should be lowering it, and vice versa.

Agents’ attitude towards the extent of belief change in the light of evidence can be summarized by the distribution $f\left(\beta \right)$ across subjects. If $f\left(\beta \right)$ has most probability mass between 0 and 1, most agents only partially adjust, and subjects converge to full adjustment at $\beta =1$ to the extent that the probability mass in $f\left(\beta \right)$ converges towards unity.

3.2.3 Inferential expectations

Cohen et al. (Reference Cohen, Henckel, Menzies, Muhle-Karbe and Zizzo2019) show that models with cost-based state-dependent sticky belief adjustment are equivalent to an inferential expectations (IE) model, where agents’ hypothesis testing generates infrequent belief adjustment (Menzies and Zizzo Reference Menzies and Zizzo2009).Footnote ¹⁴ We show this in our specific context in “Appendix 2: Relationship between inferential expectations and switching cost models”. We therefore are in a position to model the degree of state-dependent belief stickiness by the test size, where a low test size implies a higher cost of changing beliefs and therefore relatively infrequent adjustment.

We assume subjects hold a belief until enough evidence has accumulated to pass a threshold of statistical significance, at which point beliefs are updated. Agents form a belief and do not depart from that belief until the weight of evidence against the belief is sufficiently strong. Under IE, each agent is assumed to start with a belief about the probability of U (that is, $P_0=0.6$ ) and its implied probability of a white ball ( $P_0^w=0.54$ ), and conducts a hypothesis test that the latter is true after drawing a test size from his or her own distribution of $\alpha$ , namely $f_i\left(\alpha \right)$ . Agents are assumed to draw this every round during the experiment.

In the first row of the final column of Table 2, we provide a measure $z_t$ of the strength of evidence against the probability guess at the time of the last change. Agents change their guesses from time to time, and $z_t$ tells us if the value of P at the last change, denoted $P_m$ , seems mistaken in the light of subsequent evidence.

Importantly, as shown in “Appendix 1: Closeness of two strength-of-evidence measures”, the top-row $z_t$ is approximately equal to the standard test statistic for a proportion, shown in the second row of the final column of Table 2, using the maximal value of the variance of the sampling distribution (namely ${\left(\frac{1}{2}\right)}^2$ ):

(4) $\begin{array}{c}{z}_t\approx \frac{P_t^w-P_m^w}{{}^{{\left(0,.,5^2,/,t\right)}^{1/2}}}.\end{array}$

Thus, the p value for IE can be derived from (4) as the test statistic. We assume for simplicity that $z_t$ is distributed as a standard Normal.

Later in the paper we derive the full distribution of $\alpha$ so as to let the data adjudicate each agent’s attitude towards evidence. If $f_i\left(\alpha \right)$ has most probability mass near zero, agent i exhibits sticky belief adjustment. Probability mass in $f_i\left(\alpha \right)$ near unity implies a willingness to update for any evidence. In the limit, as $\alpha$ approaches unity, agents will update regardless of evidence (or, more precisely, even for zero evidence against the null). This is equivalent to (stochastic) time dependent updating. That is, if the probability mass at unity in $f_i\left(\alpha \right)$ is, say, 0.3, it implies that there is a thirty per cent chance that agent i will update regardless of what the evidence says. More formally, the decision rule in a hypothesis test is to reject $H_0$ , the status quo, if the p value $\le \alpha$ . A value for $\alpha$ of unity implies the status quo will be rejected, which is the same as updating in this context, for any p value whatsoever.

4.1 Nonparametric analysis

The baseline and complex treatments each had 82 subjects, and the inattention treatment had 81 subjects. In this sub-section, we motivate our model with nonparametric analysis.

First, we consider the number of times our subjects executed a no-change, meaning a guessed probability equal to that of the previous period. This is interesting because, given the nature of the information and comparatively small number of draws, incidences of no-change are not predicted by either Bayesian or Quasi-Bayesian updating, and so, if such observations are widespread in the data, this is the first piece of nonparametric evidence that these standard models are incomplete.

The maximum of the number of no-changes for each subject is 49: seven opportunities for no change out of eight draws, times the seven stages. The distributions over subjects separately by treatment are shown in Fig. 1. The baseline distribution shows a concentration at low values; for both Complex and Inattention, there appears to be a shift in the distribution towards higher values, as one might expect. The means for each treatment are represented by the vertical lines on the right hand side. The vertical lines on the left hand side show the mean number of no-changes that would result from subjects rounding the Bayesian probabilities to two decimal places (or, equivalently, rounding percentages). Clearly, rounding cannot account for the prevalence of no-changes found in the empirical distribution.

Fig. 1

Distributions of number of no-changes over subjects separately by treatment. In each panel, the vertical line on the right represents the mean number of no-changes per subject, while the vertical line on the left would be the outcome of Bayesian behaviour if subjects rounded their answers

The mean is higher under C (22.79) than under B (18.74) (Mann–Whitney test gives $p=0.007$ ); and higher under I (26.97) than under B ( $p<0.001$ ).Footnote ¹⁵ This is expected: complexity and inattention are both expected to increase the tendency to leave guesses unchanged. When C and I are compared, the p value is 0.06, indicating mild evidence of a difference between the two treatments.

In Fig. 1 it is clear from the nonparametric evidence of widespread incidence of no-changes that any successful model of our data will have to deal with the phenomenon of whether to adjust, before considering how much to adjust.

Second, when agents do change, it is of interest why they do. In Fig. 2 we plot the binary indicator for updating against the strength of evidence against the maintained beliefs $\left(z_{\mathrm{it}}\right)$ (the absolute change in the Bayesian posterior since the last time the subject updated; see top of row Table 2). A Lowess smoother is superimposed, and this can be interpreted as the predicted probability of an update for a given value of $\left(z_{\mathrm{it}}\right)$ . This provides good nonparametric evidence that higher values of $\left(z_{\mathrm{it}}\right)$ make change more likely, but, across subjects, agents who are more reluctant to change will exhibit both a relatively low probability of update and a higher value of $\left(z_{\mathrm{it}}\right)$ . Hence there is an econometric concern that the relationship may be affected by endogeneity bias.Footnote ¹⁶

Fig. 2

Predicted probability of updating against strength of evidence (the latter measured as the absolute value of z, defined in Table 2). The dots represent individual decisions to update (1 = update; 0 = no update). The lines are Lowess smoothers, obtained using a tricube weighting function and bandwidth 0.8 (both STATA defaults). Left panel: smoother obtained for full sample. Right panel: smoother obtained separately by treatment

Third, Fig. 3 shows the extent of any updating on receipt of a white ball and an orange ball, both as a raw change and as a proportion of the absolute change dictated by Bayes rule. The upper panels indicate that updates are often in step sizes of 0.1 or 0.05 and the 0.6 prior for $P_t$ was not so asymmetric as to generate artefacts.

Fig. 3

Change in guess: raw, and as a proportion of a Bayesian benchmark. The top panels show the raw size of the updates on receiving a ball of each color. The bottom panels show the proportional size of the updates on receiving a ball of each color: this is defined as the actual update on receiving a ball as a proportion of the absolute correct Bayesian update (assuming the subject starts from the Bayesian prediction). A vertical line is drawn in correspondence to 0 (no update) and, for the bottom panels, in correspondence to the Bayesian proportional change in guess (so 1 on receiving a white ball and − 1 on receiving an orange ball). Data from all treatments are used

In the lower panels, the updates relative to the Bayesian benchmark cluster between zero and one (for a white ball) and minus one and zero (for an orange ball).Footnote ¹⁷ This shows that when agents adjust, they tend to do so in a reasonable direction for risk averse agents, raising their probability for Urn 1 when a white ball is drawn and lowering it when an orange ball is drawn.Footnote ¹⁸ Furthermore, the clustering indicates that any Quasi-Bayesian representation of their adjustment will require a $\beta$ parameter less than unity, reflecting insufficient belief adjustment.

These nonparametric statistics are consistent with more than one theoretical approach, but it is not clear that just one approach will explain all the features of the data. With that in mind, we now turn to a model which allows the different approaches to co-exist.

4.2 A double hurdle model of belief adjustment

In this section, we develop a parametric double hurdle model which simultaneously considers the decision to update beliefs and the extent to which beliefs are changed when updates occur. The purpose of the model is to act as a testing tool for state-dependent belief adjustment, namely Bayesian belief adjustment and Quasi-Bayesian belief adjustment in the simple version previously defined, as well as (stochastic) time-dependent belief adjustment.

Our econometric task is to model the transformed implied belief $r_t^{\ast }={\Phi }^{-1}\left(g_t^{\ast },\left({\theta }_i\right)\right)$ , which in turn requires an estimate for risk aversion. We estimate this at the individual level using the technique by Offerman et al. (Reference Offerman, Sonnemans, Van De Kuilen and Wakker2009). “Appendix 3: Method for estimating CRRA risk parameter” contains the subject-level details surrounding the estimation of ${\theta }_i$ . On average, agents are risk averse with a mean $\theta$ of 0.2.Footnote ¹⁹

We will refer to $r_{\mathrm{it}}^{\ast }$ , subject i’s belief in period t, as shorthand for ‘transformed implied belief’. We will treat $r_{\mathrm{it}}^{\ast }$ as the focus of the analysis, because $r_{\mathrm{it}}^{\ast }$ has the same dimensionality as $z_{{}_{\mathrm{it}}}$ , the test statistic defined in (4). That is, both have support $(-\infty ,\infty )$ . Sometimes $r_{\mathrm{it}}^{\ast }$ changes between $t-1$ and t; other times, it remains the same. Let $\Delta r_{\mathrm{it}}^{\ast }$ be the change in belief of subject i between $t-1$ and t. That is, $\Delta r_{\mathrm{it}}^{\ast }=r_{\mathrm{it}}^{\ast }-r_{it-1}^{\ast }$ .

In the following estimation we exploit the near equivalence between (4) and the scaled difference since the last update $2({\Phi }^{-1}\left(P_t\right)-{\Phi }^{-1}\left(P_m\right))/\sqrt{t}$ (from Table 2). In round 1, $P_m$ equals the prior 0.6 and the movement of the guess for a given subject is $\Delta r_{i1}^{\ast }=r_1^{\ast }-{\Phi }^{-1}\left(0.6\right)$ . That is, both the objective measure of the information change and the subjective guess of the agent are assumed to anchor onto the prior probability that Urn 1 is chosen, 0.6, in the first period.

4.2.1 First hurdle

The probability that a belief is updated (in either direction) in period t is given by:

(5) $\begin{array}{c}P\left(\Delta ,r_{\mathrm{it}}^{\ast },\ne ,0\right)=\Phi \left({\delta }_i,+,x_i^{\prime },{\Psi }_1,+,\gamma ,\left(z_{\mathrm{it}}\right)\right),\end{array}$

where $\Phi \left(·\right)$ is the standard Normal cdf and ${\delta }_i$ represents subject i’s idiosyncratic propensity to update beliefs, and therefore models random probabilistic belief adjustment (time-dependent belief adjustment). The probability of an update is assumed to depend (positively) on the absolute value of $z_{\mathrm{it}}$ , the test statistic. The vector $x_i$ contains treatment and gender dummy variables together with an age variable and a score on two questions from the comprehension questionnaire administered after the main part of the experiment was explained,Footnote ²⁰ all of which are time invariant and can be expected to affect the propensity to update.

One econometric issue flagged in the last sub-section is the endogeneity of the variable $\left(z_{\mathrm{it}}\right)$ : subjects who are averse to updating tend to generate large values of $\left(z_{\mathrm{it}}\right)$ while subjects who update regularly do not allow it to grow beyond small values. This could create a downward bias in the estimate of the parameter $\gamma$ in the first hurdle. To deal with this concern we use an instrumental variables (IV) estimator which uses the variable $\widehat{\left(z_{\mathrm{it}}\right)}$ in place of $\left(z_{\mathrm{it}}\right)$ , where $\widehat{\left(z_{\mathrm{it}}\right)}$ comprises the fitted values from a regression of $\left(z_{\mathrm{it}}\right)$ on a set of suitable instruments.Footnote ²¹

4.2.2 Second hurdle

Conditional on subject i choosing to update beliefs in draw t, the next question relates to how much they do so. This is given by:

(6) $\begin{array}{c}\Delta r_{\mathrm{it}}^{\ast }=\left({\beta }_i,+,x_i^{\prime },{\Psi }_2\right)\frac{\sqrt{t}{z}_{\mathrm{it}}}{2}+{\epsilon }_{it,}{\epsilon }_{\mathrm{it}}\sim N\left(0,,,{\sigma }^2\right).\end{array}$

As a reminder, the Quasi-Bayesian belief adjustment parameter ${\beta }_i$ represents subject i’s idiosyncratic responsiveness to the accumulation of new information: if ${\beta }_i=1$ , subject i responds fully; if ${\beta }_i=0$ , subject i does not respond at all. Remember that ${\beta }_i$ is not constrained to [0, 1]. In particular, a value of ${\beta }_i$ greater than one would indicate the plausible phenomenon of overreaction. Again, treatment variables are included: the elements of the vector ${\Psi }_2$ tell us how responsiveness differs by treatment.

Considering the complete model, there are two idiosyncratic parameters, ${\delta }_i$ and ${\beta }_i$ . These are assumed to be distributed over the population of subjects as follows:

(7) $\begin{array}{c}\left(\begin{array}{c}{\delta }_i\\ {\beta }_i\end{array}\right)\sim N\left(\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right),,,\left(\begin{array}{cc}{\eta }_1^2& \rho {\eta }_1{\eta }_2\\ \rho {\eta }_1{\eta }_2& {\eta }_2^2\end{array}\right)\right).\end{array}$

In total, there are seventeen parameters to estimate: ${\mu }_1$ , ${\eta }_1$ , ${\mu }_2$ , ${\eta }_2$ , $\rho$ , $\gamma$ , $\sigma$ , four treatment effects (two in each hurdle); two gender effects (one in each hurdle); two scores from the comprehension questionnaire (one in each hurdle); and two age effects (one in each hurdle). Estimation is performed using the method of maximum simulated likelihood (MSL), with a set of Halton draws representing each of the two idiosyncratic parameters appearing in (7). Following estimation of the model, Bayes rule is used to obtain posterior estimates (denoted $\widehat{{\delta }_i}$ and $\widehat{{\beta }_i}$ ) of the idiosyncratic parameters for each subject.Footnote ²²

The results are presented in Table 3 for four different models. The last column shows the preferred model. Model 1 estimates the QB benchmark, in which it is assumed that the first hurdle is crossed for every observation—that is, updates always occur. Zero updates are treated as zero realizations of the update variable in the second hurdle, and their likelihood contribution is a density instead of a probability. Because of this difference in the way the likelihood function is computed, the log-likelihoods and AICs cannot be used to compare the performance of QB to that of the other models.

Table 3

Results of hurdle model with risk adjustment

	QB	IE	IE-QB ( $\rho =0$ )	IE-QB
	(1)	(2)	(3)	(4)
Propensity to update
${\mu }_1$	$+\infty$	0.171 (0.216)	0.143 (0.118)	0.061 (0.109)
[Baseline time-dependent update probability $\Phi \left({\mu }_1\right)$ ]	[1]	[0.57]	[0.56]	[0.52]
${\eta }_1$	0	1.243** (0.063)	1.315** (0.059)	1.28** (0.057)
$\gamma$	0	0.603** (0.102)	0.598** (0.102)	0.597** (0.102)
Complex	0	− 0.199 (0.181)	− 0.273** (0.081)	− 0.211** (0.075)
Inattention	0	− 0.276 (0.174)	− 0.398** (0.077)	− 0.305** (0.076)
Male	0	− 0.324** (0.137)	− 0.419** (0.088)	− 0.382** (0.079)
Comprehension	0	− 0.126 (0.107)	− 0.147** (0.064)	− 0.133** (0.056)
Age-22	0	− 0.037** (0.009)	− 0.026** (0.004)	− 0.025** (0.005)
Extent of update
${\mu }_2$	0.522** (0.085)	1	0.609** (0.099)	0.819** (0.123)
${\eta }_2$	0.498** (0.022)	0	0.807** (0.033)	0.876** (0.044)
Complex	− 0.022 (0.078)	0	0.049 (0.085)	0.116 (0.093)
Inattention	− 0.074 (0.075)	0	0.240** (0.076)	− 0.254** (0.092)
Comprehension	− 0.007 (0.046)	0	0.014 (0.054)	− 0.023 (0.062)
Male	− 0.027 (0.068)	0	0.271** (0.072)	0.203** (0.078)
Age-22	0.021** (0.005)	0	0.019** (0.005)	0.019** (0.007)
$\sigma$	0.516** (0.003)	0.731** (0.006)	0.660** (0.006)	0.660** (0.006)
$\rho$	N/A	N/A	0	− 0.18* (0.081)
LogL	− 8,790	− 12,328	− 11,924	− 11,923
AIC $(=2\text{k}-2\text{LogL})$	N/A	24,690	23,882	23,880
Wald test (df, p value)	66,599 (9, 0.000)	430 (8, 0.000)	4.36 (1, 0.037)	N/A
ROC (out of sample)	Fail by definition	0.551	0.588	0.588
$R^2$ (out of sample)	0.090	0.082	0.081	0.083
n (subjects)	245	245	245	245
T (observations per subject)	56	56	56	56

LogL for QB cannot be compared with that of other columns

Model 2 estimates the IE benchmark, in which the update parameter ( ${\beta }_i$ ) is fixed at 1 for all subjects. Consequently the extra residual variation in updates is reflected in the higher estimate of $\sigma$ . The parameters in the first hurdle are free.

Model 3 combines IE and QB, but constrains the correlation ( $\rho$ ) between $\delta$ and $\beta$ to be zero. Model 4 is the same model with $\rho$ unconstrained.

The overall performance of a model is judged initially using the AIC; the preferred model being the one with the lowest AIC. Using this criterion, the best model is the most general model 4 (model 1 not being subject to the AIC criterion): IE-QB with $\rho$ unrestricted, whose results are presented in the final column of Table 3.

To confirm the superiority of the general model over the restricted models, we conduct Wald tests of the restrictions implied by the three less general models. We see that, in all three cases, the implied restrictions are rejected, implying that the general model is superior. Note in particular that this establishes the superiority of the general model 4 (IE-QB with $\rho$ unrestricted) over the QB model 1 (a comparison that was not possible on the basis of AIC).

Further confirmation is furnished by measuring the sample predictive accuracy on a subsample of data (the ‘cross validation’ approach). ROC (Receiver Operating Characteristic) is the model’s out of sample predictive accuracy for hurdle 1 (the frequency of updates) and $R^2$ (out of sample) is the model’s predictive accuracy for hurdle 2 (the extent of updates).Footnote ²³ Model 4 is no worse than model 3 on an ROC criterion, but predicts the extent of adjustment better. Model 1 is best of all at predicting the extent of adjustment, but it fails to predict ‘no-change behavior’, by construction. Thus, on both in and out of sample criteria, model 4 is best overall.Footnote ²⁴

We interpret the results from model 4 as follows. Consider the first hurdle (propensity to update). The intercept parameter in the first hurdle ( ${\mu }_1$ ) tells us that a typical subject has a predicted probability of $\Phi \left(0.061\right)=0.524$ of updating in any task, in the absence of any evidence (i.e. when $\left(z_{\mathrm{it}}\right)=0$ ). We note that this estimate is not significantly different from zero, which would imply a 50% probability of updating. The Inattention treatment effect is significant and negative, suggesting that the probability of update is lower when subjects are not paying attention. So is the Complexity treatment but not by as much. The effect of the questionnaire score is negative and significant, though it is not large. The negative coefficient is consistent with Cohen et al.’s (Reference Cohen, Henckel, Menzies, Muhle-Karbe and Zizzo2019) model if subjects have cognitive costs. This gives us our first result:

Result 1

There is evidence of time-dependent (random) belief adjustment. Subjects update their beliefs idiosyncratically around half the time.

The large estimate of ${\eta }_1$ tells us that there is considerable heterogeneity in the propensity to update (see Fig. 4), something we will explore further in Sect. 4.3. The parameter $\gamma$ is estimated to be significantly positive, and this tells us, as expected, that the more cumulative evidence there is, in either direction, the greater the probability of an update:

Result 2

There is evidence of state-dependent belief adjustment. Subjects are more likely to adjust if there is more evidence to suggest that an update is appropriate (thus making it costlier not to update).

In the second hurdle, the intercept ( ${\mu }_2$ ) is estimated to be 0.819 in our preferred model 4: when a typical (baseline) subject does update, she updates by a proportion 0.819 of the difference from the Bayes probability. The large estimate of ${\eta }_2$ tells us that there is considerable heterogeneity in this proportion also (see Fig. 4). Interestingly, on the basis of the posterior estimates from model 4, only 13 out of 245 subjects appear to have $\beta <0$ , which indicates noise or confused subjects who adjusted in the wrong direction. Moreover, 87 out of 245 subjects (around one third) display overreaction to the evidence. We summarize this in the following result:

Result 3

There is evidence of Quasi-Bayesian partial belief adjustment. On average, subjects who adjust do so by around 80%. There is evidence of prior information under-weighting: around one third of the subjects overreact to evidence once they decide to adjust.

The estimate of $\rho$ is negative, indicating that subjects who have a higher propensity to update, tend to update by a lower proportion of the difference from the Bayes probability. Inattention is important for both hurdles:

Result 4

Inattention lowers the probability of updating from 50 to 40% and lowers the extent of update from 80 to 56% of the amount prescribed by Bayes rule. Complexity also lowers the probability of update in the first hurdle.

Fig. 4

Jittered scatter of posterior QB parameter against posterior probability of updating for the four models whose estimates are reported in Table 3. Scatters based on Models 1 (top left), 2 (top right), 3 (bottom left) and 4 (bottom right) in Table 3 respectively. Each dot corresponds to a subject

4.3 The empirical distribution of $\alpha$ and $\beta$

To get a better sense of the population heterogeneity in belief adjustment, this subsection maps out the empirical distribution of the IE ${\alpha }_i$ and QB ${\beta }_i$ parameters across subjects against each other. The estimated distribution $f\left(\beta \right)$ can be seen from the distribution of the posterior estimates $\widehat{{\beta }_i}$ from Model 4, and this distribution is the marginal distribution of the extent of update on the vertical axis on the bottom-right panel of Fig. 4. We next use the first hurdle information to generate $f_i\left(\alpha \right)$ , the empirical distribution of ${\alpha }_i$ .

As we flagged earlier, each agent has a full distribution of $\alpha$ and so we need a representative ${\alpha }_i$ to summarize the extent of sticky belief adjustment for agent i, to then relate to their ${\beta }_i$ . As will be clear below, the choice that permits analytic solutions is the median ${\alpha }_i$ from $f_i\left(\alpha \right)$ .

The econometric equation for the first hurdle is equivalent to the probability of rejecting the null under IE. We omit the dummy variables and begin by re-writing the first hurdle, namely (5):

(8) $\begin{array}{c}Pr{\left(\text{reject},H_0\right)}_{\mathrm{it}}=\Phi \left({\delta }_i,+,\gamma ,\left(z_{\mathrm{it}}\right)\right),\end{array}$

where ${\delta }_i$ and $\gamma$ are estimated parameters and $\left(z_{\mathrm{it}}\right)$ is the test statistic based on the proportion of white balls:

(9) $\begin{array}{c}{z}_{\mathrm{it}}=\frac{P_{\mathrm{it}}^w-P_{\mathrm{im}}^w}{\sqrt{0.5^2/t}}.\end{array}$

For any $\left(z_{\mathrm{it}}\right)$ it is possible to work out an implied p value and we do so by assuming that (9) is approximately distributed N(0, 1). This in turn allows us to work out $f_i\left(\alpha \right)$ from the econometric equation for the first hurdle. When $\left(z_{\mathrm{it}}\right)=0$ , the p value for a hypothesis test is unity, and so the equation says that a fraction of agents will reject $H_0$ if the p value is unity. Since the criterion for rejecting $H_0$ in a hypothesis test is always $\alpha \ge p$ -value, the observed behavior of rejecting $H_0$ when $\left(z_{\mathrm{it}}\right)=0$ implies that there must be a non-zero probability mass on $f_i\left(\alpha \right)$ at the value of $\alpha$ exactly equal to 1. The pdf of ${\alpha }_i$ will thus have a discrete ‘spike’ at unity and be continuous elsewhere. We know what that spike is from Eq. (8) with $\left(z_{\mathrm{it}}\right)=0$ substituted in, namely $\Phi \left({\delta }_i\right)$ .

The probability of rejecting $H_0$ depends on the probability that the test size is greater than the p value, but this is also equal to the econometric equation for the first hurdle.

(10) $\begin{array}{c}Pr{\left(\text{reject},H_0\right)}_{\mathrm{it}}={\int }_{p{\text{-value}}_{\mathrm{it}}}^1{f}_i\left(\alpha \right)d{\alpha }_i=1-F_i\left(p,{\text{-value}}_{\mathrm{it}}\right)=\Phi \left({\delta }_i,+,\gamma ,\left(z_{\mathrm{it}}\right)\right)\end{array}$

Upper case F in the last equality is the anti-derivative of the density. We define $F_i\left(1\right)$ to be unity since 1 is the upper end of the support of $\alpha$ but we also note that there is a discontinuity such that F jumps from $1-\Phi \left({\delta }_i\right)$ to 1 at $\alpha =1$ , as a consequence of the non-zero probability mass on $f_i\left(\alpha \right)$ at unity. To solve the equation we use an expression for the p value of $\left(z_{\mathrm{it}}\right)$ on a two-sided Normal test.

(11) $\begin{array}{c}p{\text{-value}}_{\mathrm{it}}=2\left(1,-,\Phi ,\left(\left(z_{\mathrm{it}}\right)\right)\right).\end{array}$

We use a ‘single parameter’ approximation to the cumulative Normal (see Bowling et al. Reference Bowling, Khasawnch, Kaewkuekool and Cho2009). For our purposes $\sqrt{3}$ is sufficient for the single parameter.

(12) $\begin{array}{c}\Phi \left(\left(z_{\mathrm{it}}\right)\right)\approx \frac{1}{1+exp\left(-,\sqrt{3},\left(z_{\mathrm{it}}\right)\right)}.\end{array}$

We can now write down $\left(z_{\mathrm{it}}\right)$ as a function of the p value using (11) and (12):

(13) $\begin{array}{c}\left(z_{\mathrm{it}}\right)\approx \frac{1}{\sqrt{3}}ln\left(\frac{2-p{\text{-value}}_{\mathrm{it}}}{p{\text{-value}}_{\mathrm{it}}}\right).\end{array}$

Intuitively, a p value of zero implies an infinite $\left(z_{\mathrm{it}}\right)$ and a p value of unity implies $|z_{\mathrm{it}}|$ is zero, and (13) confirms this. We can now use the relationship between $F_i\left(p,{\text{-value}}_{\mathrm{it}}\right)$ and our estimated first hurdle to generate $F_i\left(\alpha \right)$ .

(14) $\begin{array}{cc}1-F_i\left(p,{\text{-value}}_{\mathrm{it}}\right)& =\Phi \left({\delta }_i,+,\gamma ,\left(z_{\mathrm{it}}\right)\right)\\ \therefore F_i\left(p,{\text{-value}}_{\mathrm{it}}\right)& =1-\Phi \left({\delta }_i,+,\gamma ,\left(z_{\mathrm{it}}\right)\right)\\ & \approx 1-\Phi \left({\delta }_i,+,\gamma ,\left(\frac{1}{\sqrt{3}},ln,\left(\frac{2-p{\text{-value}}_{\mathrm{it}}}{p{\text{-value}}_{\mathrm{it}}}\right)\right)\right).\end{array}$

In the above expression the variable ‘ $p{\text{-value}}_{\mathrm{it}}$ ’ is just a place-holder and can be replaced by anything with the same support leaving the meaning of (14) unchanged. Thus, it can be replaced by $\alpha$ giving the cumulative density of $\alpha$ .

(15) $\begin{array}{cc}{F}_i\left(\alpha \right)& \approx 1-\Phi \left({\delta }_i,+,\gamma ,\left(\frac{1}{\sqrt{3}},ln,\left(\frac{2-\alpha }{\alpha }\right)\right)\right)\\ & \approx 1-\frac{1}{1+{\left(\frac{\alpha }{2-\alpha }\right)}^{\gamma }exp\left(-,\sqrt{3},{\delta }_i\right)}.\end{array}$

Substitution of $\alpha =1$ does not give unity, which is what we earlier assumed for the value of $F_i\left(1\right)$ . However, it does give $1-\Phi \left({\delta }_i\right)$ , which of course concurs with the econometric equation for the first hurdle when $\left(z_{\mathrm{it}}\right)=0$ . This discontinuity in $F_i$ is consistent with a discrete probability mass in $f_i\left(\alpha \right)$ at unity, as we noted earlier. It now just remains to differentiate $F_i$ to obtain the continuous density $f_i\left(\alpha \right)$ for $\alpha$ strictly less than unity. The description of the function at the upper end of the support (unity) is completed with a discrete mass at unity of $\Phi \left({\delta }_i\right)$ .

(16) $\begin{array}{c}\left(\begin{array}{cc}{f}_i\left(\alpha \right)\approx \frac{2\gamma {\alpha }^{\gamma -1}exp\left(-,\sqrt{3},{\delta }_i\right)}{{\left(2,-,\alpha \right)}^{1+\gamma }{\left(1,+,{\left(\frac{\alpha }{2-\alpha }\right)}^{\gamma },exp,\left(-,\sqrt{3},{\delta }_i\right)\right)}^2},& \alpha <1\\ {Pr}_i\left(\alpha ,=,1\right)\approx \frac{1}{1+exp\left(-,\sqrt{3},{\delta }_i\right)},& \alpha =1\end{array}\right)\end{array}$

Figure 5 illustrates the distribution $f_i\left(\alpha \right)$ for ${\delta }_i=0.1$ and $\gamma =0.6$ together with the distributions one standard deviation either side of ${\delta }_i$ . The former is the mean of $\delta$ across subjects, from our estimation (from the last column of Table 3, rounded). On the right-most of the chart is the probability mass when $\alpha =1$ . As discussed earlier, this corresponds to the proportion of agents who update on vanishingly small evidence ( $\left(z_{\mathrm{it}}\right)=0$ ). There is clearly a great deal of interesting heterogeneity. One distribution has a near-zero probability of a random update (10%) and when the agent uses information they are very conservative, with $\alpha$ close to zero. We might call them ‘classical statisticians’ given the large probability mass around 1%, 5% and 10%. Another distribution has a virtually certain probability of a random update (90%) and we might call these agents ‘fully attentive’. The central estimate of $\delta$ describes an agent who updates roughly half the time, and otherwise has a more or less uniform distribution over $\alpha$ .

Fig. 5

Distribution of $f_i\left(\alpha \right)$ for ${\delta }_i=0.1$ and $\gamma =0.6$ together with the distributions one standard deviation either side of ${\delta }_i$

Since there are idiosyncratic values of ${\delta }_i$ there will be a separate distribution for every subject varying over ${\delta }_i$ . So we must use a summary statistic for $f_i\left(\alpha \right)$ , and the one which comes to hand is the median $\alpha$ value, obtained by solving $F_i\left(\alpha \right)=0.5$ in Eq. (15). In Fig. 6, we plot the collection of subject i’s (median $\alpha$ , $\beta$ ) duples for model 4, our preferred equation. Table 4 lists the percentage of subjects in each (median ${\alpha }_i$ , ${\beta }_i$ ) 0.2 bracket.Footnote ²⁵

Fig. 6

Median $\beta$ against median $\alpha$ for Model 4. Each point represents a subject

Table 4

Percentage of subjects in each ( ${\alpha }_i,{\beta }_i$ ) bracket in model 4

				$\beta$
${\alpha }_{\mathrm{med}}$	below 0 (%)	0–0.2 (%)	0.2–0.4 (%)	0.4–0.6 (%)	0.6–0.8 (%)	0.8–1 (%)	Above 1 (%)	Total (%)
0–0.2	2	0	4	3	3	4	12	28
0.2–0.4	0	1	1	1	1	1	3	8
0.4–0.6	0	2	0	2	1	1	2	7
0.6–0.8	0	0	1	0	1	1	3	7
0.8–1	2	3	5	11	7	5	16	49
Total	5	6	11	17	13	12	36	100

Roughly half the subjects update regardless of evidence, so the median $\alpha$ ’s cluster at unity along the bottom axis with half of them (49%) in the range at or above 0.8. Just under one quarter (22%) of agents could be described as classical statisticians with median $\alpha$ ’s around the 1–10% level and a similar figure (28%) have ‘conservative belief adjustment’, with $\alpha$ -values no more than 0.20.

Regarding the size of updating, we already know from Result 3 that it is less than complete. In Table 4, 22% update no more than 40 per cent of what they should.