2 The free-choice paradigm and its formal analysis

2.1 The free-choice paradigm

A typical implementation of the free-choice paradigm, or FCP (for the ratings case; the case of rankings is analogous) runs as follows. The experimenter has collected a set of comparable options to be used as stimuli. Those can be, for instance, holiday destinations (as in Sharot et al., 2009, 2010) or artistic paintings and posters (as in Reference Gerard and WhiteGerard & White, 1983; Reference Jordens and Van OverwalleJordens & Van Overwalle, 2005; Liberman et al., 2001; Schultz et al., 1999). Each participant in the experiment is confronted with these options in three different phases.

In the first phase, the participant provides a rating r _k ¹ for each option x _k in the set, in a pre-determined scale, e.g., 1 to 8 or 0 to 100, with higher values indicating higher desirability. In the second phase, the participant is confronted with pairs of objects ( x _i, x _j) and asked to choose among them, i.e., indicate whether, if faced with the possibility of receiving either x _i or x _j, he or she would choose x _i or rather x _j. In the third phase, the participant is presented with the same set of options and asked to provide ratings r _k ³ again, for each option x _k, according to how he or she feels at that point. That way, for each relevant option x _k, the experimenter obtains three pieces of information: the pre-choice rating r _k ¹, the post-choice rating r _k ³, and whether the option was hypothetically chosen in the corresponding pair or not.

Crucially, and unbeknownst to the participant, the choice pairs ( x _i, x _j) are not randomly extracted from the set of available options. Rather, the experimenter, or a computer program on his behalf, selects pairs that have been rated at a pre-determined distance D≥ 0 from each other in the first-phase rating. That is, each pair ( x _i, x _j) will be such that r _i ¹− r _j ¹= D. A few different values of D might be used in a single experiment, but then each possible value of D is used as a within-subject condition, and hence a given study might present results for D=0,1 and 2 (and later pool all of them). That is, the analysis is always conditional on the given value of D. To be clear, the experimenter previously decides on a value D of interest and then, within the experiment, selects pairs of alternatives previously rated at a distance D from each other. For notational convenience, within a given pair we denote by x _i the alternative rated weakly better, i.e. r _i ¹= r _j ¹+ D.

Experimental and formal statements then refer to the spread of alternatives. The spread Δ of a pair ( x _i, x _j) constructed as indicated is defined as follows. First, construct the pre- and post-choice rating differences as the rating of the option chosen in the second phase minus the rating of the other (unchosen) option. Note that the pre-choice rating difference can hence be either D or − D, depending on which option is chosen. The spread is then the post-choice rating difference minus the pre-choice rating difference. That is,

Experimental evidence starting with Brehm [1956] overwhelmingly shows that measured values of Δ, conditional on, e.g., D=0, D=1, or D=2, are strictly positive, indicating that the rating of the chosen option goes up and/or the rating of the not chosen option goes down in the third phase with respect to the first one. Cognitive dissonance theory (Festinger [1957]) and self-perception theory (Bem [1967]a) give different interpretations of this rating reevaluation. The construction of Δ, which conditions on which is the chosen option, aims to measure precisely this effect, and hence reflects the changes in the rating difference between the chosen option and the unchosen one.

As in any other experiment, due to noise in the decision-making and experimental processes, all experimental observations can and should be treated as random variables. It is important to specify right away what this means for the FCP. The participants’ decisions are to be treated as random variables. This means the ratings in Phases 1 and 3, i.e. all r _k ^t for t=1,3, and also the choices in Phase 2, i.e. the binary variable indicating whether x _i or x _j is chosen. The distance D, however, is exogenously fixed by the experimenter and has to be treated as a fixed value. Hence, the quantity that experiments aim to measure is the following expectation.

2.2 The formal analysis of experiments

The key to understand the formal analysis of the FCP or any other decision-making experiment is that one can formally study the results that the experimental procedure would elicit from a population of artificial, hypothetical participants affected by a given psychological phenomenon. Likewise, and even more easily, one can formally study the results elicited from a population of hypothetical participants not affected by such a phenomenon. In particular, if an experimental paradigm is to be used to collect evidence for the existence of choice-induced preference change, the paradigm should not generate any such evidence when applied to a set of artificial participants with immutable preferences, i.e., whose choices never feed back into their preferences. In the terms of the FCP, for such artificial participants, one should obtain E[Δ| D]=0 for any value of D. In other words, if the experiments aim to reject the null hypothesis that the spread is equal to zero, then a hypothetical control group not displaying choice-induced preference change should not lead to such a rejection.

In order to generate such populations of artificial agents, Chen and Risen [2010] postulate a random utility model.Footnote ³ Specifically, each agent is endowed with a randomly determined “true” preference, which remains fixed for the duration of the experiment. The random generation of preferences reflects random sampling of participants and the inherent randomness in the experiment (differences in participants, options, etc). Hence, both Chen and Risen (2010, footnote 8) and our model below treat preferences, ratings, and choices as random variables. There is a fundamental difference between preferences and other variables, though. Suppose participants from a hypothetical control group whose preferences never change are “recruited.” Choices and ratings will vary randomly within the experiment. In contrast, participants’ preferences remain fixed and will not change in the course of the experiment. Hence, for these participants no preference change should be found.

An actual analytical computation of the expected value of Δ, however, is not straightforward. One needs to be clear about the modeling assumptions and spell out the full expression of the mathematical expectation. Chen and Risen [2010] postulate a setting where ratings and choices are generated as noisy observations of a true underlying preference. They then postulate a series of (reasonable) assumptions and state a theorem claiming that the expected value of Δ is strictly positive. To examine this claim, we need to spell out the relevant assumptions of Chen and Risen [2010].

Let V be the set of possible rating values and let u _k∈ V denote the actual, true rating (reflecting the actual, immutable preference) of a given participant for alternative x _k. We start with the assumptions on ratings. Intuitively, the assumptions aim to reflect that ratings are noisy perturbations of the true underlying preferences. We keep the original assumption numbering from Chen and Risen [2010].

Assumption 1: For each alternative x _k and each possible rating value v _k, for both t=1 and t=3,

(1)

This assumption means that the ratings in Phases 1 and 3 are guided by the actual preferences of the participant, with zero-mean noise.

Assumption 3:For each two alternatives x _k, x _ℓ and for both rating phases t=1 and t=3,

(2)

and

(3)

This assumption indicates that a preference for one alternative ( u _k> u _ℓ) results in ratings consistent with the preference ( r _k> r _ℓ) with a large probability (larger than 1/2), but with some residual noise (hence smaller than 1). That is, preferences do guide ratings, but imperfectly. Note that (3) is also referred to as Assumption 1a in Chen and Risen [2010].

We now turn to choices. Let c: X ² → X be the choice function of the participant in Phase 2, i.e., c( x _i, x _j)= x _i means that the participant, when presented with the possibility of choosing either x _i or x _j, does choose the former, while c( x _i, x _j)= x _j means that he chooses the latter. The assumption is that this choice is based on preferences (utilities) but is also subject to some stochastic noise.

Assumption 2: For each two alternatives x _k, x _ℓ,

and

(4)

That is, the choice in Phase 2 is partially guided by the preference, in the sense that there is a larger probability of choosing the alternative with the largest true utility.

Given these assumptions, Chen and Risen [2010] state the following claim, which we reformulate for clarity.

Chen and Risen’s Claim: For any model of random utilities, ratings, and choices fulfilling Assumptions 1, 2, and 3, and for any value of D, the value of E[Δ| D] is strictly positive.

To summarize, classical studies using the free-choice paradigm might have naïvely assumed that E[Δ| r _i ¹− r _j ¹= D]=0 in the absence of choice-induced preference change. As experiments obtained strictly positive values for Δ given D, results were interpreted as evidence for choice-induced preference change. Chen and Risen [2010] argue that, in models where ratings and choices are just noisy realizations of immutable preferences, E[Δ| r _i ¹− r _j ¹= D]>0 even in the absence of choice-induced preference change, and hence experimental evidence is meaningless.Footnote ⁴

Anticipating our objective here, we will show below that, actually, even though E[Δ| r _i ¹− r _j ¹= D]≠ 0 in general (which is enough to show that the FCP is partially flawed), it is not true that E[Δ| r _i ¹− r _j ¹= D]>0 always holds, and even E[Δ| r _i ¹− r _j ¹= D]<0 might occur. Hence, experimentally observed strictly positive values still most likely point to a real phenomenon.

3 The problems with intuitive arguments

The essence of the problems with the FCP is that the construction of the spread Δ relies on identifying which option is “chosen” and which is “unchosen”. This, in turn, is based on the participant’s decision in Phase 2. Hence, options are not randomly assigned to the roles of chosen or unchosen, and a statistical bias appears. For instance, Sharot et al. [2010] randomly assign options to such roles by deceiving participants into believing they made a certain choice or another (“blind choice”), hence fully avoiding any possible bias. The idea behind the “implicit choice paradigm” (Alós-Ferrer et al. [2012]) is also to randomly assign the role of chosen or unchosen to the alternatives, but then use the participant’s own preferences to induced the desired choices, i.e., the (randomly selected) chosen item is paired with a low-preference item, so that it will very likely be chosen, and the (randomly selected) unchosen item is paired with a high-preference item; positive spreading must then be attributed to the two choices and not to non-random selection of items. The FCP pays no attention to this issue and hence contains a subtle form of non-random assignment.

Once this problem is made explicit, it is obvious that the claim that E[Δ| D]=0 in the absence of choice-induced preference change is unwarranted. Imagine again a hypothetical participant with immutable preferences, which correspond to a “true rating”, but whose answers in each phase of the experiment are based on ratings that are affected by zero-mean noise, independently across phases. The fact that an option has been chosen over the other one is informative about the true preferences and hence influences the distribution of the rating differences. Since the first rating difference is constrained by the value D, it is clear that the expected value of Δ will in general not be zero, because conditioning has an effect on the expectation. It is, however, far from obvious what exactly the expected value of Δ should be in the absence of effects along the lines of cognitive dissonance or self-perception theory. A computation of the expected value of Δ is not trivial. One needs to actually be clear about the modeling assumptions and spell out the full expression of the mathematical expectation.

The experimental design and the concepts underlying the FCP are not straightforward. While it is tempting to rely on intuitive arguments, intuitions can easily be led astray. We present here a short discussion of why this is the case.

Let us consider a first example, which specifies merely one of many cases that ought to go into the computation of the actual expected value of Δ. Let us go back momentarily to the case of rankings, and suppose that in an initial ranking of options (Phase 1), a certain alternative A has been ranked two levels above another alternative B ( D=2). Suppose further that we do not even know whether the participant has chosen A or B in the choice task (Phase 2). Consider two hypothetical possibilities for the later re-ranking (Phase 3). In the first possibility (re-ranking X), A is ranked two levels higher than it was ranked in Phase 1 and B two levels lower than it was ranked in Phase 1, so that A is again ranked above B, by a larger amount. In the second possibility (re-ranking Y), A is ranked two levels lower and B two levels higher than before, so that now B is ranked above A. Which of these two possible re-rankings is more likely? It is easy to jump to the conclusion that, since those re-rankings are in some sense symmetric with respect to the initial one, they should be equally likely. For instance, Chen and Risen (2010, p. 580) claim that “without choice information, re-rankings X and Y are equally likely” and then go on to use this example in order to build intuition on their result. This statement sounds very intuitive; however, it is incorrect, and so is any intuition based on it. Suppose that, as assumed in Chen and Risen [2010], rankings imperfectly reflect underlying preferences. The subsequent re-rankings are not noisy versions of the initial ranking. They are noisy versions of the true ranking which corresponds to the preferences; and, indeed, also the initial ranking is just a noisy version of the true one. If the initial ranking places A and B in the 5th and 7th positions, respectively, but the true ranking would have them at the 3rd and 9th, then a re-ranking exchanging their positions (so that A becomes 7th and B becomes 5th) will be much less likely than a re-ranking which puts A 3rd and B 9th (and hence actually agrees with the true preferences).

The point we want to make here is not that, for this particular example, one or the other re-ranking is more likely. The point is simply that one needs to consider all possibilities and spell out all probabilities. An actual computation of Δ conditions on the additional information revealed by the choices in Phase 2. If we consider this information for the example above, the situation becomes even less transparent. It is not warranted to claim that the fact that, say, B has been chosen (even though the initial ranking favored A) makes a re-ranking where B is placed higher more likely. In order to compare the likelihood of two re-rankings, one needs to compare the probability that the initial ranking contradicted the true preferences but the choice agreed with them with the probability that the initial ranking agreed with the true preferences but the choice contradicted them. In order to precisely balance those possibilities, one needs to spell out a formal framework allowing for exact computations. Intuitive shortcuts can be misleading. The only way to make sure that all relevant cases have been considered and our intuition is not based on a partial analysis is to make the formal analysis precise. This is what we will do below for a particular model.

A second argument more or less explicit in the critique against the FCP is that choices should be a better predictor of re-rankings than initial rankings. Chen and Risen (2010, Table 2) discuss an example where both the choice and the re-ranking contradict the initial ranking and argue that this needs not reflect any cognitive dissonance, but merely the fact that choices are informative on the underlying preferences. However, as pointed out by Sagarin and Skowronski [2009]a, empirically observed choices only imperfectly reflect preferences. Hence, it is also easy to consider examples where the choice contradicts the initial ranking and also the true preferences, and the fact that the re-ranking agrees with the choice reflects a reduction of cognitive dissonance. Of course, one needs to precisely balance the likelihood of these (and other) possibilities. Again, this can be done only in the framework of a full-fledged formal model of choice.

4 A sample model

4.1 A formal behavioral model

This section presents a formal behavioural model of the free-choice paradigm which adheres to Assumptions 1–3 of Chen and Risen [2010] introduced earlier. We concentrate on the case of ratings. Our model is an example of the framework developed above. Specifically, both choices and ratings are based on preferences, but both are noisy. In a nutshell, the ratings reported in Phases 1 and 3 will reflect underlying preferences, i.e., a true rating, with a large probability, but occasionally deviate from them slightly; specifically, the deviation will be just one position above or below with respect to the true rating. Likewise, in Phase 2 the participant will choose the option prescribed by his preferences with a large probability, but still occasionally decide against the preferences and choose the other option.

let us turn to the formal specification of the model. as above, let u _k be the participant’s utility level or true rating for option x _k, i.e., a numerical representation of his preferences. for each k=1,…, n, we assume that u _k follows a finite discrete distribution (in the example below, we will use a uniform distribution with 2 m+1 levels) with full support on a finite set v _k ≡ {µ_k− m,…, µ_k−1, µ_k, µ_k+1, …, µ_k+ m } ( µ_k, m ∈ ℕ). in other words, the actual preference of an individual for an option is determined randomly “around” u _k=µ_k, but it remains fixed for the duration of the experiment. we further assume that, for any two alternatives k,ℓ, |µ_k−µ_ℓ|≤ 2 m−1, which guarantees that options are not ex ante too different and there is always some (possibly small) probability that u _k= u _ℓ, u _k> u _ℓ, and u _k< u _ℓ.

Actual behavior is determined as follows. let r _k ^t be the participant’s rating of option x _k in phase t=1,3. let v _k denote a typical element of v _k. we assume that, for any k=1,2,…, n, if u _k= v _k, then

where β < 1/4 measures the amount of noise in the participant’s ratings. That is, the revealed rating is numerically equal to the actual one, but occasionally differs from it by one unit.

Under this modeling assumption, E[ r _k ^t| u _k= v _k]=β ( v _k−1) + (1−2β) v _k + β ( v _k+1) = v _k, and hence Assumption 1 holds. Let us turn to Assumption 3. It is immediate that (2) is fulfilled due to the condition |µ_k−µ_ℓ|≤ 2 m−1. For then there is positive probability that (for example) u _k= u _ℓ+1 but r _k ^t= u _k−1 and r _ℓ ^t= u _ℓ+1, hence the conditional probability Pr{ r _k ^t> r _ℓ ^t| u _k> u _ℓ} has to be strictly smaller than one. To see (3), note that, conditional on u _k> u _ℓ, the events { r _k ^t≥ u _k, r _ℓ ^t≤ u _ℓ} (which has probability (1−β)²), { r _k ^t= u _k+1, r _ℓ ^t= u _ℓ+1} (with probability β²), and { r _k ^t= u _k−1, r _ℓ ^t= u _ℓ−1} (also with probability β²) all lead to r _k ^t> r _ℓ ^t. Hence, the conditional probability Pr{ r _k ^t> r _ℓ ^t| u _k> u _ℓ} is larger than or equal to (1−β)²+2β²=1−2β+3β², which is always strictly larger than 1/2.

Consider now Phase 2’s choices. We directly specify the choice function c: X ² → X as follows.Footnote ⁵

If u _i> u _j,

if u _i = u _j,

and, if u _i< u _j,

where 0< ε < 1/2 measures the amount of noise in the participant’s choices.

Under this modeling assumption, it follows that

That is, Assumption 2 holds.

4.2 A Counterexample

We now spell out an example in the framework of the behavioural model above which does not lead to positive spreading, despite fulfilling Assumptions 1–3. It is a simple matter to construct other examples generating positive spreading; however, for our purposes we only need to exhibit an example with strictly negative spreading. Let x ₁ and x ₂ be the two options in the choice task in Phase 2. Let u ₁ follow a uniform distribution with support {1,2,3} and u ₂ follow a uniform distribution with support {2,3,4}. That is, µ₁=2, µ₂=3, and m=1.

Remark 1. We are, of course, free to choose any values of µ₁,µ₂, and m in order to exhibit a counterexample to a claimed result; a single counterexample suffices to establish that the claimed result is false. However, we also provide a second counterexample in Appendix A where µ₁=µ₂, i.e. under the additional (and unwarranted) assumption that utilities are a priori identically distributed for the two involved options.

Suppose that in Phase 1, the ratings of the subjects for x ₁ and x ₂ are r ₁ ¹ and r ₂ ¹ respectively with r ₂ ¹− r ₁ ¹ ≡ D=2. According to Chen and Risen [2010], there must be a positive rating spread in Phase 3, even though ratings in that stage are generated from exactly the same preferences as those in Phase 1. We now proceed to show that this implication is false.

The rating spread for x ₁ and x ₂ is Δ=( r ₂ ³− r ₁ ³)−( r ₂ ¹− r ₁ ¹)= r ₂ ³− r ₁ ³− D if x ₂ is chosen, and Δ=( r ₁ ³− r ₂ ³)−( r ₁ ¹− r ₂ ¹)= r ₁ ³− r ₂ ³+ D if x ₁ is chosen. We will show that the expected value of Δ given that D=2 is smaller than zero. Let r ¹=( r ₁ ¹, r ₂ ¹) denote a pair of ratings for x ₁ and x ₂ in Phase 1. First note that we can decompose the expected value of Δ as follows:

(5)

This equality follows from the fact that, in our example, there are exactly four possible pairs of values for r1 giving rise to D = 2, which are (0, 2), (1, 3), (2, 4), and (3, 5). We are going to compute the expected rating difference conditional on each specific r ¹ with D = 2, and show that this expected rating difference is strictly smaller than 0 given each r ¹. Hence, it will follow from (5) that E[Δ|D = 2]< 0.

In Phase 2, the participant may choose x ₁ or x ₂. Since Δ depends on the actual choice in Phase 2, we need to further decompose each E[Δ| r ¹] as follows:

(6)

Since D=2 is given, it is enough to compute the expectations of the post-choice rating differences r _i ³− r _j ³. Let the set of possible utility pairs be given by U ={(1,2),(1,3),(1,4),(2,2),(2,3),(2,4),(3,2),(3,3),(3,4)}. It is useful to further decompose the computations as follows:

(7)

where the denominator is given by Pr{ c( x ₁, x ₂)= x _i| r ¹}=∑_{u∈ U} Pr{ c( x ₁, x ₂)= x _i| u, r ¹} Pr{ u| r ¹}. Hence, we can obtain the conditional expectations by methodically computing the quantities Pr{ c( x ₁, x ₂)= x _i| u, r ¹}, Pr{ u| r ¹}, and E[( r _i ³− r _j ³)| u] for each u∈ U.

The last quantity is trivial to obtain since E[( r _i ³− r _j ³)| u]= u _i− u _j by (1). Further, since choices are stochastically independent of ratings given the utilities, independently of r ¹, we have that Pr{ c( x ₁, x ₂)= x ₁| u=(3,2), r ¹}=1−ε, Pr{ c( x ₁, x ₂)= x ₁| u=(2,2), r ¹}=Pr{ c( x ₁, x ₂)= x ₁| u=(3,3), r ¹}=1/2, and Pr{ c( x ₁, x ₂)= x ₁| u, r ¹}=ε for every u∈ U, u≠ (3,2),(2,2),(3,3); the probabilities for c( x ₁, x ₂)= x ₂ are complementary.

We now distinguish four cases according to whether r ¹=(0,2), (1,3), (2,4), or (3,5).

Case 1: r ¹=(0,2).

In this case, note that Pr{ u| r ¹=(0,2)}=0 for all u∈ U except (1,2) and (1,3). We have that Pr{ u=(1,2)| r ¹=(0,2)}= and Pr{ u=(1,3)| r ¹=(0,2)}=. Given the choice probabilities computed above, we obtain Pr{ c( x ₁, x ₂)= x ₁| r ¹=(0,2)}=ε and Pr{ c( x ₁, x ₂)= x ₂| r ¹=(0,2)}=1−ε, and adding terms together yields

Hence, substituting in (6),

which is smaller than 0 for any 0<ε<1/2 and 0<β<1/4.

Case 2: r ¹=(1,3).

In this case, Pr{ u ₁=3| r ¹=(1,3)}=0, Pr{ u=(1,2)| r ¹=(1,3)}=Pr{ u=(1,4)| r ¹=(1,3)}=Pr{ u=(2,3)| r ¹=(1,3)}=, Pr{ u=(2,2)| r ¹=(1,3)}=Pr{ u=(2,4)| r ¹=(1,3)}=, Pr{ u=(1,3)| r ¹=(1,3)}=.

Given the choice probabilities, we have

Then, using (7), we have

Substituting in (4.2) yields

which is always strictly smaller than 0 for any 0<β<1/4 and 0<ε<1/2.

Case 3:r ¹=(2,4).

This case is symmetric to the case r ¹=(1,3); all computations are identical and hence

for any 0<ε<1/2 and 0<β<1/4.

Case 4: r ¹=(3,5).

This case is symmetric to the case r ¹=(0,2); all computations are identical and hence

for any 0<ε<1/2 and 0<β<1/4.

Finally, combining the four cases above, equation (5) implies that

which completes the counterexample.

4.3 Simulations

A different way to illustrate the main point is to conduct simulated experiments using our model to describe hypothetical participants with immutable preferences. Izuma and Murayama [2013] followed precisely this path to show that the FCP will generate positive spreading for a particular model (different from ours) and the particular value D=0. Here we use the same approach to illustrate our results.

Figure 1 presents simulations for the options used in the counterexample above, i.e. µ₁=2, µ₂=3, and m=1, with different values of β and ε. In each of one million draws, the actual utilities u ₁, u ₂ are drawn from uniform distributions on {µ_i−1,µ_i,µ_i+1}. The ratings in Phases 1 and 3 and the choices in Phase 2 follow the model described above. Accordingly, in the simulations we obtain spreads for all possible values of D. Figure 1 shows that positive spread is observed for very small values of D, and negative spread is observed for larger values. Our previous counterexample corresponds to the values for D=2.

Figure 1: Simulation with two options, µ₁=2, µ₂=3, m=1, one million draws for each (β,ε) combination.

Figure 2 presents the results of simulations for a symmetric example where µ₁=µ₂=2 and m=1. These are the options used in the counterexample reported in Appendix A. Again, both positive and negative spread can be observed.

Figure 2: Simulation with two options, µ₁=µ₂=2, m=1, one million draws for each (β,ε) combination.

Finally, Figure 3 presents the average results of 60,000 simulated experiments with 80 possible options and 40 participants each. The preferences of the participants (i.e.  the values u _k) are generated randomly by first obtaining random values for µ_k according to a uniform distribution on the set {7,…,13}, then obtaining u _k from a uniform distribution on {µ_k−5,…,µ_k+5}. Hence, realized utilities range from 2 to 18 and realized ratings in Phases 1 and 3 range from 1 to 19. In Phase 2, each participant is presented with 14 choices. A computer algorithm attempts to select two pairs each with distances D=0,1,… 6 according to the initial ratings (depending on the participant’s ratings, it might not always be possible to obtain 14 such pairs), in such a way that no option is used for two different choice pairs. The spread conditional on D is computed for each simulated experiment, and the figure presents average results over 10,000 experiments for each of six possible combinations of β and ε (the simulation took around 8 hours of computing time). As illustrated in the figure, the realized value of the spread Δ can be both positive (for small values of D) and negative (in this example, for D≥ 2). Note that the case D=0 replicates the results of Izuma and Murayama [2013] (who considered only D=0) for a different model.

Figure 3: Average of 60,000 simulated experiments with 80 options, 40 participants, µ_k∈{7,…,13}, and m=5. 10,000 experiments for each (β,ε) combination.

It needs to be noted that FCP experiments inspired by cognitive dissonance theory typically rely on small values of D, since dissonance is generated by closeness to indifference (this would not necessarily be true if the motivation for an experiment arose from self-perception theory). In the simulations presented here, a value of D=0 always result in positive spreading. In any case, we remind the reader that whether a value of D indicates closeness to indifference or not depends on how fine the actual scale used in an experiment is. For a set of options as those in Figure 3, realized ratings vary from 1 to 19, and a distance of 2 (for which negative spreading is already observed) is rather “close”.

5 The mistake(s) in Chen and Risen’s proof

This section explains the mistake made in the proof in Chen and Risen [2010], which leads to the incorrect prediction of positive rating spread in the post-choice rating. Remember that one considers two fixed alternatives x _i and x _j such that r _i ¹= r _j ¹+ D with D>0. The claim is that, using exclusively Assumptions 1, 2, and 3, it is possible to deduce that E[Δ| D]>0. In view of our counterexample and simulations above, this claim is incorrect.

The key mistake appears on p. 578 of Chen and Risen [2010], when the authors argue that because of (1), E[ r _i ³− r _j ³| r _i ¹− r _j ¹= D]= D. In words, this says that, given the realized, possibly biased first rating difference, the expected second rating difference is equal to the first, biased one. This claim is incorrect in general.Footnote ⁶

Consider the example detailed above, i.e., µ₁=2, µ₂=3, m=3, and D=2, exactly as in our counterexample in the previous section. We proceed to compute E[ r _i ³− r _j ³| D=2].

which is not equal to 2 for any 0<β<1/4, showing that the claim E[ r _i ³− r _j ³| r _i ¹− r _j ¹= D]= D is incorrect.

To clarify matters further, consider an extreme case where µ₁=2, µ₂=3, and m=3 as above, but D=5. If the argument of Chen and Risen [2010] were to hold, we would have that E[ r _i ³− r _j ³| r _i ¹− r _j ¹=5]=5. This is not true. Indeed, there is only one possibility for D=5 to be realized in our model, corresponding to ratings r _j ¹=0 and r _i ¹=5. This implies that u _j=1 and u _i=4 for sure. Hence,

Remark 2. Chen and Risen (2010, footnote 6) argue that it is equivalent to state Assumption 3 in terms of Pr{ r _k ^t> r _ℓ ^t| u _k> u _ℓ} or in terms of Pr{ u _k> u _ℓ| r _k ^t> r _ℓ ^t}. This is incorrect, but inconsequential for our analysis. Appendix B discusses this point and shows that our model also fulfills the additional assumption that 1/2 < Pr{ u _k> u _ℓ| r _k ^t> r _ℓ ^t}< 1.

6 Regression to the mean

The Appendix of Chen and Risen [2010] generalizes the approach considering a possible “regression to the mean” effect. In order to account for this possible effect, it is argued that instead of comparing the expected rating spread E[Δ| D] with 0, one should compare it with the quantity − R defined by − R= E[ r _i ³− r _j ³| D]− D, which measures the expected increase of the rating difference in a hypothetical control group where participants rate twice without choice tasks in between. Chen and Risen [2010] claim that, with this adjustment, the FCP will exhibit rating spread strictly larger than the control group’s − R, even if the preferences are stable.

Of course, since this generalization includes the case R=0 as a particular case, it follows from our analysis above that the generalization is also incorrect. However, in this section we provide a more direct illustration.

Consider again our numerical example where u ₁ follows a uniform distribution on {1,2,3} and u ₂ follows a uniform distribution on {2,3,4} again. We compute the rating spread E[Δ| D] and compare it with E[ r _i ³− r _j ³| D]− D for D=0.

Straightforward but cumbersome computations, analogous to the ones illustrated above, allow us to obtain the rating spread for D=0.

Similarly, we can compute E[ r ₂ ³− r ₁ ³| D=0], which yields

According to Chen and Risen [2010], we should obtain that E[Δ| D=0]− E[ r ₂ ³− r ₁ ³| D=0]>0. Straightforward computations show that

which is smaller than zero for ε>1−2β/4−6β. Note that 1−2β/4−6β is smaller than 1/2 for any β<1/4. That is, when ε fulfills the condition above, the rating spread adjusted for the “regression to the mean” effect for D=0 in this numerical example is negative. Hence, the claim of Chen and Risen [2010] is still incorrect after adjusting for a “regression to the mean” effect.

7 A proof of positive spreading for D=0

Our simulations and those of Izuma and Murayama [2013] suggest that in the extreme case with D=0, positive spreading might occur in the absence of choice-induced preference change, i.e. the result stated by Chen and Risen [2010] might be correct in this particular case. This would be intuitive, because when D=0, if actual preferences are close to indifference, we would expect the rating difference in Phase 3 to be also close to zero, but if actual preferences are not close to indifference, the fact that an option is chosen is a signal on the actual preference and we would expect positive spreading. Intuitions, however, can be misleading, and hence we set out to provide a formal proof of this fact. Since the case D=0 is especially prominent in the literature, this proof is of independent interest.

The result we prove is not limited to our model. Rather, it applies generally to any model satisfying a few elementary assumptions. Those include Assumptions 1-2 from Chen and Risen [2010] (Assumption 3, interestingly, is not needed) and a new, minimal assumption dealing with knife-edge cases of exact indifference.

Formally, we consider any model where, given the true utilities, the random variables capturing ratings (in Phases 1 and 3) and choices are all independent. that is, although of course ratings and choices are derived from preferences, the corresponding error terms are independent. Further, Assumptions 1 and 2 are fulfilled, and the following assumption also holds.

Assumption 4: For each two alternatives x _k, x _ℓ,

(8)

This assumption, which is fulfilled in our model, means that even if the observed ratings in the first phase are identical ( D=0), it is not necessarily true that the options are actually indifferent. This does not follow from Assumptions 1-3, because those refer to strict preference only. It is, however, a natural consequence of the idea that ratings are noisy versions of preferences, and it is immediately fulfilled in our model.

Consider now the case D=0. That is, we consider only pairs such that r _i ¹− r _j ¹=0. We want to show that E[Δ| D=0]>0 using only Assumptions 1,2, and 4, rather than relying on a particular model.

To simplify the following equations, let p _i=Pr{ u _i> u _j| D=0}, p _j=Pr{ u _i< u _j| D=0}, and p ₀=Pr{ u _i= u _j| D=0} denote the probabilities that x _i is actually strictly preferred to x _j, that x _j is strictly preferred to x _i, and that the two options are truly indifferent, respectively, all conditional on having observed equal ratings in Phase 1.

First we decompose E[Δ| D=0] with respect to the realizations of utilities as follows.

By Assumption 1, E[Δ| D=0, u _i= u _j]=0. That is, if the participant is truly indifferent, the expected difference in ratings is zero). Hence, the second term above is zero. Note that, if Assumption 4 did not hold, we would have p ₀=1 and hence E[Δ| D=0]=0. The remainder of the analysis hinges on Assumption 4, i.e. p ₀<1.

We now decompose the expression depending on the actual choices in Phase 2. Since D=0, we further simplify the spread by noting that Δ= r _i ³− r _j ³ if x _i is chosen and Δ= r _j ³− r _i ³ if x _j is chosen.

That is, the first of the two major terms (starting with p _i) represents the case where x _i is actually strictly preferred to x _j, subdivided into the cases where x _i is chosen and x _j is chosen. Analogously, the second major term (starting with p _j) corresponds to the case where x _j is strictly preferred to x _i, subdivided into the same cases.

Since, given the utilities, ratings are independent of the choices, we have that E[ r _i ³− r _j ³| D=0, u _i> u _j, c( x _i, x _j)= x _i]= E[ r _i ³− r _j ³| D=0, u _i> u _j, c( x _i, x _j)= x _j], because which option has been actually chosen in Phase 2 does not change the expectation of the ratings in Phase 3. Further, conditional on a given event, E[ r _j ³− r _i ³]=− E[ r _i ³− r _j ³]. Factoring out terms, we can simplify the expression as follows.

Let us consider the terms in this expression separately. By Assumption 1, E _i= E[ r _i ³− r _j ³| D=0, u _i> u _j]= u _i− u _j>0 and E _j= E[ r _j ³− r _i ³| D=0, u _i< u _j]= u _j− u _i>0. By Assumption 2, Q _i=Pr{ c( x _i, x _j)= x _i| D=0, u _i> u _j} − Pr{ c( x _i, x _j)= x _j| D=0, u _i> u _j}>0 (the probability of choosing an option is larger than the complementary if that option is actually the preferred one); analogously, Q _j=Pr{ c( x _i, x _j)= x _j| D=0, u _i< u _j} − Pr{ c( x _i, x _j)= x _i| D=0, u _i< u _j}>0.

We have that

Since by (8) either p _i or p _j are strictly positive (and of course both are weakly positive) and we have just argued that E _i, E _j, Q _i, and Q _j are all strictly positive, it follows that E[Δ| D=0]>0. In conclusion, we have proven the following.

Theorem: For any model of random utilities, ratings, and choices (where ratings in phases 1 and 3 and choices are independent given the utilities) fulfilling Assumptions 1, 2, and 4, the value of E[Δ| D=0] is strictly positive.

We conclude that positive spreading always obtains in the FCP for D=0, for a broad family of models. This result explains the simulation results of Izuma and Murayama [2013] and also our own simulations for the case D=0. Beyond that, it means that in any FCP study, if only pairs with D=0 are considered, one will obtain positive spreading even in the absence of preference change. However, as our previous counterexample shows, this result cannot be extended to the case D>0. If a model incorporates reasonable continuity assumptions, the result can obviously be extended to small values of D. However, whether a particular value of D>0 is small or large depends on the scale (recall, e.g., our simulation in Figure Figure 3). Hence, the result above has limited bearing on studies with D>0 or on those using rankings (where exact indifference is excluded).