Reference BirnbaumBirnbaum (2020) presents a reanalysis of the data in Reference Butler and PogrebnaButler & Pogrebna (2018) using his ‘true and error’ model as the test. This test focuses on choice patterns rather than tests based on binary choice probabilities, which he claims is a better test of transitivity when individuals have different true preference patterns. Not everyone would agree that the true and error model is uniquely suited to tests of transitivity, see for example the remarks in Reference Cavagnaro and Davis-StoberCavagnaro and Davis-Stober (2014), though their position is also strongly contested (Reference Birnbaum and WanBirnbaum & Wang, 2020). However, we believe that his model is a logically coherent alternative to tests of transitivity based on choice probabilities and we view his analysis as a useful supplement to the tests reported in our paper.

Reassuringly, Birnbaum’s reanalysis of our data using the true & error model provides evidence largely consistent with the results reported in Reference Butler and PogrebnaButler and Pogrebna (2018), obviating any need to reconcile differences. It would be useful to understand how to apply or extend the true and error model for the comparisons between an individual’s binary and ternary choices, as that is a key part of our study of intransitivity in the paper. There are two other observations we would like to make regarding Reference BirnbaumBirnbaum (2020). The first concerns his remarks on the evidence for most probable winner (MPW) cycles, and the second, on the significance of predictably intransitive choice patterns.

Reference Butler and PogrebnaButler and Pogrebna (2018) explain that the inspiration for their experiment was a statistical paradox in applied probabilities, best known as the paradox of intransitive dice. This paradox, which we labelled the STP (Steinhaus-Trybula Paradox), occurs over three, paired lottery comparisons, when the relation ‘stochastically greater than’ leads to an intransitive cycle. In this context, the relation ‘stochastically greater than’ is equivalent to ‘most probable winner’ cycles. For three lotteries to generate the paradox each of the three lotteries comprising the binary comparisons needs to take on a particular form; that is, it needs to follow a particular ‘recipe’.

We noticed that this ‘recipe’ for lottery design bore a striking resemblance to the structure of the famous ‘preference reversal’ (PR) $- and P-bets (Lichtenstein & Slovic, 1971), where the “$” bet has a higher maximal prize and the “P” bet has the higher probability of winning, and where the (new) third object is a degenerate sum obtained with certainty. However, there were key differences between the new STP set and those from the standard preference reversal gambles, which motivated our interest in designing an experiment. In Reference Butler and PogrebnaButler and Pogrebna (2018), we tailor made lottery triples to this new recipe and labelled them ‘STP triples’. For details of our recipe and the reasoning behind them, see Reference Butler and PogrebnaButler and Pogrebna (2018) and Reference Butler and BlavatskyyButler and Blavatskyy (2020).

As we noted in Reference Butler and PogrebnaButler and Pogrebna (2018, p. 219), we suspected our choice of lottery triples might produce a tension as to the expected direction any cycles might take. While the STP structure may normally nudge choices toward probable winner cycles, i.e., P>CE>$>P, we also chose to mimic the ranking of expected values typical of the PR lotteries, to allow for risk aversion. For PR lotteries, intransitive cycles typically take the form of P>$>CE>P, sometimes called ‘regret’ cycles. Birnbaum comments that evidence for regret cycles in our data is the more convincing of the two, which we accept. We were agnostic as to which direction of cycles might predominate given the two offsetting influences. Our conjecture was simply that lottery designs constructed from the STP structure might prove to be fertile territory to locate preference cycles and our experiment was a small first attempt to explore this idea. We encourage interested readers to build on our initial exploration of STP lotteries following the general principles we outlined; we suspect that territory that is even more fertile than we found awaits discovery.

Secondly, Birnbaum previously asked whether violations of transitivity “are akin to friction in physics labs, real phenomena that can be modelled but which merely complicate the study of the main principles that govern decision making”. Our answer to this is ‘no’; rather than complicating the study of these principles, it is through unusual phenomena such as this paradox that new insights can be obtained. The STP shows how reliance on transitivity over three binary comparisons can lead to a Pareto-inferior choice in the third comparison; is there nothing decision theory can learn from that? As an analogy, observe that the frequency of lottery triples, of the kind studied, meeting the STP is up to 1/56 of all unique triples (Reference Butler and BlavatskyyButler & Blavatskyy, 2020). This proportion is a little less than one-third that for the comparable Condorcet ‘paradox of voting’ cycles, which is 1/18. No one questions the import of the voting paradox. We believe the STP is of similar importance. As Saari has noted (1995), “one must expect that many of the mathematical paradoxes from the decision and statistical sciences have been manifested by groups unknowingly selecting inferior alternatives.” Understanding the STP may help individuals too to avoid this fate.

As a further analogy, consider 2×2 games; just one out of 78, unique configurations of rankings of 2×2 payoffs produces the famous ‘prisoner’s dilemma’. The best response, dominant strategy choice rule can lead to Pareto inferior outcomes, if followed given this configuration of consequences. However, this is a rule that if available works perfectly well for any games comprised of the other 77 rank orderings of consequences; it fails for only 1/78 them. Should social scientists have responded that this example is a mere blemish, a friction that can arise when using the ‘best response, dominant strategy’ choice rule? Surely, our understanding of society would today be much the poorer if they had.

What else do we see as the importance of the STP for understanding decision-making? We think the STP triples identified in Reference Butler and PogrebnaButler and Pogrebna (2018) cast a shadow of doubt over current transitive models of choice under risk. A core utility theory, whether EU, prospect theory or some other theory, represents an individual’s preferences over all sets of lotteries. However, the STP shows that we can construct pairs of lotteries for which three binary comparisons should violate transitivity. Assuming that the decision maker’s preference is to maximise the probability of earning the greater return, to obey transitivity in these cases would produce a less advantageous outcome (Reference Butler and BlavatskyyButler & Blavatskyy, 2020). (In contrast, not obeying transitivity outside the STP structure would result in the less advantageous outcome.) For this reason, we see no point proposing non-transitive core preference theories, such as most probable winner theory (Reference BlavatskyyBlavatskyy, 2006) or regret theory (Reference Loomes and SugdenLoomes & Sugden, 1982). However, this same reason must apply also to theorists’ transitive core utility theories. At the least, they should state the domain of preference profiles for which their theories do not apply, such as for the lottery pairs we identify. Rather than being a distraction from the main game, the STP may prove to be the catalyst for the next generation of choice models.