1. Introduction
Decades of research in psychology, economics, marketing, and neuroscience have been dedicated to understanding context effects in multi-alternative, multi-attribute choice. The overwhelming majority of this work has focused on three specific context effects–the attraction (Huber et al., Reference Huber, Payne and Puto1982), similarity (Tversky, Reference Tversky1972), and compromise (Simonson, Reference Simonson1989) effects. These effects occur when choices between two alternatives can vary based on the presence of a third alternative. The three effects are important because they violate the principles of classic economic theories of choice including simple scalability, independence of irrelevant alternatives, and regularity (Roe et al., Reference Roe, Busemeyer and Townsend2001). Hence, these effects cannot be explained by standard normative models of utility where every option is evaluated independently of each other (but see Farmer et al., Reference Farmer, Warren, El-Deredy and Howes2017; Howes et al., Reference Howes, Warren, Farmer, El-Deredy and Lewis2016; Kruis et al., Reference Kruis, Maris, Marsman, Bolsinova and van der Maas2020 for rational explanations of context effects). Therefore, researchers have looked for psychological explanations for these effects. See Busemeyer et al. (Reference Busemeyer, Gluth, Rieskamp and Turner2019); Trueblood (Reference Trueblood2022); Wollschlaeger and Diederich (Reference Wollschlaeger and Diederich2020) for a review of recent findings and computational models.
Research investigating context effects has revealed an intriguing pattern of results (Evans et al., Reference Evans, Holmes, Dasari and Trueblood2021b; Spektor et al., Reference Spektor, Bhatia and Gluth2021; Trueblood et al., Reference Trueblood, Brown and Heathcote2015). On the one hand, context effects, and in particular the attraction effect, seem to be robust and ubiquitous. That is, they occur across a broad variety of different domains, including perception (Trueblood, Reference Trueblood2015; Trueblood et al., Reference Trueblood, Brown, Heathcote and Busemeyer2013, Reference Trueblood, Brown and Heathcote2015), risky decision making (Farmer et al., Reference Farmer, Warren, El-Deredy and Howes2017), inference (Choplin and Hummel, Reference Choplin and Hummel2005; Trueblood, Reference Trueblood2012), motor-planning decisions (Farmer et al., Reference Farmer, Warren, El-Deredy and Howes2017), and consumer choice (Dhar et al., Reference Dhar, Nowlis and Sherman2000; Huber et al., Reference Huber, Payne and Puto1982; Mishra et al., Reference Mishra, Umesh and Stem1993; Noguchi and Stewart, Reference Noguchi and Stewart2014; Ratneshwar et al., Reference Ratneshwar, Shocker and Stewart1987; Sen, Reference Sen1998; Wedell and Pettibone, Reference Wedell and Pettibone1996). In addition, research in developmental psychology and behavioral ecology has shown that the attraction effect occurs in children (Zhen and Yu, Reference Zhen and Yu2016), monkeys (Parrish et al., Reference Parrish, Evans and Beran2015), cats (Scarpi, Reference Scarpi2011), hummingbirds (Bateson et al., Reference Bateson, Healy and Hurly2003), frogs (Lea and Ryan, Reference Lea and Ryan2015), honeybees (Shafir et al., Reference Shafir, Waite and Smith2002) and even slime molds (Latty and Beekman, Reference Latty and Beekman2011). Thus, there is a sense that contextual sensitivity is a universal property of multi-alternative choice behavior. On the other hand, these effects are also fragile, in that they may disappear or even reverse under certain conditions. The pattern of findings where context effects appear, vanish or reverse has sparked a lively debate about the underlying mechanisms responsible for multi-alternative, multi-attribute decision-making (see Cataldo and Cohen, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021a; Frederick et al., Reference Frederick, Lee and Baskin2014; Huber et al., Reference Huber, Payne and Puto2014; Spektor et al., Reference Spektor, Kellen and Hotaling2018, Reference Spektor, Kellen and Klauer2022; Trendl et al., Reference Trendl, Stewart and Mullett2021; Trueblood et al., Reference Trueblood, Brown and Heathcote2015; Tsetsos et al., Reference Tsetsos, Chater and Usher2015; Yang and Lynn, Reference Yang and Lynn2014 for examples). Hence, understanding both the presence and absence of these effects is crucial to theory development.
Of the three effects, the attraction effect also known as the asymmetrically dominated alternative effect or the decoy effect (Huber and Puto, Reference Huber and Puto1983; Huber et al., Reference Huber, Payne and Puto1982; Simonson, Reference Simonson1989) has received the most attention. In the attraction effect, when choosing between two alternatives—a target and a competitor, the presence of a third decoy alternative that is similar but inferior to the target increases the choices of the target option. The attraction effect has been important both theoretically (Dhar and Glazer, Reference Dhar and Glazer1996; Huber et al., Reference Huber, Payne and Puto1982) as well as in applied settings, such as policy issues (Herne, Reference Herne1997; Stoffel et al., Reference Stoffel, Yang, Vlaev and von Wagner2019) and understanding gender bias in hiring decisions (Keck and Tang, Reference Keck and Tang2020).
Recent research investigating the attraction effect has revealed an interesting pattern of findings with it appearing, disappearing or reversing under different conditions based on the way information is displayed (Cataldo and Cohen, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021a; Frederick et al., Reference Frederick, Lee and Baskin2014; Spektor et al., Reference Spektor, Kellen and Klauer2022; Trueblood et al., Reference Trueblood, Brown and Heathcote2015; Yang and Lynn, Reference Yang and Lynn2014). These findings indicate that the way information is displayed, such as the mode of presentation (e.g., numeric vs. visual, Frederick et al., Reference Frederick, Lee and Baskin2014; Trendl et al., Reference Trendl, Stewart and Mullett2021; Yang and Lynn, Reference Yang and Lynn2014), presentation order (Evans et al., Reference Evans, Holmes, Dasari and Trueblood2021b), and presentation format (Cataldo and Cohen, Reference Cataldo and Cohen2019; Chang and Liu, Reference Chang and Liu2008), influences the presence of the attraction effect.
Understanding the appearance and disappearance of the attraction effect can aid theory development and ultimately shed light on the processes that govern multi-alternative, multi-attribute choice behavior. For example, Trueblood et al. (Reference Trueblood, Brown and Heathcote2015) and Berkowitsch et al. (Reference Berkowitsch, Scheibehenne and Rieskamp2014) showed that the compromise and attraction effects often co-occur while the similarity effect is negatively related to them. This pattern of correlations is now a benchmark for theories of context effects. Going forward, findings in Cataldo and Cohen (Reference Cataldo and Cohen2019) and Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) showing that the attraction effect depends on presentation layout and presentation order of the options might also become benchmarks for theories of context effects. However, more empirical work is needed to establish these and other moderators of the attraction effect before they are adopted as modeling benchmarks. The present work aims to accomplish this goal.
Current models of multi-alternative, multi-attribute decision-making do not make predictions about the dependence of context effects on presentation formats (Bhatia, Reference Bhatia2013; Busemeyer et al., Reference Busemeyer, Gluth, Rieskamp and Turner2019; Noguchi and Stewart, Reference Noguchi and Stewart2018; Roe et al., Reference Roe, Busemeyer and Townsend2001; Trueblood, Reference Trueblood2022; Trueblood et al., Reference Trueblood, Brown and Heathcote2014; Turner et al., Reference Turner, Schley, Muller and Tsetsos2018; Usher and McClelland, Reference Usher and McClelland2004). These models use attribute values as inputs and describe how these values are transformed and compared to each other over time to produce a choice. They do not explain how the presentation format might affect these mechanisms.
For example, one leading model of context effects is the Multiattribute Linear Ballistic Accumulator (MLBA) (Evans et al., Reference Evans, Holmes and Trueblood2019; Trueblood et al., Reference Trueblood, Brown and Heathcote2014; Turner et al., Reference Turner, Schley, Muller and Tsetsos2018). According to the MLBA, preference for an option is calculated as a weighted sum of pairwise comparisons between that option and the other options in the choice set. The weights (serving as a proxy for attention) on each pairwise comparison are related to the similarity of the attributes being compared, with more weight placed on options with similar attributes. When a decoy option that is similar but inferior to the target is introduced in the choice set, more weight is placed on that difference. Thus, comparisons of the target and decoy (which are favorable to the target) are weighted more when calculating preference. This leads to the target being chosen more often, resulting in the attraction effect (Trueblood et al., Reference Trueblood, Brown and Heathcote2014). This model could be extended in various ways to explain the impact of presentation formats on context effects. For example, Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) extended the MLBA to explain how the temporal presentation of options influences the attraction effect. With regard to spatial order, the attention paid during a comparison might also be a function of the spatial proximity of different alternatives, where alternative pairs that are spatially close to each other receive more attention. However, in its original form, the MLBA makes no predictions about how attention might depend on the way the information is presented.
To understand the complex dependence of context effects on presentation formats, theorists will have to elaborate on the processes that convert information from the external displays to internal representations (Spektor et al., Reference Spektor, Bhatia and Gluth2021). One might for example theorize about attentional processes that select information from external displays. These attention processes have been identified to play a central role in decision making (Krajbich, Reference Krajbich2019; Krajbich et al., Reference Krajbich, Armel and Rangel2010; Noguchi and Stewart, Reference Noguchi and Stewart2018) and multi-alternative, multi-attribute choice (Trueblood, Reference Trueblood2022). This may result in modifying a model, as described above in the case of the MLBA, where alternative pairs are selected based on their spatial proximity and other visual factors. Eye tracking studies have suggested that the processes that select information from external displays are important and of theoretical interest since they are interwoven with decision-making processes such as comparison and evidence accumulation (Krajbich, Reference Krajbich2019; Krajbich et al., Reference Krajbich, Armel and Rangel2010; Noguchi and Stewart, Reference Noguchi and Stewart2014, Reference Noguchi and Stewart2018).
Theory building requires one to have a robust set of empirical findings that lays the groundwork for model development. As we shall see in the following section, existing literature reports conflicting empirical results regarding the impact of presentation format on the attraction effect. Past studies have also used a wide range of tasks and stimuli. For example, some studies have focused on perceptual decision making (Evans et al., Reference Evans, Holmes, Dasari and Trueblood2021b; Spektor et al., Reference Spektor, Kellen and Hotaling2018, Reference Spektor, Kellen and Klauer2022; Trueblood et al., Reference Trueblood, Brown, Heathcote and Busemeyer2013) as compared to preferential choice (Cataldo and Cohen, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021b; Frederick et al., Reference Frederick, Lee and Baskin2014). Further, some studies have shown that the attraction effect fails to replicate in certain situations (Frederick et al., Reference Frederick, Lee and Baskin2014; Xiao et al., Reference Xiao, Zeng and Feldman2020). This paper builds on recent studies examining the dependence of multiattribute choice on presentation format and provides a comprehensive account of how presentation mode, layout, and order impact the attraction effect, providing rich empirical results for future theory development.
1.1. Presentation format factors
In this section, we review the different presentation factors that are known to affect the attraction effect. We have identified three important factors that are thought to affect the attraction effect—mode of presentation, presentation layout, and presentation order.
1.1.1. Mode of presentation
Frederick et al. (Reference Frederick, Lee and Baskin2014) argued that the attraction effect only occurs for numerically presented stimuli. They showed that most of the previous experiments demonstrating the attraction effect used information presented in a numerical format. They found that the attraction effect failed to generalize to naturalistic stimuli that did not use numbers. For example, in one scenario, they asked participants to choose between apples and oranges shown in pictures, where the decoy option was a moldy apple or a bruised orange. In another image based choice scenario, they asked participants to choose between Tropicana orange juice and milk by targeting the Tropicana orange juice by adding an additional option of a ‘stop and shop orange juice’ as a decoy. They also showed that graphical information, such as pie-charts, did not bring about the attraction effect. These results were further bolstered by Yang and Lynn (Reference Yang and Lynn2014), who independently found that images, verbal descriptions, and qualitative descriptions failed to produce the attraction effect. For example, in one scenario, participants chose between ‘Coca-Cola’ and ‘Sprite’ with either the decoy ‘Cola’ or ‘Refreshe’, respectively. In another example, participants chose between three named food dishes based on their calorie content and price. The options were ‘Grilled Chicken Caesar Salad’ and a ‘Grilled Steak Caesar Salad’ with either the decoy ‘Crispy Chicken Caesar Salad’ or ‘Flat Iron Steak Frites’, respectively. More recently, Trendl et al. (Reference Trendl, Stewart and Mullett2021) showed that the attraction effect does not occur for naturalistic stimuli, specifically choices among movies based on their title. Hence, they argued that the attraction effect is very limited and only occurs in highly stylized experiments with numerical stimuli.
This is in contrast to findings that the attraction effect holds with qualitative descriptions (Kivetz et al., Reference Kivetz, Netzer and Srinivasan2004; Sen, Reference Sen1998) and physical objects (Ratneshwar et al., Reference Ratneshwar, Shocker and Stewart1987; Simonson and Tversky, Reference Simonson and Tversky1992). In fact, studies have shown that the attraction effect persists even when participants are asked to perform low-level perceptual tasks such as judging the similarity of shapes (Choplin and Hummel, Reference Choplin and Hummel2005) or choosing rectangles with the largest area (Trueblood et al., Reference Trueblood, Brown, Heathcote and Busemeyer2013). All of these studies failed to replicate in (Frederick et al., Reference Frederick, Lee and Baskin2014) when picnickers were used as participants (except for Choplin and Hummel (Reference Choplin and Hummel2005) which was not tested). Since then, other studies have recreated the attraction effect using different presentation modes for the options. For instance, Cataldo and Cohen (Reference Cataldo and Cohen2019) have recreated the attraction effect using bar graphs. Dimara et al. (Reference Dimara, Bezerianos and Dragicevic2016) have recreated it using scatter plot display formats, and Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b), Farmer et al. (Reference Farmer, Warren, El-Deredy and Howes2017), Trueblood et al. (Reference Trueblood, Brown and Heathcote2015), Turner et al. (Reference Turner, Schley, Muller and Tsetsos2018) recreated it with perceptual stimuli similar to Trueblood et al. (Reference Trueblood, Brown, Heathcote and Busemeyer2013). Taken together, Spektor et al. (Reference Spektor, Bhatia and Gluth2021) identified the ‘concreteness’ of representation as an important factor moderating the attraction effect. In our experiment, we use both graphical displays as in Cataldo and Cohen (Reference Cataldo and Cohen2019) along with numerical displays to test the importance of presentation mode (see Figure 1 for examples of graphical and numerical stimuli used).
1.1.2. Presentation layout
The presentation layout of information has been shown to influence the presence of context effects (Cataldo and Cohen, Reference Cataldo and Cohen2018, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021b; Chang and Liu, Reference Chang and Liu2008; Liu and Trueblood, Reference Liu and Trueblood2023). In Chang and Liu (Reference Chang and Liu2008) and Cataldo and Cohen (Reference Cataldo and Cohen2019) the authors showed that changing the display of options from ‘1by-attribute’ to ‘by-alternative’ impacted context effects. The by-attribute format lists alternatives in rows and attributes in columns, which may facilitate easy comparison of the same attribute for different alternatives. The by-alternative format lists alternatives in columns and attributes and rows and is designed to impede comparisons between alternatives, thus different alternatives are compared more holistically rather than by their individual attributes. Cataldo and Cohen (Reference Cataldo and Cohen2019) showed that when the alternatives are compared in a by-attribute format, the attraction effect appears. However, when alternatives are compared in a by-alternative format, then the attraction effect disappears. Hence, by-attribute and by-alternative display formats have been identified as strong modulators of the attraction effect (Spektor et al., Reference Spektor, Bhatia and Gluth2021).
The same choice set in different presentation formats used in our experiment. The panels on the left show the graphical formats and the panels on the right show the numerical formats. The top row contains examples of the by-alternative layout. The bottom row contains examples of the by-attribute layout. In each condition, Apartments A and B are the target and competitor, respectively. Apartment C is the decoy option that is dominated by A but not by B.
However, other studies examining by-attribute and by-alternative formats have found conflicting results. For example, Cataldo and Cohen (Reference Cataldo and Cohen2021b) used both graphical (two experiments) and numerical (three experiments) formats. We reanalyzed their data and found a complex pattern of attraction and repulsion effects (full details and results are presented in the Supplementary Materials). Briefly, similar to Cataldo and Cohen (Reference Cataldo and Cohen2019), our reanalysis found that graphical formats showed an attraction effect in the by-attribute format but a weak repulsion effect in one of their two experiments in the by-alternative format. Regarding the numerical format, in one experiment where attributes were presented along rows and alternatives along columns, we found a net null effect. In a second experiment, when alternatives were presented in rows and attributes in columns, we observed a weak attraction effect. Finally, in a third experiment, with a similar format as the second experiment, when boxes were drawn around the attributes, potentially to facilitate more attribute-wise comparison, once again, we observed a net null effect. Liu and Trueblood (Reference Liu and Trueblood2023) conducted experiments where participants had to choose between tasks based on difficulty and length. They used numerical by-attribute and by-alternative formats similar to the ones in this paper. They observed a weak repulsion effect and no attraction effect in all of the formats among a large variety of manipulations. These results challenge the ideas that concreteness of representation (Spektor et al., Reference Spektor, Bhatia and Gluth2021) and numerical formats (Frederick et al., Reference Frederick, Lee and Baskin2014; Yang and Lynn, Reference Yang and Lynn2014) are essential to the emergence of the attraction effect.
In our experiment, we follow Cataldo and Cohen (Reference Cataldo and Cohen2019) and present information in a by-alternative and by-attribute format to test the importance of the presentation format. We cross this manipulation with presentation mode to examine how presentation format impacts graphical vs. numerical stimuli. See Figure 1 for examples of by-alternative and by-attribute formats used in our experiment.
1.1.3. Presentation order
Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) showed that the order in which the three options, target (T), competitor (C) and decoy (D), are presented can modulate the attraction effect. Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) reanalyzed data from Trueblood et al. (Reference Trueblood, Brown and Heathcote2015) where the spatial order (i.e., the left to right placement) of the three options was varied. When all orderings were considered together, these stimuli showed a small attraction effect. However, of the 6 orders, 3 orders (CTD, DTC, and TDC, where the order of the letters corresponds to the left, middle, and right placement of options) showed a strong attraction effect, 2 orders (DCT and TCD) showed a strong repulsion effect, and 1 order (CDT) showed a null/weak attraction effect. Their conclusion was that the close proximity of the target and the decoy facilitated a series of comparisons that brought about the attraction effect.
In a reanalysis of Cataldo and Cohen (Reference Cataldo and Cohen2021a), we also examined the impact of presentation order on attraction and repulsion effects (full results are presented in the Supplementary Materials). This experiment used graphical formats similar to Cataldo and Cohen (Reference Cataldo and Cohen2019) shown in Figure 1. To summarize, the impact of presentation order differed between by-attribute and by-alternative formats for graphically represented options. As mentioned above, the by-alternative format had a weak repulsion effect on average. However, this repulsion effect was only significant in two of the 6 orders—TCD and TDC. In contrast, in the by-attribute format, there was a significant attraction effect on average. However, it was strongly significant in 3 of the 6 orders (CTD, DTC and TDC) and weakly significant in the TCD order. As shown in the Supplementary Materials, these results change slightly when the exclusion criterion is removed.
Hence, in the by-attribute graphical formats, the attraction effect is the strongest with the same orders as those of the perceptual decision-making from Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b). However, none of the orders show a repulsion effect. For the by-alternative format, none of the orders showed an attraction effect. However, TCD showed a repulsion effect as in the perceptual decision-making task. Intriguingly, compared to the perceptual decision-making task and the by-attribute graphical display, the reversal with TDC suggests an interaction between presentation layout and presentation order.
In our experiment, we vary the order in which the options are presented to participants to test whether the presentation order continues to influence the attraction effect in all of the conditions mentioned above.
1.2. Experimental Aims
Previous studies have independently examined the impact of various presentation factors on the attraction effect. As mentioned above, these studies have used a vast set of stimuli and tasks and have come to different conclusions. Yang and Lynn (Reference Yang and Lynn2014) and Frederick et al. (Reference Frederick, Lee and Baskin2014) showed that the attraction effect is weaker with graphical stimuli than with numerical stimuli. However, since then, the attraction effect has been recreated using graphical stimuli (Cataldo and Cohen, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021a; Spektor et al., Reference Spektor, Kellen and Hotaling2018; Trueblood et al., Reference Trueblood, Brown and Heathcote2015). It has also been reversed for numerical stimuli (Liu and Trueblood, Reference Liu and Trueblood2023) or found to be weaker in our re-analysis of data from Cataldo and Cohen (Reference Cataldo and Cohen2021b) with numerical stimuli. There have only been a few studies (Cataldo and Cohen, Reference Cataldo and Cohen2019, Reference Cataldo and Cohen2021a) that examine the dependence of the attraction effect on presentation layout (by-alternative vs. by-attribute). These studies show that presentation layout modulates the attraction effect in graphical displays. The influence of presentation layout on the attraction effect with numerical stimuli has been either null or weakly positive or negative (Cataldo and Cohen, Reference Cataldo and Cohen2021b; Liu and Trueblood, Reference Liu and Trueblood2023) (but see Chang and Liu (Reference Chang and Liu2008) for modulation of the compromise effect with numerical stimuli). The effect of changing presentation order has been shown for perceptual decision-making (Evans et al., Reference Evans, Holmes and Trueblood2019), but it remains unknown how order impacts numerical stimuli. In our reanalysis of Cataldo and Cohen (Reference Cataldo and Cohen2021a), we showed that the impact of changing presentation order may be different for consumer decision-making in different presentation formats. Hence, empirically, it is not clear which of these effects are robust. Further, as shown with our reanalysis of Cataldo and Cohen (Reference Cataldo and Cohen2021a), presentation layout and presentation order might interact with each other. Thus, it is important to systematically vary all of these factors in a single large-scale experiment.
At this point, we reiterate the theoretical positions and predictions made by different studies. Frederick et al. (Reference Frederick, Lee and Baskin2014) and Yang and Lynn (Reference Yang and Lynn2014) predict that the attraction effect will be stronger or only occur when information is represented in numerical formats as compared to graphical formats. Spektor et al. (Reference Spektor, Bhatia and Gluth2021) hypothesized that this might be due to the fact that numerical formats are represented more concretely, making the domination of the decoy by the target clearer. Cataldo and Cohen (Reference Cataldo and Cohen2019) predict that the attraction effect will be stronger in by-attribute formats than by-alternative formats since this facilitates more comparisons between options. This is consistent with the idea that certain comparisons are facilitated in certain formats and that these comparisons are integral to the construction of choice (Cataldo and Cohen, Reference Cataldo and Cohen2019; Noguchi and Stewart, Reference Noguchi and Stewart2014, Reference Noguchi and Stewart2018; Spektor et al., Reference Spektor, Kellen and Klauer2022). For presentation order, the results from Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) indicate that the attraction effect is enhanced when the decoy and the target occupy neighboring positions. This might also be due to an ease of comparison between the target and the decoy.
In order to lay the groundwork for further theory development, we need a comprehensive understanding of how different aspects of presentation format impact multi-alternative, multi-attribute choice. However, it is not clear which of these findings are robust and generalizable. Since existing theories do not naturally (without modification) predict or explain these presentation effects, we think it is important to lay a firm foundation for future theory building. That is the development of new models or extensions to existing models that can account for more complex patterns of behavior. Thus, there is a need to evaluate which of these formats facilitate the attraction effect and which ones potentially reverse it. We address this need in this paper by using a factorial design to carefully manipulate the different presentation factors that are thought to affect the size and direction of the attraction effect.
Further, the combined effects of these factors are not known. For example, Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) argued that the presentation order facilitates certain comparisons between options that are spatially closer to each other. Similarly Cataldo and Cohen (Reference Cataldo and Cohen2019) argued that by-alternative and by-attribute displays facilitate certain comparisons between the alternatives and attributes. However, when both manipulations are implemented simultaneously, it is not immediately clear what the results will be without an empirical test. Our re-analysis of Cataldo and Cohen (Reference Cataldo and Cohen2021a) suggests that there might be interesting ways in which these factors interact with each other.
Our approach differs from the recommendation made by proponents of the crucial experimentation hypothesis (Lakatos, Reference Lakatos1974) or the strong induction hypothesis (Platt, Reference Platt1964) where experiments are designed to test competing models. Instead, we systematically vary factors that are of relevance (Almaatouq et al., Reference Almaatouq, Griffiths, Suchow, Whiting, Evans and Watts2022; Dubova et al., Reference Dubova, Moskvichev and Zollman2022; Peterson et al., Reference Peterson, Bourgin, Agrawal, Reichman and Griffiths2021) to multi alternative, multiattribute decision making and the attraction effect more specifically. As mentioned above, the current models of multiattribute choice do not make strong predictions about how presentation formats impact decision-making. This experiment will serve as a high-powered benchmark and provide researchers with a robust set of empirical findings that will aid in future model development (Noguchi and Stewart, Reference Noguchi and Stewart2018; Trueblood et al., Reference Trueblood, Brown and Heathcote2014; Turner et al., Reference Turner, Schley, Muller and Tsetsos2018).
2. Method
In our experiment, we examine the presence of the attraction effect among 6 different consumer goods (3 novel goods and 3 goods from Noguchi and Stewart (Reference Noguchi and Stewart2014)) by varying the mode of presentation, presentation layout, and presentation order in which the options are displayed. We use 2 different presentation modes (numerical and graphical), 2 different presentation layouts (by-alternative and by-attribute), and 6 different presentation orders. This gives a total of 24 ternary choice conditions. The data and analysis code are available on the Open Science Framework at https://osf.io/wzct3/.
2.1. Participants
We aimed to recruit a total of 2300 participants from Amazon Mechanical Turk using CloudResearch in exchange for a cash payment of $
$0.50$
per person. We included participants who were over 18 years of age and based in the United States with matching IP addresses. To maintain data quality, we only allowed individuals who had a 95% or higher HIT approval rate and were approved by Cloud Research’s attention and engagement measures to participate. Using CloudResearch, we also blocked participants based on their list of suspicious geocode locations and with duplicate IP addresses. We pre-registered our manuscript after first stage acceptance on the Open Science Framework (OSF): https://doi.org/10.17605/osf.io/sgj3w. The experiment was approved by the Institutional Review Board at Indiana University. We only consider the data of the individuals who completed the entire experiment.
Since the goals of our experiment are exploratory, we will test the robustness of our results with and without exclusions. We apply two exclusion criteria that are often used in the literature. We included two attention checks in our experiment to exclude participants who were not attentive during the experiment. In previous experiments, for example, Cataldo and Cohen (Reference Cataldo and Cohen2019) and Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b), participants were excluded for failing catch trials. In the same vein, we will exclude participants who chose the decoy option for more than 3 out of 12 trials since they selected an option that was dominated.
Ultimately, a total of 2,293 participants completed the experiment (gender: 903 males, 1357 females, 20 non-binary/third gender, 13 preferred not to say; age in years: M =
$43.3$
, SD =
$13.4$
, IQR:
$33 - 53$
; race: 1791 White, 218 Black or African American, 136 Asian, 74 Multiracial, 14 American Indian or Alaska Native, 6 Native Hawaiian or Other Pacific Islander, 54 preferred not to say; income: 163 less than $15,000, 211 $15,000–$25,000, 561 $25,000-$50,000, 525 $50,000–$75,000, 494 $75,000–$120,000, 273 more than $120,000, 66 preferred not to say). As described below, there were four between-subject conditions. In the binary, by-alternative condition we had 147 participants. In the binary, by-attribute condition we had 150 participants. In the ternary, by-alternative condition we had 999 participants. In the ternary, by-attribute condition we had 997 participants. The experiment took an average of
$6.8$
(IQR:
$4.1-7.6$
) minutes to complete.
Sample size calculation: We analyze the attraction effect using the relative choice share of the target (RST) determined by the ternary choice sets (Berkowitsch et al., Reference Berkowitsch, Scheibehenne and Rieskamp2014). The RST is defined as the ratio of the number of times the target has been selected to the number of times the target and the competitor have been selected:
$$\begin{align*}RST = \frac{n_{\text{target}}}{n_{\text{target}}+n_{\text{competitor}}}.\end{align*}$$
If the RST is significantly greater than 0.5, then there is evidence of an attraction effect. RST values significantly less than 0.5 provide evidence of a repulsion effect. To estimate the sample size (n), we conducted the following power analysis. We assumed that each trial is a Bernoulli variable with an underlying probability of p of selecting the target. To show evidence of the attraction effect, we need to show that the probability of selecting the target is significantly greater than
$0.5$
on trials where either the target or competitor was selected (that is, we exclude the trials where the decoy was selected). This is equivalent to testing whether the RST value is significantly greater than 0.5. We tested this using an exact binomial test.
Suppose that we observe that the target has been selected
$n_{\text {target}}$
out of n (=
$n_{\text {target}}$
+
$n_{\text {competitor}}$
) times. The exact binomial test will be significant if
$n_{\text {target}}$
is greater than the threshold t that is given by the
$97.5^{th}$
percentile of the binomial distribution generated with a probability of
$0.5$
with n repetitions. Suppose our desired power is P, we would want this result to be significant with a probability of at least P. We find the smallest n such that the outcome is
$n_{\text {target}}$
of a binomial random variable generated with a probability p with n trials greater than t with a probability of P. We plot these values for different probabilities of selecting the target p and power P in Figure 2.
This figure shows the sample size (n) required for the exact binomial test to be significant for a given power (
$P)$
. The sample size (
$n)$
required is plotted on the y-axis and the unknown underlying probability (p) of selecting the target is plotted on the x-axis. The different lines show the sample size required for different powers P after accounting for the Bonferroni correction at
$p=0.05/24$
. The vertical lines allow us to see the sample size required for different values of p. The horizontal line at
$n=900$
shows that we would be able to detect an effect with
$p=0.58$
with a high power
$P=0.95$
. When old stimuli and novel stimuli are considered separately, we would be able to detect an effect with
$p=0.61$
with a power of
$P=0.95$
.
To select an estimate of the probability p of selecting the target, we used publicly available data from Noguchi and Stewart (Reference Noguchi and Stewart2014) to calculate RST values. We present all of our results in the Supplementary Materials. As described in the following section, these stimuli used identical attribute names and values as some of the stimuli used in our experiment. The RST values were
$82.0\%$
,
$80.0\%$
, and
$64.4\%$
for the stimuli considered in our experiment. However, other studies find much lower values, such as
$50.4\%$
and
$55.2\%$
in Cataldo and Cohen (Reference Cataldo and Cohen2019).
As shown in the figure, the number of trials required changes drastically when the probability of selecting the target is around
$0.55$
. At a reasonably high power of
$0.95$
, we would require about
$900$
trials to detect an effect with an underlying probability of selecting the target of
$0.58$
after accounting for the Bonferroni correction at
$p=0.05/24$
. Since we have a total of 6 (order) x 2 (layout) x 2 (format) = 24 ternary choice conditions, we need a total of
$21,600$
ternary choice trials. Since each participant does
$12$
trials, we obtain about
$1,800$
participants. Participants choose the decoy option about
$5\%-10\%$
of the time (Cataldo and Cohen, Reference Cataldo and Cohen2021b; Frederick et al., Reference Frederick, Lee and Baskin2014; Noguchi and Stewart, Reference Noguchi and Stewart2014). These trials will need to be removed from our RST analysis. We account for this in our estimate and obtain our estimate of
$2,000$
participants. These calculations assume that our analyses are conducted by combining the 6 different consumer goods in our experiment. We also plan to analyze old stimuli (i.e., the three consumer goods from Noguchi and Stewart (Reference Noguchi and Stewart2014)) and novel stimuli separately to test for reliability and generalizability. If we consider each of the 6 goods separately, we obtain about
$150$
trials per condition. Thus, we will have 450 trials per condition after combining the three old consumer goods and 450 trials per condition after combining the three new consumer goods. This will allow us to detect an effect when the underlying probability of selecting the target is about
$0.61$
with a power of
$0.95$
when we separately analyze old and new consumer goods.
We conducted a power analysis using nested model comparison to test the strength of the interaction effects. Here we summarize the key results and provide the details in the Supplementary Materials. We quantify the effect size e as the increase in the probability of selecting the target by e. We see that with a power of
$0.95$
, for the mode and layout interaction, we can observe an interaction effect of size
$0.05$
when all the consumer goods are analyzed together and an effect of size
$0.07$
when old and novel goods are analyzed separately.
For the order and mode interaction, the effect can be driven by different levels of order. The effect can be driven by a single level of order (e.g., DTC: decoy, target, competitor in the numerical mode) or by several different levels of orders interacting with the mode. In the Supplementary Materials, we present the results of the power analysis when there are
$1$
,
$2$
, and
$3$
possible levels of order interacting with the mode. When we combine all consumer products, with a power of
$0.95$
, we will be able to detect an effect of size
$0.09$
when there is a single level of order interacting with the mode and
$0.07$
when two or three levels of order interact with the mode. When old and new goods are analyzed separately, with a power of
$0.95$
, we will be able to detect an effect of size
$0.11$
when it is driven by one level of order interacting with the mode and an effect of size
$0.08$
when it is driven by two or three levels. These results are the same for the interaction between order and layout since both mode and layout have the same number of levels.
We only show the results of the three way interaction when a single level of order interacts with the other two variables. When we combine the consumer goods, with a power of
$0.95$
, we will be able to detect an effect of size
$0.16$
. When we analyze old and new goods separately, with a power of
$0.95$
, we will be able to detect an effect of size
$0.22$
.
Apart from the ternary trials mentioned above, to observe choice proportions between the target and the competitor in the absence of the decoy option, we decided to include binary choice trials. These will be presented in 2 different layouts and 2 different formats giving us 4 conditions. Since we will not analyze the binary trials based on order, we do not account for that factor in this calculation (but note that order is randomized). Since we need
$900$
trials per condition, we will need
$3,600$
trials. Since each individual will do
$12$
trials, we obtain our estimate of
$300$
additional participants. This takes our total to
$2,300$
participants.
2.2. Materials
We used six different consumer goods for our experiment. The consumer goods, attribute names, and attribute values are presented in Table 1. To construct a binary choice set, we present the two alternatives in one of two possible orders
$XY$
and
$YX$
. For a ternary choice set where the options are shown in one particular order (e.g., DTC: Decoy, Target, Competitor from left to right), there are 6 choice sets with one decoy (e.g.,
$X_d$
) and another 6 choice sets with the other decoy (e.g.,
$Y_d$
), yielding a total of 12 ternary choice sets. For each stimulus, we generated four different question formats in a 2x2 design [mode: graphical vs. numerical] x [layout: by-alternative vs. by-attribute]. Examples of these different formats can be seen in Figure 1.
The attribute values for the 6 stimuli used in the experiment. For each stimulus, there were two different alternatives X and Y
Note: Two additional decoy alternatives were
$X_d$
and
$Y_d$
such that these were strictly dominated by X and Y, respectively. The ternary choice set for each consumer good consisted of X, Y and one of the two decoy options
$X_d$
or
$Y_d$
. For every participant, for each consumer good, the decoy option was selected randomly. The binary choice set consisted of a choice between X and Y in the two orders.
Since we were primarily interested in studying the importance of presentation format, we uniformly chose attributes that had a positive valence. That is, larger values were considered to be more desirable. As shown in the Supplementary Materials, three of these stimuli - paper towels, highlighters, and walking shoes and their numerical values - were identical to stimuli used in Noguchi and Stewart (Reference Noguchi and Stewart2014) and demonstrated strong significant attraction effects (Paper Towels: RST =
$82.0\%$
,
$p<0.001$
; Highlighter: RST =
$80.0\%$
,
$p<0.001$
; Walking Shoes: RST =
$64.4\%$
,
$p=0.008$
). In the experimental program, one of the attribute values for the walking shoes was erroneously copied from Noguchi and Stewart (Reference Noguchi and Stewart2014). Specifically, the comfort for
$Y_d$
should have been 58 instead of 26. Despite the error,
$Y_d$
is still dominated by Y but not X and serves as an attraction decoy. Hence, we retain the walking shoes in our analyses but do not treat them as an old product. We note that walking shoes show a standard attraction effect when analyzed separately with an RST=
$61.7\%$
,
$p<0.001$
, as shown in the Supplementary Materials. However, since
$X_d$
is from the original Noguchi and Stewart (Reference Noguchi and Stewart2014), we do not treat it as a new product either.
The other three consumer goods - apartments, cars, and laptops - have also been used in context effects research (Cataldo and Cohen, Reference Cataldo and Cohen2019; Evans et al., Reference Evans, Holmes and Trueblood2019; Simonson and Tversky, Reference Simonson and Tversky1992). For each product, the two attributes were on a 0-10 scale. In order for X and Y to have similar preference levels (assuming equal weight on the two attributes), the sum of the two attributes of X was similar to the sum of the two attributes of Y. Further, these ratings were mostly in the 4-10 range, so neither option would be too unfavorable on one of the attributes. Here we use novel numerical attribute values for these items to test the generalizability of our results.
2.3. Procedure
During the course of the experiment, every participant answered a total of 12 questions over two blocks. The small number of trials per participant is based on the suggestion made by Lichters et al. (Reference Lichters, Sarstedt and Vogt2015) to avoid learning processes in repeated choice and since context effects are thought to be attenuated with a high frequency of repeated choices. In between the two blocks, we included ten filler trials in which participants were asked to name different state capitals in the United States of America. This distractor task was added based on the suggestion by Lichters et al. (Reference Lichters, Sarstedt and Vogt2015) and Hutchinson et al. (Reference Hutchinson, Kamakura and Lynch2000) to avoid carry-over effects and psychological reactivity from previous trials.
At the start of the experiment, each participant was assigned to one of 2x2 between-subjects conditions [choice type: binary, ternary] x [layout: by-alternative vs. by-attribute]. The layout of the options (as shown in Figure 1) remained constant for participants throughout the experiment. The two blocks of the experiment were the graphical block and the numerical block. The order of the two blocks was randomized across participants. For each question, participants, depending on the condition they were assigned, saw two or three options of a consumer good and were asked to select the one that they preferred. The graphical block presented questions in a graphical format using bar graphs and the numerical block presented questions in a numerical format using tables. See Figure 1 for examples of questions in different formats.
Each block consisted of 6 questions with the 6 different consumer goods from Table 1. Each consumer good appeared once in each block. For each consumer good, the decoy (
$X_d$
or
$Y_d$
) was randomly selected for each question in the first block. Since there were only 2 binary presentation orders (XY and YX), these were randomized for every participant. For the ternary choice condition, the presentation order (of the target, competitor and decoy) was pseudo-randomly varied such that each participant saw all 6 orders. In the second block, the choice sets and order of alternatives were the same as in the first block. The only difference between the first and second blocks was the presentation mode (numerical vs. graphical). The randomization was designed so that each of the 24 ternary conditions - 2 presentation modes, 2 presentation layouts, 6 orders, and 2 decoy types (i.e.,
$X_d$
or
$Y_d$
) - were counterbalanced. The two binary conditions were shown as often as 1 of the 24 ternary conditions, balanced equally in the two presentation orders XY and YX.
There were two attention checks (e.g., the true/false question ‘I have never used a computer’) at different points in the experiment. At the end of the experiment, participants were asked for their demographic information.
3. Planned analyses
In this section, we describe the planned analysesFootnote 1 . First, we analyzed each of our 24 ternary choice conditions separately, then we conducted a single factor analysis to test for the attraction effect in different conditions, and finally, we used nested logistic regression model comparisons to test for main and interaction effects. We also planned to conduct the same analyses combining the three stimuli from Noguchi and Stewart (Reference Noguchi and Stewart2014) to test for reliability. However, we departed from our planned analysis and excluded the shoes from the old stimuli due to a difference in our stimulus values as compared to the original values as described in Noguchi and Stewart (Reference Noguchi and Stewart2014). Additionally, we ran the analyses combining the three novel stimuli to test for generalizability.
3.1. Single condition analysis
We were first interested in a model-free test where we make no underlying assumptions about how the different conditions give rise to the attraction effect. To this end, as described in our power analysis, we conducted an exact binomial test for each of the 24 ternary choice conditions separately to test if the RST value was significantly different from 0.5. We corrected for multiple comparisons using a Bonferroni correction to the p-value, resulting in a significance cutoff of
$p=0.05/24=0.002$
for each of the ternary conditions.
3.2. Single factor analysis
We were interested in how the different presentation factors impacted the attraction effect. First, we were interested in testing whether the presentation layout influences the attraction effect (e.g., does the by-attribute layout produce an attraction effect?) To this end, we aggregated all decisions made in the by-attribute layout condition. We then conducted an exact binomial test against a probability of 0.5 to test if the RST value was significantly different. We repeated this for the by-alternative layout. We corrected for multiple comparisons using a Bonferroni correction, setting the significance cutoff at
$p=0.05/2=0.025$
.
We performed similar analyses for the mode variables and order variables. Since there were two different modes, the Bonferroni correction for multiple comparisons was set at
$p=0.05/2=0.025$
. Since there were six different orders, the Bonferroni correction was set at
$p=0.05/6=0.0083$
.
3.3. Interaction effects
To test for interaction effects, we conducted a nested logistic regression model comparison. We set the selection of the target as the dependent variable. We compared the added advantage of including various interaction terms in the model to a null model with only the main effects. We planned to include by-subject random intercepts barring any model convergence issues. We used the likelihood ratio test for our nested model comparison with significance at the
$\alpha =0.05$
level.
We tested for the interaction between mode and format by comparing the null model Mode + Format to the model with the interaction term Mode + Format + Mode*Format. We tested for the interaction between mode and order by comparing the null model Mode + Order to the model with the interaction term Mode + Order + Mode*Order. If the interaction tests were significant, we looked at individual contrasts for the mode and order interaction terms. When testing for the interaction between layout and order, our null model was Layout + Order and our model with the interaction term was Layout + Order + Layout*Order. If the interaction tests were significant, we looked at individual contrasts for the layout and order interaction terms.
To test for three-way interactions, we compared the null model Mode + Layout + Order+ Mode*Layout+ Mode*Order+ Layout*Order to the model with the additional three-way interaction term Mode + Layout + Order+ Mode*Layout+ Mode*Order+ Layout*Order+ Mode*Layout*Order. If the interaction test was significant, we looked at individual contrasts for the Mode, Layout, and Order interaction terms.
4. Results
We first present the results of the planned analysis, which was designed to test the robustness of the attraction effect to different conditions. After this, we conducted a comparative study across the different conditions. Analyses using the binary choice data are presented in the Supplementary Materials.
4.1. Single condition Analysis
The results of the single condition analyses are summarized in Table 2. First, we analyzed the data by collapsing across all six product types. Standard attraction effects, indicated by RST values significantly greater than 0.5, were observed in 23 out of 24 ternary choice conditions, after the Bonferroni correction (
$p=0.05/24=0.002$
). The graphical
$\times $
by-alternative
$\times $
TCD condition was not significant after the Bonferroni correction but was in the direction of a standard attraction effect
$(RST = 0.544, p=0.008$
). Next, we partitioned the data into old products (i.e., Highlighters and Paper Towels from Noguchi and Stewart (Reference Noguchi and Stewart2014)) and new products (i.e., Apartments, Laptop, Cars). As described in the methods, we excluded them from the new and old products. After the Bonferroni correction (
$p=0.05/24=0.002$
), there was a significant standard attraction effect in 20 out of 24 ternary choice conditions for the old products and 11 out of 24 ternary choice conditions for the new products. The results of analyses performed on data sets after exclusions were similar and are reported in the Supplementary Materials.
Results of exact binomial tests for each of the 24 ternary choice conditions to test if the RST value was significantly different from 0.5
Note: The analyses were performed on data from (1) all consumer goods (left), (2) old consumer goods from Noguchi and Stewart (Reference Noguchi and Stewart2014) (middle), and (3) novel consumer goods (right). p-values less than
$p=0.05/24=0.002$
are in bold font.
4.2. Single factor analysis
The results of single factor analyses are summarized in Table 3. Standard attraction effects (i.e., RST values significantly greater than 0.5) were observed across all levels of the three factors of interest (i.e., Mode, Layout, and Order). These results held when the old and new product groups were analyzed separately. The results of analyses performed on data sets after exclusions are similar and are reported in the Supplementary Materials.
Results of exact binomial tests for the three factors of interest to test if the RST value was significantly different from 0.5
Note: The analyses were performed on data from (1) all consumer goods (left), (2) old consumer goods from Noguchi and Stewart (Reference Noguchi and Stewart2014) (middle), and (3) novel consumer goods (right). p-values less than
$p = 0.025$
for Mode and Layout, and
$p = 0.0083$
for Order are in bold.
4.3. Interaction effects
The results of analyses testing the interaction effects are summarized in Table 4. As described in the planned analysis, we treated the subject as a random effect to account for individual differences. Significant 2-way interactions between Mode and Layout and between Layout and Order were observed when collapsing across all six consumer goods and when analyzing the three new consumer goods separately. None of the interaction terms were significant when analyzing the two old consumer goods. The results of analyses performed on data sets after exclusions are similar and are reported in the Supplementary Materials.
Results of nested logistic regression model comparison for each interaction term
Note: Individual contrasts for each significant interaction term are reported. The coding scheme for categorical variables is deviation coding, which compares each level to the grand mean. p-values less than
$p = 0.05$
are in bold font.
4.4. Main effects
In the single-factor analysis, our goal was to test whether the attraction effect was robust to different experimental manipulations. This did not directly compare the different conditions to each other. To compare the different presentation modes, layouts, and orders to each other, we plotted the change in the RST under the different factors in Figure 3.
The comparative dependence of RST on the different factors varied in the experiment. The dependence on order, mode, and layout are shown in the top, bottom left, and bottom right panels, respectively.
To test for significance, we conducted a mixed effects logistic regression with the selection of the target as the dependent variable and tested each of the individual factors using a logistic regression Target Selection
$\sim $
Mode + Layout + Order + (1|subject). As in the previous analysis, we use deviation coding so that the comparisons are made to the mean and not to the contrast class. We present our results in Table 5 (left model) (Hlavac, Reference Hlavac2022). We also tested models with higher-order interaction terms and presented them in the Supplementary Materials.
Coefficients of regression models of target selection as predicted by different presentation formats and product types
Note: Standard Errors are presented below the coefficient in rounded brackets.
p-values are Bonferroni corrected to account for the number of coefficients.
p-values less than p = 0.05/7 for the left model and p = 0.05/8 for the right model are in bold.
$^{*}p<0.05$
;
$^{**}p<0.01$
;
$^{***}p<0.001$
From the results in Table 5 and Figure 3, we observed that the different orders modulated the strength of the attraction effect. We also observed the attraction effect is significantly stronger in the DTC condition. It was significantly weaker in the DCT and TCD conditions where the target and the decoy were not adjacent to each other. For the conditions CDT and DTC, the strength of the effect is similar to the mean effect. These results indicate that the strength of the attraction effect is modulated by the order in which the three choice options were presented.
We observed that the strength of the attraction effect was significantly weaker in the graphical format than in the numerical format. There was no difference in the strength of the attraction effect in the by-attribute and by-alternative conditions. These results show that the strength of the attraction effect was modulated by the mode. However, there was no main effect of the layout.
We also tested whether the strength of the attraction effect was different for the old consumer products from Noguchi and Stewart (Reference Noguchi and Stewart2014) as compared to the new consumer products. We added an extra variable ‘product type’ and tested another logistic regression: Target Selection
$\sim $
Mode + Layout + Order + Product Type + (1|subject). As described in the methods, the products from Noguchi and Stewart (Reference Noguchi and Stewart2014) - highlighter and paper towels—were coded as old. Further, apartments, laptops, and cars were coded as new. Shoes were coded separately. The variables used deviation coding, so the comparison is made to the mean effect. We present our results in Table 5 (right model). We observed that the strength of the attraction effect was modulated by the product type variable, with older stimuli producing a stronger attraction effect. Thus, the specific attributes and attribute values that were used in the experiment were also important in modulating the attraction effect.
5. General discussion
In this paper, we conducted a high-powered pre-registered experiment where we varied the mode (numerical vs. graphical), layout (by-attribute vs. by-alternative), and order as presentation factors thought to modulate the attraction effect (Frederick et al., Reference Frederick, Lee and Baskin2014; Spektor et al., Reference Spektor, Bhatia and Gluth2021; Yang and Lynn, Reference Yang and Lynn2014). Previous literature had suggested that the attraction effect was fragile and required the stimuli to be presented in some specific formats, without which the attraction effect would disappear or reverse. We observed that the attraction effect was robust to each of these factors, often producing strong attraction effects where the target was chosen at an average
$1.5$
times as often as the competitor (i.e., RST
$= 0.6$
). While we observed a robust attraction effect across our various manipulations, the strength of the effect was modulated by the display factors. It was consistently stronger for some factors as compared to others. Specifically, we observed a dependence on order and mode. This indicates that the format in which information is displayed plays a role in decision-making processes.
Regarding order, we found the weakest attraction effect when the target and the decoy were not adjacent to each other. This might be because their adjacency potentially facilitates a comparison between the two, which brings out the dominance of the target relative to the decoy. This is possibly hindered when the competitor is between the target and the decoy. Further, we noticed that the attraction effect is the strongest for the order DTC. This might be because when the decoy is presented first (i.e., left or top), it forms the baseline for future comparisons. The dominance of the target compared to this baseline potentially results in it being perceived as a superior option and finally results in it being chosen. Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) also found that the order DTC produced a strong attraction effect in the reanalysis of Trueblood et al. (Reference Trueblood, Brown and Heathcote2015). However, Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) also examined the impact of temporal order on the attraction effect and found a different pattern of results. For example, the temporal order TDC (where T is presented first, D second, and C last) produces a strong repulsion instead of an attraction effect. Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) model this using a memory component that leads to slower evidence accumulation for a previously presented option. In our task, we do not anticipate any role of memory and hence do not need to account for this effect. Overall, order might interact with other cognitive processes (e.g., memory in Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) and attention in the current experiment) and play an important role in the kind of evaluations and comparisons that are made during the decision-making process.
Order produces a remarkably consistent influence on the attraction effect across many different settings. This pattern was similar across the different modes and layouts as depicted in Figure 3. This pattern is similar to the one observed in the analyses reported in the Supplementary Materials using data from Cataldo and Cohen (Reference Cataldo and Cohen2021b). Evans et al. (Reference Evans, Holmes, Dasari and Trueblood2021b) found a similar pattern in the reanalysis of Trueblood et al. (Reference Trueblood, Brown and Heathcote2015), which used a perceptual decision-making task. These results indicate potential common mechanisms between different kinds of decisions. Hence, we conclude that the influence of order on the attraction effect is a robust finding.
The influence of mode (i.e., numerical vs. graphical) on the attraction effect is in accordance with Frederick et al. (Reference Frederick, Lee and Baskin2014), Spektor et al. (Reference Spektor, Bhatia and Gluth2021), and Yang and Lynn (Reference Yang and Lynn2014), where numerical stimuli are predicted to produce stronger attraction effects. Spektor et al. (Reference Spektor, Bhatia and Gluth2021) suggests that this is driven by the concreteness of the representation, where numerical stimuli bring out the dominance relationship between the target and the competitor. As shown in Figure 3, the influence of mode seems to be robust to the layout and order in which the information is displayed. In an exploratory analysis provided in the Supplementary Materials, we show that the dependence on mode does not hold for all of the consumer goods tested in the experiment. Specifically, it seems to hold for the old consumer goods from Noguchi and Stewart (Reference Noguchi and Stewart2014) but not for the novel stimuli. Hence, further exploration is needed to understand how specific attributes and attribute values might interact with the mode.
We observed a null effect of the layout (i.e., by-alternative vs. by-attribute) in our experiment. In an additional analysis presented in the Supplementary Materials, we only used data from the graphical mode, which is very similar in format to the stimuli used in Cataldo and Cohen (Reference Cataldo and Cohen2019), and still observed a null effect. This is a failure to replicate the results of Cataldo and Cohen (Reference Cataldo and Cohen2019). It is worth noting that there are differences in our experiment and Cataldo and Cohen (Reference Cataldo and Cohen2019), such as the number of trials and differences in participants, which could potentially explain the discrepancy in results. The number of trials could modulate the attraction effect since decision-makers might develop decision strategies to reduce mental effort during the course of a long experiment with a large number of trials (Ahn et al., Reference Ahn, Kim and Ha2015; Lichters et al., Reference Lichters, Sarstedt and Vogt2015). Such strategies might be more sensitive to differences in the ease of comparing options and thus reveal differences between presentation layout as found in Cataldo and Cohen (Reference Cataldo and Cohen2019). Strategies to reduce mental effort are less likely to be adopted in short experiments with a small number of trials (such as the present one). Our findings call into question the unified impact of layout on the attraction effect.
We observed that the attraction effect is at an average stronger for the consumer goods from Noguchi and Stewart (Reference Noguchi and Stewart2014) compared to novel consumer goods tested in our experiment. This might be because the consumer goods were chosen from Noguchi and Stewart (Reference Noguchi and Stewart2014) based on their ability to produce strong attraction effects, resulting in a selection bias. This might also be due to many potentially confounding differences between the two sets of consumer goods. For one, the consumer goods—paper towels, highlighters, and walking shoes in Noguchi and Stewart (Reference Noguchi and Stewart2014) were everyday low-value consumer goods. On the other hand, the novel consumer goods—apartments, laptops, and cars were high-value goods that are purchased rarely. Second, the attributes for the novel stimuli used a 1–10 scale and had similar units. The attributes for the older stimuli used different scales and had different units. Specifically, Hayes et al. (Reference Hayes, Holmes and Trueblood2024) showed that when attributes are on a similar scale, individuals can make inter-attribute comparisons, which can impact the presence of the attraction effect. Thus, we find that the specific attributes and attribute values can modulate the strength of the attraction effect.
As discussed in the introduction, interactions between attentional processes and decision making have been invoked to explain context effects (Cataldo and Cohen, Reference Cataldo and Cohen2019; Spektor et al., Reference Spektor, Bhatia and Gluth2021; Trueblood, Reference Trueblood2022). Trueblood et al. (Reference Trueblood, Liu, Murrow, Hayes and Holmes2022) proposed a promising approach to model multi-alternative, multi-attribute choice using an attentional process that informs a preference accumulation process. In this model, a Markov attentional process probabilistically selects the options and attributes that are compared at any given moment in time. Preference is accumulated for the attended options until a threshold is reached, triggering a decision. Trueblood et al. (Reference Trueblood, Liu, Murrow, Hayes and Holmes2022) showed that the dependence of the attraction effect on spatial order can be explained by the Markov attention process by allowing the probabilities with which options are selected and compared to depend on their spatial proximity. For example, the model allows for larger transition probabilities when options are spatially adjacent. Thus, when the target and decoy are adjacent (such as in the order DTC), the model predicts that these two options will be compared more, thus leading to a stronger attraction effect. Further, Hayes et al. (Reference Hayes, Holmes and Trueblood2024) expanded this framework and allowed for between-attribute comparisons when attributes are on a common 0–10 scale, like the new stimuli in our experiment. They found that between-attribute comparisons reduced the strength of the attraction effect. Formats such as graphical displays might also allow for between-attribute comparisons since an individual can easily compare the size of bars for different attributes. The facilitation of between-attribute comparisons in graphical displays could explain the observed reduction in the attraction effect. Thus, the theory of attentional dynamics described in Hayes et al. (Reference Hayes, Holmes and Trueblood2024) and Trueblood et al. (Reference Trueblood, Liu, Murrow, Hayes and Holmes2022) is potentially a unifying framework to explain our findings.
From a practical and applied perspective, decision-making biases can be used to design choice architectures to improve the quality of decisions (Lichters et al., Reference Lichters, Bengart, Sarstedt and Vogt2017; Thaler et al., Reference Thaler, Sunstein and Balz2013). The attraction effect has been suggested as a method to help individuals make better decisions such as selecting more healthy food choices (van den Enden and Geyskens, Reference van den Enden and Geyskens2021) or buying eco-friendly goods (Guath et al., Reference Guath, Stikvoort and Juslin2022). Our findings have a direct implication for the format in which information about the choices can be displayed to produce the strongest effect. Our findings suggest that the information should be presented numerically, and the target and the decoy should be presented adjacent to each other to maximize the choice share of the target. Choice architects can utilize the presentation format as an additional dimension to aid in improving decisions.
5.1. Constraints on generality
In our study, we varied a large number of presentation factors and observed how they modulated the attraction effect. Despite our efforts to comprehensively test a large number of formats, there were additional presentation formats that we did not test in our experiment. For instance, we did not arrange the options in a triangle as in Noguchi and Stewart (Reference Noguchi and Stewart2014). Apart from presentation formats, additional experimental factors can modulate the strength of the attraction effect like time pressure (Cataldo and Cohen, Reference Cataldo and Cohen2021b; Evans et al., Reference Evans, Holmes and Trueblood2019; Spektor et al., Reference Spektor, Bhatia and Gluth2021) or the number of trials (Lichters et al., Reference Lichters, Sarstedt and Vogt2015). Future experimenters might vary some of these factors to understand which ones might reverse the attraction effect.
In our experiment, we only used a single set of attribute values for the target and competitor options for each product (see Table 1). Future experiments might vary the attribute values in order to more comprehensively test how attribute values influence the attraction effect. Experimenters that vary these values will face an additional challenge since it is more difficult to calculate the RST when the target and competitor options can take on a larger range of values. Innovative experimental designs and forms of inference will need to be considered (see for example, Dumbalska et al. (Reference Dumbalska, Li, Tsetsos and Summerfield2020)).
All our options were presented with explicit attribute values. In some studies, the attributes are not presented explicitly to the participants (Dumbalska et al., Reference Dumbalska, Li, Tsetsos and Summerfield2020; Frederick et al., Reference Frederick, Lee and Baskin2014; Trendl et al., Reference Trendl, Stewart and Mullett2021). These kinds of stimuli (e.g., pictures of different fruits) might require the decision maker to generate attributes and convert these attributes to internal representations which might not be accounted for by the way we varied the ‘mode’ (numerical vs. graphical).
We only studied value-based consumer decision-making. However, there are additional decision-making paradigms such as perceptual decisions (Trueblood et al., Reference Trueblood, Brown and Heathcote2015), inter-temporal choice (Gluth et al., Reference Gluth, Hotaling and Rieskamp2017), risky gambles (Frederick et al., Reference Frederick, Lee and Baskin2014), and even memory tasks (Maylor and Roberts, Reference Maylor and Roberts2007) that have been used to study the attraction effect. While we observed choice patterns that were consistent with findings from other paradigms, it is still an important theoretical question of whether people use the same mechanisms for attribute comparisons and integration across domains (Frederick et al., Reference Frederick, Lee and Baskin2014; Shimojo et al., Reference Shimojo, Simion, Shimojo and Scheier2003; Trueblood et al., Reference Trueblood, Brown, Heathcote and Busemeyer2013). Future experiments might vary the task (e.g., consumer choice vs. perceptual choice) along with the presentation format to test how the presentation format impacts choice across different tasks.
5.2. Future directions
Overall, our findings challenge researchers to provide explanations to account for the dependence of the attraction effect on presentation formats. As discussed in the introduction, the existing multi-alternative multi-attribute models in their current form do not give an account as to how presentation might impact choice. Cataldo and Cohen (Reference Cataldo and Cohen2019) and Trueblood et al. (Reference Trueblood, Liu, Murrow, Hayes and Holmes2022) suggest that attention and comparisons are critical for multiattribute choice. Spektor et al. (Reference Spektor, Bhatia and Gluth2021) argue that the stimulus representation also plays a key role. While all of these ideas are related, they could be theoretically and computationally instantiated in different ways. Our data can help inform such theoretical developments.
One explanation for the dependence of the attraction effect on presentation formats relates to the interaction of attention and decision-making processes (Trueblood et al., Reference Trueblood, Liu, Murrow, Hayes and Holmes2022). Future research could further investigate the interactions among presentation formats, attention, and decision-making. For example, previous studies using process tracing have revealed that transitions between alternatives often predict the final decision (Marini et al., Reference Marini, Sapienza and Paglieri2023; Noguchi and Stewart, Reference Noguchi and Stewart2014). Additionally, research indicates that different multiattribute tasks exhibit distinct between and within-attribute transitions (Yang and Krajbich, Reference Yang and Krajbich2023), as suggested in Hayes et al. (Reference Hayes, Holmes and Trueblood2024). Interestingly, a bottom-up manipulation of attention did not alter the comparison process (Hasan and Trueblood, Reference Hasan and Trueblood2024), suggesting that attention allocation may be influenced by decision strategies (Callaway et al., Reference Callaway, Rangel and Griffiths2021) and top-down features like value (Towal et al., Reference Towal, Mormann and Koch2013). By coupling process tracing methods with presentation-format-based manipulations, researchers might discover significant differences in attention allocation and choices across different presentation formats.
Registered reports were introduced to increase transparency in the scientific process, validate experimental designs, and reduce publication bias against research outcomes (Chambers, Reference Chambers2013; Chambers and Tzavella, Reference Chambers and Tzavella2022; Henderson and Chambers, Reference Henderson and Chambers2022). While registered reports are primarily used for confirmatory research (Henderson and Chambers, Reference Henderson and Chambers2022) (e.g., Evans et al., Reference Evans, Kogler and Sleegers2021a; Saribay et al., Reference Saribay, Yilmaz and Körpe2020; Zgxonnikov et al., Reference Zgonnikov, Atiya, O’hora, Rañò and Wong-Lin2019), these processes can also improve the quality and trust of exploratory research (Dirnagl, Reference Dirnagl2020; McIntosh, Reference McIntosh2017). Some journals have introduced a separate publication category called ‘exploratory reports’ to encourage researchers to conduct exploratory research, independently from the conventions of hypothesis-testing methods (McIntosh, Reference McIntosh2017). We hope that findings, such as ours, from exploratory registered reports will be able to serve as a strong basis for fostering theoretical development.
6. Conclusion
We undertook a registered report to conduct a high-powered exploratory study to understand the dependence of the attraction effect on presentation formats. We varied the mode (numerical vs. graphical), the layout (by-alternative vs. by-attribute) and the order (e.g., DTC - Decoy-Target-Competitor) in which options were displayed. We recruited over 2,000 people and recorded over 24,000 choices on value-based consumer decisions. We observed a robust attraction effect across each of these factors. The strength of the effect was modulated by the order and the mode but not by the layout. These empirical findings are not directly predicted by any of the modeling paradigms and can be used as a foundation for theory development.
Acknowledgments
We thank William Hayes for his assistance with the power analysis and his helpful comments. We also thank Hoyoung Doh for his useful comments.
Funding statement
This work was supported by the Estes summer fellowship from Indiana University to EH and NSF grant 1846764 / 2305559 to J.S.T.
Data Availability Statement
The data and analysis code are available on the Open Science Framework at https://osf.io/wzct3/.
Open access
