1.1. Eye Movements and Visual Attention

We use the term (visual) attention as a container for various cognitive processes that are involved in the selection of relevant and the suppression of irrelevant information during search and choice, for which according to James (Reference James1890, pp. 403–404): “Focalization, concentration, and consciousness are of […] essence.” The next section describes these processes and their links to eye movements. It is important to point out that in most natural decision-making tasks, eye-movement patterns are valid and accurate indicators of the hidden attention processes of interest (Findlay, Reference Findlay2005). There are also successful examples of inferences on higher order cognitive states from eye movements, such as on tasks and goals (Borji & Itti, Reference Borji and Itti2014; Haji-Abolhassani & Clark, Reference Haji-Abolhassani and Clark2014; Kardan, Berman, Yourganov, Schmidt, & Henderson, Reference Kardan, Berman, Yourganov, Schmidt and Henderson2015) and on the emotional valence of images (Nummenmaa, Hyönä, & Calvo, Reference Nummenmaa, Hyönä and Calvo2006; R.-Tavakoli et al., Reference R.-Tavakoli, Atyabi, Rantanen, Laukka, Nefti-Meziani and Heikkilä2015), although these tend to require additional experimental control, measures, and theoretical assumptions.

At any point in time, the human eye-vision system processes only about one percent of the field of vision with high acuity (Rayner, Reference Rayner1998; Wedel & Pieters, Reference Wedel and Pieters2008a). This is due to the density of photosensitive cells in the human retina decreasing rapidly from a high concentration in a small region in its center (fovea) toward the parafovea and periphery. Therefore, people need to move their eyes to explore their environment in sufficient detail. In practice, people make on average three to six larger eye movements per second during tasks such as reading, scene perception, or decision making (Rayner, Reference Rayner1998). These saccades are fast jumps of the eye’s point of regard (POR) between spatial locations in the environment. Saccades typically last for about 20–40 ms. Fixations are periods of about 100–400 ms when the eyes hardly move. While during a saccade visual perception is suppressed to prevent blurring, during an eye fixation detailed visual information is obtained from a small region of about two degrees of visual angle around the POR.

A scan-path is the spatiotemporal sequence of eye fixations and saccades for a particular task, stimulus, and person. Figure 1 summarizes the scan-paths of four participants who were engaged in a target search task: they searched on a shopping website for a brand of perfume to choose as a gift. The figure (based on data from study 2 in van der Lans, Pieters, & Wedel, Reference van der Lans, Pieters and Wedel2021) reveals several key features of the scan-paths which inform theories and psychometric models. First, the first fixation commonly lands close to the center of the screen. This location may be optimal because it has the smallest distance to all other locations. Second, some fixations land on visually salient objects, i.e., objects that stand out from their surroundings in terms of luminance, color, or shape. Other parts of individuals’ scan-paths reflect systematic left-right movements. That pattern may be induced by the organization of the image (scene layout). Third, the scan-paths consist of clusters of fixations on objects, interspersed with saccades toward more distant objects. Those clusters reflect refixations which are used to accumulate information to determine the identity or value of objects of interest. Longer saccades reflect search for where new objects are located. Fourth, a cluster of fixations is specifically directed at the final, chosen object. Such a pattern may reflect preference formation or a final verification of the object before implementing a decision, or it might be that looking at an object longer increases one’s preference for it. Fifth, scan-paths show large individual variation and differ markedly among distinct tasks and stimuli. In addition, the scan-paths of participants who had a different search goal (in the top versus the bottom row of the figure) are noticeably different. We return to all those features of scan-paths in the sequel. The scan-paths observed when people make choices among alternative options are similar to those in Fig. 1 but are often considerably more elaborate.

Next to fixations and saccades, other types of eye movements include micro-saccades which are involuntary movements with small amplitude that occur during fixations, smooth pursuits which are fixations on moving objects, vergence movements which occur to maintain the POR on an object that moves toward or away from the viewer, and pupil movements to regulate the amount of incoming light, amongst others. While the analysis of these movements has been useful for specific problems, the present review focuses on fixations and saccades.

1.2. Eye-Movement Recording

The dominant method of recording eye movements in research on search and choice uses an infrared light source directed at one or both eyes, which creates (invisible) reflections on the cornea, the hard outer layer of each eye, called Purkinje reflections (Duchowski, Reference Duchowski2003). After a short calibration task, video cameras record one or more of these reflections, even while participants move their heads. Based on the calibration results, algorithms infer the POR in x-y stimulus coordinates from these reflections. Commonly used eye trackers are built into desktop monitors or into small stand-alone devices that can be attached to laptops and other objects. In addition, mobile eye trackers embedded in glasses allow free body movements while recording the entire field of view as well as the POR within it. Commercial infrared eye trackers typically sample the POR with a sampling frequency of 50 or 60 Hertz and a spatial accuracy of 0.5 degrees of visual angle, or better. Eye trackers with a higher spatiotemporal precision make use of multiple Purkinje reflections and sampling frequencies of 120 Hertz or more, for which the participant’s head sometimes needs to be fixed during recording with a chin or forehead rest. Further, eye movements can be recorded with regular front-facing (web) cameras integrated into digital devices, by using eye landmarks, such as the center and shape of the pupil and the location of eye corners, as input to computer vision algorithms that estimate gaze direction from images of the eyes. Those eye-tracking solutions still have lower accuracy than infrared eye tracking.

1.3. Eye-Movement Data

Eye tracking during decision making results in large amounts of raw data. For example, an eye-tracking study in which participants make 20 decisions, with 10 s available for each decision and a 50 Hertz sampling frequency, results in some ten thousand samples comprised of the x-y coordinates of the POR of each eye per participant. This amounts to one million data points for a study with one hundred participants. A proportion of the data may involve outliers or may be missing due to eye blinks and tracking problems, however. From the raw samples of the POR, algorithms are used to identify fixations and saccades based on their spatial proximity or the velocity of their movement (Salvucci & Goldberg, Reference Salvucci and Goldberg2000). This reduces the dimensionality of the data by a factor 20 or more. For example, the BIT (Binocular Individual Threshold) algorithm (van der Lans, Wedel, & Pieters, Reference van der Lans, Wedel and Pieters2011) automatically determines fixations using individual-specific eye-velocity thresholds for both eyes, automatically removing eye blinks and other recording abnormalities. Due to blinks, saccades and recording errors, missing data points are common in raw samples of the POR. As a rule of thumb, van der Lans and Wedel (Reference van der Lans and Wedel2017) propose that one may consider removing a participant’s data if less than 80 percent of their raw samples are classified as fixations. Once the fixation data have been prepared, various characteristics of the eye-movement patterns can be examined, such as x-y coordinates of fixations over time, scan-paths, or aggregate measures such as fixation counts, gaze or dwell times, and fixation selection indicators, on so-called Regions of Interest (ROIs) (Holmqvist et al., Reference Holmqvist, Nyström, Andersson, Dewhurst, Jarodzka and van de Weijer2011). The spatial aggregation of fixations on those ROIs, for example by calculating gaze times on the individual perfume bottles in Fig. 1, reduces the size of the data further (the gaze time on a ROI is defined as the sum of the durations of all fixations that land on that ROI). Fixation patterns are often graphically explored via heatmaps, which represent the density of fixations using colors, and fixation plots which depict scan-paths as shown in Fig. 1.

Figure 1.

Scan-paths of four participants searching for a brand of perfume to choose on a shopping website. The dots indicate fixations, the lines saccades. Red rectangular boxes indicate the first and last fixation. The top, respectively, bottom, panels show scan-paths for two participants searching for the Jimmy Choo, respectively, the Dolce & Gabana, perfumes (based on data from van der Lans et al., Reference van der Lans, Pieters and Wedel2021).

2.1. Decision-Making Tasks

In Table 1, we propose a classification that involves six fundamental decision-making tasks that people commonly engage in. These tasks have in common that they require eye movements. They therefore involve exposure durations longer than the duration of a single fixation (100–300 ms). The tasks elicit different types of uncertainties which people try to resolve through visual information acquisition.

Table 1.

Six fundamental decision-making tasks, the types of uncertainty they induce, and the dominant cognitive process involved when engaging in the task.

Table 1 makes a main distinction between perceptual decision making and preferential decision making (e.g., Dutilh & Rieskamp, Reference Dutilh and Rieskamp2016; Summerfield & Blangero, Reference Summerfield and Blangero2017). Perceptual decision-making tasks have an objective (external) performance criterion, mostly speed and/or accuracy. Search tasks are perceptual decision-making tasks. Preferential (or value-based) decision-making tasks have one or more subjective (internal) performance criteria, such as perceived costs and benefits, decision justification, or post-choice satisfaction. The completion of these tasks is based on personal preference or utility functions. Choice tasks are preferential decision-making tasks. The dominant processes involved in the tasks in Table 1 are perception, cognition, and evaluation, respectively (Table 1, column IV). Although cognition is sometimes used in a broader sense to encompass perception and evaluation, here perception refers to the process of acquiring new information via the senses, cognition refers to the processes of storing, retrieving, interpreting, and combining new and existing information, and evaluation refers to assessing the personal value of the information (and acting upon it).

We distinguish six basic tasks, each having its own specific type of uncertainty that people need to reduce (Table 1, columns I and II), and each being at least partly observable from eye movements. 1. Localization aims to reduce uncertainty about the spatial location of a known object; 2. Identification aims to reduce uncertainty about the identity of an object vis-à-vis other objects and/or some object template in memory; 3. Specification aims to reduce uncertainty about the presence of specific attributes or features of an object by associating the object with information on these attributes in memory; 4. Inference aims to reduce uncertainty about the consequences of these attributes and their decision outcomes; 5. Valuation aims to reduce uncertainty about the value or meaningfulness of an object’s attributes and features to the person’s goals; 6. Integration aims to reduce uncertainty about the overall utility of an object (choice alternative) to the person relative to other objects. We use the term “integration,” because people need to integrate expected outcomes and their valuation of these outcomes into their overall assessment of the utility of a stimulus (Anderson & Shanteau, Reference Anderson and Shanteau1970). Integration can take place via some (optimal) subjective expected utility mechanism, as traditionally assumed in economics, or via satisficing mechanisms (Simon, Reference Simon1955). The order of uncertainties in the table from top to bottom corresponds to the hierarchical organization of knowledge structures from bottom-up features of objects to top-down values and preferences of people. It reflects the structure of the human motivation system from concrete means (objects and their attributes), which have expected consequences (outcomes), which satisfy certain abstract ends (values) (Kruglanski, Chernikova, Babush, Dugas, & Schumpe, Reference Kruglanski, Chernikova, Babush, Dugas and Schumpe2015), which contribute to utility.

The tasks that people engage in in real-life contexts are commonly combinations of these six basic tasks. For example, localization (task 1) and identification (task 2) are subtasks of a target search task (van der Lans et al., Reference van der Lans, Pieters and Wedel2008b). Localization (task 1) and specification (task 3) are subtasks of an information search task (Moorthy, Ratchford, & Talukdar, Reference Moorthy, Ratchford and Talukdar1997). Identification (task 2) and specification (task 3) are subtasks of a categorization task (Rosch, Reference Rosch1978), in which people differentiate prototypes from exemplars in a category or exemplars in one category from exemplars in another. It is crucial to note that whereas controlled experiments can be used to isolate a single task or uncertainty, the tasks that people perform in natural settings are mostly comprised of several subtasks, each with their specific uncertainty (Droll & Hayhoe, Reference Droll and Hayhoe2007; Hayhoe & Ballard, Reference Hayhoe and Ballard2005; Locke & Latham, Reference Locke and Latham2002).

2. Theoretical Framework

The classification in Table 1 informs our framework, summarized in Fig. 2, of how people use eye movements to reduce uncertainties inherent in perceptual and preferential decision-making tasks. The framework relates eye movements to the underlying processes of interest. It shows from left to right 1) inputs, 2) processes, and 3) decision outputs. These concepts are organized from bottom to top by level of processing, from features, to scene, objects, and the person. Symbols indicate the links between these concepts.

First, the framework distinguishes five observable exogenous inputs: a) person characteristics (i.e., age, prior knowledge), b) task instructions (i.e., search goal, incentives, time constraints), c) object attributes (i.e., description of objects/products on dimensions such as price, brand name and other qualities), d) scene layout (i.e., how the objects are arranged on a row-column shopping website), and e) perceptual features (i.e., detailed visual characteristics, such as colors and edges of objects). Second, the outputs of decision tasks consist of observable micro- (fixation locations, fixation durations) and macro-decision outcomes (search accuracy, choice, and response time), which are represented at the right-hand side of Fig. 2. Third, in between these observable inputs and outputs are unobserved, or latent, processes, depicted in the center of the figure (connected via P1-P5). These latent processes are affected by exogenous inputs (via links I1-I5), and in turn impact both eye movement (via O1-O2) and choice decisions (via O3-O4).

Task and strategy switching take a central role in the framework. Formally, task switching concerns switching between two or more tasks which each have their own unique goals and uncertainties (Kiesel et al., Reference Kiesel, Steinhauser, Wendt, Falkenstein, Jost, Philipp and Koch2010), whereas strategy switching concerns switching between different ways of pursuing the same task or goal (Shi et al., Reference Shi, Wedel and Pieters2013). In this review, we use the terms interchangeably, because in real-life contexts, empirically the distinction between task and strategy switching is often ambiguous. Decision making in real-life contexts typically involves the deployment of attention to resolve multiple uncertainties over time (Table 1). Therefore, people break up complex decision tasks into multiple simpler subtasks, each of which requires a unique strategy that involves the deployment of attention to resolve the associated uncertainty (Stewart, Hermens, & Matthews, Reference Stewart, Hermens and Matthews2016; Stojić, Orquin, Dayan, Dolan, & Speekenbrink, Reference Stojić, Orquin, Dayan, Dolan and Speekenbrink2020). People switch between these subtasks over time to assess the utility of decision alternatives and meet the overall decision goal, under cognitive cost and effort constraints. They monitor goal progress within tasks, and switch when a task nears completion and/or incremental expected utility drops below or cost rises above a threshold (Gutzwiller, Wickens, & Clegg, Reference Gutzwiller, Wickens and Clegg2019). Such task/strategy switching characterizes goal pursuit and decision making in complex, realistic contexts (Locke & Latham, Reference Locke and Latham2002). The next sections describe the attention processes during decision making. Appendix 1 summarizes the neuroscientific basis of attention and eye movements.

Figure 2.

Eye movements and decision task performance. Framework for how people use eye movements to reduce uncertainties inherent in perceptual and preferential decision-making tasks. Observable exogenous inputs to the decision processes are on the left side; unobservable perceptual, cognitive, and evaluative processes are in the center; observable decision outcomes are on the right-side. Inputs and processes are organized from top to bottom by level of visual processing: person, object, scene, and feature. Arrows depict the links between inputs and processes (I1 to I5), between unobserved processes (P1 to P5), and between processes and decision outputs (O1 to O4).

3.1. Eye-Movement Models of Target Search

In Hidden Markov Models (HMM), unobserved discrete states that evolve over time according to a Markov Process are assumed to generate the observed eye-movement time series according to some probabilistic model. Over-time variations in the observed eye movements are reflections of these unobserved states and the switches between them. Liechty et al. (Reference Liechty, Pieters and Wedel2003) were among the first to model latent attention states via HMMs to identify shifts between attention strategies that people use during exploratory search upon advertising exposure.

During search for complex targets, people aim to reduce two types of uncertainties: location and identity uncertainties (Ungerleider & Mishkin, Reference Ungerleider and Mishkin1982; Appendix 1; Table 1). They do so by switching over time between two attention states in which one of the two subtasks takes precedence (see Liechty et al., Reference Liechty, Pieters and Wedel2003). The two hidden states, localization and identification, are assumed to follow a Markov process over time. Van der Lans et al. (Reference van der Lans, Pieters and Wedel2008a; Reference van der Lans, Pieters and Wedelb) developed HMMs to uncover the time path of localization and identification states from eye-movement patterns during target search. The models in question specify the location of each fixation, in pixel coordinates, as a spatial point process. The two attention states are identified via parametric assumptions on that process that are specific to each state. The localization state causes long-amplitude saccades and the identification state short-amplitude saccades (Bullier et al., Reference Bullier, Schall and Morel1996; Thompson, Reference Thompson2005; Appendix 1). Eye movements in the localization state are driven by the salience of locations in the visual display (Itti & Koch, Reference Itti and Koch2001; Koch & Ullman, Reference Koch and Ullman1985; Appendix 1). In the model, salience is defined in terms of pixel-level perceptual features, including colors, brightness, and edges. Computer vision techniques are used to extract these features from digital images of the search display, resulting in (RGB or CIELAB) feature values for each pixel. Because the eye only processes detailed information from a region of about two degrees of visual angle around the point of fixation (Appendix 1), a Normal spatial kernel with a width of two degrees is used to smooth these pixel-level variables. The salience map is then represented as a weighted combination of perceptual features, the weights being represented by individual-level model parameters. These parameters are assumed to follow a Normal distribution and are estimated based on the eye-movement data. They capture the effect of top-down factors on attention (Fig. 2, P5). Although some consider salience to be a purely bottom-up stimulus property (Itti & Koch, Reference Itti and Koch2001), its operationalization as a weighted sum of basic features by Van der Lans et al. (Reference van der Lans, Pieters and Wedel2008a; Reference van der Lans, Pieters and Wedelb) is in line with the idea that salience results from activation or inhibition of visual information that is deemed, respectively, relevant or irrelevant, which is a top-down process (Gaspelin & Luck, Reference Gaspelin and Luck2018; Sawaki & Luck, Reference Sawaki and Luck2010). Similar probabilistic formulations of salience had been previously proposed (Baddeley & Tatler, Reference Baddeley and Tatler2006; Rutishauser & Koch, Reference Rutishauser and Koch2007). The Van der Lans et al. target search model provides estimates of individual-level salience maps. Further, in the localization state, factors measured at the level of the display (scene organization or display architecture) and the person (task schemas or scanning habits) drive eye movements (Fig. 2, I2, I5). Systematic eye-movement strategies (Spalek & Hammad, Reference Spalek and Hammad2005) are captured via Markov transition probabilities on the ROIs containing objects to the left and right, or top and bottom, respectively, of the previously fixated ROI. The identification state accommodates the tendency to re-fixate on an object to collect more information on its identity (Smith & Henderson, Reference Smith and Henderson2009; Tatler & Vincent, Reference Tatler and Vincent2009). This model and its extensions (van der Lans et al., Reference van der Lans, Pieters and Wedel2021) incorporates a range of well-documented neuro-psychological processes (Appendix 1).

Top-down modulation of low-level visual processes involves selectively enhancing visual features that are diagnostic for a search task and suppressing features that are non-diagnostic (Desimone & Duncan, Reference Desimone and Duncan1995; Fig. 2, P5; Einhäuser, Rutishauser, & Koch, Reference Einhäuser, Rutishauser and Koch2008; Rutishauser & Koch, Reference Rutishauser and Koch2007). Bayesian formulations that encapsulate top-down information via a hierarchical prior reflect the mechanism involved (Borji, Sihite, & Itti, Reference Borji, Sihite and Itti2014; Lee & Mumford, Reference Lee and Mumford2003; Torralba et al., Reference Torralba, Oliva, Castelhano and Henderson2006). Van der Lans et al. (Reference van der Lans, Pieters and Wedel2008a) manipulate task instructions to enable the decomposition of bottom-up and top-down influences according to such a Bayesian hierarchical structure. Estimation of the model on eye-movement data, collected while participants were searching for brands of laundry detergent on a simulated shopping shelf, revealed that about two thirds of the variation in the salience of products on the shelf was due to bottom-up factors and one third due to top-down factors.

Several components of these search models, such as top-down effects and salience-based and systematic processing, can be used in stand-alone models applied in experiments in which other uncertainties have been strictly controlled for. For example, the relative salience of an object fixated during search can be estimated as the average fixation-weighted Euclidean distance between that object and the target object in the CIELAB color space (van der Lans et al., Reference van der Lans, Pieters and Wedel2021). Such an operationalization of salience encapsulates the extent to which participants fixate on distractors that are similar to the target, with shared color being a dominant source of visual similarity (Rutishauser & Koch, Reference Rutishauser and Koch2007). Results across a range of products (sunglasses, sneakers, perfumes) revealed that search efficiency gains occur via top-down suppression of perceptual features of competing, distractor products rather than by enhancement of the features of the target product (van der Lans et al., Reference van der Lans, Pieters and Wedel2021; Fig. 2, P5).

Recent work has captured the time-dependency of eye movements during search and related perceptual decision tasks via first-order autoregressive (AR) model formulations, rather than via HMMs. In AR models, the observed variable (fixations) depends probabilistically on its own past values. AR formulations have been used to capture time-dependencies in eye movements in Generalized Linear Mixed models by Cho, Brown-Schmidt and Lee (Reference Cho, Brown-Schmidt and Lee2018), and in tree-based Item Response Theory models by Cho, Brown-Schmidt, Boeck and Shen (Reference Cho, Brown-Schmidt, Boeck and Shen2020). These models describe the POR (point of regard) of the eyes during search as a sequence of binary micro-decisions. The neuro-psychological support for such formulations, although not used to motivate the original modeling, may come from such phenomena as inhibition of return (IOR; Castel, Pratt, & Craik, Reference Castel, Pratt and Craik2003), attentional momentum (Pratt, Spalek, & Bradshaw, Reference Pratt, Spalek and Bradshaw1999), and systematic oculomotor tendencies (Spalek & Hammad, Reference Spalek and Hammad2005; Fig. 2, P5). Looking at time dependencies via a hierarchical Bayes logistic regression, Shi and Trusov (Reference Shi and Trusov2021) study how people explore search engine (Google) results. They investigate what part of the results page people look at, whether they scroll, which items they look at, and which item they click on. The results show that the prior scan-path affects these micro-decisions, along with top-down effects of search goals and bottom-up effects of the semantic context and the spatial layout of the page.

3.2. Eye-Movement Models of Specification Search

Research in agricultural and food economics has examined specification search for price, nutrition, sustainability and other attributes of products, packaging, and food labels, as reviewed by Van Loo et al. (Reference Van Loo, Grebitus, Nayga, Verbeke and Roosen2018). The purpose of that research is to understand which perceptual features and abstract attributes of objects influence perceptual (“find the most sustainable products on display”) and preferential decisions (“choose the products that fit your preferences best”). Research in this tradition has explored bottom-up and top-down factors, the latter including for example having “no goal” versus having “general health” or more specific “reduce sodium intake” goals (Oswald, Adhikari, & Mohan, Reference Oswald, Adhikari and Mohan2022). For instance, van Herpen and van Trijp (Reference van Herpen and van Trijp2011) found that pictorial “traffic light” labels improved search for attributes of food more than textual labels did, even when people were not under time pressure. These studies provide descriptions of specification search as it unfolds in daily life.

Eye movements during specification search have also been studied as part of other than pure specification tasks (see Table 1). For example, research has examined visual exploration of so-called retail feature advertisements, which display multiple (price discounted or otherwise featured) products in a single advertisement. Visual exploration of these ads is a mixed target/specification search task (Table 1), during which people aim to reduce uncertainty about whether a preferred product is present and if so, what its attributes are. Pieters, Wedel and Zhang (Reference Pieters, Wedel and Zhang2007) examined such a task with a hierarchical Bayes model to capture the multilevel nature of multiple ads nested in such ad displays. They used measures of the visual distinctiveness of the target (perceptual difference between a target ad and its competitors in terms of the sizes of their ROIs -regions of interest) and the heterogeneity of distractors (perceptual differences among the other, competing ads) as predictor variables. Their results showed, in support of fundamental psychological theories (Duncan & Humphreys, Reference Duncan and Humphreys1989), that target distinctiveness facilitated and distractor heterogeneity hampered search. The modelling results enabled optimizing the organization of the ad display to improve search performance for each of the advertised products. Strategy switching during specification search tasks (Liechty et al., Reference Liechty, Pieters and Wedel2003) has also been studied, for instance during exploration of theme advertisements (Liechty et al., Reference Liechty, Pieters and Wedel2003; Wedel et al., Reference Wedel, Pieters and Liechty2008), word-sentence processing (Simola, Salojärvi, & Kojo, Reference Simola, Salojärvi and Kojo2008), picture viewing (Haji-Abolhassani & Clark, Reference Haji-Abolhassani and Clark2013, 2014), and tracking moving targets (Kim, Singh, Thiessen, & Fisher, Reference Kim, Singh, Thiessen and Fisher2020).

In view of the ubiquity of search and choice behavior by consumers on attribute-by-product displays and the potential costs of errors, the paucity of eye-movement research in this domain is surprising. But, for instance, tracking eye movements on ”balanced score cards” that managers rely on to assess employees (Chen, Jermias, & Panggabean, Reference Chen, Jermias and Panggabean2016) revealed factors that support better accounting decisions. Eye-movement research into search and choice on row-column displays has led to qualitatively different insights than obtained by more traditional process tracing methodologies. Research relying on traditional process tracing methodologies, such as information display boards (e.g., Lohse & Johnson, Reference Lohse and Johnson1996) or one-way mirrors to observe people’s eye movements during decision making (e.g., Russo & Leclerc, Reference Russo and Leclerc1994) initially identified two or three well-ordered sequential stages in preferential decision making (from screening to evaluation, and then to verification). Likewise, eye-tracking studies initially found that in repeated conjoint choice tasks, participants switch from using attribute-based strategies first to product-based strategies later on (Meißner & Decker, Reference Meißner and Decker2010). Research with Hidden Markov Models (HMMs) has amended this orderly, sequential, two or three-stage perspective on search and choice processes. Shi et al. (Reference Shi, Wedel and Pieters2013) used a three-layer Hierarchical Hidden Markov Model (HHMM) to represent eye movements made to acquire information on attribute-by-product matrices. Their model has two hierarchically connected layers of unobserved states, where the transitions between states in each layer are governed by a Markov process. Given the states of the first hidden layer, the output layer describes the eye movements using Markov transition probabilities between ROIs that are defined via a row (attribute)-column (product) spatial grid. The first hidden layer consists of two states that represent the latent by-attribute or by-product information acquisition strategies, given the states of the second layer. That second layer contains states that allow for switching between these first-layer strategies. The second-layer states, the authors speculated, might reflect top-down processes that activate a specific information-acquisition strategy. Notably, findings from the HHMM applied to data about choices between laptops presented on a 12 by 4 row-column display are at variance with some of the received knowledge obtained from traditional process tracing methodologies and descriptive modeling. First, rather than starting with attribute-based search, people tend to start and end with product-based information search, while in between relying on attribute-based search. Second, rather than switching only once or twice between attribute-based and product-based search, people switched many times. Third, the switching patterns between attribute- and product-based processing depended on the orientation of the display (attributes-by-products or products-by-attributes).

In another application, Chuk et al. (Reference Chuk, Chan, Shimojo and Hsiao2020) use a similar two-level HHMM to describe visual information acquisition during choice of one out of two faces. The purpose of the analysis was to identify regions of interest (ROIs) that are unknown in terms of their number, locations, and boundaries, and the saccade patterns between them. The states in the first hidden layer are interpreted as individual-specific ROIs, that is regions on the human face that a participant uses in deciding which face they find more appealing. The second layer of the HHMM captures the transitions between states of the first hidden layer. The fixation locations at the output layer are modelled via a spatial Gaussian distribution. The HHMM was estimated for each individual separately using a (variational) Bayesian approach that determines the optimal number of states automatically. The results reveal two dominant patterns of eye movements. In the first pattern, people fixate on the preferred face sooner, but the accuracy of inferring their final choice from the eye movements is lower. In the second, people transition from exploration to fixating on the preferred face later, but the tendency to fixate on their preferred face is stronger.

4.1. Preferential Decision Models with Eye-Movements as Explanatory Variables

Eye movements have been used as explanatory variables in choice models to capture the information that people use or ignore in making decisions. Inattentional blindness is an automatic phenomenon that occurs when certain objects or attributes of a scene are not noticed (even though they might receive eye fixations) because attention is devoted to another task or object (Mack & Rock, Reference Mack and Rock1998). In the context of multi-attribute choice, attribute inattention (Fig. 2, I3, P5) occurs when certain attributes are not processed (Hensher, Rose, & Greene, Reference Hensher, Rose and Greene2005), and object inattention occurs when one or more alternatives are not considered in the choice decision (Roberts & Lattin, Reference Roberts and Lattin1997). But such inattention can be mitigated. When choice tasks in eye-tracking studies are designed to be incentive compatible, participants utilize up to twenty percent more of the attribute information presented to them (Yang, Toubia, & de Jong, Reference Yang, Toubia and de Jong2018). Nonetheless, to accommodate inattention, eye-movement selection indicators (yes-no fixated) have been used as explanatory variables in choice models, as direct effects (Pieters & Warlop, Reference Pieters and Warlop1999), as moderators of the attributes (part-worths) of products (Balcombe, Fraser, & McSorley, Reference Balcombe, Fraser and McSorley2015; Yegoryan, Guhl, & Klapper, Reference Yegoryan, Guhl and Klapper2020), or as mediators to explain decision outcomes (Chandon et al., Reference Chandon, Hutchinson, Bradlow and Young2009; Pieters & Warlop, Reference Pieters and Warlop1999).

4.2. Psychological Process Models of Preferential Decision Making

A stream of literature in psychology has developed process models of decision making that are calibrated on response times and choice outcomes. Such Sequential Sampling Models (SSM) (Ratcliff, Reference Ratcliff1978; Ratcliff, Van Zandt, & McKoon, Reference Ratcliff, Van Zandt and McKoon1999; Smith & Ratcliff, Reference Smith and Ratcliff2004) assume that evidence (perceptual choice) or utility (preferential choice) on two or more choice alternatives accumulates stochastically, until one alternative has accumulated sufficient evidence to cross a decision threshold. A larger value of the threshold results in more accurate but slower responses. Drift Diffusion Models (DDM) are a special case. In the basic DDM version, the accumulation of evidence on each alternative is described by a Wiener diffusion process (which has independent Gaussian increments). DDMs were originally developed to describe accuracy and latency of two-alternative perceptual decision tasks (Ratcliff, Reference Ratcliff1978; Smith & Ratcliff, Reference Smith and Ratcliff2004; Voss, Nagler, & Lerche, Reference Voss, Nagler and Lerche2013), but have later also been applied to examine preferential decision making (Dutilh & Rieskamp, Reference Dutilh and Rieskamp2016; Summerfield & Tsetsos, Reference Summerfield and Tsetsos2012; Voss, Rothermund, & Voss, Reference Voss, Rothermund and Voss2004).

Commonly, DDMs are fit to preferential decision outcomes assuming an underlying diffusion process according to which a unitary subjective value for each alternative accumulates over time. For example, DDMs have been used to describe choices between pairs of food products using preference ratings for these products provided by participants prior to the choice task as exogenous input (Milosavljevic, Malmaud, Huth, Koch, & Rangel, Reference Milosavljevic, Malmaud, Huth, Koch and Rangel2010). DDMs have been fit to choices among multiple alternatives as well, based on the assumption that the differences in accumulated evidence among them favors one of the options and determines choice. Other SSMs describe multi-alternative choice with multiple diffusion processes, each with their own drift rate, which race until the first one crosses a threshold (Usher & McClelland, Reference Usher and McClelland2001; Usher, Olami, & McClelland, Reference Usher, Olami and McClelland2002).

A critical assumption of traditional DDMs is that visual information uptake is randomly distributed across alternatives. Therefore, these models imply that eye movements are uniformly distributed over alternatives (and attributes), which is similar to the implicit assumption in compensatory decision models that people attend equally to all information in the choice task (Glöckner & Herbold, Reference Glöckner and Herbold2011; Orquin et al., Reference Orquin, Lahm and Stojić2021). Despite some challenges in generalizing DDMs, and SSMs more generally, to multi-alternative preferential choice in realistic contexts (e.g., Mormann & Russo, Reference Mormann and Russo2021), the models provide an important step toward connecting neuro-physiological processes to unobserved preference formation. With a few exceptions (e.g., Satomura, Wedel, & Pieters, Reference Satomura, Wedel and Pieters2014), sequential sampling models have yet received limited attention in the marketing and economics literature as process models for decision making. Nonetheless, the Poisson race model, which also rest on sequential sampling of information during stimulus exposure, has been applied to conjoint choice tasks (Ruan, MacEachern, Otter, & Dean, Reference Ruan, MacEachern, Otter and Dean2008). That model assumes that information signals arrive for each alternative according to a Poisson process (with time-varying rate), until an (integer) threshold is crossed. While that model was fit to only macro-outcomes, choice and latency, Poisson race models for perceptual decisions have used eye-tracking data to express how random chunks of information arrive via eye fixations according to a Poisson process until a threshold is crossed that determines the decision outcome (Wedel & Pieters, Reference Wedel and Pieters2000).

Extensions of DDMs have included eye movements to describe perceptual (Rangelov & Mattingley, Reference Rangelov and Mattingley2020; Tavares, Perona, & Rangel, Reference Tavares, Perona and Rangel2017) and preferential decision-making (Callaway et al., Reference Callaway, Rangel and Griffiths2021; Cavanagh, Wiecki, Kochar, & Frank, Reference Cavanagh, Wiecki, Kochar and Frank2014; Towal, Mormann, & Koch, Reference Towal, Mormann and Koch2013). The key feature of the attentional DDM (aDDM) by Krajbich, Rangel and colleagues (Krajbich et al., Reference Krajbich, Armel and Rangel2010; Krajbich, Lu, Camerer, & Rangel, Reference Krajbich, Lu, Camerer and Rangel2012; Krajbich & Rangel, Reference Krajbich and Rangel2011) is that the information on any alternative accumulates only while the eyes fixate on that alternative. The parameters of the model capture the speed of value accumulation, the bias toward the fixated option, and the error variance.

The initial aDDM predicts that alternatives fixated first and fixated more frequently are more likely to be chosen. Still, it does not allow for effects of bottom-up factors (Fig. 2, I1, I2, P3), nor for effects of the accumulating utility on eye movements (Fig. 2, P2; Mormann & Russo, Reference Mormann and Russo2021). Thus, it assumes the eye-movement process to be exogenous to the preference formation and choice processes, that is, saccades between choice options are assumed to be random, which is at variance which evidence reviewed here. Recent extensions of the aDDM (Gluth et al., Reference Gluth, Kern, Kortmann and Vitali2020) allow the accumulated value of an option to increase the likelihood that it will be fixated (Fig. 2, P2). This extended model accounts for various eye-movement patterns, including the first fixation and the evolution of fixations on choice alternatives over time. Likewise, Song, Wang, Zhang and Li (Reference Song, Wang, Zhang and Li2019) have proposed a sequential sampling model in which alternatives with higher (value/utility) uncertainty are more likely to be fixated (Table 1). In their model, eye fixations deliver additional information that reduces the uncertainty about choice alternatives via a Bayesian updating mechanism. It has been shown (Callaway et al., Reference Callaway, Rangel and Griffiths2021) that with Bayesian updating the selection of which alternative to fixate and when to terminate the process can be formulated as a dynamic optimization problem, in which (costly) eye movements are allocated optimally over time to acquire information that reduces uncertainty about the values of the alternatives.

4.3. Economic Models of Preferential Decision Making

Recent models of preferential decision-making aim to predict future choice between multiple alternatives while endogenizing the complete eye-movement process. For example, Martinovici et al. (Reference Martinovici, Pieters and Erdem2021) use not just the final fixations, as in research on the gaze cascade (Shimojo et al., Reference Shimojo, Simion, Shimojo and Scheier2003), or the sum of fixations as in aDDMs, but the entire trajectory of eye movements for each of the choice alternatives to predict preferential decision making in a naturalistic choice task. They develop a joint model of eye movements and choice outcomes that describes the gaze on product ROIs over time as a set of covarying higher-order polynomials. It enables prediction of the final choice from gaze patterns well before the actual choices are implemented, out-of-sample. It is a predictive model, however, that is agnostic about the underlying interplay between attention and preference formation prior to choice.

Researchers have recently applied economic modeling frameworks (Gabaix, Laibson, Moloche, & Weinberg, Reference Gabaix, Laibson, Moloche and Weinberg2006; Moorthy et al., Reference Moorthy, Ratchford and Talukdar1997; Weitzman, Reference Weitzman1979) to eye movements, in order to describe information search and choice for high-involvement products (e.g., laptops, cell phones). These frameworks describe information search as a sequential process with an optimal solution. At each step in the process, a decision maker examines the alternative with the highest utility and stops when the realized utility of that alternative is larger than the expected utility of the next alternative (Ursu et al., Reference Ursu, Zhang and Erdem2021; Yang et al., Reference Yang, Toubia and de Jong2015). Information search in these models is thus determined by the expected value that the next piece of information provides. The models in question simultaneously embed micro- (fixation) and macro- (choice) decisions in a utility maximization, multinomial logit, modeling framework. The models assume that people use Bayesian updating of their prior uncertainty about the value of an attribute level (Table 1) with a noisy chunk of information obtained by fixating a product attribute, resulting in an updated product value. The expected utility is modeled as a weighted sum of the part-utilities of the attribute levels. The utility of search (and thus implicitly its cost) is specified as a function of eye-movement characteristics, such as the saccade length and direction (Ursu et al., Reference Ursu, Zhang and Erdem2021; Yang et al., Reference Yang, Toubia and de Jong2015) to allow for local clusters of fixations and for horizontal and vertical saccade tendencies. Unobserved heterogeneity in the models’ parameters across individuals is accommodated via a Normal distribution. The results of that research show that integrating eye movements into models of search and choice improves parameter estimates, the understanding of the choice process, as well as out-of-sample predictions.

The appeal of such economic approaches is that they (1) integrate choice and eye-movement decisions in a single framework, (2) postulate a plausible economic (cost-benefit) mechanism for eye-movement decisions (Callaway et al., Reference Callaway, Rangel and Griffiths2021), and (3) assume that utility/value evolves during the decision process and affects fixations (Gluth et al., Reference Gluth, Kern, Kortmann and Vitali2020; Ursu et al., Reference Ursu, Zhang and Erdem2021). However, a limitation of current implementations is that known influences of visual factors and attentional processes (Fig. 2, bottom up: I1, I2, P3, and top down: I4, P4, P5) are not accommodated or only in a basic way. Also, the assumed (one-step) forward-looking mechanism remains as yet unverified.

Research for low-involvement value-based decisions (e.g., snacks) has also relied on an utility framework, but has specified different underlying processes (Reutskaja et al., Reference Reutskaja, Nagel, Camerer and Rangel2011; Stüttgen et al., Reference Stüttgen, Boatwright and Monroe2012). For low-involvement decisions, target search is more likely a main component of the choice task (Table 1), with people relying on the perceptual salience of objects, habitual decision making, and satisficing choice rules. For instance, in modeling choice of one out of a large set of products under time pressure, it was assumed that people may stop and choose after having fixated all alternatives at least once or after running out of time (Reutskaja et al., Reference Reutskaja, Nagel, Camerer and Rangel2011). This assumption is in line with a satisficing choice heuristic where search stops after the value of the currently fixated alternative exceeds a reservation value.

Stüttgen et al. (Reference Stüttgen, Boatwright and Monroe2012) extended the target search model of van der Lans et al. (Reference van der Lans, Pieters and Wedel2008a) by including a third state in the HMM, in addition to the localization and identification states. The model specifies a satisficing rather than maximizing process (Simon, Reference Simon1955). Specifically, in the third, termination state an individual chooses a product that is satisfactory, as follows (products that are undetermined are chosen with a probability close to zero). The individual assigns the status satisfactory, unsatisfactory, or undetermined to each product at each fixation. If a product (here, instant noodles) has not been fixated, its status is undetermined. To determine whether a product is satisfactory, the individual needs to fixate all attributes of that product and the value of each attribute needs to be acceptable. The model accounts for strategy switching, where the switching probabilities depend on the status of the products and the number of prior fixations. It allows people to proceed to a new choice option before returning and making a final choice, however, which is at variance with a pure satisficing heuristic (Gilbride & Allenby, Reference Gilbride and Allenby2004, Reference Gilbride and Allenby2006; Jedidi & Kohli, Reference Jedidi and Kohli2008) or the process assumed in aDDMs (Gluth et al., Reference Gluth, Kern, Kortmann and Vitali2020; Krajbich et al., Reference Krajbich, Armel and Rangel2010; Song et al., Reference Song, Wang, Zhang and Li2019).