2.1 Preferences and RT

2.2 Decision processes and RT

Sequential-sampling models of decision making (also referred to as information-accumulation, or drift-diffusion models) have become one of the main paradigms in the mathematical/cognitive psychology literature (Busemeyer Reference Busemeyer2002; Ratcliff and Smith Reference Ratcliff and Smith2004; Smith Reference Smith2000; Smith and Ratcliff Reference Smith and Ratcliff2004; Usher and McClelland Reference Usher and McClelland2001)—see also the extensive discussion in Sect. 3.6. These models assume that cognitive systems are inherently noisy and that the process of arriving at a decision occurs through the accumulation (integration) of noisy samples of evidence until a decision threshold is reached. An important prediction of these models is that the smaller the difference in the values of the options, the longer the RT. Krajbich et al. (Reference Krajbich, Lu, Camerer and Rangel2012)—see also similar work in Krajbich et al. (Reference Krajbich, Armel and Rangel2010) and Krajbich and Rangel (Reference Krajbich and Rangel2011)—extend standard sequential-sampling models to explicitly incorporate the allocation of attention and show that their model can simultaneously account for the triptych of information lookups, choice and RT. Importantly, their model predicts that the time spent on information lookups can influence choice, and that time pressure can lead to noisier valuations, thereby increasing the probability of an error.

Similar conclusions have been reached in the economics literature, albeit derived from different models. Wilcox (Reference Wilcox1993) finds that subjects exhibit longer RT–a proxy for effort–in a lottery choice task when monetary incentives are higher and the task is complex. Gabaix and Laibson (Reference Gabaix and Laibson2005) and Gabaix et al. (Reference Gabaix, Laibson, Moloche and Weinberg2006) also derived the above-mentioned relationship between RT and the difference between option valuations under the assumption that valuations are noisy, but improve the more time is devoted to the task—more details on their modeling can be found in Sect. 3.5. Chabris et al. (Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2009) tested the optimal allocation of time in decision tasks and reported empirical evidence that the closer the expected utility of the competing options is, the longer the response time. Similarly, Chabris et al. (Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2008) find that the same principle can be used to recover discount rates from RT data without observing choices.

Another important theme in the literature is the explicit consideration of heuristics (including the use of focal points) versus compensatory, and more complex, decision rules. Guida and Devetag (Reference Di Guida and Devetag2013) combine eye-tracking and RT analysis in normal-form games, and find that RT was shorter for games with a clear focal point, and longer for Nash equilibrium choices. Fischbacher et al. (Reference Fischbacher, Hertwig and Bruhin2013) find that participants’ behavior, although heterogeneous, is consistent with the sequential application of three motives in lexicographic fashion. The more motives that are considered, the longer the RT, e.g., a selfish type only examines own payoffs, whereas a pro-social type must also examine others’ payoffs. Coricelli et al. (Reference Coricelli, Diecidue and Zaffuto2016), on the other hand, argue that choices between lotteries–whenever possible–may be driven by a simplifying heuristic based on aspiration levels. Such an aspiration-based heuristic can be executed more quickly than the compensatory processes that subjects revert to when this heuristic is not applicable. Spiliopoulos et al. (Reference Spiliopoulos, Ortmann and Zhang2015) found that subjects under time pressure shifted to simpler–yet still effective–heuristics, namely the Level-1 heuristic that simply best responds to the belief that an opponent randomizes with equal probability over his/her action space. Spiliopoulos (Reference Spiliopoulos2016) examines repeated constant-sum games and finds that RT is dependent on the interaction of two different decision rules: the win-stay/lose-shift heuristic and a more complex pattern-detecting reinforcement learning model. While the former is executed faster than the latter, response time was longer when the two decision rules gave conflicting recommendations regarding which action to choose in the next round.

Research on the impact of emotions is less common. Grimm and Mengel (Reference Grimm and Mengel2011) delay participants’ decisions whether to accept/reject an offer in an Ultimatum game for ten minutes. In line with their hypothesis that negative emotions are attenuated as time passes, they find higher acceptance rates after the time delay. Although regret and disappointment have been found to play a role in choices under risk (e.g., Bault et al. Reference Bault, Wydoodt and Coricelli2016; Coricelli et al. Reference Coricelli, Critchley, Joffily, O’Doherty, Sirigu and Dolan2005; Coricelli and Rustichini Reference Coricelli and Rustichini2009), their relationship with RT has not been thoroughly investigated.

2.3 Classification

RT is also used to classify subjects into different types, above and beyond possible classifications according to choice behavior. For example, Rubinstein (Reference Rubinstein2007, Reference Rubinstein2013) show that a typology based on RT is more predictive than a typology based on the estimated level of risk aversion. Rubinstein (Reference Rubinstein2016) objectively defines contemplative (instinctive) actions in ten different games as those actions with longer (shorter) RT than the average RT in the game for all actions. The contemplative index of a player derived from subsets of nine of the ten games was positively correlated to the probability of the same player choosing a contemplative action in the tenth game.

Devetag et al. (Reference Devetag, Di Guida and Polonio2016) find that the time spent looking up each payoff in $3\times 3$ normal form games is predictive of final choices and the level of strategic reasoning of players. Schotter and Trevino (Reference Schotter and Trevino2014b) use RT in global games to distinguish between two types of players with respect to their learning process. Intuitionists who have a eureka moment when they realize which strategy is effective and learners who acquire an effective strategy through a slower trial-and-error (or inductive) process. A striking result is that RT was more predictive of out-of-sample behavior than the equilibrium solution.

These findings show that RT can be used either alone or in conjunction with choice data to sharpen the classification of subjects into types, thereby increasing our ability to predict the behavior of decision makers across different tasks. This suggests that models including both choice and RT predictions have greater scope and are more generalizable to new situations (Busemeyer and Wang Reference Busemeyer and Wang2000), thereby increasing the predictive content of behavioral models.

2.4 Speed–performance profile

Another theme in the literature relates time pressure and the opportunity cost of RT to the quality of decision making, i.e., the speed–performance relationship (discussed at length in Sect. 4.2). Kocher and Sutter (Reference Kocher and Sutter2006) found that time pressure reduced the quality of individual DM, but time-based incentives led to faster decisions without a decrease in decision quality. Arad and Rubinstein (Reference Arad and Rubinstein2012) discover that higher average payoffs are achieved by subjects with longer (endogenous) RT. We believe that this theme, which is closely related to the adaptive decision maker hypothesis is the least studied so far in strategic DM. The allocation of time between a set of tasks has been studied by Chabris et al. (Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2009). Subjects allocated more time to those tasks that were more difficult, defined as tasks where alternative options had more similar valuations. Recall that Spiliopoulos et al. (Reference Spiliopoulos, Ortmann and Zhang2015) find that roughly one-third of subjects adapt strategically to time pressure without sacrificing performance (here, payoffs) despite switching to less sophisticated heuristics. There is much work to be done in understanding the speed–performance relationship in strategic DM, and examining whether it is robust to context and tasks. We conjecture it is not, therefore further work will be required to map out how and why this relationship changes—we return to this in more detail in Sect. 4.2.

2.5 Summary

Our review of the existing literature revealed significant evidence that RT matters in decision making. Decision makers typically adapt to time constraints leading to significantly different behavior. Consequently, the generalizability of empirical findings from the lab and the scope of existing models of behavior may need to be revised. Future work should be directed toward rigorously testing the robustness of some of the main findings in experimental economics and enriching our models with procedural components that can predict how decision makers adapt to the temporal aspect of decision environments—the following section is devoted to the latter.

3.1 Dual-system models

Dual-system (or dual-process) theories, based on the assumption that the human brain is figuratively comprised of two different systems, are increasingly being applied to decision making (Kahneman Reference Kahneman2011). For an overview of the implications of dual-system models for economic behavior, see also Alós-Ferrer and Strack (Reference Alós-Ferrer and Strack2014) and other articles in the special issue on dual-system models in the Journal of Economic Psychology of which it was a part. System 1, the intuitive system, is conceptualized as being influenced by emotions, instinct and/or simple cognitive computations occurring below the level of consciousness. Decisions are made quickly and do not require vast reams of information. This system is viewed as part of the earlier evolution of the human brain and tends to be associated with “older” areas of the brain, e.g., the fight-or-flight system. System 2, the deliberative system, is conceptualized to operate on the conscious level and involves higher-level cognitive processes. Decisions are made more slowly and can involve conscious information search. This system is viewed as a more recent evolution of the human brain and its usefulness involves the ability to override the instinctual responses of System 1 when necessary, or to plan a cognitive response in a new environment. Although there is evidence of some degree of localization of these systems, the double-dissociation studies often presented as evidence of two literally distinct systems at the neural level is not without controversy—see Keren and Schul (Reference Keren and Schul2009), Rustichini (Reference Rustichini2008) and Ortmann (Reference Ortmann2008) for critiques and comparisons of unitary versus dual system models.

We consider standard dual-system models to be primarily descriptive models of behavior rather than procedural models. We base this assessment on how dual-system models are applied rather than their potential. Typically they are used to classify behavior as instinctive or deliberative. The inherent freedom in classifying behaviors as instinctive or deliberative is an important issue with the dual-system approach, particularly for strategic DM. Rubinstein (Reference Rubinstein2007) uses the following possible classifications for an instinctive response, depending on the strategic structure of the game.

1. 1. The number of iterations required to reach the subgame perfect NE.

2. 2. The strategy associated with the highest own payoff.

3. 3. The number of steps of iterated dominance required to solve a game.

4. 4. The strategy selected by self-interested individuals.

5. 5. The strategy that yields a “fair” outcome.

There are other criteria that could define an instinctive response. In one-shot games, Guida and Devetag (Reference Di Guida and Devetag2013) find that RT is smaller for games with a focal point compared to those without. In sum, definitions of instinctive responses can be very task- and context-dependent. The contradictory findings for games where social preferences are dominant provide striking evidence of this. Some studies conclude that RT is lower for self-interested choices (Brañas-Garza et al. Reference Brañas-Garza, Meloso and Miller2016; Fischbacher et al. Reference Fischbacher, Hertwig and Bruhin2013; Matthey and Regner Reference Matthey and Regner2011; Piovesan and Wengström Reference Piovesan and Wengström2009), whereas other studies find that the equitable or “fair” split is associated with a lower RT (Cappelletti et al. Reference Cappelletti, Güth and Ploner2011; Halali et al. Reference Halali, Bereby-Meyer and Meiran2011; Lotito et al. Reference Lotito, Migheli and Ortona2013). Under the auxiliary assumption that instinctive choices require less time, these studies arrive at opposing conclusions of what behavior has evolved to be instinctive. Furthermore, as already briefly indicated, the use of reverse inference–observing which choices are faster and declaring them to be intuitive–has been contested (Krajbich et al. Reference Krajbich, Bartling, Hare and Fehr2015). The basic idea of these critical authors makes use of people’s well-documented heterogeneity, for example in social preferences, and they propose essentially that one’s basic disposition (being selfish, or being altruistic for example), determines what one considers intuitive. An alternative to the instinctive versus deliberative dichotomy, relates the computational complexity of different (procedural) decision rules to endogenous or exogenous RT (Spiliopoulos et al. Reference Spiliopoulos, Ortmann and Zhang2015).

Extending the currently primarily descriptive models to include procedural sub-models for each system, and an explicit mechanism for how the two systems interact, would transform them into procedural models. Since System 2 can override System 1, a complete theory would require a specification of how, and when, this occurs. Empirical findings suggest that System 2 is less likely to control the decision if there is time pressure, cognitive load, scarcity of information, etc. (Kahneman Reference Kahneman2003). However, the multitude of switching mechanisms currently proposed combined with the dual systems, which individually can account for different behavior, leads to the possibility of ad-hoc explanations of empirical findings.

A new generation of dual-system type models address these concerns by explicitly modeling the interaction of the systems. Models of dual selves do so by explicitly defining the role of each self and imposing structure on their strategic interactions (Fudenberg and Levine Reference Fudenberg and Levine2006, Reference Fudenberg and Levine2012; Fudenberg et al. Reference Fudenberg, Levine and Maniadis2014). The long-run self cares about future payoffs, whereas the short-run self cares only about short-run, typically immediate, payoffs. The short-run self is in control of the final decision made. The long-run self seeks to influence the utility function of the short-run self, but incurs a self-control cost. Such an explicit representation of the dual selves and their interaction permits sharper predictions of behavior than standard dual-system models. While these models do not explicitly account for time, it is possible to operationalize RT with the auxiliary assumption that it is increasing in the cost function of self-control. Achtziger and Alós-Ferrer (Reference Achtziger and Alós-Ferrer2014) and Alós-Ferrer (Reference Alós-Ferrer2016) propose a dual-process model in which the interaction between a faster, automatic process and a slower, controlled process is explicitly defined. The model’s RT predictions, for both erroneous and correct decisions conditional on the degree of conflict or agreement of the two processes, were empirically verified in a belief-updating experiment. Spiliopoulos (Reference Spiliopoulos2016) similarly validates the model’s qualitative RT predictions in repeated games, where the automatic process is specified as the win-stay/lose-shift heuristic and the controlled process as the pattern-detecting reinforcement learning model introduced in Spiliopoulos (Reference Spiliopoulos2013). Conflict between the two processes led to longer RT, and also influenced RTs conditional on the interaction between conflict and which process the chosen action was consistent with.

3.2 Heuristics and the adaptive toolbox

Heuristics–often referred to as fast and frugal–in the tradition of the ecological-rationality program (Gigerenzer et al. Reference Gigerenzer and Todd1999; Ortmann and Spiliopoulos Reference Ortmann, Spiliopoulos and Altman2017), are simple decision rules that often perform as well, if not better, than more complex decision rules for out-of-sample predictions, i.e., cross-validation. Heuristics are particularly amenable to response time analyses because their sub-processes and interactions are typically explicitly specified in the definition of the heuristic.Footnote ⁷ Consequently, RT can be defined as an increasing function of the number of elementary information processing (EIP) units required to execute a decision rule (Payne et al. Reference Payne, Bettman and Johnson1992, Reference Payne, Bettman and Johnson1993). EIPs can be thought of as the lowest level operations required for the execution of a computational algorithm. These would include retrieving units of information, and processing them, e.g, mathematical operations such as addition, multiplication, subtraction, and magnitude comparisons. While originally applied to individual DM, Spiliopoulos et al. (Reference Spiliopoulos, Ortmann and Zhang2015) calculate the EIPs of popular decision strategies for normal-form games, and find that under time pressure players shift to strategies that are less complex, i.e., are comprised of fewer EIPs. Another class of heuristics that have been applied to strategic DM are decision trees, which structure the decision processes as a series of sequential operations conditional on the history of prior operations, eventually leading to a terminal node that determines the final decision. Empirical investigations of decision trees in the ultimatum game can be found in Fischbacher et al. (Reference Fischbacher, Hertwig and Bruhin2013) and Hertwig et al. (Reference Hertwig and Hoffrage2013).

The Adaptive-Toolbox paradigm (Gigerenzer and Selten Reference Gigerenzer and Selten2002) posits that decision makers choose from a set of heuristics, and that a heuristic’s performance depends on the exploitable characteristics of the current environment. A decision maker is therefore faced with the task of how to match the appropriate heuristic to environmental characteristics. Obviously, any such choice will be affected by RT. How, in particular, are heuristics or strategies chosen if the decision maker has no prior knowledge of the relationship between heuristics’ performance and environmental characteristics? For individual DM tasks, Rieskamp and Otto (Reference Rieskamp and Otto2006) find evidence that subjects use a reinforcement-learning scheme over the available heuristics. For strategic DM, Stahl (Reference Stahl1996) concludes that subjects often apply rule-learning, which is essentially a form of reinforcement learning over a set of decision strategies. Closely related to this approach is the literature on evolution as the selection mechanism of decision rules, e.g., Engle-Warnick and Slonim (Reference Engle-Warnick and Slonim2004); Friedman (Reference Friedman1991).

3.3 Models of iterated strategic reasoning

Models of bounded rationality incorporating finite iterated best responses, such as the iterated deletion of dominated strategies, cognitive hierarchy (Camerer et al. Reference Camerer, Ho and Chong2004) and Level-k models (Costa-Gomes et al. Reference Costa-Gomes, Crawford and Broseta2001; Stahl and Wilson Reference Stahl and Wilson1995), make implicit predictions about RT. Although evidence in favor of these models has been based on choice data, there are falsifiable RT predictions that would provide further useful information. Cognitive hierarchy or Level-k models implicitly produce an ordinal ranking of RT over the degree of sophistication within a decision.Footnote ⁸ For example, since Level-k agents must solve for the actions of all prior $k-1$ level players to calculate a best response, RT is a monotone increasing function of the level, k.

3.4 Substantive models augmented with auxiliary assumptions

The joint modeling of choice and RT is not necessarily restricted to explicitly designed procedural models, but can be accomplished by redefining models of substantive rationality. For example, the EU maximization problem can be modified in the following ways:

3.5 Search and attentional models

Models in this class explicitly account for information search or acquisition either externally (directly from the environment) or internally (from memory). For example, the Directed-Cognition model of external search (Gabaix and Laibson Reference Gabaix and Laibson2005; Gabaix et al. Reference Gabaix, Laibson, Moloche and Weinberg2006) extends the agent’s objective function to include the opportunity cost of time, and is consistent with empirical evidence that subjects were partially myopic in their assessments of the future benefits and costs of additional information acquisition, thereby circumventing the intractability of a rational solution. Similarly, Bordalo et al. (Reference Bordalo, Gennaioli and Shleifer2012) find that information salience is predictive of RT through the effect of salience on the allocation of attention. Internal information acquisition from memory is also time-dependent, e.g., memories that are more likely to be needed (are more recent and/or have been rehearsed more times) are retrieved more quickly (Schooler and Anderson Reference Schooler and Anderson1997). In individual DM, Marewski and Melhorn (Reference Marewski and Melhorn2011) leverage the explicit modeling of memory using the ACT-R framework (Anderson Reference Anderson2007; Anderson and Lebiere Reference Anderson and Lebiere1998) to infer which models are appropriate. In strategic DM, forgetting is found to constrain the strategies used by players in repeated games (Stevens et al. Reference Stevens, Volstorf, Schooler and Rieskamp2011).

3.6 Sequential-sampling models

One of the main advantages of such models is the clear identification of the underlying process mechanism and the simultaneous modeling of both choices and RT. The instantaneous valuations of each available option are conceptualized as a deterministic drift component, which is a function of the expected payoff of the option, and a random component. Evidence for each option is accumulated over time, as determined by the drift rate and noise. The whole process resembles a random walk with a drift specified by the instantaneous valuations of each option. If there are no time constraints, then a decision is made when the accumulated evidence for any of the options reaches a threshold value. Intuitively, for a given threshold, a lower (higher) drift rate leads to a longer (shorter) mean RT. For a given drift rate, a higher threshold reduces the probability of erroneously choosing the option with the lower mean valuation as the effects of noise are diminished. Alternatively, if a time constraint is enforced, then rather than racing towards a threshold value, a decision is made in favor of the option that has the highest accumulated evidence at the time the constraint is reached.

Early work originated in the context of memory retrieval (Ratcliff Reference Ratcliff1978). Busemeyer and Townsend (Reference Busemeyer and Townsend1993) formalized this process for individual DM under risk (referred to as Decision Field Theory). Many variations and related models can be found in the psychology literature and more recently in economics (e.g., Busemeyer Reference Busemeyer2002; Caplin and Martin Reference Caplin and Martin2016; Clithero Reference Clithero2016; Fudenberg et al. Reference Fudenberg, Strack and Strzalecki2015; Krajbich et al. Reference Krajbich, Armel and Rangel2010, Reference Krajbich, Lu, Camerer and Rangel2012; Krajbich and Rangel Reference Krajbich and Rangel2011; Ratcliff and Smith Reference Ratcliff and Smith2004; Rieskamp et al. Reference Rieskamp, Busemeyer and Mellers2006; Smith Reference Smith2000; Smith and Ratcliff Reference Smith and Ratcliff2004; Usher and McClelland Reference Usher and McClelland2001; Webb Reference Webb2016; Woodford Reference Woodford2014). Although strategic DM can also be modeled in this manner, more complex characterizations of the decision processes are necessary. Spiliopoulos (Reference Spiliopoulos2013) examines belief formation and choice in repeated games, extending a sequential model to capture strategic processes by assuming that the instantaneous drift is driven by an expected value calculation based on payoffs and strategic beliefs—the latter are determined by the retrieval of historical patterns of play from memory.

The first sequential-sampling models proposed a unitary-system model of behavior that can produce a variety of different behaviors by conditioning the decision threshold on the task properties and environment. Consequently, they were viewed as competitors to the dual-system approach, see Newell and Lee (Reference Newell and Lee2011). However, interesting hybridizations of dual-systems models and sequential-sampling models have been presented recently. Caplin and Martin (Reference Caplin and Martin2016) propose a dual-process sequential-sampling model that first performs a cost-benefit analysis of whether accruing further information (beyond any priors) is expected to be beneficial, and then either makes an automatic decision based on the priors if the expected costs exceed the benefits or otherwise triggers a standard accumulation process. The discussion about the appropriateness of dual-system, sequential-sampling and hybrid models is ongoing and in our view deserves the attention it receives. The varying RT predictions of these competing models can be useful in model comparison and selection.

3.7 Summary

We have presented a multitude of different models, often arising from opposing schools of thought, e.g., simple heuristics versus optimization under constraints, single versus dual-system models. The presented models also differ significantly in terms of whether they explicitly incorporate decision processes or address only the functional, e.g., according to Marr (Reference Marr1982), the former operates on the algorithmic and the latter on the computational (or functional) level. We are partial to models operating at the algorithmic level or what we refer to as procedural modeling—further discussed in Sect. 6.1. However, operating at a higher level of abstraction can also have benefits, including simplicity. We suspect that the type of model chosen for RT analysis will be highly dependent on a researcher’s proclivity; however, we encourage model comparisons between these different types of models. Furthermore, it may be the case that different types of models operate at varying degrees of time constraints; in this case we argue for a better understanding of the scope of these models and under what conditions each one is triggered in human reasoning.

4.1 Improved external validity

In the Introduction, we expressed concerns regarding the external validity of standard experiments that do not account for time constraints and the opportunity cost of time by assuming virtually unlimited, costless information search and integration. We argue that external validity can be improved by increasing experimental control through RT experiments (discussed in Sect. 4.3), and that such experiments allow us to thoroughly investigate the speed–performance relationship (discussed in the following section), which is particularly relevant for decisions in the wild.

4.2 Mapping the relationship between RT and performance

An often investigated relationship is the speed–performance or speed–accuracy trade-off. The difference between accuracy and performance is subtle but important. The former is a measure in the choice space, whereas the latter in the consequence space, which is essentially measured by the payoffs derived from a choice. For example, measures of accuracy include the proportion of actions that were dominated, the proportion that were errors (when clearly defined)—note that these measures do not capture the cost to the decision maker of said errors. However, if time is scarce or costly, fast errors may be optimal if they have a relatively small consequence on payoffs, and permit the allocation of time—and therefore reduced probability of an error—to decisions with higher payoffs.

A key insight of the ecological-rationality program (Gigerenzer Reference Gigerenzer, Koehler and Harvey1988; Gigerenzer et al. Reference Gigerenzer and Todd1999, Reference Gigerenzer, Hertwig and Pachur2011; Gigerenzer and Selten Reference Gigerenzer and Selten2002; Ortmann and Spiliopoulos Reference Ortmann, Spiliopoulos and Altman2017) is that, in contrast to claims by researchers in the original adaptive decision maker tradition (Payne et al. Reference Payne, Bettman and Johnson1988, Reference Payne, Bettman and Johnson1993), more speed is not necessarily bought at the cost of lower performance. We note that this surprising result is conditional on an important methodological innovation, cross-validation, that only recently has found the appropriate appreciation in economics (e.g., Erev et al. Reference Erev, Ert, Plonsky, Cohen and Cohen2017; Ericson et al. Reference Ericson, White, Laibson and Cohen2015)—see also Ortmann and Spiliopoulos (Reference Ortmann, Spiliopoulos and Altman2017) for other references and details.

Economists seem well advised to thoroughly map the speed–performance relationship across classes of strategic games, and to do so possibly–at least for certain research questions–also by way of cross-validation. Obviously, for strategic DM it is necessary to define both the class of game and the strategies that opponents are using. Determining which classes of games we can expect realized payoffs to be negatively or positively related to time pressure or RT is an open question that seems worth investigating.

There exists less work on the speed–performance relationship compared to the speed–accuracy relationship in strategic DM, as researchers have focused on variables in the action space such as cooperation rates, error rates, degree of sophistication, or equilibrium selection. For example, Rubinstein (Reference Rubinstein2013) finds a negative relationship between RT and (clearly defined) errors, but no relationship between RT and the consistency of behavior with EUT in individual DM tasks. However, an explicit discussion of whether RT is related to the actual performance of players is notably absent, albeit easily remedied. As discussed in “Appendix 2.2”, although a positive relationship between RT and the level of sophistication in reasoning seems intuitive and supported by the available evidence, in some games decreasing sophistication may actually lead to higher payoffs for all players of a game—recall the game in Table 5. Similarly, the findings by Guida and Devetag (Reference Di Guida and Devetag2013) suggest that focal points are increasingly chosen under time pressure—in games where these focal points may help players to coordinate, this may result in higher payoffs but not necessarily so. We are aware only of three economics studies, already mentioned earlier, Kocher and Sutter (Reference Kocher and Sutter2006) in individual DM and Arad and Rubinstein (Reference Arad and Rubinstein2012); Spiliopoulos et al. (Reference Spiliopoulos, Ortmann and Zhang2015) in strategic DM that explicitly relate performance to RT—more attention to the consequence space rather than the action space seems desirable.

If decision makers explicitly consider both performance and the necessary RT to achieve various levels of it, then an important unanswered question is how they choose the exact trade-off point (assuming a negative relationship exists between performance and RT)? Do they strategically choose this point conditional on task characteristics such as task difficulty, other concurrent cognitive load, types of time constraint, etc.? We present an indicative selection of hypotheses under the assumption that speed and performance are negatively related:

1. (a) Unconstrained Expected Utility maximization: The effect of RT is completely ignored, and subjects simply aim to maximize their expected utility.

2. (b) Unconstrained Expected Rate of Utility maximization: The objective function that is maximized is the expected utility per time unit.

3. (c) Performance satisficing: An aspiration level of performance (utility) is set, and RT is adjusted to keep performance constant.

4. (d) Time-constraint satisficing: A time-pressure constraint is externally set and is exhausted, thereby determining the performance.

We present some evidence from individual DM tasks for consideration. If the decision maker has the opportunity to repeatedly engage in the same task, then there exists a closed-form solution for the decision threshold that optimizes the reward rate for choice sets with two options (Bogacz et al. Reference Bogacz, Brown, Moehlis, Holmes and Cohen2006). Hawkins et al. (Reference Hawkins, Brown, Steyvers and Wagenmakers2012) present evidence that subjects engage in performance satisficing rather than maximization. Satisficing requires the specification of how high the performance criterion is set, and how this may depend on prior experiences. Balci et al. (Reference Balci, Simen, Niyogi, Saxe, Hughes, Holmes and Cohen2010) find that subjects facing a two-alternative forced-choice task exhibit a bias towards maximizing decision accuracy rather than the reward rate initially, i.e., adopted a suboptimal speed–accuracy trade-off. However, after repeated exposure to the task subjects’ behavior moved significantly towards the maximization of the reward rate. Young subjects are more likely to seek a balance between accuracy and speed than older subjects; the former tend to maximize reward rates, especially with experience and extensive feedback, whereas the latter maximize accuracy, i.e., minimize errors (Starns and Ratcliff Reference Starns and Ratcliff2010).

4.3 Explicit experimental control of RT

At first sight, experimental studies without any explicit exogenous constraint on RT may be immune from RT considerations. However, implicit time constraints may be inadvertently imposed by the experimenter or inferred by subjects. In consequence, studies that are otherwise similar may not be directly comparable if the implicit RT pressure varies across them. We conjecture that differences in implicit time pressure may drive some of the seemingly contradictory or non-replicable results in the literature if behavior is adaptive. Implicit time constraints may exist in many studies where RT is supposedly endogenous for the following reasons:

1. (a) Recruitment materials usually mention the expected length of the experiment, which is likely to cue subjects to the experimenter’s expectation of the time it takes to complete the task.

2. (b) Experimental instructions often include information that may influence the amount of time a subject decides to allocate to tasks. Strategic interaction of subjects, for example, might imply a weak-link matching scheme where the slowest player determines the time the session takes.

3. (c) For practical reasons–such as random matching for the determination of payoffs, or to avoid disturbances from subjects exiting early–subjects might have to wait for all participants to finish before they are allowed to collect payment and leave. Similarly, subjects may be delayed whilst waiting for other subjects to enter their choices before moving on to the next round of a repeated game.

4. (d) Subjects may be affected by many subtle cues in the wording of instructions. Benson (Reference Benson, Svenson and Maule1993) and Maule et al. (Reference Maule, Hockey and Bdzola2000) are cautionary tales of the effects of instructions on perceived time pressure—behavior was significantly influenced by different (loaded) instructions describing the same objective time limit.

Concluding, the loss of experimental control associated with implicit time constraints is a potential problem. Consequently, experiments with explicit exogenous time constraints may be significantly more comparable–and less noisy within a particular experiment–as they do not run the risk of participants subjectively inferring implicit time pressure. Alternatively, the adverse impact of implicit time constraints can be reduced without imposing explicit time constraints by permitting subjects to engage in an enjoyable activity, e.g., surf the internet if they have completed all their tasks early.Footnote ⁹ We would also encourage accounting for implicit time constraints in meta-analyses of studies–to the best of our knowledge this has not been done before.

4.4 Improved model selection, identification and parameter estimation

Model selection and identification, as we have argued earlier, can be sharpened by the use of RT. Models differ in their explanation of how an adaptive decision maker will react to time constraints and, ultimately, how observed behavior will change. As mentioned, differential RT predictions are a valuable aid in comparing competing models of behavior, e.g., Bergert and Nosofsky Reference Bergert and Nosofsky2007; Marewski and Melhorn Reference Marewski and Melhorn2011. Significant information can be gleaned from the relationship between RT and candidate variables of observed behavior, such as the error rate, realized choices, adherence to theoretical concepts such transitivity, equilibrium concepts etc. In short, models that make RT predictions in addition to choice are more structured, rendering them more falsifiable as both RT and/or choice data can refute them.

4.5 Classification of heterogeneous types

RT data can sharpen the classification of subject types, particularly in cases where two or more different decision strategies prescribe the same, or very similar, observed choices. The Allais-Paradox task in Rubinstein (Reference Rubinstein2013) is a case in point—patterns of choices differed significantly between subjects with low and high RT. Another example involves distinguishing between two types of learning: (a) incremental learning, where RT is expected to be smoothly decreasing with experience, and (b) eureka or epiphany learning, where RT should abruptly fall when subjects have an important insight that has a lasting impact on play (Dufwenberg et al. Reference Dufwenberg, Sundaram and Butler2010; McKinney and Huyck Reference McKinney and Van Huyck2013; Schotter and Trevino Reference Schotter and Trevino2014b).

4.6 RT as a proxy for other variables

RT may be used as a proxy for effort (e.g., Ofek et al. Reference Ofek, Yildiz and Haruvy2007; Wilcox Reference Wilcox1993) to examine the effects of variations in important variables such as experimental financial incentives, labor productivity incentives, and other general incentive structures. For example, RT can be used as a proxy for effort in the debate regarding financial incentives in experiments. A positive relationship between RT and the magnitude of financial incentives, ceteris paribus, would support the viewpoint that incentives matter. Alternatively, RT may also be a proxy for the strength of preference for an option—see the empirical evidence (e.g., Chabris et al. Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2008, Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2009) in favor of a negative relationship between RT and the difference in the strength of preference among available options. Such a relationship is also predicted by the sequential-sampling models discussed in Sect. 3.6.

5.1 Identification

The use of RT–above and beyond choices only–is beneficial for identification purposes, however it is not a panacea. Recall the extensive discussion in Sect. 2.1.2 about reverse-inference and identification in games where social preferences are important. The interaction of players in strategic DM provides an additional layer of complexity in the identification of processes, e.g., beliefs may play an important role. Consider social-dilemma games where RT constraints are implemented to examine their causal effect on the degree of cooperation or pro-social choices. If it is common knowledge that all players face time pressure in a treatment, then players may change their beliefs about how cooperative their opponents will be. Consequently, changes in social preferences and beliefs would be confounded, rendering the attribution to either impossible. These issues can be alleviated by careful choice of experimental design and implementation details, and the concurrent collection of other process measures such as information search and beliefs. For example, Merkel and Lohse (Reference Merkel and Lohse2016) explicitly collect players’ beliefs about their opponents’ likely behavior across different time treatments.

Identification may also be hampered in cases where RT constraints have a differential effect on other treatments, i.e., when RT interacts with the other treatments. For example, consider a public good experiment played under time pressure, where the treatments manipulate the number of players (few versus many). If increasing the number of players makes the game more complex or difficult, then a specific level of time pressure may have a greater relative impact in the treatment with more players. Such cases are easily remedied with an appropriate full factorial $2\times 2$ design where RT (endogenous versus time pressure) and the number of players (few versus many) are both manipulated, as the main effects of each factor and their interaction can then be recovered.

5.2 Irregular RT distributions, outliers and non-responses

The question of whether extreme values are regarded as outliers or not, and if so, how they are handled in the data analysis is of considerable importance and consequence—recall the debate in Rand et al. (Reference Rand, Greene and Nowak2012), Tinghög et al. (Reference Tinghög, Andersson, Bonn, Böttiger, Josephson, Lundgren, Västfjäll, Kirchler and Johannesson2013) reviewed earlier. Very short RTs may arise from fast-guessing, or very long RTs from subjects that are not exerting much effort and are bored.Footnote ¹⁰ Furthermore, the use of time pressure often leads to a number of non-responses if subjects do not answer on time. This leads to a selection problem if non-responses are correlated with subject characteristics. How these RT idiosyncrasies are treated is of paramount importance.

Consequently, endogenous RT distributions tend to be non-normal (left-truncated at zero), heavily skewed and often consist of extreme (low and high) values. This renders analyses using mean RT and ANOVA problematic. Whelan (Reference Whelan2010) recommends the use of the median and inter-quartile ranges for such cases, but notes that since true population medians are strongly underestimated for small sample sizes, median RTs should not be used to compare conditions with different numbers of observations. Another common solution is to appropriately transform the RT distribution into an approximate normal distribution, usually through the use of a log-transform. Outliers can have a significant impact on parametric summary statistics; possible solutions include using (a) robust non-parametric statistics, (b) Student t-distributions that allow for fat-tailed distributions (e.g., Spiliopoulos Reference Spiliopoulos2016), and (c) hierarchical modeling (see Sect. 6.3). We refer the reader to Van Zandt (Reference Van Zandt, Pashler and Wixted2002) and Whelan (Reference Whelan2010) for an extensive discussion of RT distribution modeling.

5.3 Heterogeneity

Experimental studies of individual DM and strategic DM generally find significant between-subject heterogeneity, e.g. in learning models (Cheung and Friedman Reference Cheung and Friedman1997; Daniel et al. Reference Daniel, Seale and Rapoport1998; Ho et al. Reference Ho, Camerer and Chong2007; Rapoport et al. Reference Rapoport, Daniel and Seale1998; Rutström and Wilcox Reference Rutström and Wilcox2009; Stahl Reference Stahl1996; Shachat and Swarthout Reference Shachat and Swarthout2004; Spiliopoulos Reference Spiliopoulos2012).

Between-subject heterogeneity can be attributed to two sources. Parameter heterogeneity arises from subjects using the same type of model, i.e. identical functional forms, but with individual-specific parameters. Model heterogeneity arises from subjects using completely different models, e.g. heuristics that cannot be nested within a more general model.

It is imperative to model heterogeneity directly as pooling estimation affects parameter recovery and model selection (Cabrales and Garcia-Fontes Reference Cabrales and Garcia-Fontes2000; Cohen et al. Reference Cohen, Sanborn and Shiffrin2008; Erev and Haruvy Reference Erev and Haruvy2005; Estes and Maddox Reference Estes and Maddox2005; Estes Reference Estes1956; Wilcox Reference Wilcox2006). Consequently, econometric models of RT should also allow for different RTs across subjects and heterogeneous effects of various RT determinants (e.g., Spiliopoulos Reference Spiliopoulos2016). Modeling both parameter and model heterogeneity requires the estimation of both finite mixtures and random-effects or hierarchical econometric models presented in Sect. 6.3. See our working paper (Spiliopoulos and Ortmann Reference Spiliopoulos and Ortmann2016) for an extended discussion.

5.4 RT measurement error

Experimentalists will typically delegate RT data collection automatically to whatever software package they use to set up the experiment, or in rarer cases may code their own experiment from scratch. While the accuracy of RT data collection has not been extensively examined in economics, more work has been done in psychology. We note that in psychology response times are often on the order of hundreds of milliseconds compared to seconds in economics. Therefore the accuracy of data collection for such finer graduations is not as important in experimental economics. Variations in RT estimates can be caused by any combination of hardware, software, and network latencies (for online experiments). The importance of these variations depends on their magnitude relative to the absolute RTs in the experiment and whether they are systematic or random, i.e., whether the noise can be expected to average out for a large enough number of observations. The general conclusion is that while absolute measures of RT may not be reliable across differences in these three sources of noise, relative measures of RT remain relatively faithful. Furthermore, the standard deviation of the induced noise is very low compared to the scale of RTs that experimental economics deals with.Footnote ¹¹

The most popular experimental economics software is z-Tree (Fischbacher Reference Fischbacher2007) and it includes the ability to internally measure response times. Perakakis et al. (Reference Perakakis, Guinot, Conde, Jaber-López, García-Gallego and Georgantzis2013) propose an alternative method based on photo-sensors that capture changes in the presentation of information on screen to counter-act the possible mis-calculations arising from the computer’s internal timing. Their photo-sensor system recorded response times that were on average 0.5 s lower than those recorded internally by z-Tree. While this difference may be problematic if the study attempts to link the timing of events with other biophysical markers such as heart-rate, it did not adversely affect the conclusions drawn from RT analyses across treatments. In economics, we will typically be interested in relative RTs and changes across treatments rather than absolute RTs; therefore, z-Tree should be accurate enough for the vast majority of applications. Seithe et al. (Reference Seithe, Morina and Glöckner2015) introduced a new software package (BoXS) specifically designed to capture process measures in strategic interactions including RT. They present evidence that this software’s RT accuracy is approximately $\pm 50$ ms (when presenting information for at least 100 ms), which is more than adequate for economic applications.

Concluding, for online experiments we can expect significant variations arising from both hardware and network sources, i.e., RT measurements will be relatively noisy. However, online experiments usually have a large number of subjects so that the noise often cancels out. For experiments in the laboratory, there does not seem to be a significant problem in the accuracy of RT measurements in the most common types of applications.

6.1 Procedural modeling

The existing RT literature is dominated by non-procedural (descriptive) rather than procedural modeling. We believe that in many instances procedural models are more useful than non-procedural models, as the former allow for comparative statics or quantitative predictions regarding the joint distribution of choice and RT. Such models can be falsified either by incorrect choice or RT predictions, thereby increasing the statistical power of experiments and associated hypothesis tests. Other process measure variables discussed in the next section could increase the power even further. Procedural models also provide a coherent framework within which to organize and define exactly how behavior adapts to time constraints—various mechanisms are discussed in “Appendix 2”.

6.2 Concurrent collection of other process measures

Few existing studies in experimental economics collect other process measures beyond choice and RT, some notable exceptions can be found in Table 3. Examples of other decision or process variables include information search using Mouselab or eye-tracking techniques (Crawford Reference Crawford, Caplin and Schotter2008), response dynamics (Koop and Johnson Reference Koop and Johnson2011), provisional choice dynamics (Agranov et al. Reference Agranov, Caplin and Tergiman2015), belief elicitation (Schotter and Trevino Reference Schotter and Trevino2014a), communication between players (Burchardi and Penczynski Reference Burchardi and Penczynski2014), verbal reports (Ericsson and Simon Reference Ericsson and Simon1980), physiological responses and neuroeconomic evidence (Camerer et al. Reference Camerer, Loewenstein and Prelec2005). However, it should be kept in mind that collecting these process measures is more disruptive than RT, and therefore their collection could influence behavior. The reader is referred to Glöckner (Reference Glöckner2009) and Glöckner and Bröder (Reference Glöckner and Bröder2011) for examples of procedural models predicting multiple measures: RT, information search, confidence judgments, and fixation duration. Other examples of the value added of process measures beyond choice data include Johnson et al. (Reference Johnson, Camerer, Sen and Rymon2002), Costa-Gomes et al. (Reference Costa-Gomes, Crawford and Broseta2001) and Spiliopoulos et al. (Reference Spiliopoulos, Ortmann and Zhang2015).

6.3 Hierarchical latent-class modeling

Hierarchical latent-class models can be an effective ally in capturing heterogeneity and outliers in the data. Estimating models per individual–whilst capturing individual heterogeneity–may not be the best line of attack due to the large number of free parameters and susceptibility to overfitting. Instead, we propose hierarchical latent-class models that capture both types of between-subjects heterogeneity with a reduction in free parameters (Conte and Hey Reference Conte and Hey2013; Lee and Webb Reference Lee and Webb2005; Scheibehenne et al. Reference Scheibehenne, Rieskamp and Wagenmakers2013; Spiliopoulos Reference Spiliopoulos2012; Spiliopoulos and Hertwig Reference Spiliopoulos and Hertwig2015). The latent classes capture model heterogeneity, whereas the hierarchical structure models parameter heterogeneity.Footnote ¹² The latent-class approach yields both prior and posterior (after updating the prior with the observed data) probabilities of subjects belonging to the specified latent classes. An additional bonus to such an econometric specification is that outliers can automatically be identified as belonging to one of the classes. Furthermore, latent-class models can also be used for within-subject heterogeneity (Davis-Stober and Brown Reference Davis-Stober and Brown2011; Shachat et al. Reference Shachat, Swarthout and Wei2015), e.g., the adaptive use of heuristics.

6.4 Cross-validation

Models of choice and RT ought to be subjected to the same strict demands that we impose on existing models. Specifically, procedural models should be competitively tested on out-of-sample data to ascertain their predictive ability or generalizability (Ahn et al. Reference Ahn, Busemeyer, Wagenmakers and Stout2008; Busemeyer and Wang Reference Busemeyer and Wang2000; Yechiam and Busemeyer Reference Yechiam and Busemeyer2008), such as in the context of a large-scale model tournament (Ert et al. Reference Ert, Erev and Roth2011; Spiliopoulos and Ortmann Reference Spiliopoulos and Ortmann2014). A commonly used technique is that of cross-validation, which requires that experimental data be partitioned into an estimation and a prediction dataset. Models are then fitted on the estimation dataset, and their performance is judged on the prediction dataset. This technique is effective in comparing models of varying flexibility as it avoids the problem of over-fitting by complex models. This is, in fact, the crux of one of the main tenets of the ecological-rationality program (Gigerenzer Reference Gigerenzer, Koehler and Harvey1988; Gigerenzer et al. Reference Gigerenzer and Todd1999, Reference Gigerenzer, Hertwig and Pachur2011; Gigerenzer and Selten Reference Gigerenzer and Selten2002)—for a critical review see Ortmann and Spiliopoulos (Reference Ortmann, Spiliopoulos and Altman2017). Simple heuristics can outperform complex decision models on unseen data exactly because they have less ability to overfit to noise or uncertainty in the environment.

Interestingly, cross-validation has not been extensively used in the RT literature with the exception of a few notable studies. Chabris et al. (Reference Chabris, Laibson, Morris, Schuldt and Taubinsky2009) and Schotter and Trevino (Reference Schotter and Trevino2014b) find that RT can be predictive of intertemporal choice and behavior in global games, respectively. Importantly, Rubinstein (Reference Rubinstein2013) and Rubinstein (Reference Rubinstein2016) find evidence that RT can be predictive of both individual DM and strategic DM choices across tasks. These results suggest that the degree of mutual information in choice and RT data is significant and can be exploited—we hope to see more analyses of this kind.

6.5 Experimental implementation

We have argued that we expect significant between-subject heterogeneity in strategic DM. In conjunction with the inherent stochasticity of choice and the practical limitations on the number of observations from empirical studies, this will make inference challenging. In some cases, an effective solution in terms of experimental implementation is to design within-subject treatments, thereby eliminating the between-subjects source of variability. However, the decision to use a between- or within-subject design implies a trade-off (Charness et al. Reference Charness, Gneezy and Kuhn2012); a within-subject design will limit the number of tasks that can be examined.

The RT literature using individual DM and strategic DM tasks has so far used rudimentary designs where behavior is observed as a function of different RT constraints, thereby revealing the RT-behavior profile. Existing experiments often use a t-treatments design, where t is the number of treatments with different time constraints and is usually equal to two or three. We will argue that we can further augment the usefulness of RT data by expanding experimental designs by, firstly, increasing the number and type of time treatments and, secondly, by varying another variable, namely the difficulty of the task. What we propose is a $t\times d$ factorial design ( $d=$ number of treatments with varying difficulty) where the manipulation of the time constraint reveals the speed–performance profile and manipulation of task difficulty reveals shifts of this profile. Task difficulty can be defined in various ways, which may differentially affect the speed–performance profile. Some examples of defining characteristics of difficulty for strategic DM are:

1. 1. The size of players’ action sets

2. 2. The number of players in a game

3. 3. The distance between a player’s (subjective) expected payoffs per action

4. 4. The uncertainty about opponents’ types

5. 5. The existence of imperfect information

6. 6. The lack of focal points

7. 7. The presence of cognitive load

Moving beyond the two-dimensional speed–performance trade-off to a three-dimensional speed–performance-difficulty profile, will generate more conflicting predictions between models, thereby aiding model comparison and identification. Also, note that an experimental design that explicitly manipulates difficulty addresses one of the critiques put forth by Krajbich et al. (Reference Krajbich, Bartling, Hare and Fehr2015); namely, that differences in task difficulty and the degree that behavior is instinctive may be confounded in existing studies that do not control for the former. We are aware that this line of attack might run into hostile budgetary defense lines.

Finally, researchers should consider when and how to disclose a time-constraint to subjects. If RT is exogenous, then specifics of the constraint can be announced in advance, or revealed to subjects during the decision process. For example, subjects may not be told how long they will have to make a decision but may be alerted in real-time when they must decide. Knowledge of the constraint may induce a subject to significantly change their decision process, e.g., under extreme time pressure they may switch to a simple heuristic, or in the dual-system approach, they may switch from the deliberative and slow System 2 to automatic and fast System 1 (Kahneman Reference Kahneman2011). If subjects do not know what the time limit will be, they may respond in the following two ways: (a) be more likely to use the same decision process and simply terminate when the time limit is announced and (b) make a provisional fast decision and then search for a better alternative, as in Agranov et al. (Reference Agranov, Caplin and Tergiman2015).