2.1. The model in GHW

To formalize the idea of the depth of rationality and the hypotheses we are going to test, we introduce the model and notations used in GHW. In their model, an n-person normal form game $\gamma \in \Gamma$ is represented by $(N, S, \{u_i\}_{i \in N})$, where $N = \{1,...,n\}$ denotes the set of players, $S = S_1 \times \cdots \times S_n = \Pi_{i = 1}^n S_i$ denotes the strategy sets, and $u_i : S \to \mathrm{R}$ for $i \in N$ denotes the payoff functions.

Player i’s strategic ability is modeled by two functions $(c_i, k_i)$. Let T be the set of environmental parameters, which captures the information a player observes about their opponents’ cognitive abilities. The function $c_i : \Gamma \rightarrow \mathbb{N}_0$ represents i’s capacity for game γ, and the function $k_i : \Gamma \times T \rightarrow \mathbb{N}_0$ represents i’s (realized) level for game γ. A player’s level for a game is bounded by their capacity, so $k_i(\gamma, \tau_i) \leq c_i(\gamma)$ for all γ, $\tau_i \in T$, and $i \in N$. The goal of our experiment is to measure $c_i(\gamma)$ and to test if $c_i(\gamma)$ (or $k_i(\gamma, \tau_i)$, after controlling for τ_i) exhibits any stability across different games (see Section 5.2 for further discussion).

2.2. Higher-order rationality

To characterize a player’s behavior in the games, we define kth-order rationality (Bernheim, Reference Bernheim1984; Lim & Xiong, Reference Lim and Xiong2016; Pearce, Reference Pearce1984) in the following way. Let $R_i^k(\gamma)$ be the set of strategies that survive k rounds of iterated elimination of strictly dominated strategies (IEDS) for player i. Namely, a strategy s_i is in $R_i^1(\gamma)$ if s_i is a best response to some arbitrary s _−i, and s_i is in $R_i^{k'}(\gamma)$ if s_i is a best response to some $s_{-i} \in R_{-i}^{{k'} - 1}(\gamma)$ for $k' \gt 1$. We say that a player i exhibits kth-order rationality in γ if and only if i always plays a strategy in $R_i^k(\gamma)$. Note that given any game $\gamma \in \Gamma$, $R_i^{k+1}(\gamma) \subset R_i^k(\gamma)$ for all $k \in \mathbb{N}_0$. In other words, a player exhibiting kth-order rationality also exhibits jth-order rationality for all $j \leq k$.

We characterize boundedly rational behavior using higher-order rationality rather than the standard level-k model employed by GHW to avoid the ad hoc assumption regarding level-0 players.Footnote ¹³ In the standard level-k model, pinning down a level-k player’s strategy requires an assumption about the level-0 strategy. However, studies have reported variations in level-0 actions and level-0 beliefs across individuals (Burchardi & Penczynski, Reference Burchardi and Penczynski2014; Chen et al., Reference Chen, Huang and Wang2018). Consequently, an individual’s identified level of reasoning can be sensitive to the structural assumptions of the level-k model. Alternatively, the higher-order rationality approach avoids structural assumptions about (beliefs regarding) non-rational players’ behavior, thereby providing a structure-free method for empirically categorizing individuals into distinct reasoning levels.

3.1. Ring games

A four-player ring game is a simultaneous game characterized by four $3 \times 3$ payoff matrices. Figure 1 summarizes the structures of the two ring games, G1 and G2, used in our experiment. As shown in Figure 1, each player $i \in \{1, 2, 3, 4\}$ simultaneously chooses an action $a_i \in \{a, b, c\}$. Player 4 and Player 1’s choices determine Player 4’s payoff, and Player k and Player (k + 1)’s choices determine Player k’s payoff for $k \in \{1, 2, 3\}$.

Figure 1.

The ring games. The Nash equilibrium is highlighted with colored borders, and the secure actions are underscored

The payoff matrices for Player 1, 2, and 3 are identical in G1 and G2. However, the matrix for Player 4 differs between G1 and G2, with the rows corresponding to Player 4’s actions (a, b, c) interchanged, leading to different best replies in the subsequent matrices.

Specifically, Player 4 has a strictly dominant strategy in each ring game: b in G1 and c in G2. Given the payoff structure, a (first-order) rational individual will always choose b in G1 and c in G2 when acting as Player 4. By eliminating dominated strategies, an individual exhibiting second-order rationality will always choose c in G1 and b in G2 when acting as Player 3. Continuing this process iteratively, an individual exhibiting third-order rationality will always choose c in G1 and a in G2 when acting as Player 2, and an individual exhibiting fourth-order rationality will always choose b in G1 and c in G2 when acting as Player 1. Thus, the unique Nash equilibrium of G1 is Player 1, 2, 3, and 4 choosing b, c, c, and b, respectively, and for G2, Player 1, 2, 3, and 4 choosing c, a, b, and c, as highlighted in Figure 1.

Note that the payoff structures in our ring games are identical to those in Kneeland Reference Kneeland(2015), except that rows a and b are swapped for Player 4 in G1. This modification ensures that our equilibrium-predicted actions do not coincide with the secure actions (or max-min actions) in both G1 and G2, which maximize the total payoff sum over the opponents’ possible actions, potentially encouraging subjects to choose the equilibrium strategy based on non-payoff-maximizing motives.Footnote ¹⁴ Adopting the same payoff structure as Kneeland’s design facilitates comparability between our results and hers.

3.2. Guessing games

In our experiment, the guessing game is a simultaneous two-player game parameterized by a constant $p \in (0, 1)$. We use $p = 1/3$, $1/2$ and $2/3$ in our experiment. Each player i simultaneously chooses a positive integer s_i between 1 and 100. Player i’s payoff strictly decreases in the difference between the number chosen by i, s_i, and the number chosen by i’s opponent multiplied by a constant p, $p s_{-i}$. Specifically, player i’s payoff is equal to $0.2 \times (100 - |s_i - p s_{-i}|)$. Thus, a payoff-maximizing player’s objective is to make a guess that matches their opponent’s guess times p. Note that, given p < 1, any action (integer) greater than or equal to $\lfloor 100 p + 0.5 \rfloor + 1$ is strictly dominated by $ \lfloor 100 p + 0.5\rfloor $ since $|\lfloor 100 p + 0.5\rfloor - p s_{-i}| \lt |s'_i - p s_{-i}|$ for all $s_{-i} \in \{1,...,100\}$ and $s'_i \in \{\lfloor 100 p + 0.5\rfloor + 1,...,100\}$.Footnote ¹⁵

Given the payoff logic above, a (first-order) rational individual will choose an integer between 1 and $K_1 \equiv \lfloor 100 p + 0.5\rfloor $, since any action greater than K ₁ is strictly dominated by K ₁. A second-order rational individual will believe the other player is first-order rational and choose a positive integer between 1 and $\lfloor K_1 p + 0.5\rfloor $, and so on. The unique equilibrium of the two-person guessing game is thus both players choosing 1.

5.1. Treatments

We design a laboratory experiment to measure subjects’ higher-order rationality. In the main part of the experiment, subjects first play the ring games, followed by the guessing games, in two different scenarios: the Robot Treatment and the History Treatment. Using a within-subject design, we alternate the order of the two scenarios (RH Order and HR Order) across sessions to balance out potential spillover effects from one treatment to another.

In each scenario, each subject first plays the two four-player, three-action ring games (G1 and G2 in Figure 1) in each position in each game once (for a total of eight rounds). Each subject is, in addition, assigned a neutral label (Member A, B, C, or D) before the ring games start. The label is only used for the explanation of an opponent’s strategy in the History Treatment and does not reflect player position. To facilitate the cross-subject comparison, all the subjects play the games in the following fixed order: P1 in G1, P2 in G1, P3 in G1, P4 in G1, P1 in G2, P2 in G2, P3 in G2, and P4 in G2.Footnote ¹⁸ The order of payoff matrices is also fixed, with a subject’s own payoff matrix being fixed at the leftmost side.Footnote ¹⁹

In the Robot Treatment, the subjects play against fully rational computer players. Specifically, each subject in each round is matched with three robot players who only select the strategies that survive iterated dominance elimination (i.e., the equilibrium strategy). We inform the subjects of the presence of robot players that exhibit third-order rationality.Footnote ²⁰ The instructions for the robot strategy are as follows:Footnote ²¹

The first line of a robot’s decision rule (“The computers aim to...”) implies that a robot never plays strictly dominated strategies and thus exhibits first-order rationality. The second line (along with the first line) indicates that a robot holds the belief that other players are (first-order) rational and best responds to such belief, which implies a robot’s second-order rationality. The third line (along with the first and second lines) implies that, applying the same logic, a robot exhibits third-order rationality.

In the History Treatment, the subjects play against the data drawn from their decisions in the previous scenario. Specifically, in each round, a subject is matched with three programmed players who adopt actions chosen in the Robot Treatment by three other subjects.Footnote ²² Every subject is informed that other human participants’ payoffs would not be affected by their choices at this stage. By having the subjects play against past decision data, we can exclude the potential confounding effect of other-regarding preferences on individual actions.

After the ring games, the subjects play the two-person guessing games (in the order of $p = 2/3$, $1/3$, $1/2$) in both the Robot Treatment and the History Treatment. Instead of being matched with three opponents, a subject is matched with only one player in the guessing games. The instructions for the guessing games in both treatments are revised accordingly.

5.2. Hypotheses

The Robot Treatment is designed to convince subjects that the computer opponents they face are the most sophisticated players they could encounter. Consequently, if our Robot Treatment is effectively implemented, it should prompt subjects to employ a strategy at the highest achievable level k, i.e., $k_i(\gamma, \tau_i = Robot) = c_i(\gamma)$ for all γ and i. (Recall that k_i and c_i denote subject i’s realized level and capacity, respectively.) This observation gives rise to the first hypothesis we aim to evaluate.

In other words, we test whether subjects’ rationality levels against robots capture individual strategic reasoning capacity. The corresponding analysis is presented in Section 6.2.

If Hypothesis 1 holds, then we can evaluate several possible restrictions on c_i by forming hypotheses on $k_i(\gamma, Robot)$. In evaluating Hypothesis 2, we examine whether there are stable patterns in (revealed) individual reasoning depth across games.

This hypothesis imposes the strictest requirement on stability by testing if a player’s rationality level remains constant across games. In other words, it assesses whether playing against robots provides a measure of one’s absolute depth of reasoning. The corresponding analysis is presented in Section 6.3.

In addition to these two hypotheses, Online Appendix B explores two less stringent stability requirements, such as the stability of relative rankings between players’ rationality levels (Hypothesis 3) or the consistency of game difficulty in terms of revealed rationality across players (Hypothesis 4).Footnote ²³

5.3. Cognitive tests

Apart from the ring games and the guessing games, subjects also complete three cognitive tests to measure different aspects of their cognitive ability and strategic reasoning:

1. (1) the Cognitive Reflection Test (CRT),

2. (2) the Wechsler Digit Span Test, and

3. (3) the farsightedness task.

The CRT, proposed by Frederick Reference Frederick(2005), is designed to evaluate the ability to reflect on intuitive answers. This test contains three questions that often trigger intuitive but incorrect answers. Performance on this test has been found to be correlated with strategic abilities. For instance, GHW report that the subjects’ CRT scores have moderate predictive power on their expected earnings and level-k types.

The second test is the Wechsler Digit Span Test (Wechsler, Reference Wechsler1939), which is designed to test short-term memory. In our experiment, this test contains eleven rounds. In each round, a subject needs to repeat a sequence of digits displayed on the screen at the rate of one digit every second. The maximum length of the digit sequence a subject can memorize reflects the subject’s short-term memory capacity.Footnote ²⁴ Devetag and Warglien Reference Devetag and Warglien(2003) find a positive correlation between individual short-term memory and strategic ability.

Lastly, the farsightedness task, developed by Bone et al. Reference Bone, Hey and Suckling(2009), is an individual task to measure a subject’s ability to do backward induction, or to anticipate their own future action and make the best choice accordingly. Specifically, it is a sequential task that involves two sets of decision nodes and two sets of chance nodes (see the decision tree in Figure 2). The first and third sets of nodes are the decision nodes where a decision maker is going to take an action (up or down). The second and fourth sets of nodes are the chance nodes where the decision maker is going to be randomly assigned an action (with equal probability).

Figure 2.

The farsightedness task in Bone et al. Reference Bone, Hey and Suckling(2009)

Notice that there is one dominant action, in the sense of first-order stochastic dominance, at each of the third set of nodes (i.e., the second set of decision nodes). Anticipating the dominant actions at the second set of decision nodes, the decision maker also has a dominant action (down) at the first node. However, if a payoff maximizer lacks farsightedness and anticipates that each payoff will be reached with equal chance, then the dominated action (up) at the first node will become the dominant option from this decision maker’s perspective. Therefore, a farsighted payoff-maximizer is expected to choose down, but a myopic one is expected to choose up, at the first move (and choose the dominant actions at the second moves). Consequently, we can use their choice at the first move to evaluate the correlation between one’s farsightedness and rationality level.

5.4. Laboratory implementation

We conducted 41 sessions between August 31, 2020 and January 28, 2021 at the Taiwan Social Sciences Experiment Laboratory (TASSEL) in National Taiwan University (NTU). The experiment was programmed with the software zTree (Fischbacher, Reference Fischbacher2007) and instructed in Chinese. A total of 299 NTU students participated in the experiment, all recruited through ORSEE (Greiner, Reference Greiner2015). In our experiment, 136 subjects played the Robot Treatment before the History Treatment in both families of games (RH Order), while 157 subjects played the History Treatment first (HR Order).Footnote ²⁵

Each experimental session lasted about 140 minutes, and the protocol is summarized in Figure 3. At the beginning of the experiment, subjects first completed the CRT and the Wechsler Digit Span Test. After these tasks, subjects played the ring games in both the Robot Treatment and the History Treatment, followed by the guessing games in both treatments. In the final section of the experiment, subjects were asked to complete the farsightedness task. The experimental subjects did not receive any feedback about the outcomes of their choices until the end of the experiment.

Figure 3.

Experiment protocol

There was a 180-second time limit on every subject’s decisions in the ring games, guessing games, and farsightedness task. A subject who did not confirm their choice within 180 seconds would have earned zero payoff for that round; however, no subjects exceeded the time limit.Footnote ²⁶

The subjects were paid based on the payoffs (in ESC, Experimental Standard Currency) they received throughout the experiment. In addition to the payoff in the farsightedness task, one round in the ring games and one round in the guessing games were randomly chosen for payment. A subject also got three ESC for each correct answer in the CRT, and one ESC for each correct answer in the Digit Span Test. Including a show-up fee of NT$200 (New Taiwan dollars; approximately $7 in USD in 2020), the earnings in the experiment ranged between NT$303 and NT$554, with an average of NT$430.Footnote ²⁷

6.1. Data description

Before delving into the main results, we begin by summarizing the subjects’ choice frequencies in the ring games (Figure 4) and guessing games (Figure 5). Figure 4 reports the subjects’ choice frequencies in the two ring games (G1 and G2, see Figure 1) at each player position. From the figure, we can first observe that in both treatments, over 97% of subjects choose the equilibrium strategy (b, c) at P4 (χ ² test p-value = 0.252). This suggests that subjects are able to recognize strict dominance in the ring games.

Figure 4.

Ring game choice frequency at each position. The first and the second arguments represent the actions of G1 and G2

Figure 5.

Cumulative distribution of guesses

Second, at each player position except P4, the significance of χ ² tests suggests that subjects’ behavior is responsive to the treatments (P1: χ ² test p-value = 0.020; P2: χ ² test p-value < 0.001; P3: χ ² test p-value = 0.088). Moreover, the Robot Treatment shows a 10 to 15 percentage point higher frequency of subjects choosing the equilibrium strategy (b, c) at P1, (c, a) at P2, and (c, b) at P3 compared to the History Treatment, indicating that the Robot Treatment effectively prompts subjects to display higher rationality levels.

Third, at each player position except P4, a notable proportion of subjects choose the secure action that maximizes the minimum possible payoff among the three available actions (a at P1, b at P2, a at P3). As shown in Figure 4, a high proportion of subjects opt for secure actions as an alternative to equilibrium actions. Moreover, except at P4, the proportion of secure actions is higher in earlier positions. At P1, 38% of subjects in the Robot Treatment and 44% in the History Treatment choose the secure actions (a, a). This tendency is more pronounced in the History Treatment, where secure actions are chosen more frequently than equilibrium actions at P1 and P2.Footnote ²⁸ This evidence suggests that when players have uncertainty about their opponents’ reasoning and strategic behavior, some players may opt for a non-equilibrium strategy to avoid the possibility of experiencing the worst possible payoff.Footnote ²⁹ A detailed analysis of the behavior of choosing secure actions is provided in Section 6.4.

Figure 5 presents the cumulative distribution of subjects’ guesses across the three guessing games. We observe significant differences in the distributions between the two treatments, regardless of the value of p ( $p=2/3$: KS test p-value = 0.001; $p=1/3$: KS test p-value < 0.001; $p=1/2$: KS test p-value = 0.001). Furthermore, in the Robot Treatment, there is a 13 to 16 percentage point higher proportion of subjects making the equilibrium guess (i.e., choosing 1) across all three guessing games compared to the History Treatment. This difference leads to first order stochastic dominance of the cumulative distribution of guesses in the Robot Treatment over that in the History Treatment, indicating a higher rationality level among subjects in the Robot Treatment for the guessing games.

Furthermore, these distributional differences are driven by variations in equilibrium choices (i.e., 1). After excluding the equilibrium choice of 1, the cumulative distributions between the two treatments are not significantly different for any value of p ( $p=2/3$: KS test p-value = 0.218; $p=1/3$: KS test p-value = 0.704; $p=1/2$: KS test p-value = 0.129). This result further confirms that subjects prompted to perform at their maximum depth of reasoning when facing robots are the primary driving force behind the deeper reasoning observed in the Robot Treatment. In the next section, we will describe our approach for classifying individual rationality levels and perform statistical tests to assess whether subjects demonstrate higher rationality levels when playing against robots.

6.2. Rationality level classification

We adopt the revealed rationality approach to classify subjects into different rationality levels. Specifically, let $s_i = (s^{\gamma}_i)$ be the vector which collects player i’s actions in each family of games γ, where $\gamma \in \{\mbox{Ring}, \mbox{Guessing}\}$. In the ring games, we classify subjects based on the classification rule shown in Table 1. In both the Robot Treatment and the History Treatment, if a subject’s action profile matches one of the predicted action profiles of type R0–R4 exactly, then the subject is assigned that level. Therefore, we can obtain each subject’s rationality level in the Robot Treatment and the History Treatment, which are denoted as $k_i(\mbox{Ring}, \mbox{Robot})$ and $k_i(\mbox{Ring}, \mbox{History})$, respectively.

Similarly, for the guessing games, we classify subjects based on the rule outlined in Table 2. In both treatments, each subject makes three guesses (at $p=2/3$, $1/3$, and $1/2$). If a subject is categorized into different levels in different guessing games, we assign the subject the lower level. Thus, we can obtain the levels in both treatments, denoted as $k_i(\mbox{Guessing}, \mbox{Robot})$ and $k_i(\mbox{Guessing}, \mbox{History})$, respectively. Following this rationale, we construct the overall distribution of individual rationality levels for each treatment by assigning each subject the lower level they exhibit across the two classes of games, i.e., $k_i(\tau_i) = \min\{k_i(\mbox{Ring}, \tau_i), k_i(\mbox{Guessing}, \tau_i)\}$.Footnote ³⁰

Figure 6 reports the overall distribution of rationality levels for the Robot and History Treatments. As shown in the top figure, subjects tend to be classified into higher levels when playing against robots. There are more R1 and R2 players but fewer R3 and R4 players in the History Treatment than in the Robot Treatment. To examine if a subject’s reasoning depth is bounded by their revealed rationality level in the Robot Treatment (Hypothesis 1), at the aggregate level, we conduct the two-sample Kolmogorov-Smirnov test to compare the distributions of rationality levels in the two treatments. If Hypothesis 1 holds, we should observe either no difference in the two distributions or the distribution in the Robot Treatment dominating the distribution in the History Treatment. Our results show that the underlying distribution of individual rationality levels in the Robot Treatment stochastically dominates the one in the History Treatment (KS test p-value = 0.015), and thus provide supporting evidence for Hypothesis 1.

Figure 6.

Distributions of rationality levels. The top figure is the overall distribution of rationality levels. The bottom figures are the distributions of rationality levels in ring games (Left) and guessing games (Right)

This result is robust across different types of games. As shown in the bottom panels of Figure 6, a similar pattern of first order stochastic dominance is observed regardless of whether rationality levels are classified based on behavior in the ring games or the guessing games (Ring game: KS test p-value = 0.015; Guessing game: KS test p-value = 0.001).

Moreover, our within-subject design gives us paired data of individual rationality levels across treatments, which gives us another way to test Hypothesis 1. Overall, 85 percent of subjects (249/293) exhibit (weakly) higher rationality levels in the Robot Treatment than in the History Treatment. We further conduct the Wilcoxon signed-rank test to examine whether the subjects’ rationality levels in the Robot Treatment are significantly greater than the History Treatment. Consistent with Hypothesis 1, we observe higher rationality levels in the Robot Treatment (Wilcoxon test p-value < 0.0001). Therefore, we conclude that the rationality levels in the Robot Treatment can serve as a proxy of individual strategic reasoning capacity. In Online Appendix B, we separate the data by different games, finding a robust pattern in both.

It is noteworthy that, contrary to previous findings, we observe very few R0 players in the ring games in both treatments (Robot: 0.68%; History: 2.04%).Footnote ³¹ In our experiment, the subjects do not interact with each other in both treatments. Thus, our observation suggests that, when human interactions exist, social preferences may play some roles in a ring game and lead to (seemingly) irrational behavior, though we cannot exclude the possibility that this discrepancy in the prevalence of R0 players is due to different samples.

Yet in the guessing games, our classification results display a typical distribution pattern of estimated levels as documented in Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006) and GHW. First, the modal type is R1 (Level 1), with more than 35 % of subjects classified as R1 players in both treatments (Robot: 38.23%; History: 47.78%; Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006): 48.86%; GHW: 50.00%). In particular, the proportion of R1 players reported in the History treatment of our guessing games is very close to the proportion of level-1 players reported in Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006) and GHW. Second, R3 (Level 3) represents the least frequently observed category among the rational types (i.e., R1–R4), with fewer than 10 percent of subjects classified as R3 players in both treatments, a proportion that aligns with findings in the literature. (Robot: 6.14%; History: 4.10%; Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006): 3.41%; GHW: 10.34%). Third, the percentage of R4 players in our History Treatment falls within the range of equilibrium-type player proportions reported in Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006) and GHW (Robot: 30.03%; History: 16.04%; Costa-Gomes and Crawford Reference Costa-Gomes and Crawford(2006): 15.91%; GHW: 27.59%). Noticeably, in our Robot Treatment, we observe a relatively high frequency of R4 players compared to previous literature.Footnote ³² This finding underscores the significant impact of non-equilibrium belief about opponents on non-equilibrium behavior.

While our subjects’ revealed rationality levels are comparatively higher when playing against robots, most do not exhibit more than two steps of reasoning. In the Robot Treatment, around 70 percent of subjects still show an overall rationality level below the third order. This result supports the long-standing idea in the level-k literature: humans have a relatively low cognitive ceiling for strategic thinking, often below level four.

6.3. Consistency of rationality levels across games

In this section, we evaluate whether controlling for beliefs about the opponent’s depth of reasoning leads individuals to reveal consistent rationality levels across games. There are different notions of consistency. As a first exercise, we assess the strictest form of consistency: an individual reveals constant rationality levels across games (Hypothesis 2).

To examine this hypothesis, we generate a Markov transition matrix of rationality levels between the ring games and the guessing games in the Robot Treatment. Table 3 reports the frequency with which an individual moves from each rationality level in the ring games to each rationality level in the guessing games in the Robot Treatment. If the observed individual rationality level is the same across games, then every diagonal entry of each transition matrix in Table 3 will be 100%. Alternatively, if subjects’ rationality levels in the ring games and guessing games are uncorrelated, every row in a transition matrix will be the same and equals the overall distribution in the guessing games.

Table 3.

Markov transition for rationality levels in the robot treatment

The number of observations is reported in parentheses.

The most frequently observed transitions are highlighted in bold.

The transition matrix shows that most R1 and R4 players in the ring games remain as the same level in the guessing games. Most R2 ring game players, however, only exhibit first-order rationality in the guessing games. We do not observe any subjects consistently classified into R3 for both ring and guessing games, possibly because we have relatively low numbers of R3 subjects in either games. Overall, there is a relatively high proportion of subjects (38.23%) that exhibit the same rationality level across games.Footnote ³³ Also, note that we observe a relatively high proportion (52/293 = 17.74%) of subjects classified as R4 players in both games,Footnote ³⁴ suggesting that subjects in our experiment understand the instruction for robots’ decision rules and try to play the best response to such rules.

Moreover, to formally test whether the rationality levels from the ring games and guessing games are independent, we conduct a Monte Carlo simulation in Online Appendix B, following the approach of GHW. Contrary to their findings, we observe that the null hypothesis of independence in rationality levels across games is rejected in the Robot Treatment but not in the History Treatment. Furthermore, we observe that rationality levels in the Robot Treatment are more stable across games than in the History Treatment under alternative notions of consistency compared to Hypothesis 2. These findings provide supportive evidence for our belief control approach.

Although the null hypothesis of independence in rationality levels across games is statistically rejected in the Robot Treatment, the Markov transition matrix reveals that R2 and R3 types appear to be relatively unstable across games. Both R2 and R3 types identified in the ring games tend to “cluster” at R1 in the guessing games, a pattern that is also evident in the History Treatment.

There are two possible explanations for this pattern. First, the revealed rationality approach identifies an individual’s level by the maximum order of rationalizability of their strategy without imposing additional structural assumptions. Since a kth-order rational player’s strategy can also be rationalized by lower-order rationality (see Section 2.2 and Footnote 13), this approach could mechanically “inflate” the identified levels compared to the standard level-k model, which imposes specific structures on beliefs. More importantly, this inflation may be sensitive to the structure of the games, suggesting that the inconsistency of R2 and R3 types could result from this mechanical effect. Second, rather than engaging in higher-order rationality inferences, R2 and R3 types may simply be “avoiding dominated strategies.” To test these hypotheses, in Section 7.3, we compare the revealed rationality approach with the standard level-k model and a diagnostic model, Random, Avoiding Dominated, and Equilibrium (RADE), and report the corresponding results.

6.4. Cognitive tests, secure actions and strategic sophistication

6.4.1. Cognitive tests and strategic sophistication

In addition to analyzing consistency across games, this section investigates whether an individual’s performance in other cognitive tests can predict their strategic reasoning ability. To explore this, we regress subjects’ revealed rationality levels on their CRT scores, short-term memory task scores, and farsightedness task scores.

The definitions of the independent variables are as follows: CRT Score (ranging from 0 to 3) represents the number of correct answers a subject gets in the three CRT questions. Memory Score (ranging from 0 to 11) is defined as the number of correct answers a subject provides before making the first mistake. Farsightedness is an indicator variable that equals one if a subject chooses to go down at the first move in the farsightedness task (see Section 5.3). Lastly, the dependent variables is the individual revealed rationality levels (ranging from 0 to 4) within each class of games and treatment.

Table 4 presents the OLS regression results for revealed rationality levels. The analysis shows a positive correlation between a subject’s CRT performance and their revealed rationality levels across all game types and treatments. Overall, the CRT score is a stronger predictor of rationality levels in the guessing games and the Robot Treatment. In the Robot Treatment, each additional correct answer on the CRT is associated with an average increase of 0.298 (p-value < 0.001) in revealed rationality levels for the ring games, and 0.566 (p-value < 0.001) for the guessing games. In comparison, in the History Treatment, each additional correct answer on the CRT corresponds to a smaller average increase of 0.239 (p-value = 0.002) for the ring games and 0.461 (p-value < 0.001) for the guessing games—approximately 80% of the effect size observed in the Robot Treatment.

Table 4.

OLS regressions for revealed rationality levels

The standard errors are clustered at the session level.

* Significance level: p < 0.05,

** p < 0.01,

*** p < 0.001.

In contrast to the previous finding, our results show no significant correlation between short-term memory and strategic sophistication. The coefficient estimates of Memory Score are all below 0.03, and all the corresponding p-values are above 0.3. Notably, these findings are in line with those of GHW, who also observe that CRT scores hold some predictive power over subjects’ strategic thinking types, whereas short-term memory capacity does not.

Lastly, an individual’s performance on the farsightedness task also significantly predicts their revealed rationality level across all game types and treatments. Similar to the CRT score, we observe a stronger correlation between farsightedness and individual rationality levels in the guessing games and in the Robot Treatment. In the Robot Treatment, a farsighted subject’s revealed rationality level is, on average, 0.569 (p-value = 0.002) and 0.842 (p-value < 0.001) levels higher than that of a myopic subject when playing ring games and guessing games, respectively. Comparatively, in the History Treatment, a farsighted subject’s revealed rationality level is, on average, 0.339 (p-value = 0.050) and 0.631 (p-value < 0.001) levels higher than that of a myopic subject when playing ring games and guessing games, respectively. Both of these coefficients are smaller in size compared to the estimates reported for the Robot Treatment. These results indicate a strong correlation between an important strategic thinking skill in a dynamic game—backward induction ability—and the revealed rationality levels in one-shot interactions.

6.4.2. Secure actions in the ring gamesFootnote ³⁵

Another feature of our modified ring games is that, except at P4, the secure actions differ from the equilibrium actions. This distinction allows us to explore whether players opt for secure actions when they have reached their rationality capacity.

In this section, we analyze the behavior of choosing secure actions by decomposing the revealed rationality levels identified from the ring games into secure and non-secure types. Specifically, for any rationality level k, a player is classified as Rk-Secure (or Rk-S) if they exhibit rationality level k and choose secure actions in earlier positions.Footnote ³⁶ Conversely, a player is classified as Rk-Non-Secure (or Rk-NS) if they exhibit rationality level k but do not choose secure actions in earlier positions. Based on this classification, players are divided into eight possible types: R0, R1-S, R1-NS, R2-S, R2-NS, R3-S, R3-NS, and R4. The distributions for the Robot and History Treatments are shown in Figure 7.

Figure 7.

Distribution of rational levels with secure actions in the ring games

From this figure, we can first observe that there are more secure-type players in the Robot Treatment than in the History Treatment. Among the R1 players, 25.2% are classified as R1-S in the Robot Treatment, while 19.7% are classified as R1-S in the History Treatment. Furthermore, among the R2 players, 42.5% are R2-S in the Robot Treatment, compared to 24.4% in the History Treatment. This result suggests that instead of betting on risky actions, players are more likely to choose a secure action when facing robot players rather than human players.Footnote ³⁷

Given this result, we can further explore the behavior of these secure-type players in the guessing games. This is an interesting exercise because there is no secure action in the guessing games, and one might reasonably hypothesize that secure-type players will exhibit higher rationality levels, as their choice of secure actions in the ring games suggests a degree of deliberate, thoughtful decision-making. However, from the Markov transition matrices in Online Appendix A, we find that, in neither the Robot Treatment nor the History Treatment, for any $k \in \{1,2,3\}$, are the transition probabilities between Rk-S and Rk-NS significantly different.Footnote ³⁸ This suggests that the existence of secure actions is indeed a unique feature of our modified ring games. Given any rationality level, choosing secure actions when reaching their rationality capacity does not imply significantly different behavior in the guessing games, where secure actions are absent.

7.1. Validity of robot treatment

The validity of the Robot Treatment in eliciting individual strategic thinking capacity relies on our Hypothesis 1 (i.e., that individual rationality levels are higher in the Robot Treatment). An implicit assumption behind this hypothesis is that a subject has an incentive to play at the highest level they can achieve when encountering fully rational opponents playing at their maximum reasoning level. This statement is trivially true for equilibrium-type subjects, as they know their opponents will play the equilibrium strategy and are able to best respond to it. However, for a bounded rational player, this may or may not hold.

If we assume that an iterative reasoning model describes an individual’s actual decision-making process, two scenarios explain why a player might only perform k steps of iterative reasoning. First, they may incorrectly believe that other players can exhibit (at most) $(k - 1)$th-order of rationality and best respond to that belief. Second, they may correctly perceive that other players can exhibit (at least) kth-order of rationality but fail to best respond to it. While our statement regarding incentive compatibility holds in the first case, it becomes unclear how a bounded rational player would respond when facing opponents with rationality levels above k.

Nevertheless, this scenario does not pose a problem under the identification strategy of the revealed rationality approach. Notice that a player exhibiting kth-order rationality would also exhibit mth-order rationality for all $m \leq k$. Thus, a level-k player i who perceives other players as exhibiting at least kth-order rationality also perceives them as exhibiting $(k - 1)$th-order rationality. That is, the player knows that their robot opponents’ strategies will survive k − 1 rounds of IEDS. Therefore, a payoff-maximizing player i capable of k steps of iterative reasoning will choose a strategy in $R_i^k(\cdot)$, which contains all undominated strategies after k − 1 rounds of IEDS. Under the revealed rationality approach, player i will then be classified as a kth-order revealed-rational player.

Indeed, whether subjects follow the hypothesis and exhibit higher rationality levels when facing fully rational robots is an empirical question. In our setting, we have provided supportive evidence for Hypothesis 1 in Section 6.2. However, it remains an open question whether this increased rationality consistently emerges when individuals encounter robot players in other strategic environments. For instance, in complex games (e.g., Go), individuals might lower their effort and opt for random actions if they perceive highly intelligent robot opponents as unbeatable. Accordingly, exploring how information about robot opponents may influence people’s strategic responses in various settings could deepen our understanding of human-robot interactions, especially as AI increasingly shapes human decision-making processes.

7.2. Choice of robot strategy instruction

To elicit individual strategic thinking capacity, our Robot Treatment instructions inform subjects that the computer player is third-order rational (i.e., the computer is rational, knows its opponent is rational, and knows its opponent knows it is rational) to control for their beliefs about the sophisticated robot. Previous experimental studies have used different approaches to inform subjects about the strategy of a fully rational, equilibrium robot player, such as explaining the concept of equilibrium (e.g., Costa-Gomes & Crawford, Reference Costa-Gomes and Crawford2006) or fully disclosing the computer player’s exact strategy (e.g., Meijering et al., Reference Meijering, Van Rijn, Taatgen and Verbrugge2012; Hanaki et al., Reference Hanaki, Jacquemet, Luchini and Zylbersztejn2016). However, both approaches may introduce a coaching effect: providing background knowledge about the robot’s exact strategy or the concept of equilibrium could directly teach subjects how to play and succeed in the specific game, potentially inflating our estimate of their true strategic thinking capacity. On the contrary, we describe the robot players’ rationality in a multi-layered, recursive manner without providing specific details about their actions (Johnson et al., Reference Johnson, Camerer, Sen and Rymon2002), aiming to reduce the risk of over-coaching while still conveying the robot’s strategic sophistication.

Despite our efforts, we acknowledge that our instruction strategy might not fully eliminate the possibility of instruction effects, and some subjects might still be influenced by the way the robot players’ rationality is described. For instance, some subjects might pick up hints on how to apply the logic of IEDS in our dominance-solvable games, thereby enhancing their depth of strategic thinking. Conversely, others may find the verbal representation of iterative, self-referential logic confusing, which could hinder deeper reasoning. In future experiments, one could evaluate these effects by running a treatment where subjects read the robot instructions but still play against human opponents, then compare their estimated rationality levels to those in a treatment against humans without robot instructions.

Notably, the goal of our design is to capture individual strategic thinking capacity with respect to iterative reasoning by aligning subjects’ beliefs about the robot’s higher-order rationality. As a result, a limitation of our design is that we do not aim to measure how well subjects form beliefs about the overall distribution of the population’s strategic reasoning depth and best respond accordingly, which is a key aspect of strategic sophistication in the sense of Stahl and Wilson Reference Stahl and Wilson(1995)’s “worldly type.” An interesting future direction could be to introduce such a “worldly” robot player and examine whether subjects could outsmart this strategically sophisticated type when playing against the robot.Footnote ³⁹

7.3. Diagnosis of the inconsistency of R2 and R3 with alternative modelsFootnote ⁴⁰

Following up on the discussion at the end of Section 6.3, we diagnose the inconsistency of R2 and R3 types by comparing the revealed rationality approach with two alternative models: the standard level-k model and a diagnostic model: Random, Avoiding Dominated, and Equilibrium (RADE). The comparison between the revealed rationality approach and the level-k model aims to determine whether the inconsistency arises from a mechanical effect inherent in the revealed rationality approach. Furthermore, the differences between RADE and the revealed rationality approach help clarify whether R2 and R3 types are simply “avoiding dominated strategies” rather than engaging in higher-order rationality inferences.

In our diagnostic analysis, we consider the standard level-k model where level-0 players are assumed to uniformly randomize across all strategies and level-k players best respond to level k − 1 players. The revealed rationality approach, in contrast, does not impose structural assumptions on how irrational players behave or how higher-order rational players respond to lower types. Instead, it classifies individuals based on the highest rationalizable order of their strategy, identifying irrational (R0) players as those who choose strictly dominated strategies. In our ring games, the standard level-k model predicts that a level-1 player best responds to uniformly random opponents by consistently choosing secure actions at all player positions, aligning with the R1-S type (see Footnote 36). Moreover, Player 1’s and Player 2’s secure actions are also best responses to those of Player 2 and Player 3, respectively (see Figure 1). Consequently, level-2 and level-3 players correspond to R2-S and R3-S, respectively, while level 4 (and above) players are classified as R4.Footnote ⁴¹ In the guessing games, level-k players choose $50p^{k}$ for $k \geq 1$. To ensure a fair comparison between the revealed rationality approach and the level-k model, we consider five level types: L0, L1, L2, L3, and L4+.

On the other hand, the RADE model can be viewed as a simplified version of Stahl and Wilson Reference Stahl and Wilson(1995), featuring three behavioral types: Random (R), Avoiding Dominated (AD), and Equilibrium (E). Players of the equilibrium type consistently select equilibrium strategies. Consequently, they choose equilibrium actions at all positions in the ring games and select 1 in all three guessing games. Players of the avoiding dominated type never choose a dominated strategy, even though they do not always select equilibrium strategies. Thus, they behave as R1, R2, or R3 in the ring games and avoid strictly dominated numbers in the guessing games. Lastly, random players are assumed to make choices randomly and are the only type that would ever select a strictly dominated strategy.

To evaluate whether the level-k model and the RADE classification offer greater consistency in type classification than the revealed rationality approach, we first classify subjects into revealed rationality types, level-k types and RADE categories based on their behavior in the ring games. These classifications are then used to predict their behavior in the guessing games. Since both the revealed rationality approach and the RADE model generally lack precise predictions for choices in the guessing games, we make quantitative comparisons by assuming that each type of player selects any number consistent with their behavioral type with equal probability.Footnote ⁴² Under this assumption, we evaluate the consistency of the three models by comparing their mean squared deviation (MSD) scores.

Table 5 presents the mean squared deviations (MSDs) for the revealed rationality approach, the level-k model and the RADE model. When comparing the revealed rationality approach with the level-k model, we observe that the MSDs of the level-k model are slightly lower than those of the revealed rationality approach. This suggests that the way reasoning levels are classified in the revealed rationality approach only marginally contributes to the inconsistency of the R2 and R3 types.

Table 5.

MSD of revealed rationality, level-k and RADE

Note: Under the assumption of uniform randomization, all three models provide precise predictions about the percentage of each number chosen. The MSD is computed by summing the squared deviations of these percentages, and the mean is determined by averaging across games.

Furthermore, as shown in the table, the RADE model yields 30%–60% lower MSDs than the other two approaches across all three guessing games in both the Robot and History Treatments. This result suggests that the RADE model achieves greater consistency across games compared to the revealed rationality approach. It also implies that the R2 and R3 types may simply be avoiding dominated strategies rather than engaging in higher-order rationality inferences.Footnote ⁴³

Notably, the MSDs under RADE in the Robot Treatment are smaller than those in the History Treatment, highlighting the effectiveness of our belief control approach. This finding aligns with another notable contrast between the two treatments: the stability of the E types.Footnote ⁴⁴ In the Robot Treatment, 66% of the 74 subjects classified as E types in the ring games remain classified as such in the guessing games. In contrast, in the History Treatment, only 38% of the E types in the ring games retain their classification in the guessing games. Additionally, 58% of the E types in the ring games switch to AD types in the guessing games, suggesting that the inconsistency of intermediate rationality levels may stem from the heuristic of avoiding dominated strategies when faced with a novel game and uncertainty about opponents’ strategies.

In summary, the revealed rationality approach is a classification method that does not rely on any ad hoc assumptions about beliefs and can be universally applied to any dominant-solvable game in the same way. However, these appealing properties come at a cost. As demonstrated in this diagnostic analysis, the consistency of the revealed rationality classification is sensitive to the structure of iterative rationalizable reasoning. Moreover, this consistency can be potentially distorted if players are simply avoiding dominated strategies.

Finally, to further investigate the correlation between the behavior of avoiding dominated strategies and other cognitive abilities, we regress subjects’ RADE types on their performance in three cognitive tasks. Since RADE type is a categorical dependent variable, we use multinomial logistic regression with the avoid dominated (AD) type as the reference category.Footnote ⁴⁵ Table 6 shows that, whether in the Robot Treatment or the History Treatment, E types identified from the ring games exhibit significantly higher CRT scores and better performance in the farsightedness task than AD types. Additionally, AD types do not show significantly different performance on these cognitive tests compared to R types. In contrast, AD types identified in the guessing games have significantly higher CRT scores than R types. Furthermore, in the Robot Treatment, E types demonstrate significantly better performance across all three cognitive tasks compared to AD types. Yet in the History Treatment, no significant differences in performance are observed between E and AD types. These results complement our main findings in Section 6.4.1 and contrast the two treatments from a different perspective.

Table 6.

Multinomial logistic regressions for RADE types

The standard errors are clustered at the session level.

* Significance level: p < 0.05,

** p < 0.01,

*** p < 0.001.