2.1 Assessment of planning with the tower of London test

The original version of the ToL test (Shallice, Reference Shallice1982) consists of three differently colored beads placed on three vertical pegs of different heights that may hold at maximum either one, two, or three beads, respectively. The task requires to transform the start state into a goal state with a sequence of actions, each of which consists of moving one bead from one peg to another, in accordance with specific rules and given the number of moves. A problem is correctly solved when the goal state is achieved within the minimum number of moves.

During the execution of the ToL task, many indices can be recorded—such as accuracy, solution time, number of moves, sequences of moves, and rule violations. The most frequently used behavioral measures are accuracy, pre-planning time, and execution time (e.g., Anderson et al., Reference Anderson, Anderson and Lajoie1996; Culbertson & Zillmer, Reference Culbertson and Zillmer1998; Krikorian et al., Reference Krikorian, Bartok and Gay1994; Unterrainer et al., Reference Unterrainer, Rahm, Leonhart, Ruff and Halsband2003).

Evidence from developmental studies using accuracy as a performance indicator shows a gradual growth in planning skills from childhood to adolescence and early adulthood (Albert & Steinberg, Reference Albert and Steinberg2011; Asato et al., Reference Asato, Sweeney and Luna2006; Crone & Steinbeis, Reference Crone and Steinbeis2017; De Luca et al., Reference De Luca, Wood, Anderson, Buchanan, Proffitt, Mahony and Pantelis2003; Korkman et al., Reference Korkman, Kemp and Kirk2001; Luciana & Nelson, Reference Luciana and Nelson1998; Luciana et al., Reference Luciana, Collins, Olson and Schissel2009). Specifically, traditional and computerized versions of ToL test indicate that planning ability improves with age, reaching adult-like levels by 14 years (De Luca et al., Reference De Luca, Wood, Anderson, Buchanan, Proffitt, Mahony and Pantelis2003; Krikorian et al., Reference Krikorian, Bartok and Gay1994; Luciana & Nelson, Reference Luciana and Nelson1998, Reference Luciana and Nelson2002; Malloy-Diniz et al., Reference Malloy-Diniz, Cardoso-Martins, Nassif, Levy, Leite and Fuentes2008). This trend is consistent with findings using the Tower of Hanoi, which show progressive planning improvement during childhood (Díaz et al., Reference Díaz, Martín, Jiménez, García, Hernández and Rodríguez2012).

Findings on age-related changes in planning time are mixed. Some studies report increased planning time with age, particularly in complex tasks (Asato et al., Reference Asato, Sweeney and Luna2006; Filippi et al., Reference Filippi, Ceccolini, Periche-Tomas and Bright2020; Steinberg et al., Reference Steinberg, Albert, Cauffman, Banich, Graham and Woolard2008), while others show a decrease during childhood and early adolescence (Huizinga et al., Reference Huizinga, Dolan and Van der Molen2006; Injoque-Ricle et al., Reference Injoque-Ricle, Barreyro, Calero and Burin2014). Task difficulty appears to modulate these effects across age groups.

The aforementioned studies used pre-planning time as an index of planning skills. However, it is conceivable that, in some cases, planning during the ToL task does not end when the first move is made, but extends throughout the task. Indeed, the execution phase likely encompasses not only the motor actions of bead movement, but also the cognitive processes involved in ongoing online planning and error correction (Berg & Byrd, Reference Berg and Byrd2002; Luciana et al., Reference Luciana, Collins, Olson and Schissel2009). Notably, to our knowledge, there is a paucity of research that attempts to isolate, differentiate, or quantify, within the execution phase, the time spent on movement and the time spent thinking. Indeed, there have been attempts to reduce the planning before the first move (Owen et al., Reference Owen, Sahakian, Hodges, Summers, Polkey and Robbins1995; Phillips et al., Reference Phillips, Wynn, Gilhooly, Della Sala and Logie2001; Ward & Allport, Reference Ward and Allport1997); however, no study has measured planning time within the execution phase. In the traditional paper-and-pencil versions of the ToL, the distinction between movement and planning time is not typically provided, and would pose a challenge for the experimenter to measure it. Nevertheless, computerized versions offer the possibility to dissect these processes easier, potentially revealing valuable information and insights into the diverse strategies individuals employ to complete the task.

2.2 Knowledge space theory

KST (Doignon & Falmagne, Reference Doignon and Falmagne1985, Doignon & Falmagne, Reference Doignon and Falmagne1999, Falmagne & Doignon, Reference Falmagne and Doignon2011) is a mathematical framework developed for the efficient and accurate assessment of knowledge. At its core is the concept of a knowledge state, denoted as a set $K \subseteq Q$ , representing the problems that an individual masters within a specific collection of problems Q named the knowledge domain. The entire collection of possible knowledge states for a population is a knowledge structure $\mathcal {K}$ . When a knowledge structure is closed under unionFootnote ¹ it is called a knowledge space.

Various methods have been developed over the years to construct a knowledge structure. These methods include exploiting the relation between the problems (e.g., Dowling, Reference Dowling1993; Koppen, Reference Koppen1993; Stefanutti & Koppen, Reference Stefanutti and Koppen2003), assuming a competence level along with a set of underlying skills (e.g., Heller, Augustin, et al., Reference Heller, Augustin, Hockemeyer, Stefanutti and Albert2013; Heller, Ünlü, et al., Reference Heller, Ünlü and Albert2013; Korossy, Reference Korossy, Albert and Lukas1999), employing data-driven procedures (e.g., de Chiusole et al., Reference de Chiusole, Stefanutti and Spoto2017, Reference de Chiusole, Spoto and Stefanutti2020; Spoto et al., Reference Spoto, Stefanutti and Vidotto2016), or deriving a knowledge space from a problem space underlying the tasks (Stefanutti, Reference Stefanutti2019). This last approach is the one adopted in this article.

The most common probabilistic model in KST is the basic local independence model (BLIM; Falmagne & Doignon, Reference Falmagne and Doignon1988). The BLIM is a restricted latent class model where the latent classes represent the knowledge states. In this model, what is observed is a response pattern, denoted as $R \subseteq Q$ , which indicates the subset of items correctly solved by each individual. Given a population of individuals, a probability distribution $\pi : \mathcal {K} \rightarrow (0,1)$ on the knowledge states is assumed. Moreover, two parameters estimated for each item $q \in Q$ are considered: the conditional probability of observing a wrong answer for item q given that $q \in K$ , denoted $\beta _q$ ; the conditional probability of observing a correct answer for item q given that $q \notin K$ , denoted $\eta _q$ . Both parameters have a typical interpretation in the BLIM: $\beta _q$ is a careless error probability for the item q and $\eta _q$ is a lucky guess probability for the same item.

In the BLIM, the restriction $\beta _q + \eta _q < 1$ for each item is a required property for obtaining a consistent assessment of the knowledge state, that is, for identifying the knowledge state of an individual based on the observed item responses of this individual. Specifically, the probability of making a careless error on item q should be lower than the probability of failing q due to lack of mastery. On the other hand, the probability of correctly solving an item because it is mastered must exceed that of guessing it. The above restriction is called the “monotonicity condition.”

The $\beta _q$ , $\eta _q$ , and $\pi _K$ parameters of the BLIM can be estimated by ML via the expectation–maximization algorithm (Stefanutti & Robusto, Reference Stefanutti and Robusto2009), by minimum discrepancy (MD, Heller & Wickelmaier, Reference Heller and Wickelmaier2013) or by an estimation procedure that is a mix of the two (MDML, Heller & Wickelmaier, Reference Heller and Wickelmaier2013). Methods are also available for computing ML estimates of the model parameters in the presence of missing data (Anselmi et al., Reference Anselmi, Robusto, Stefanutti and de Chiusole2016; de Chiusole et al., Reference de Chiusole, Stefanutti, Anselmi and Robusto2015). Once the BLIM based on a specific knowledge structure $\mathcal {K}$ has been estimated and validated on an appropriate sample of individuals, the model parameter estimates can be used to identify the knowledge state $K \in \mathcal {K}$ of each individual based on their item responses. This allows for the assessment of individual differences. Since, in general, knowledge states are partially ordered, the assessment is multidimensional.

2.3 Procedural knowledge space theory

The notion of problem space introduced by Newell & Simon (Reference Newell and Simon1972) is a commonly used concept in the study of human problem solving. The connection between KST and the problem space theory of Newell & Simon (Reference Newell and Simon1972) was first pointed out by Stefanutti & Albert (Reference Stefanutti and Albert2003). The theoretical foundations were established by Stefanutti (Reference Stefanutti2019), and take the name of PKST. The remainder of the section delves into the fundamental concepts of PKST.

Let $\Omega $ be a set of operations. A string of length n of elements in $\Omega $ is a sequence $\omega _1\omega _2\ldots \omega _n \in \Omega ^n$ . The concatenation between two strings $\omega _1\omega _2\ldots \omega _m$ and $\omega _{m+1}\omega _{m+2}\ldots \omega _{n}$ is the string $\omega _1\omega _2\ldots \omega _m\omega _{m+1}\omega _{m+2}\ldots \omega _{n}$ . The collection of all the strings of operations of finite length is

$$\begin{align*}\Omega^* = \bigcup_{n \in \mathbb{Z}^+} \Omega^n, \end{align*}$$

where $\mathbb {Z}^+$ is the set of non-negative integer numbers. The empty sequence $\epsilon $ belongs to $\Omega ^*$ (i.e., $\epsilon \in \Omega ^*$ ).

In PKST, a problem space is defined as a triple $\mathbf {P}=(S,\Omega ,\cdot )$ , where S is a nonempty set of problem states, $\Omega $ is a nonempty set of operations, and $\cdot : S \times \Omega ^* \to S$ is an operator satisfying, for all $s \in S$ and $\pi ,\sigma \in \Omega ^*$ , the two axioms:

1. (P1) $s \cdot \epsilon = s$ ;

2. (P2) $(s \cdot \pi ) \cdot \sigma = s \cdot \pi \sigma $ .

The operator $\cdot $ is named operation application. Problem states and operations are primitive concepts in PKST.

A problem within the problem space $\mathbf {P}$ is a pair $(s,t),$ where $s, t \in S$ and there exists a string $\pi \in \Omega ^*$ such that $s \cdot \pi = t$ . The collection of all the problems in $\mathbf {P}$ is thus

$$\begin{align*}Q = \{(s,t) \in S \times S: s \ne t \text{ and } s \cdot \pi = t \text{ for some } \pi \in \Omega^*\}. \end{align*}$$

Notably, the set Q corresponds to the domain of knowledge in KST. In the problem space $\mathbf {P}$ , a solution path is any pair $s\pi \in \Pi = S \times (\Omega ^* \setminus \{\epsilon \})$ . In addition, a solution path $s\pi \in \Pi $ is said to solve the problem $(s,t) \in Q$ if $s \cdot \pi = t$ . It is worth mentioning that the same problem may be solved by multiple solution paths. A solution path $s\pi $ is a subpath of another solution path $t\sigma $ (denoted by $s\pi \sqsubseteq t\sigma $ ) if there are $\alpha ,\beta \in \Omega ^*$ such that $\sigma = \alpha \pi \beta $ and $t \cdot \alpha = s$ . The relation $\sqsubseteq $ , called sub-path relation, is a partial order for the set $\Pi $ . A subset $C \subseteq \Pi $ is said to respect path inclusion if the condition

$$\begin{align*}s\pi \sqsubseteq t\sigma, t\sigma \in C \implies s\pi \in C \end{align*}$$

is respected for all $s\pi ,t\sigma \in \Pi $ . A subset of solution paths respecting path inclusion is named a competence state of the problem space $\mathbf {P}$ . The collection $\mathcal {C}$ of all the competence states is the competence space.

The function $\tau : Q \to 2^{\Pi \setminus \{\emptyset \}}$ maps each problem $(s,t) \in Q$ to the collection $\tau (s,t)$ of all solution paths that solve it.

The collection of all the problems in Q that can be solved by an individual whose competence state is $C \in \mathcal {C}$ is

$$\begin{align*}p(C) = \{(s,t) \in Q: \tau(s,t) \cap C \ne \emptyset\}. \end{align*}$$

This collection $p(C) $ is named the knowledge state delineated by competence state C. It is evident that a problem $(s, t)$ belongs to $p(C)$ if and only if C includes at least one solution path solving $(s, t)$ . The collection $\mathcal {K}$ of all the knowledge states is the knowledge space derived from the problem space $\mathbf {P}$ . Stefanutti (Reference Stefanutti2019) introduced an algorithm for the automatic derivation of a knowledge space from a problem space. Like KST, PKST also classifies individuals based on their knowledge states, allowing for the assessment of individual differences in problem-solving skills.

A particular kind of problem space called “goal space” has been introduced in Stefanutti et al. (Reference Stefanutti, de Chiusole and Brancaccio2021) and it is central in this work. In a goal space, each step of the solution process of a problem is classified as “correct-so-far” or “incorrect.” A goal space is defined as a problem space $(S, \Omega , \cdot )$ with two distinct states $f,g \in S$ such that:

1. (GS1) for all $\omega \in \Omega $ , $f \cdot \omega = f$ and $g \cdot \omega = g$ ;

2. (GS2) for each $s \in S \setminus \{f\},$ there is a string $\pi \in \Omega ^*$ such that $s \cdot \pi =g$ .

The problem states f and g are called the failure and the goal states, respectively, and a goal space is denoted by the 5-tuple $(S,f,g,\Omega ,\cdot )$ . A solution process is deemed “correct” when it reaches g and “incorrect” when it reaches f.

According to property (GS1), both the failure and the goal states are considered final states, meaning that any operation applied to either of them is ineffective. Property (GS2) stipulates that the goal state g is reachable from any problem state except f. Consequently, any pair $(s,g)$ with $s \in S \setminus \{f,g\}$ constitutes a problem within the goal space.

2.4 An example with the tower of London

In this section, an example based on the problem space of the ToL test is presented to provide a practical illustration of the theoretical concepts introduced in the previous section. In the ToL test, a problem state consists of three beads of different colors placed on three pegs of different heights mounted on a wooden support. The three beads can be moved, one by one, from one peg to another. The number of possible problem states in the ToL test problem space is 36, and they coincide with the configurations of the beads in the pegs. Figure 1 depicts the 36 problem states of the ToL. Each of them is labeled by a pair $xy$ of numbers, where x stands for one of the six spatial arrangements (rows in the figure), whereas y stands for one of the color permutations (columns in the figure). An operation consists of moving a ball from one of the three pegs to another. Naming the three pegs as left, center, and right, six operations are entailed: left to center $(a)$ , center to left $(\bar a)$ , center to right $(b)$ , right to center $(\bar b)$ , left to right $(c)$ , and right to left $(\bar c)$ . Therefore, in the ToL, the set of operations is $\Omega _{1} = \{a,b,c,\bar a,\bar b,\bar c\}$ .

Figure 1

The $6 \times 6$ different problem states of the ToL test.

Figure 2 depicts the directed graph of a portion of the ToL test problem space in which the collection of problem states is $S_{1} = \{ s_1, s_2, s_3, s_4, s_5,s_6\}$ . Each problem state corresponds to a label based on the coding used in Figure 1. Specifically, $s_1=32$ , $s_2=22$ , $s_3=52$ , $s_4=51$ , $s_5=21,$ and $s_6=31$ . The triple $\mathbf {P}_{1}=(S_1,\Omega _1,\cdot )$ is the problem space for this portion of the ToL test problem space. We can trivially derive a goal space $\mathbf {P}_{1g}=(S_{1g},f,g,\Omega _1,\cdot ),$ where $S_{1g}= S_1 \cup \{f\}$ and $g=s_6$ . In addition, all problem states different from the goal are connected to the failure state f by some operations in $\Omega _1$ .

Figure 2

Directed graph of a portion of the ToL test problem space.

In the example, only the subset of problems from goal space $\mathbf {P}_{1g}$ which have $s_6$ as goal state is considered, namely, $Q_{1}=\{(s_1,s_6),(s_2,s_6),(s_3,s_6),(s_4,s_6),(s_5,s_6)\}$ . Since all the problems in $Q_{1}$ have form $(s_i, s_6)$ , for lightening the notation, each of them is just represented by the initial state $s_i \in S_{1g}$ . To solve a problem, one needs to know at least one of the solution paths of that problem. For instance, problem $s_1$ has two possible solution paths, namely, $s_1c \bar a \bar b$ and $s_1 b a \bar c$ . The set of all solution paths that solves anyone of the problems in $Q_1$ is

$$\begin{align*}\Pi_1=\{s_1c \bar a \bar b,s_1 b a \bar c, s_3\bar a \bar b, s_2 a \bar c, s_5\bar b, s_4 \bar c \}. \end{align*}$$

The mapping $\tau _1$ for the subset $\Pi _1$ is constructed instead of deriving the mapping $\tau $ for the whole set $\Pi $ of solution paths. Thus, the mapping $\tau _1$ is defined as follows:

$$ \begin{align*} \tau_1(s_1) &= \{s_1c \bar a \bar b,s_1 b a \bar c\}, \\ \tau_1(s_2) &= \{ s_2 a \bar c\}, \\ \tau_1(s_3) &= \{s_3\bar a \bar b\}, \\ \tau_1(s_4) &= \{ s_4 \bar c\}, \\ \tau_1(s_5) &= \{ s_5\bar b\}. \end{align*} $$

The collection of solution paths that respects path inclusion in $\Pi _{ig}$ is shown in column 1 of Table 1.

Table 1

The 16 competence states obtained from the goal space $\Pi _{ig}$ and the knowledge states delineated from them are presented in the first and second columns, respectively

Let us consider the competence states $C=\{s_3 \bar a \bar b, s_5 \bar b, s_4 \bar c \} \in \mathcal {C}_1$ , since $s_5 \bar b \sqsubseteq s_3 \bar a \bar b$ and both $ s_5 \bar b$ , and $s_4 \bar c$ do not have any subpath indeed C respects path inclusion. The knowledge state delineated by C is obtained by a simple application of the problem function:

$$\begin{align*}p(\{s_3 \bar a \bar b, s_5 \bar b, s_4 \bar c \})=\{s_3, s_5, s_4\}. \end{align*}$$

On the whole, applying the problem function p to each of the competence states yields the knowledge states in column $2$ of Table 1. It is worth noting that certain knowledge states, specifically $\{s_5, s_4, s_3, s_2, s_1\}$ , are repeated in the table. This repetition arises because the problem function p is not injective.

It is worth recalling that each problem in this example is expressed only by its initial state. In general, problems in the knowledge state would be expressed as ordered pairs of initial and final states. In this example, since $\mathbf {P}_{1g}$ is a goal space, every knowledge state can be obtained from some competence state by just dropping the sequence of operations and retaining the initial state for each of the solution paths in the competence state.

3.1 Parameter estimation

The parameters of the MSPMs can be estimated by ML via an adaptation of the expectation–maximization algorithm (Dempster et al., Reference Dempster, Laird and Rubin1977). The details concerning the derivation of the estimation equations are provided in the Appendix section. It should be observed that (unconstrained) maximization of the model’s likelihood does not guarantee that the monotonicity conditions of the form given in (2) are satisfied. If they are violated for one or more pairs of transition probabilities $\eta _{ij}$ and $\beta _{ij}$ (with $j \ne f$ ), then the interpretation of these last parameters as “lucky guess” and “non-careless error” is not credible anymore, and the interpretation of the whole model may become problematic. Furthermore, if the model parameters are applied for carrying out adaptive assessment, convergence to the true state of the problem solver is not guaranteed when the monotonicity conditions (2) are violated (see, in this respect, Brancaccio et al., Reference Brancaccio, de Chiusole and Stefanutti2023). For these reasons, a constrained ML procedure was derived for both MSPM1 and MSPM2 models, where the constraints have the form of inequalities $\eta _{ij} < \beta _{ij}$ .

The unconstrained estimation of the MSPM1 was derived by Stefanutti et al. (Reference Stefanutti, de Chiusole and Brancaccio2021) and the MATLAB code can be found at https://osf.io/qa8mg/?view_only=8b4e148300de40a6941df4a102067fc1/.

Concerning the unconstrained estimation of the MSPM2 and the constrained estimation of both the MSPM1 and MSPM2, the corresponding algorithms are available at: https://osf.io/m658x/?view_only=25c8e23729554a1a84e74ed8e50d484b.

Once estimated and validated on an appropriate sample of individuals, both the MSPM1 and MSPM2 allow for the identification of the knowledge state of an individual based on their observed response pattern. For example, the posterior probability of a knowledge state K given an observed jump matrix J can be computed using Bayes’ rule:

$$\begin{align*}P(K|J) = \frac{P(J|K)P(K)} {\sum_{K' \in \mathcal{K}}P(J|K')P(K')}. \end{align*}$$

The knowledge state that maximizes $P(K|J)$ is taken as the most plausible estimate for the individual who produced the response pattern represented by J. This enables the assessment of individual differences. However, since $P(J|K)$ takes on different forms under assumptions (A1) and (A2), as shown by Equations (A.1) and (A.2) in the Appendix, the two models might assign different knowledge states to the same individual. Consequently, it is important to identify the best fitting model for an accurate assessment (Brancaccio et al., Reference Brancaccio, de Chiusole and Stefanutti2023).

4.1 Simulation design and data set generation

The present simulation study is based on the goal space $\mathbf {P} =(S,f,g,\Omega ,\cdot )$ obtained from the ToL problem space. It was defined according to two criteria: (i) the goal state is problem state 11 and (ii) the solution path length ranges from 1 to 5 moves. The resulting goal space $\mathbf {P}$ contains 20 problem states plus the goal and the failure states. Thus, 20 problems can be derived from $\mathbf {P}$ , which are all the pairs $(s,g)$ with $s \in S \setminus \{g,f\}$ . Knowledge space $\mathcal {K}$ derived from $\mathbf {P}$ contained 1,573 knowledge states.

The data were generated under 12 different conditions displayed in Table 3. The 12 conditions result from the combination of the two assumptions (A1) and (A2) used to simulate the data, three sample sizes (300, 1,500, and 6,000), and two error levels. The three sample sizes were chosen to represent situations with small, medium, and large data sets, respectively, compared to 1,573, which is the cardinality of the structure. The two error levels were selected to represent low and medium-high error conditions.

Table 3

Design of the simulation study used for generating the data

Note: Column 1 displays the condition number, column 2 displays the assumption underlying the generative model, columns 3 and 4 show the error interval used for generating the data, and column 5 displays the sample size (see Section 4.1).

The error level in the data was manipulated through the $\beta _{ij}$ and $\eta _{ij}$ transition probabilities of the model. In particular, for each $ i \in S \setminus \{f,g\}$ , the $\beta _{if}$ parameters were drawn at random from a uniform distribution with intervals $(0,0.1]$ and $(0,0.3]$ depending on the conditions. On the other hand, the $\eta _{if}$ parameters were randomly drawn from a uniform distribution in the intervals $[0.7,1)$ and $[0.9,1)$ . Different intervals were chosen for the $\beta _{if}$ and $\eta _{if}$ parameters because they have different interpretations. In particular, the $\beta _{if}$ are interpreted as “careless errors” and, thus, they are expected to be small, whereas the $\eta _{if}$ are interpreted as “non-lucky guesses” and, hence, they are expected to be close to one. The remaining $\beta _{ij}$ and $\eta _{ij}$ were drawn at random from a uniform distribution and then normalized to sum up to $1- \beta _{if}$ and $1- \eta _{if}$ , respectively. For these reasons, conditions where $\beta _{if} \in (0,0.1]$ and $\eta _{if} \in [0.9,1)$ represent a low error scenario, whereas conditions where $\beta _{if} \in (0,0.3]$ and $\eta _{if} \in [0.7,1)$ represent a medium-high error scenario. Notice that, in each scenario, each pair $(\beta _{ij},\eta _{ij})$ of parameters with $j \neq f$ also satisfies the inequality $\beta _{ij}-\eta _{ij} \ge 0$ (see the Appendix).

The data were simulated under the two different assumptions (A1) and (A2). Specifically, for each subject with knowledge state $K \in \mathcal {K}$ and for each problem $(s,g)$ , a sequence of problem states terminating with either the failure state or the goal state is generated using the $\beta _{ij}$ and $\eta _{ij}$ parameters. The procedure for simulating each response pattern follows the method described in Stefanutti et al. (Reference Stefanutti, de Chiusole and Brancaccio2021, Figure 5). The only variation between (A1) and (A2) is that, in the former case, the transition probabilities used to generate the sequence are all $\beta _{ij}$ or $\eta _{ij}$ if problem $(s,g)$ , respectively, belongs or does not belong to K, whereas in (A2), each transition probability is a $\beta _{ij}$ or $\eta _{ij}$ if problem $(i,g)$ belongs or does not belong to K.

For each of the 12 conditions, 500 data sets were generated, obtaining a total of $12 \times 500 =6,000$ simulated data sets. Every single data set consists of a given number of response patterns, each of which is a collection of jump matrices $J=\{J_q: q \in Q\}$ , one for each of the problems in Q.

4.2 Methods

The identifiability of the MSPM1 and the MSPM2 has been tested using an informal method consisting of the following two steps: (i) the parameters of each model were re-estimated $n \ge 50$ times on the same data set, each time starting from a different initial set of parameters values, until 50 different estimations with equal likelihood were obtained and (ii) the standard deviations of the parameter estimates obtained in the 50 estimations were computed. It is expected that the standard deviation of the estimates is zero up to round-off when parameters are identifiable. Typically, this procedure is applied in those situations in which a formal test of (local) identifiability is not available (see, e.g., Stefanutti et al., Reference Stefanutti, de Chiusole, Anselmi and Spoto2020, Reference Stefanutti, de Chiusole and Brancaccio2021).

In each of the 12 conditions, 500 data sets were simulated and both the MSPM1 and MSPM2 were fitted to the data. The goodness of the parameter recovery of the MSPM2 was tested in those conditions in which (A2) was the assumption under which the data sets were generated. In particular, the bias and the variance of the parameter estimates across the 500 simulated data sets were computed. The estimated bias for each $\beta _{ij}$ was computed as follows:

(3) $$ \begin{align} bias(\hat\beta_{ij})= \bar\beta_{ij} - \beta_{ij}, \end{align} $$

where $\beta _{ij}$ is the parameter value used for generating the data and $\bar \beta _{ij}$ is the arithmetic mean of the 500 estimates of the $\beta _{ij}$ parameters. The standard deviation of the estimate $\hat \beta _{ij}$ was obtained as follows:

(4) $$ \begin{align} \sigma(\hat\beta_{ij})=\sqrt{\frac{1}{500} \sum_{k=1}^{500} (\hat \beta^{(k)}_{ij} - \bar\beta_{ij})^2}, \end{align} $$

where $\hat \beta ^{(k)}_{ij}$ is the estimated value of the transition probability for the k-th simulated sample. It is worth noticing that $\sigma (\hat \beta _{ij})$ is also the standard deviation around $bias(\hat \beta _{ij})$ . The bias and variance for $\hat \eta _{ij}$ were computed in the same way.

Moreover, as an index of the expected amount of information in the data to estimate the $\beta _{ij}$ parameters, the “true” marginal problem probability was computed. Let $\mathcal {K}_q=\{K \in \mathcal {K}: q \in K\}$ be the collection of all the knowledge states containing problem q. Then, for each $q=(i,g) \in Q$ , the “true” marginal problem probability $P(\mathcal {K}_q)$ is given by

$$\begin{align*}P(\mathcal{K}_q)=\sum_{K \in \mathcal{K}_q}P(K). \end{align*}$$

Similarly, the expected amount of information in the data to estimate the $\eta _{ij}$ parameters is given by the probability

$$\begin{align*}P(\mathcal{K}_{\overline{q}})=1- P(\mathcal{K}_q). \end{align*}$$

For simplicity, both $P(\mathcal {K}_q)$ and $P(\mathcal {K}_{\overline {q}})$ are called “true” marginal probability of the problem all along the manuscript, because it is the probability of the states used to generate the data sets.

To test the capability of the model selection criteria to identify the assumption under which the data were generated, the Akaike information criteria (AIC; Akaike, Reference Akaike, Petrov and Csaki1973) computed for the two estimated models were compared to one another. The proportion p of times over the 500 replications in which the AIC selects the model that is consistent with the generative assumption was computed in all the conditions. For instance, a $p=1$ in Condition 1 means that the AIC selects the MSPM1 for all the 500 data sets, whereas a $p=1$ in Condition 7 means that the MSPM2 is selected for all the 500 data sets.

4.4 Discussion

Overall, MSPM2 demonstrates good parameter recovery performance, particularly for the majority of transitions. Even under relatively small sample sizes (i.e., $N = 300$ ), estimates for both $\beta _{ij}$ and $\eta _{ij}$ parameters are generally unbiased, with deviations emerging only in those transitions where failure is intrinsically rare (i.e., with very low variability in the data). Importantly, the two parameters susceptible to bias—namely, $\eta _{31,g}$ and $\eta _{21,g}$ —refer to transitions toward the goal state in single-move problems, which are of little behavioral importance as these transitions are very rarely failed in the general population. As such, these aspects do not compromise the interpretability or practical utility of the model.

A potential limitation of this simulation study should also be mentioned. Indeed, data sets were obtained by using only 12 generating parameter sets. While this ensures internal consistency, it could not adequately capture the full variability of parameters encountered in real-world contexts. Future work could address this by employing a broader and more systematically varied set of generating conditions to enhance generalizability.

5.1 The assessment tool: Adap-ToL

This study is part of a research project founded by the Italian Ministry of Education and Research aimed at the development of an intelligent system, named PsycAssist (de Chiusole et al., Reference de Chiusole, Spinoso, Anselmi, Bacherini, Balboni, Mazzoni, Brancaccio, Epifania, Orsoni, Giovagnoli, Garofalo, Benassi, Robusto, Stefanutti and Pierluigi2024), for the adaptive assessment of executive functions and fluid intelligence among general and clinical populations. The study presented here shows the results obtained from the application of the MSPM1 and MSPM2 to the data collected with one of the tools available on PsycAssist, which is Adap-ToL.

Adap-ToL is a test for the assessment of planning skills in individuals aged four and above. It resembles the ToL test, where participants move beads from an initial setup to a goal configuration, within a set number of moves and following specific rules. Unlike the traditional ToL test, where the initial state is constant among the problems but the goal state varies, Adap-ToL maintains a fixed goal state while varying the initial configuration. This was designed to maintain coherence with the PKST definition of a problem in a goal space.

Different versions of the test were designed. Among all the 35 problems having the same tower configuration (i.e., configurations in the first row of Figure 1) as the goal state, those with a number of moves 1–5 were used for creating a version of the test designed for individuals aged 4–8; those with a number of moves 1–6 were used for creating a version of the test designed for individuals aged 9–13; and all the 35 problems were used in the version for individuals older than 14.

The test is administered by using a tablet (IoS operating system with a 10.9-inch screen) oriented vertically. Each problem of the test consists of two images, one above the other (Figure 7). The upper image represents the configuration of the goal state, whereas the lower image represents the configuration of the initial state of the problem. The task is to replicate the goal state from the initial configuration, within a specified minimum number of moves (specified at the top of the upper image). Acting on the lower image, participants tap the screen twice to make a move: first to select the bead, and then to choose the peg in which to move the bead. The problem is successfully solved when the goal state is reached within the minimum number of moves.

Figure 7

One of the problems in Adap-Tol.

Note: Each problem of the test consists of two images, one above the other. The upper image represents the configuration of the goal state, whereas the lower image represents the initial configuration of the problem. The task is to replicate the goal state from the initial configuration, within a specified minimum number of moves (specified at the top of the upper image). In the depicted configuration, “Figura da uguagliare in 2 mosse” translates to “Figure to be matched in 2 moves”). Acting on the lower image, participants tap the screen twice to make a move: first to select the bead, and then to choose the peg in which to move the bead.

The test is introduced by a brief animated video (95 seconds), which presented the test, explained the task, and gives the following two rules: (a) the highest peg can hold three beads, the central peg can hold a maximum of two beads, and the shortest peg can hold only one bead and (b) a bead cannot be moved if it has another bead on its top. The system gives to the test-taker a feedback in case of the violation of one rule, an excessive number of moves made to complete the task, and when the task is correctly executed. Each feedback is presented within a colored box (red, orange, or green, respectively) by using a simple text and an emoticon. For children who cannot read yet or who may face challenges in reading, the evaluator reads the message aloud.

After the video presentation and before the test administration, the test-takers were presented with four practice problems. If the test-taker struggled to find any answer to a problem, the evaluator could decide to skip the problem and move to a next one. It is worth noticing that the decision to skip a problem was made solely by the evaluator and only when the test-taker was unable to reach the goal configuration and showed signs of frustration. The problems were presented one at a time in random order.

As in traditional ToL tests, the “pre-planning time” (i.e., the time elapsed between the presentation of the problem and the first click), the “execution time” (i.e., the time elapsed between the first click and the completion of the problem), the “total time” (i.e., the sum of the pre-planning and execution times), and the accuracy of each problem were recorded. Concerning accuracy, the problem is considered correctly solved when the goal state is reached within the minimum number of moves; otherwise, it is wrong.

Different from other ToL tests, where at maximum three attempts are given to solve a problem, Adap-ToL prescribes only one attempt. Moreover, Adap-Tol registers the whole solution path made by a problem-solver. This aspect is fundamental for applying MSPM1 and MSPM2.

The computerized version of the test allows for registering the time at each click made by a test-taker. Consequently, other two types of sub-times can be registered, that is: (i) the “move-execution time,” which is the time spent moving a bead, is computed as the time elapsed from the click on a bead to the click on a peg and (ii) the “move-planning time,” which is time spent selecting a new bead to move, is computed as the time from the click on a peg to the click on a new bead. Figure 8 shows how the entire solution process of a 2-move problem is decomposed among the “traditional” pre-planning and execution times and the new move-execution and move-planning times.

Figure 8

Decomposition of the solution process of a 2-move problem in the “traditional” pre-planning and execution times and the new move-execution and move-planning times.

5.2 Participants

Adap-Tol was administered to a sample, extracted from the general population, of $N = 1,466$ participants (Female $= 52.46$ %, Age range: $4.20$ – $71.05$ years, Mean age: $19.14 \pm 16.35$ years). Data were collected in different regions of the North, Centre, and South of Italy (i.e., Abruzzo, Basilicata, Calabria, Emilia-Romagna, Lazio, Liguria, Lombardia, Marche, Puglia, Toscana, Umbria, and Veneto) and schools of all grades as well as adults in the general population were involved. Before participating in the study, participants and/or their parents received detailed information about the study purposes and procedures, and they (or their parents/legal guardians) provided a signed informed consent, in accordance with the Declaration of Helsinki.

Data from 69 people with a diagnosis (e.g., ADHD, autism spectrum disorder, and intellectual disability) were removed. The removal of individuals with a diagnosis is due to the fact that this study focuses on assessing the sensitivity of the two models to age-related differences in the general population. In fact, including individuals with specific difficulties could bias the results, potentially making older participants perform more similarly to younger ones. No problems were skipped, thus the data set does not have any missing answers. Moreover, a maximum age limit was set at age 57. Thus, data from 105 participants aged 57 and above were excluded. This age limit was chosen to (i) prevent performance decline due to age-related factors and (ii) ensure a sample size comparable to the other age groups. The final sample was composed of 1,053 respondents (Female $= 51.85$ %, Age range: $4.41$ – $56.97$ years, Mean age: $16.62 \pm 11.50$ years). Age group 4 to 8 was composed of 269 children (Female = $45$ %, Mean age: $7.08\pm 1.36$ ), age group 9 to 13 was composed of 343 children (Female = $51$ %, Mean age: $11.70\pm 1.51$ ), age group 14 to 57 was composed of 441 individuals (Female = $56.70$ %, Mean age: $26.20\pm 12.10$ ).

A two-proportion z-test was used to investigate any significant difference in the proportion of correct responses between female and male children in each age group (Type I error probability $\alpha = 0.05$ ). The proportion of correct responses was computed considering the 20 problems that were presented to all respondents regardless of their age. The Benjamini–Hockberg (Benjamini & Hochberg, Reference Benjamini and Hochberg1995) correction for multiple comparison was applied. The results are reported in Table 5. After correcting the p-values for multiple comparisons, the difference in the proportion of correct responses in female and male participants remained significant in the 4–8 group and in the 30–57 group. In the former case, female participants performed significantly better than male participants, while the opposite effect was observed in the latter case.

Table 5

Proportion of correct responses of female (F) and male (M) participants in each age group

Note: The p-values reported in the last column are already corrected according to the Benjamini–Hockberg correction.

5.3 Data analysis

For maximizing the number of problems shared by the three age groups, only problems with 1–5 moves were considered.

Both MSPM1 and MSPM2 were fitted to the data of each age group by using the knowledge space $\mathcal {K}$ and goal space $\mathbf {P}$ of the ToL introduced in Section 4.

Hypotheses (H1)–(H3) were tested by comparing both the absolute and the relative goodness-of-fit of the models. The absolute goodness-of-fit of the models was tested by the Pearson Chi-square statistic. In particular, a parametric bootstrap procedure (Efron, Reference Efron1979) over 500 replications was used to compute the p-value of the Chi-square. When both models fitted the data, the AIC index and Bayes Factor were used to select the best model. The Bayes factor was approximated using the Bayesian information criterion (BIC; Schwarz, Reference Schwarz1978).

Concerning the response time analysis, the entire time spent to complete a problem was split into three sub-times. The first one is the “pre-planning time,” which is the time spent from the presentation of the task to the first click on a bead. The second one is the “move-execution time,” which is the time spent from the click on the bead to the click on the peg. The third one is the “move-planning time,” which is the time spent from the click on a peg to the click on a new bead. It follows that the sum of these three times equals the total time spent to complete the problem. Thus, for each participant, a standardized version of the three sub-times can be obtained by dividing each of them by the total time. Only the response times of problems correctly solved were considered in this analysis. Moreover, average values across test-takers were obtained separately for each age group.

Standardizing sub-times in this way allows for obtaining indexes whose range is in the interval $[0,1]$ , and that express the average proportion of time spent on the specific “action.” Since the total time spent to solve a problem strictly depends on the number of moves, these indexes might be misleading if used for comparing the response behavior of the same group (as well as of different groups) to different problems. Nonetheless, these indexes should provide useful insights when used for comparing the response behavior of different age groups while keeping the problem fixed.

5.5 Discussion

MSPM1 and MSPM2 are based on different assumptions about how planning occurs during problem solving. MSPM1 assumes that planning takes place predominantly at the onset of the solution process, leading to a more rigid execution of an initial strategy. In contrast, MSPM2 allows for multiple instances of planning throughout the task, supporting a more flexible and adaptive approach.

The superior fit of MSPM2 in the 4–8 age group suggests that younger children do not rely solely on an initial plan but rather adjust their strategy dynamically as they progress. In contrast to this age group, the results for older participants indicate a shift in planning behavior. For individuals aged 9–13, both MSPM1 and MSPM2 exhibit a good fit to the data, but model comparison criteria favor MSPM1. The lower AIC value and the strong Bayes factor in favor of MSPM1 suggest that, in this age range, planning tends to occur primarily at the onset of the solution process, with less reliance on dynamic adjustments during task execution. A similar pattern emerges in the 14–57 age group, where MSPM1 remains the preferred model based on AIC and Bayes factor comparisons.

These findings align with the idea that, as individuals develop greater cognitive control and experience with problem solving, they rely more on pre-planned strategies and execute solutions with reduced need for mid-task adjustments. This developmental trend supports the interpretation that younger children engage in more iterative and corrective planning, whereas older individuals increasingly favor an upfront, goal-directed approach to problem solving.