3.1 Notation and problem formulation

For the entirety of the article, we assume that the calibration dataset $\boldsymbol {Y}$ is binary, where each element $y_{ij}$ represents whether subject i answered item j correctly. We consider a general two-parameter MIRT framework with K latent factors (Bock & Gibbons, Reference Bock and Gibbons2021). Let $\boldsymbol {B} \in \mathbb R^{J \times K}$ denote the factor loading matrix, and $\boldsymbol {D} \in \mathbb R^J$ denote the intercept vector. Throughout the article, boldface notation is used to denote vectors and matrices (e.g., $\boldsymbol {\theta }, \boldsymbol {B}$ , and $\boldsymbol {D}$ ), while scalar quantity, such as $D_{j}$ , is written in standard font. For each examinee i with multivariate latent trait $\boldsymbol {\theta }^{(i)} \in \mathbb R^K$ , the data-generating process for $y_{ij}$ is given by

(3.1) $$ \begin{align} y_{ij} \mid \boldsymbol{\theta}^{(i)}, \boldsymbol{B}_j, D_j \sim \text{Bernoulli}(\Phi(\boldsymbol{B}_j'\boldsymbol{\theta}^{(i)} + D_j)), \end{align} $$

where $\boldsymbol {B}_j$ is the jth row of $\boldsymbol {B}$ , $D_j$ is the jth entry of $\boldsymbol {D}$ , and $\Phi (\cdot )$ denotes the standard normal cumulative distribution function. The item parameters can be compactly expressed as $\{\boldsymbol {\xi }_j\}_{j=1}^J := \{(\boldsymbol {B}_j, D_j)\}_{j=1}^J$ . This two-parameter MIRT framework is highly general, imposing no structural constraints on the loading matrix and requiring no specific estimation algorithms for item parameter calibration.

Given the estimated item parameters, we need to design an item-selection algorithm that efficiently tests a future examinee with unobserved latent ability $\boldsymbol {\theta } \in \mathbb R^K$ . The sequential nature of the CAT problem makes Bayesian approaches particularly appealing. Without loss of generality, we assume a standard multivariate Gaussian prior on $\boldsymbol {\theta } \sim \mathcal {N}(0, \mathbb I_K)$ and introduce the following notation:

• • For any positive integer n, let $[n]$ denote all the positive integers no greater than n. Let $j\in [J]$ be the index for the items in the item bank.

• • Let $j_t$ denote the index of the item selected at step t based on an arbitrary item-selection rule. Define $\mathcal {I}_{t}$ as the set of the first t administered items, and let $R_{t} := [J] \setminus \mathcal {I}_{t-1} $ , the set of available items before the tth item is picked.

• • We use the shorthand notation $f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:T})$ to represent the latent factor posterior distributions $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:T}, \boldsymbol {\xi }_{1:T})$ after T items have been selected. Here, the response history is denoted by $\boldsymbol {Y}_{1:T}:= [y_{j_1}, \dots , y_{j_T}]'$ , and the item parameters are given by $\boldsymbol {\xi }_{1:T} := (\boldsymbol {B}_{1:T}, \boldsymbol {D}_{1:T})$ , where $\boldsymbol {B}_{1:T}:=[\boldsymbol {B}_{j_1}, \dots , \boldsymbol {B}_{j_T}]'$ and $\boldsymbol {D}_{1:T}:=[D_{j_1}, \dots , D_{j_T}]'$ .

At time $(t+1)$ , the algorithm takes the current posterior $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:t})$ as input and outputs the next item selection $j_{t+1}$ . This process continues until at time $T'$ , either when the posterior variance of $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:T'})$ falls below a predefined threshold $\tau ^2$ , or when $T'=H$ , where $H \leq J$ is the maximum number of items that can be administered. Hence, the goal of CAT is to minimize $T'$ given the estimated item parameters of the item bank.

3.2 Reviews of KL information item-selection rules

Given that our proposed deep CAT system is built on a general Bayesian MIRT framework (Li et al., Reference Li, Gibbons and Rockova2025), we briefly revisit common Bayesian item-selection rules and show in Section 4.1 how our framework can be used to accelerate these baseline methods. The popular KL expected a priori (EAP) rule selects the tth item based on the average KL information between response distributions on the candidate item at the EAP estimate $\hat {\boldsymbol {\theta }}_{t-1} = \int _{\boldsymbol {\theta }} \boldsymbol {\theta } f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:(t-1)}) d\boldsymbol {\theta }$ , and random factor $\boldsymbol {\theta }$ sampled from the posterior distribution $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)})$ (Veldkamp & van der Linden, Reference Veldkamp and van der Linden2002):

(3.2) $$ \begin{align} \arg\max_{j_t \in R_t} \int_{\boldsymbol{\theta}} \bigg\{ \sum_{l=0}^{1} P(y_{j_t}=l | \hat{\boldsymbol{\theta}}_{t-1}) \log \frac{ P(y_{j_t}=l | \hat{\boldsymbol{\theta}}_{t-1})}{ P(y_{j_t}=l | \boldsymbol{\theta})} \bigg\}\ f(\boldsymbol{\theta} | \boldsymbol{Y}_{1:(t-1)}) d\boldsymbol{\theta}. \end{align} $$

Rather than focusing on the KL information based on the response distributions, Mulder and van der Linden (Reference Mulder and van der Linden2010) propose the MAX Pos approach by maximizing the KL information between two subsequent latent factor posteriors $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)})$ and $f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:t})$ . Intuitively, this approach prioritizes items that induce the largest shift in the posterior, formalized as

(3.3) $$ \begin{align} \arg\max_{j_t \in R_t} \sum_{y_{j_t}=0}^{1} f(y_{j_t}| \boldsymbol{Y}_{1:(t-1)}) \int_{\boldsymbol{\theta}} f(\boldsymbol{\theta}|\boldsymbol{Y}_{1:(t-1)}) \log \frac{f(\boldsymbol{\theta}|\boldsymbol{Y}_{1:(t-1)})}{f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:t})} d\boldsymbol{\theta}, \end{align} $$

where $f(y_{j_t}| \boldsymbol {Y}_{1:(t-1)})$ represents the posterior predictive probability:

(3.4) $$ \begin{align} f(y_{j_t}| \boldsymbol{Y}_{1:(t-1)}) = \int_{\boldsymbol{\theta}} f(y_{j_t}| \boldsymbol{\theta}) f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) d\boldsymbol{\theta}. \end{align} $$

A third Bayesian strategy, the MI approach, maximizes the MI between the current posterior distribution and the response distribution of new item $y_{j_{t}}$ (Weissman, Reference Weissman2007). Rooted in experimental design theory (Rényi, Reference Rényi1961), we can also interpret MI as the entropy reduction of the current posterior distribution of $\boldsymbol {\theta }$ after observing new response $y_{j_t}$ . More formally,

(3.5) $$ \begin{align} \arg\max_{j_t \in R_t} I_M (\boldsymbol{\theta}, y_{j_t}) = \arg\max_{j_t \in R_t} \sum_{y_{j_t}=0}^{1} \int_{\boldsymbol{\theta}} f(\boldsymbol{\theta}, y_{j_t} | \boldsymbol{Y}_{1:(t-1)}) \log \frac{f(\boldsymbol{\theta}, y_{j_t} | \boldsymbol{Y}_{1:(t-1)})}{f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) f(y_{j_t} | \boldsymbol{Y}_{1:(t-1)})} d\boldsymbol{\theta}. \end{align} $$

Given the impressive empirical success of the MI method (Wang & Chang, Reference Wang and Chang2011), we also introduce a competitive heuristic item-selection rule that chooses items with the highest predictive variance under the current posterior estimates, serving as an additional benchmark method. Formally, write $c_{j_t} := f(y_{j_t} | \boldsymbol {Y}_{1:(t-1)})$ as defined in (3.4), and consider the following criterion:

(3.6) $$ \begin{align} \arg\max_{j_t \in R_t} \int_{\boldsymbol{\theta}} (\Phi(\boldsymbol{B}_{j_t}' \boldsymbol{\theta} + D_{j_t})-c_{j_t})^2 f(\boldsymbol{\theta}|\boldsymbol{Y}_{1:(t-1)}) d\boldsymbol{\theta}. \end{align} $$

This rule selects the item $j_t$ that maximizes the variance of the predictive means, weighted by the current posterior $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ . In Appendix A of the Supplementary Material, we establish a connection between this selection rule and the MI method, showing that both favor items with high prediction uncertainty.

3.3 Reinforcement learning formulation

Existing item-selection rules face several limitations: first, these methods rely on one-step-lookahead selection, choosing items that provide the most immediate information without considering their impact on future selections, which can result in suboptimal policies (Sutton & Barto, Reference Sutton and Barto2018). Second, they are heuristically designed to balance information across all latent factors, rather than emphasizing the most essential factors of interest. Finally, these rules do not directly minimize test length, potentially increasing test duration without proportional gains in accuracy.

A more principled approach is to formulate item selection as an RL problem, where the optimal policy is learned using Bellman optimality principles rather than relying on heuristics that ignore long-term planning. Beyond addressing myopia, an RL-based formulation provides a direct mechanism to minimize the number of items required to reach a predefined posterior variance reduction threshold, explicitly guiding item selection toward accurately measuring the primary factors of interests.

More generally, consider a general finite horizon setting where each examinee can answer at most $H \leq J$ items, and the CAT algorithm terminates whenever the posterior variance of all factors of interest is smaller than a certain threshold $\tau ^2$ , or when H is reached. Since H can be set sufficiently large, it serves as a practical secondary stopping criterion to prevent excessively long tests. Formally:

• • State space $\ \mathcal {S}$ : The space of all possible latent factor posteriors $\mathcal {S} := \{f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:t}): t \in \{1, \ldots , H\}\}.$ The state variable in CAT represents the Bayesian estimate of the examinee’s latent traits as a multivariate distribution at time t. We may write $s_t := f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:t})$ , with $s_0 := \mathcal {N}(0, \mathbb {I}_K)$ . The state space is discrete and can be exponentially large. Since there are J items in the item bank and the responses are binary, the total number of possible states is $\sum _{t=1}^H \binom {J}{t} \cdot 2^t$ .

• • Action space $\ \mathcal {A}$ : The item bank $[J]:= \{1, \ldots , J\}$ . However, at the tth selection time, the action space is the remaining items in the test bank that have not yet been selected $R_t:= [J] \setminus \mathcal {I}_{t-1}$ , as we do not select the same item twice.

• • Transition kernel $\ \mathcal {P}: \mathcal {S} \times \mathcal {A} \rightarrow \Delta (\mathcal {S})$ , where $\Delta (\mathcal {S})$ denotes the space of probability distributions over $\mathcal {S}$ . The transition kernel $\mathcal {P}$ specifies the stochastic rule that governs how the state evolves after an action is taken. In the CAT context, the kernel maps the current latent-trait estimate $s_t$ and the selected item $a_t$ to the next posterior state $s_{t+1}$ . Conditional on $(s_t, a_t)$ , the next estimate $s_{t+1}$ depends on the examinee’s binary response $y_t \in \{0,1\}$ to item $a_t$ , which yields only two possible posterior updates. Because the true ability of the examinee and thus the response probability $p(y_t = 1 \mid s_t, a_t)$ are unknown, the transition kernel in CAT is also unknown and must be approximated.

• • Reward function: To directly minimize the test length, we assign a simple 0–1 reward structure, where we assign negative rewards whenever more items are needed to reduce posterior variance to a given threshold. In a K-factor MIRT model, we often prioritize a subset of factors $\mathcal {K} \subset [K]$ (e.g., $\mathcal {K} = \{1\}$ for pCAT-COG), with the test terminating once the posterior variances of all factors in $\mathcal {K}$ fall below the predefined threshold $\tau ^2$ :

  (3.7) $$ \begin{align} R^{(t)}(s_t, a, s_{t+1}) = \begin{cases} -1 & \text{if } V_{t+1}> \tau^2, \\ 0 & \text{otherwise}, \end{cases} \end{align} $$
  where $V_{t+1} = \max _{k \in \mathcal {K}} \operatorname {Var}(\boldsymbol {\theta }_k \mid \boldsymbol {Y}_{1:(t+1)}) $ is the maximum marginal posterior variance among the prioritized factors $\mathcal {K}$ . This also simplifies the learning of the value function, as the rewards are always bounded integers within $[-H, -1]$ . We further illustrate the advantages of adopting this reward structure in Appendix E of the Supplementary Material.

Rather than following a heuristic rule, we learn a policy $\pi $ that maps the current state (latent factor estimates) to a distribution over potential items:

$$\begin{align*}\pi:\ \mathcal{S}\to\Delta(\mathcal{A}),\qquad a_t \sim \pi(\cdot \mid s_t). \end{align*}$$

Given an initial state $S_0=s_0$ and the discount factor $\gamma \in (0,1]$ , the value function is

(3.8) $$ \begin{align} v_{\pi}(s_0) := \mathbb{E}_{\pi} \left[\sum_{t=0}^{H-1} \gamma^t R^{(t)}(s_t, \pi(s_t), s_{t+1}) | S_0 = s_0 \right]. \end{align} $$

The expectation $\mathbb {E}_\pi [\cdot ]$ is taken over trajectories generated by drawing $a_t$ from $\pi (\cdot \mid s_t)$ and then $s_{t+1}$ from the transition kernel $\mathcal {P}(\cdot \mid s_t,a_t)$ . Because direct evaluation of (3.8) is challenging, it is more convenient to consider the action-value function

(3.9) $$ \begin{align} Q_{\pi}(s_0, a) := \mathbb E_{\pi} \left[\sum_{t=0}^{H-1} \gamma^{t}R^{(t)} (s_t, \pi(s_t), s_{t+1}) | S_0=s_0, A_0= a \right], \end{align} $$

with $v_\pi (s)=\mathbb {E}_{a\sim \pi (\cdot \mid s)}Q_\pi (s,a)$ . The optimal policy $\pi ^{\star }$ satisfies the Bellman equation (Bertsekas & Tsitsiklis, Reference Bertsekas and Tsitsiklis1996):

(3.10) $$ \begin{align} Q_{\pi^\star}(s,a)\;=\;\mathbb{E}_{S'\sim\mathcal{P}(\cdot\mid s,a)}\!\left[\,R(s,a,s')+\gamma\,\max_{a'} Q_{\pi^\star}(s',a')\,\right], \end{align} $$

where $s'$ is the next posterior estimate after applying action a to the current posterior s, and $v_{\pi ^\star }(s)=\max _{a} Q_{\pi ^\star }(s,a)$ . Since the state space grows exponentially and the transition kernel is unknown, solving for $\pi ^*$ using traditional dynamic programming approaches becomes intractable (Sutton & Barto, Reference Sutton and Barto2018).

In practice, our deep Q-learning approach (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013) does not evaluate these expectations analytically. Let the transition $(s,a,r,s')$ represent one testing step, where item a is selected under state s, the reward r is observed, and the posterior is updated to $s'$ through the Bayesian MIRT model. We approximate the Bellman fixed point in (3.10) by fitting a parametric function $Q_w$ that minimizes the squared temporal-difference loss:

(3.11) $$ \begin{align} \mathcal{L}(w) = \mathbb{E}_{(s,a,r,s')\sim \mathcal{D}} \big[\big(Q_w(s,a) - y(s,a,r,s')\big)^2\big], \end{align} $$

where

(3.12) $$ \begin{align} y(s,a,r,s') = r + \gamma \max_{a'} Q_{\bar{w}}(s',a'). \end{align} $$

Here, $\mathcal {D}$ (replay buffer) denotes a collection of previously observed testing steps $(s,a,r,s')$ generated during simulation, which is used to approximate the expectation above by empirical averaging. The parameter $\bar {w}$ corresponds to a delayed copy of the model parameters w, updated less frequently to stabilize the numerical optimization. We describe the full Q-learning algorithm in Section 5.

4.1 Accelerating existing rules

While Bayesian CAT criteria require multidimensional integration, these quantities are typically evaluated in practice via Monte Carlo approximation rather than analytic quadrature in moderate to high dimensions. In such Bayesian CAT implementations, the dominant computational cost lies in obtaining valid samples from the latent factor posterior. By providing exact posterior samples without Markov chain construction, our approach removes this primary bottleneck and renders the remaining Monte Carlo integration computationally efficient and easily parallelizable. For example, in computing the KL-EAP item-selection rule (3.2), we directly sample from $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ instead of resorting to MCMC and evaluate the integral via Monte Carlo integration. Since the EAP estimate $\hat {\boldsymbol {\theta }}_{t-1}$ remains fixed at time step t, evaluating the KL information term is straightforward. For the Max Pos item-selection rule in Equation (3.3), we again obtain i.i.d. samples from $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ and compute the posterior predictive probabilities in (3.4). Although the density ratio $\frac {f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})}{f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:t})}$ is difficult to evaluate, we can leverage the conditional independence assumption of the MIRT model. In particular, we can express the joint distribution of $(\boldsymbol {\theta }, y_{j_t})$ as

(4.2) $$ \begin{align} f(\boldsymbol{\theta}, y_{j_t} | \boldsymbol{Y}_{1:(t-1)}) = f(\boldsymbol{\theta} | \boldsymbol{Y}_{1:(t-1)}, y_{j_t}) f(y_{j_t} | \boldsymbol{Y}_{1:(t-1)}) = f(y_{j_t}| \boldsymbol{\theta}) f(\boldsymbol{\theta} | \boldsymbol{Y}_{1:(t-1)}). \end{align} $$

Using Equation (4.2), we rewrite the KL information term in Equation (3.3) as

$$ \begin{align*} \int f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) \log \frac{f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)})}{f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}, y_{j_t})} d\boldsymbol{\theta} &\ = \int f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) \log \frac{f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) f(y_{j_t}| \boldsymbol{Y}_{1:(t-1)})}{f(\boldsymbol{\theta}, y_{j_t} | \boldsymbol{Y}_{1:(t-1)})} d\boldsymbol{\theta} \\ &\ = \int f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}) \log \frac{f(y_{j_t}| \boldsymbol{Y}_{1:(t-1)})}{f(y_{j_t} | \boldsymbol{\theta})} d\boldsymbol{\theta}. \end{align*} $$

Since $f(y_{j_t}|\boldsymbol {\theta })$ can be easily computed for each $\boldsymbol {\theta } \sim f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)})$ , the online computation of Max Pos remains efficient.

Although the MI selection rule in Equation (3.5) has demonstrated strong empirical performance (Wang & Chang, Reference Wang and Chang2011), its computational complexity remains a significant challenge. By applying Equation (4.2), we can rewrite MI as

(4.3) $$ \begin{align} \arg\max_{j_t \in R_t} \sum_{y_{j_t}=0}^{1} f(y_{j_t}| \boldsymbol{Y}_{1:(t-1)}) \int_{\boldsymbol{\theta}} f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}, y_{j_t} ) \log \frac{f(y_{j_t} | \boldsymbol{\theta})}{ f(y_{j_t} | \boldsymbol{Y}_{1:(t-1)})} d\boldsymbol{\theta}. \end{align} $$

This formulation reveals that maximizing MI is structurally similar to Max Pos but requires much more computational effort. Unlike Max Pos, where sampling is performed from the current posterior $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ , MI requires sampling from future posteriors $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)}, y_{j_t})$ . Since each candidate item $j_t \in R_t$ has two possible outcomes ( $y_{j_t} = 0$ or $y_{j_t} = 1$ ), evaluating Equation (4.3) requires obtaining samples from $|R_t| \times 2$ distinct posterior distributions. Even if sampling each individual posterior is computationally efficient, this approach becomes impractical for large item banks.

We hence propose a new approach to dramatically accelerate the computation of the MI quantity using the idea of importance sampling and resampling and bootstrap filter (Gordon et al., Reference Gordon, Salmond and Smith1993; Smith & Gelfand, Reference Smith and Gelfand1992). Rather than explicitly sampling the future posterior $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)}, y_{j_t})$ for each $j_t \in R_t$ and $y_{j_t} \in \{0, 1\}$ , we can simply sample from the current posterior $f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:(t-1)})$ once, and then perform proper posterior reweighting. Under Equation (3.1), let $p(\boldsymbol {\theta })$ be the prior on the latent factors $\boldsymbol {\theta }$ , $l_1(\boldsymbol {\theta })$ be the current data likelihood, and $l_2(\boldsymbol {\theta })$ denote the future data likelihood after observing $y_{j_t}$ . We have the future posterior density as follows:

(4.4) $$ \begin{align} f(\boldsymbol{\theta}| \boldsymbol{Y}_{1:(t-1)}, y_{j_t}) & \propto l_2(\boldsymbol{\theta}) p(\boldsymbol{\theta}) \propto \frac{l_2(\boldsymbol{\theta})p(\boldsymbol{\theta})}{l_1(\boldsymbol{\theta})p(\boldsymbol{\theta})} f(\boldsymbol{\theta} | \boldsymbol{Y}_{1:(t-1)}) \notag \\ & = \Phi(\boldsymbol{B}_{j_t}'\boldsymbol{\theta} + D_{j_t})^{y_{j_t}}(1- \Phi(\boldsymbol{B}_{j_t}'\boldsymbol{\theta} + D_{j_t}))^{1- y_{j_t}} f(\boldsymbol{\theta} | \boldsymbol{Y}_{1:(t-1)}). \end{align} $$

Equation (4.4) suggests that we can generate samples from $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:(t-1)}, y_{j_t})$ via reweighting and resampling from the current posterior samples $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ . Specifically, given a sufficiently large set of posterior samples $(\boldsymbol {\theta }_1, \ldots , \boldsymbol {\theta }_M) \sim f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)})$ , we assign distinct weights for each sample $m \in \{1, \ldots , M\}$ and future item $j_t \in R_t$ :

$$ \begin{align*}q_m = \frac{w_m}{\sum_{i=1}^M w_i}, \quad \text{where } w_i= \Phi(\boldsymbol{B}_{j_t}'\boldsymbol{\theta}_i + D_{j_t})^{y_{j_t}}(1- \Phi(\boldsymbol{B}_{j_t}'\boldsymbol{\theta}_i + D_{j_t}))^{1- y_{j_t}}.\end{align*} $$

To sample from $f(\boldsymbol {\theta }|\boldsymbol {Y}_{1:(t-1)}, y_{j_t})$ , we then draw from the discrete distribution over $(\boldsymbol {\theta }_1, \ldots , \boldsymbol {\theta }_M)$ , placing weight $q_m$ on $\boldsymbol {\theta }_m$ . This approach eliminates the need to sample from $|R_t| \times 2$ distinct posteriors directly, further accelerating MI computation and making it scalable for large item banks.

5.1 Deep Q-learning neural network design

A potential reason that a deep RL approach has not been proposed in the CAT literature is the ambiguity arising from unidentified latent factor distributions. However, by Theorem 4.1, it is straightforward to compactly parameterize the posterior distribution at each time step t using the parameters $\boldsymbol {C}_1 \in \mathbb R^{t \times K}$ , $\boldsymbol {C}_2 \in \mathbb {R}^t$ , and $\text {diag}(\boldsymbol {C}_3) \in \mathbb R^t$ , where $\text {diag}(\boldsymbol {C}_3)$ represents the diagonal vector of $\boldsymbol {C}_3$ . This structured representation of the posterior enables item-selection policy learning via deep neural networks, which takes posterior parameters as input and outputs item selection.

Neural networks can be regarded as flexible function approximators that learn nonlinear mappings between inputs and outputs through a composition of simple transformations (Goodfellow et al., Reference Goodfellow, Bengio and Courville2016). In our framework, the deep neural network consists of two key components: an encoder and a classifier. At each time step t, the encoder maps the collection of posterior parameters to a latent representation in $\mathbb {R}^{L}$ , where L is a hyperparameter and represents the dimension of the latent feature space. The classifier then outputs a J-dimensional vector of estimated Q-values and selects the next item from the item bank that is expected to yield the highest Q-value.

Define the collection of the posterior parameters $\tilde {\boldsymbol {\xi }}_t:= \{(\boldsymbol {C}_{1h}, C_{2h}, C_{3h})\}_{h=1}^{t}$ , where $\boldsymbol {C}_{1h} \in \mathbb {R}^{K}$ is the hth row of $\boldsymbol {C}_1$ , $C_{2h} \in \mathbb {R}$ is the hth element of $\boldsymbol {C}_2$ , and $C_{3h} \in \mathbb {R}$ is the hth elements in the diagonal of $\boldsymbol {C}_3$ . Since the size of the posterior parameters $\tilde {\boldsymbol {\xi }}_t$ grows over time, and permuting the tuples within $\tilde {\boldsymbol {\xi }}_t$ still describes the same posterior, we have to design a neural network that can take inputs of growing size, and can provide output that is permutation invariant of the inputs. One solution is to consider the weight sharing idea from the Bayesian experimental design literature (Foster et al., Reference Foster, Ivanova, Malik and Rainforth2021). Let $\phi _1(.): \mathbb {R}^{K+2} \rightarrow \mathbb {R}^{L_1}$ denote an encoder component that maps each tuple $(\boldsymbol {C}_{1h}, C_{2h}, C_{3h}) \in \mathbb {R}^{K+2}$ to an $L_1$ -dimensional latent space, and consider the operation $g_1(.)$ as follows:

(5.1) $$ \begin{align} g_1(\tilde{\boldsymbol{\xi}}_t) := \sum_{h=1}^t \phi_1 \{(\boldsymbol{C}_{1h}, C_{2h}, C_{3h})\}. \end{align} $$

Observe that $g_1(.)$ is capable of handling a growing number of inputs through summations of $\phi _1(.)$ functions over t. More importantly, permuting the order of the tuples in $\tilde {\boldsymbol {\xi }}_t$ does not change the value of $g_1(.)$ , since summation is permutation invariant. Although the form of $g_1$ may look restrictive, any function $f(.)$ operating on a countable set can be decomposed into the form $\rho \circ g_1(.)$ , where $\rho (.)$ is a suitable transformation that can be learned from another neural network (see Theorem 2 of Zaheer et al., Reference Zaheer, Kottur, Ravanbhakhsh, Póczos, Salakhutdinov and Smola2017).

In principle, the network design $\rho \circ g_1(.)$ , coupled with the Q-learning algorithm, is sufficient to learn the optimal policy. However, to enhance learning efficiency, we further enrich the state representation $\tilde {\boldsymbol {\xi }}_t$ with a matrix of prediction quartiles $\boldsymbol {\Psi }_t \in \mathbb R^{J \times Q}$ , where each row contains a vector of quantiles of the predictive distribution for item j. We form these quantiles by first drawing samples $\boldsymbol {\theta }_{1}, \ldots , \boldsymbol {\theta }_M \sim f(\boldsymbol {\theta } | \boldsymbol {Y}_{1:t})$ as in Section 4, and then computing the quantiles of the prediction samples $\{\Phi (\boldsymbol {B}_j'\boldsymbol {\theta }_i + D_j)\}_{i=1}^M$ for each item j. Since sampling from $f(\boldsymbol {\theta }| \boldsymbol {Y}_{1:t})$ only needs to be done once, computing the matrix $\boldsymbol {\Psi }_t$ online is computationally efficient.

This practice of augmenting the raw state variable with additional contextual features of the states ( $\boldsymbol {\Psi }_t$ ) echoes the common strategy in the RL literature (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglou, King, Kumaran, Wierstra, Legg and Hassabis2015; Schaul et al., Reference Schaul, Horgan, Gregor, Silver, Bach and Blei2015): learning tends to be more stable and efficient when the state representation captures not only the current ability estimate but also characteristics of the item bank. In our setting, $\boldsymbol {\Psi }_t$ offers a richer description of the predictive distributions across items given the current estimates, effectively serving as additional covariates for item selection. Empirically, we find that incorporating $\boldsymbol {\Psi }_t$ substantially accelerates convergence toward the optimal item-selection policy. Including $\boldsymbol {\Psi }_t$ in the state representation is harmless, as it is a deterministic function of the posterior distribution; identical posteriors will always produce identical $\boldsymbol {\Psi }_t$ , ensuring that the policy remains consistent across equivalent states.

In summary, Figure 2 describes the architecture of the policy network used in our approach. The same network is trained offline and then deployed during live CAT with fixed weights, so only the posterior state is updated sequentially during online administration. To select the $(t+1)$ th item, our proposed neural net takes two inputs: the posterior parameters $\tilde {\boldsymbol {\xi }}_t$ and the prediction matrix $\boldsymbol {\Psi }_t$ , and outputs the selected item. Let $\phi _2: \mathbb {R}^{J \times Q} \to \mathbb {R}^{L_2}$ represent the encoder component that maps the matrix $\boldsymbol {\Psi }_t \in \mathbb R^{J \times Q}$ to $L_2$ -dimensional space. Write $L=L_1+L_2$ and the concatenated outputs of $g_1(.)$ and $\phi _2(.)$ as $[g_1(\tilde {\boldsymbol {\xi }}_t)', \phi _2(\boldsymbol {\Psi }_t)']' \in \mathbb {R}^L $ , we then define the classification component of neural network as $\rho (.): \mathbb {R}^L \rightarrow \mathbb {R}^J$ , which maps the concatenated outputs of $g_1(.)$ and $\phi _2(.)$ to a J-dimensional logit vector. Denote the final policy network as $\pi _{\phi }$ and recall that $\rho (.)$ represents the classification layer, our proposed network selects the item at time t corresponding to the maximum value of the following function:

$$ \begin{align*} \pi_{\phi}(\tilde{\boldsymbol{\xi}}_t, \boldsymbol{\Psi}_t) := \rho \circ \left\{ [g_1(\tilde{\boldsymbol{\xi}}_t)', \phi_2(\boldsymbol{\Psi}_t)' ]' \right\}. \end{align*} $$

We implement both primary and target Q-networks as simple feed-forward multilayer perceptron and find that their performance is largely insensitive to the exact choice of $L_1$ and $L_2$ , and does not require very large depth. All architectural details, such as layer sizes, activations, and optimizer settings, are provided in Appendix D of the Supplementary Material. This robustness suggests that our empirical improvements stem from the CAT-specific state design and augmentation rather than network complexity.

Figure 2

High-level architecture of the Q-network. The shared encoder $\phi _1$ maps each tuple of posterior parameters to $\mathbb {R}^{L_1}$ and the sum yields the permutation invariant representation $g_1(\tilde {\boldsymbol {\xi }}_t)$ . The matrix $\boldsymbol {\Psi }_t$ is encoded by $\phi _2$ . The concatenated vector in $\mathbb {R}^{L}$ is passed to the classifier $\rho $ to select the jth item (largest value in the J logits). This network is trained offline using Algorithm 1; during live CAT, its weights are fixed and only the posterior state is updated sequentially.

5.2 Double deep Q-learning for CAT

In classical tabular Q-learning, the action-value function is stored as a Q-matrix whose rows correspond to states and whose columns correspond to actions. Such a representation is infeasible in CAT because the state space is combinatorially large and the state variable $s_t$ is represented by continuous posterior summaries rather than a small finite index. We therefore replace the tabular Q-matrix by a neural-network approximator $Q_w(s,a)$ . For a given state $s_t$ , the network in Figure 2 outputs the full vector of estimated Q-values before selecting the optimal item, which plays the same role as one row of a tabular Q-matrix, but is produced through function approximation rather than table lookup.

The standard Q-learning algorithm often overestimates Q-values because the same action-value approximation is used both to select the maximizing action and to evaluate it when constructing the temporal-difference target. To address this issue, we adopt the double Q-learning approach (van Hasselt et al., Reference van Hasselt, Guez and Silver2016), which replaces the single action-value approximation by two Q-networks with distinct roles. The primary network $Q_w$ is updated by gradient descent and is used to select the action with the largest estimated Q-value, while the target network $Q_{\bar w}$ is a delayed copy used only to evaluate that selected action when forming the target. This is the neural-network analog of maintaining two Q-matrices in double Q-learning. The separation reduces overestimation bias and leads to more stable training. Specifically, compared with standard Q-learning, double Q-learning minimizes the same TD loss in (3.11) but replaces the target in (3.12) with

$$\begin{align*}y(s,a,r,s') \;=\; r + \gamma\, Q_{\bar{w}}\!\big(s',\, \operatorname*{arg\,max}_{a'} Q_{w}(s',a')\big). \end{align*}$$

Our proposed algorithm is presented in Algorithm 1. The algorithm trains, entirely offline, the policy network illustrated in Figure 2. Once training is complete, the learned network is deployed during live CAT with fixed weights, and only the posterior state is updated sequentially as new responses are observed. Importantly, the offline training phase in Algorithm 1 relies only on the item parameters $(\boldsymbol {B},\boldsymbol {D})$ and simulated examinees drawn from a specified latent factor distribution (e.g., $\boldsymbol {\theta }_i \sim \mathcal {N}(0, \mathbb {I}_K)$ ), rather than on observed item response data from the item bank. If prior information about the distribution of future online examinees is available, the simulation distribution can be adapted accordingly. In this manuscript, we consistently use a standard multivariate normal distribution to ensure comparability across methods and experiments.

Note that for the Q-learning to converge to an optimal policy, it is essential to adopt an $\epsilon $ -greedy policy, where the algorithm makes random item selections with probability $\epsilon $ and gradually decreases $\epsilon $ over the course of training. For simplicity, the state variable $s_t$ in Algorithm 1 is a compact representation of $(\tilde {\boldsymbol {\xi }}_t, \boldsymbol {\Psi }_t)$ defined in Section 5.1. In practice, we can terminate the training when both the rewards and the validation loss stabilize, indicating that the neural network has well approximated the optimal policy.

6.1 Simulation design

To evaluate our approach in a challenging and realistic simulation setting, we randomly generated a $5$ -factor, $150$ -item factor loading matrix $\boldsymbol {B} \in \mathbb R^{150 \times 5}$ . Each column of $\boldsymbol {B}$ was initialized by randomly permuting $150$ equally spaced values, with the magnitude constrained to lie within $[0.3, 3]$ . We then imposed a lower triangular structure to ensure identifiability. To better reflect practical datasets, where items rarely load on all five factors, each item was required to load on the first factor and on at most two additional factors beyond the first. A visualization of the loading matrix can be found in Appendix F of the Supplementary Material. Item intercepts were independently drawn from $\text {Unif}(-1.5, 1.5)$ .

The goal of the simulation is to accurately estimate the first three factors ( $ \mathcal {K} = \{1,2,3\} $ ), while accounting for the presence of factors $4$ and $5$ . Leveraging our proposed direct sampling approach outlined in Section 4, we implemented our double Q-learning algorithm alongside all existing methods discussed in Section 3.2. These include the EAP approach (Equation (3.2)), the Max Pos approach (Equation (3.3)), the MI approach (Equation (3.5)), and the Max Var approach (Equation (3.6)). We further modified all baseline information-based criteria so that they also target only the prioritized subset of factors $\mathcal {K}=\{1,2,3\}$ . Specifically, the EAP, Max Pos, and MI rules are now written as integrals over $\boldsymbol {\theta }_{\mathcal {K}}$ rather than all five factors. These adjustments ensure that all competing CAT algorithms are tuned to the same estimation target, making the simulation comparisons fair and directly comparable.

For performance evaluation, we generated $N=500$ online examinees with latent traits $\boldsymbol {\theta }_i \stackrel {\mathrm {i.i.d.}}{\sim } \mathcal {N}(0, \mathbb {I}_5)$ . For each examinee, we administered a $50$ -item test using each of the CAT algorithms. We then compared performance in terms of posterior variance reduction, mean-squared error (MSE), termination efficiency, and item exposure rates based on these $500$ adaptive testing sessions.

For the Q-learning algorithm, we trained the Q-network for 80,000 episodes, with exploration parameter $\epsilon $ decreasing linearly from $0.99$ to $0.01$ over 700,000 steps. We set a sufficiently large $H=60$ items, with discount factor $\gamma =0.95$ . To verify that the double Q-learning algorithm indeed converges to the optimal policy $\pi ^*$ , we provide further details on the training dynamics of our deep Q-learning algorithm, illustrating the increase in rewards until convergence in Appendix F.2 of the Supplementary Material.

6.2 Simulation results

To evaluate termination efficiency, we start with a standard multivariate Gaussian prior with unit marginal variances and terminate the test once the maximum posterior variance across all three target factors in $\mathcal {K}$ falls from $1$ to below $\tau ^2 < 0.16$ . In practice, practitioners may select the threshold $\tau ^2$ based on the specific requirements of their application. Based on $500$ simulated adaptive testing sessions, Figure 3 illustrates how the percentage of completed test sessions increases as more items are administered. A faster growth rate of the completion percentages indicates faster posterior variance reduction. Notably, the purple Q-learning curve consistently rises more quickly than the others, demonstrating significant testing efficiency gains. As summarized in the second column of Table 1, Q-learning requires an average of only $21.5$ items to reach the termination criterion for all three targeted factors, outperforming all other methods.

Figure 3

Number of items versus cumulative percentage of completed tests.

Table 1

Comparison of win shares (W.S), termination, and computation

To assess early-stage performance, we compute the MSEs after $20$ items for each test session and summarize results using win shares, defined as the percentage of test takers for whom each method achieves the lowest MSE after $20$ items. These win shares are reported across all three latent dimensions in Table 1. Notably, Q-learning achieves the best win shares across all three factors. This advantage at early stages is consistent with the training objective of Q-learning, which explicitly rewards rapid posterior variance reduction and early test termination, with the average stopping time occurring at around $20$ items.

Table 2 reports the evolution of MSEs as a function of test length up to $50$ administered items. Consistent with the win share analysis, Q-learning exhibits competitive performance during the early stages of testing. In particular, at $T=20$ and $T=30$ , Q-learning attains the smallest MSEs for Factors $2$ and $3$ . As the test length increases, performance differences across methods diminish, and Q-learning achieves MSEs comparable to those of MI and Max Var. This pattern is expected, as the Q-learning policy is trained primarily on short-horizon testing trajectories: since most simulated tests terminate before $30$ items, the algorithm is exposed less frequently to long-horizon simulations during training. Additional visualizations of the MSE trajectories are provided in Appendix F.3 of the Supplementary Material.

Table 2

MSEs between posterior mean and ground truth for the first three latent factors as a function of test length

Figure 4 summarizes the distribution of item exposure rates across the $150$ item bank for each CAT algorithm, where the exposure rate of an item is defined as the proportion of examinees whose test includes that item at least once. With test length $H=50$ and bank size $J=150$ , the expected average exposure is $H/J \approx 0.33$ , and all five algorithms yield mean and median exposure rates close to this benchmark. Differences across methods primarily arise in the tails of the exposure distribution. In particular, the EAP and Q-learning policies exhibit slightly higher upper quartile and maximum exposure rates, indicating a more selective use of highly informative items. Importantly, no method concentrates exposure excessively on a small subset of items, suggesting that all algorithms maintain reasonable item utilization in this simulation setting. For the Q-learning approach, incorporating explicit exposure control mechanisms into the learning objective, such as penalizing repeated use of highly exposed items, is a natural extension but is beyond the scope of the present study.

Figure 4

Distributions of item exposure rates.

The last column of Table 1 reports the average online item-selection time, highlighting the significant computational advantages of our direct sampling approach. Even the most computationally intensive MI approach requires only $0.082$ seconds per item selection, via our proposed posterior reweighting strategy. Q-learning adds only a single feed-forward pass per selection, keeping online latency under a few milliseconds on standard hardware without GPU acceleration. While training the Q-network offline using Algorithm 1 took approximately $30$ hours for this exercise, this is a one-time investment that can be accelerated if GPUs are available; thereafter, the virtually instantaneous online policy makes the approach highly practical for real-time CAT deployment.

7.1 pCAT-COG data and experiment design

We revisit the problem of designing a deep CAT system for the pCAT-COG study, as outlined in Section 2. Since item response data for all $N = 730$ examinees across $J = 57$ items are available, we can directly use real item responses during evaluation rather than simulating testing sessions. In this experiment, all $N=730$ examinees are treated as online examinees. All CAT algorithms, including Q-learning, have access only to the estimated item parameters from the pCAT-COG study and do not observe the examinees’ binary item responses beyond those revealed sequentially during adaptive administration.

Given that pCAT-COG is designed to measure global cognitive ability (first column in Figure 1) while accounting for five cognitive subdomains, we specified the Q-learning reward function to reduce posterior variance for the primary factor ( $\mathcal {K} = \{1\}$ ). This demonstrates the flexibility of our CAT system, as it can be tailored to the specific cognitive assessment needs. As in Section 6, we also modified all baseline CAT algorithms so that they also target only the primary factor by integrating only over the primary dimension for fair comparison.

Finally, this real-data experiment also serves as a robustness check for the proposed Q-learning approach. As described in Algorithm 1, the policy is trained using simulated examinees with latent factors drawn from a standard multivariate normal distribution. In contrast, the empirical factor correlations in pCAT-COG need not follow this distributional assumption. Evaluating Q-learning on real response data therefore provides evidence that the learned policy remains effective when the latent factor structure deviates from the training distribution used in simulation.