Model CM1a: Randomization Device

In model CM1a, the baseline item in the data collection concerns the outcome of a randomization device available to the survey participant, as typically done with RRTs (Mirzazadeh et al., Reference Mirzazadeh, Shokoohi, Navadeh, Danesh, Jain, Sedaghat, Farnia and Haghdoost2018). For instance, the baseline item may be whether “the outcome of a die roll (available to survey participants) is a 2”. Hence, the responses to the two statements in the CM are conditionally independent, i.e. $P(U_i=1|Z_i)=P(U_i=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1 | Z_{i} ) = {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1 ) $$\end{document} in Eq. 1. Thus,

$\begin{array}{cc}P(Y_i=1)& =P(U_i=1)P(Z_i=0)+P(U_i=0)P(Z_i=1)\\ P(Y_i=0)& =P(U_i=1)P(Z_i=1)+P(U_i=0)P(Z_i=0)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1)&= {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1 ) {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 0)+{{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 0 ) {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1) \\ {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 0)&= {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1 ) {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1)+{{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 0 ) {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 0) \end{aligned}$$\end{document}

Given prevalence of the target item $\pi =P(U_i=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi = {{\,\mathrm{\mathbb {P}}\,}}(U_i=1)$$\end{document} , the likelihood for individual response $Y_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i$$\end{document} is:

(2) $\begin{array}{c}L\left(Y_i\right|\pi ,\delta )=P{(Y_i=1|\pi ,\delta )}^{Y_i}\times {(1-P(Y_i=1|\pi ,\delta ))}^{1-Y_i}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}(Y_i|\pi , \delta ) = {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1|\pi , \delta )^{Y_i} \times (1-{{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1|\pi , \delta ) )^{1-Y_i} \end{aligned}$$\end{document}

where the prevalence of the baseline item $P(Z_i=1)=\delta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1) = \delta $$\end{document} is the probability of the specified outcome of the randomization device. Then, $\delta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta $$\end{document} is a fixed parameter known to the analyst. For instance, in the example given above $\delta =1/6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta =1/6$$\end{document} . The probability of having the sensitive target item can be computed as follows:

(3) $\begin{array}{c}\pi =P(U_i=1)=\frac{P(Y_i=1)-P(Z_i=1)}{1-2P(Z_i=1)}=\frac{P(Y_i=1)-\delta }{1-2\delta }\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \pi = {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1) = \frac{{{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1) - {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1)}{1 - 2 {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1)} = \frac{{{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1) - \delta }{1 - 2 \delta } \end{aligned}$$\end{document}

The probability of affirming the sensitive item can then be estimated. A caveat is that the target response is not identified if $P(Z_i=1)=\delta =50\%$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1) = \delta = 50 \%$$\end{document} (under Eq. 3). Additionally, analysts are often interested in relating the sensitive target item to observed covariates denoted as $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_i$$\end{document} , that is, they are interested in $P(U_i=1|x_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1| \textbf{x}_i)$$\end{document} . The probability of affirming the target item then varies with the survey participant’s characteristics. Using a probit link function, where $\Phi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi (\cdot )$$\end{document} is the normal cumulative distribution function, this becomes:

(4) $\begin{array}{c}P(U_i=1|x_i,\beta )=\Phi ({\beta }_0+x_i^{⊺}{\beta }_X)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1| \textbf{x}_i, \varvec{\beta }) = \Phi (\beta _0 + \textbf{x}_i^\intercal \varvec{\beta }_X) \end{aligned}$$\end{document}

Past research has documented various downsides of using randomization devices to protect privacy, such as trust and understanding issues (Landsheer et al., Reference Landsheer, Van Der Heijden and Van Gils1999; John et al., Reference John, Loewenstein, Acquisti and Vosgerau2018; Wolter and Preisendörfer, Reference Wolter and Preisendörfer2013). This motivates model CM1b, using baseline items not involving a randomization device.

Model CM1b: Assumed Prevalence

Model CM1b is used, to the best of our knowledge, in almost all applications of the CM (Sagoe et al., Reference Sagoe, Cruyff, Spendiff, Chegeni, De Hon, Saugy, van der Heijden and Petróczi2021; Schnell and Thomas, Reference Schnell and Thomas2021). Typical baseline items for data collection of model CM1b are “My mother’s birthday is in January or February” or “My address number begins with 6”. The responses to these baseline items aim to be equivalent to the outcomes of a randomization device. Thus, although this approach is statistically equivalent to model CM1a, the implementation differs.

Model CM1b is straightforward, since the probability of affirming the item can be treated as known. However, the model presents several limitations (Sayed et al., Reference Sayed, Cruyff, van der Heijden and Petróczi2022). First, if survey participants do not perceive items such as one’s day or month of birth, or number of the street address as privacy protecting, then they might not comply with the instructions. For instance, they might worry that the analyst might have access to this, for instance, via web-scraping or earlier waves in case the study is part of longitudinal research (Jerke et al., Reference Jerke, Johann, Rauhut and Thomas2019). This can be prevented by asking impersonal items like “An acquaintance’s birthday is in January or February” (Höglinger and Jann, Reference Höglinger and Jann2018; Jann et al., Reference Jann, Jerke and Krumpal2011). However, these baseline items might appear weird (Kuha and Jackson, Reference Kuha and Jackson2014), thus increasing participants’ likelihood of answering randomly and thereby the measurement error of the baseline item (Höglinger and Diekmann, Reference Höglinger and Diekmann2017). Second, researchers may often be interested in using more than one CM question, for instance when they want to collect data on multiple sensitive issues or when using an anchoring item to correct for non-compliance (Sayed et al., Reference Sayed, Cruyff, van der Heijden and Petróczi2022; Atsusaka and Stevenson, 2021). If multiple CMs are administered repeatedly with similar baseline items, as is common (Höglinger and Jann, Reference Höglinger and Jann2018; Roberts and John, Reference Roberts and John2014)), participants might worry that analysts can deduce the answers to the target item from the combination of answers to these repeated items. Finally, the assumed birthday probability may be incorrect for a specific sample, for instance when birthday dates are clustered in time or unknown (Sayed et al., Reference Sayed, Cruyff, van der Heijden and Petróczi2022). To prevent these issues, it is important to use baseline items that participants perceive as (1) private, (2) neither salient nor weird, and (3) that do not repeatedly involve the same domain. For instance, Sayed et al. (Reference Sayed, Cruyff, van der Heijden and Petróczi2022) propose using a number sequence randomizer that improves much upon use of birthday items.

Model CM3a: Item Response Theory Specification

This section provides the formal details of statistical model CM3a. The model accommodates the fact that the response variables are binary (items in the CM) and binary or ordered (items outside-the-CM). We assume that H outside-the-CM polytomous items are asked directly elsewhere in the questionnaire. The outside-the-CM items and the baseline item reflect some individual latent trait, denoted as ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _{i}$$\end{document} . We employ a specific parametric function to link the trait and the responses, but the model can also be customized for alternative distributions. The observed score to the outside-the-CM items $S_{\mathrm{ih}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{ih}$$\end{document} is modeled as: (Samejima, 1969):

(5) $\begin{array}{c}P(S_{\mathrm{ih}}=c|{\theta }_i,{\alpha }_h,{\gamma }_h)=\Phi ({\alpha }_h{\theta }_i-{\gamma }_{h,c-1})-\Phi ({\alpha }_h{\theta }_i-{\gamma }_{h,c})\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(S_{ih} = c| \theta _{i}, \alpha _{h}, \varvec{\gamma }_{h}) = \Phi ( \alpha _{h} \theta _{i} - \gamma _{h, c-1}) - \Phi ( \alpha _{h} \theta _{i} - \gamma _{h, c}) \end{aligned}$$\end{document}

where $\Phi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \Phi (\cdot )$$\end{document} denotes the normal cumulative distribution function. The model specifies the conditional probability of a graded response $S_{\mathrm{ih}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{ih}$$\end{document} in category $c\in \{1,\cdots ,C\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c \in \{1,\cdots , C \}$$\end{document} . The specification in Eq. 5 maps a latent trait parameter ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \theta _{i}$$\end{document} of individual i and item-specific parameters ${\alpha }_h$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \alpha _{h}$$\end{document} and ${\gamma }_h$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{\gamma }_{h}$$\end{document} into an observed pattern of responses. A “trait” here is a general term to indicate some underlying latent construct, such as a personality trait, value, norm or attitude. ${\alpha }_h$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \alpha _{h}$$\end{document} denotes the discrimination parameter and ${\gamma }_h$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{\gamma }_{h}$$\end{document} denotes the threshold or difficulty parameter (Fox, 2010). The response to the baseline item in the pair is modeled as:

(6) $\begin{array}{c}P(Z_i=1|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}})=\Phi ({\alpha }_{\mathrm{bas}}{\theta }_i-{\gamma }_{\mathrm{bas}})\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1 | \theta _{i}, \alpha _{bas}, \gamma _{bas}) = \Phi (\alpha _{bas} \theta _{i} - \gamma _{bas}) \end{aligned}$$\end{document}

If no covariates are available to predict the target item and assuming conditional independence of $Z_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} and $U_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} , then $P(U_i=1|Z_i)=P(U_i=1)=\pi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1|Z_i) = {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1) = \pi $$\end{document} . As earlier, we define $Y_i=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i = 1$$\end{document} if individual i replies that either statement is true when answering the CM and $Y_i=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i = 0$$\end{document} if individual i replies that both or none of the two statements is true. The probability of answering $Y_i=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i = 1$$\end{document} is thus:

(7) $\begin{array}{c}P(Y_i=1|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}},\pi )=(1-\pi )\times P(Z_i=1|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}})+\pi \times P(Z_i=0|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}})\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1| \theta _{i}, \alpha _{bas}, \gamma _{bas}, \pi ) = (1-\pi ) \times {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1|\theta _{i}, \alpha _{bas}, \gamma _{bas})+\pi \times {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 0|\theta _{i}, \alpha _{bas}, \gamma _{bas}) \end{aligned}$$\end{document}

Notice that model CM3a does not require the assumption of invariance of the baseline item when estimating the prevalence $\pi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} , in contrast to model CM2. This is because with model CM2 the analyst simply plugs in $P(Z_i=1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(Z_i = 1)$$\end{document} in Eq. 1, after having estimated it in a separate control group. In contrast, with model CM3a the analyst jointly estimates the most likely answers to both the target and the baseline items, conditional on the information obtained from the outside-the-CM items. The likelihood for individual responses $Y_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i$$\end{document} and $S_{\mathrm{ih}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{ih}$$\end{document} is:

$\begin{array}{cc}& L(Y_i,S_{\mathrm{ih}}|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}},\pi ,{\alpha }_h,{\gamma }_h,\mu ,\sigma )\\ & =\underset{\Theta }{\int }P{(Y_i=1|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}},\pi )}^{Y_i}{(1-P(Y_i=1|{\theta }_i,{\alpha }_{\mathrm{bas}},{\gamma }_{\mathrm{bas}},\pi ))}^{1-Y_i}\\ & \left(\prod _{h=1}^H,\prod _{c=1}^C,P,{(S_{\mathrm{ih}}=c|{\theta }_i,{\alpha }_h,{\gamma }_h)}^{1[S_{\mathrm{ih}}=c]}\right)\varphi \left({\theta }_i\right|\mu ,\sigma )d\theta \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathcal {L}(Y_i, S_{ih}|\theta _{i}, \alpha _{bas}, \gamma _{bas}, \pi , \alpha _{h}, \varvec{\gamma }_{h},\mu ,\sigma ) \\ {}&\quad = \int \limits _{\Theta } {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1|\theta _{i}, \alpha _{bas}, \gamma _{bas}, \pi )^{Y_i} (1-{{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1|\theta _{i}, \alpha _{bas}, \gamma _{bas}, \pi ))^{1-Y_i} \\&\quad \left[ \prod _{h=1}^H \prod _{c=1}^C {{\,\mathrm{\mathbb {P}}\,}}(S_{ih} = c| \theta _{i}, \alpha _{h}, \varvec{\gamma }_{h})^{1[S_{ih} = c]} \right] \phi (\theta _i | \mu , \sigma ) d \theta \end{aligned}$$\end{document}

where $\varphi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi (\cdot )$$\end{document} is the pdf of a Gaussian distribution. Because we rely on variation in the latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} to identify the threshold parameter ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} , stronger identification assumptions are required on the discrimination parameter ${\alpha }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{bas}$$\end{document} , ensuring that ${\alpha }_{\mathrm{bas}}\gg 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{bas}\gg 0$$\end{document} . As explored in the simulations section, the use of flat or non-informative priors results in a small bias in the estimate of the prevalence $\pi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} . This is resolved by eliciting more informative priors, which can be formulated based on existing scale information (Mikkola et al., 2021). Informative priors can be constructed based on external information on the items, such as discrimination parameters of published scales. Alternatively, the items can be administered directly to a separate sample and the resulting IRT estimates can be used to formulate a suitable prior. This is conceptually similar to Model CM4a (presented later), but does not make any assumption on the threshold parameter ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} .

The model can be extended to allow for dependence between the target item and the baseline item as follows. Suppose that the analyst is interested in relating the latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} , as well as other covariates of interest $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_i$$\end{document} to the sensitive target item $U_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_{i}$$\end{document} (“antecedents”). Using a probit formulation, the probability of answering the target item $P(U_i=1|{\theta }_i,x_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1| \theta _i, \textbf{x}_i)$$\end{document} affirmatively is:

(8) $\begin{array}{c}P(U_i=1|{\theta }_i,x_i,\beta )=\Phi ({\beta }_0+{\beta }_{\theta }{\theta }_i+x_i^{⊺}{\beta }_X)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1| \theta _i, \textbf{x}_i, \varvec{\beta }) = \Phi (\beta _0 + \beta _\theta \theta _i + \textbf{x}_i^\intercal \varvec{\beta }_X ) \end{aligned}$$\end{document}

Then, $P(U_i=1|Z_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\mathrm{\mathbb {P}}\,}}(U_{i} = 1 | Z_{i} )$$\end{document} from Eq. 1 can be estimated, since the answer to $Z_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} depends on ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} (De Jong and Pieters, Reference De Jong and Pieters2019). We examine later using simulations to what extent the model parameter ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} can be correctly estimated. Figure 2a presents model CM3a in a directed acyclic graph when assuming conditional independence, and Fig. 2b presents model CM3a in a directed acyclic graph when relating some antecedents covariates $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_{i}$$\end{document} and the latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} to the sensitive target response. Extensions and model modifications are possible. Whereas we use a Gaussian link for Eqs. 5, 6 and 8, different response functions may be used, such as logistic or partial credit models. Further, one could alternatively predict the baseline item based on other covariates, which is examined in the next section.

Figure 2 DAGs for model CM3a. Note The latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} underlies the responses to the outside-the-CM items 1,..., H and to the baseline item $Z_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} . The baseline item $Z_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z_i$$\end{document} and the target item $U_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} jointly determine the CM outcome $Y_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i$$\end{document} . In the left model the latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} and other covariates $X_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_i$$\end{document} predict the target item $U_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} . DAGs for model CM3a.

Model CM3b: Binary Response Model

Model CM3a assumes that baseline and outside-the-CM items are selected from a validated scale. However, the baseline item might not necessarily come from such a scale, yet may be predicted on the basis of some observed set of covariates $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_i$$\end{document} (Kuha and Jackson, Reference Kuha and Jackson2014). Given a link function $g(·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g(\cdot )$$\end{document} , such as $g(·)=\Phi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g(\cdot ) = \Phi (\cdot )$$\end{document} :

(9) $\begin{array}{c}P(Z_i=1|x_i,\delta )=g({\delta }_0+x_i^{⊺}{\delta }_{\mathrm{bas}})\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1 | \textbf{x}_i, \varvec{\delta }) = g(\delta _0 + \textbf{x}_i^\intercal \varvec{\delta }_{bas}) \end{aligned}$$\end{document}

As with Model CM3a, the baseline item is administered only within the CM, thus by using an indirect question. The probability of answering $Y_i=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_i = 1$$\end{document} is:

(10) $\begin{array}{c}P(Y_i=1|\pi ,x_i,\delta )=(1-\pi )\times P(Z_i=1|x_i,\delta )+\pi \times P(Z_i=0|x_i,\delta )\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {{\,\mathrm{\mathbb {P}}\,}}(Y_i = 1| \pi , \textbf{x}_i, \varvec{\delta }) = (1-\pi ) \times {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 1 | \textbf{x}_i, \varvec{\delta }) + \pi \times {{\,\mathrm{\mathbb {P}}\,}}(Z_{i} = 0 | \textbf{x}_i, \varvec{\delta }) \end{aligned}$$\end{document}

The model can be easily extended to predict the target item the using available covariates. As with Model CM3a, alternative parametric functions may be used. In the simulation section, we discuss to what extent Models CM3a and CM3b can correctly estimate the prevalence of the target item.

p-Groups Design to Minimize Variance

If the response invariance assumption is not a concern, a p-groups design is the most efficient way to minimize the estimated variance of the target item in the CM. The use of p-groups designs has been proposed for List Experiments (Glynn, Reference Glynn2013; Blair and Imai, Reference Blair and Imai2012). In a p-group design, the analyst splits the sample in p groups of equal size. Each group receives the same target item but a different baseline item. Each baseline item is then asked directly in the other groups. If based on model CM4a, the baseline and the outside-the-CM items should all come from the same validated scale. If the analyst does not wish to use a baseline item from a validated scale, model CM4b can be used for analysis, relating the baseline item to other observed variables. The p-group design improves efficiency over all other models and also allows detecting potential violations of model assumptions. This is explored more in detail in the next four MC simulations, examining the performance of the various models.

Analysis of Models CM3a and CM3b

In this section, we show that models CM3a and CM3b correctly recover the target prevalence, although their performance hinges on how accurately the outcomes of the baseline item are predicted. The experimental design is 3 (prevalence $\pi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi $$\end{document} ) by 3 (model CM3a with either non-informative or informative priors, CM3b) by 3 (reliability / pseudo- $R^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} ). We vary the reliability of the scale for model CM3a as $\{.6,.7,.8\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{.6,.7,.8\}$$\end{document} and the pseudo $R^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} for model CM3b as $\{.1,.2,.3\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{.1,.2,.3\}$$\end{document} . The baseline threshold parameter ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} /intercept ${\delta }_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _0$$\end{document} is fixed to 1.

We examine two separate implementations of model CM3a: one with non-informative prior for the discrimination parameter ( ${\alpha }_{\mathrm{bas}}\sim \text{Uniform}(0,4)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{bas} \sim \text {Uniform}(0, 4)$$\end{document} , Fig. 3a) and one with informative priors, based on the scale reliability (Fig. 3b)Footnote 4. Informative priors for the baseline discrimination parameters can be assumed since the items are selected from validated scales with known reliability. Plausible ranges for the discrimination parameters are often available as well in published studies. Results for Model CM3b are shown in Fig. 3c. The figures on the left show the 95 % confidence interval of estimated prevalences across simulations, assuming a true prevalence of 10 % (subfigures i), 20 % (subfigures ii) and 30 % (subfigures iii). Thus, all models seem generally suitable to correctly estimate the assumed prevalence if the reliability is above .6–.7 or the pseudo- $R^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} is above .2–.3. There seems to be a small bias with model CM3a when the reliability is low; this is mitigated by using more informative priors on the discrimination parameter ${\alpha }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{bas}$$\end{document} . We emphasize that we do not use an informative prior for the location parameter ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} , and hence, the model makes weaker assumptions than model CM4a. Subfigures iv show a decrease of the mean-squared error if the reliability or the pseudo- $R^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} increases.

Figure 3 Estimation of prevalence of target item with models CM3a and CM3b.

Comparison of Models CM3a and CM4a: Violations of Response Invariance and Bias

The previous simulations assume that survey participants evaluate the baseline item similarly, regardless of whether it is administered directly or in a CM (that is, jointly with the target item). In the third MC simulation, the bias is examined when this assumption of response invariance is violated. We focus on the IRT models CM3a and CM4a. The following model parameters are used: ${\gamma }_{\mathrm{bas}}=-1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas} = -1$$\end{document} , reliability.7 (Fig.  4a) and.8 (Fig.  4b). In the x-axis of the figures, ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} gradually changes only in the separate control (DQ) group by increments of .2. A larger change (e.g., $\pm .6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm .6$$\end{document} ) corresponds to a more severe violation of response invariance.

Figure 4 Average bias squared, PPC and DIC when violating invariance with models CM3a and CM4a.

In subfigures i, the estimation bias in model CM4a increases as the deviation from response invariance worsens. This is because the prevalence estimate of the baseline item in the control group differs from its estimate in the CM. In contrast, model CM3a correctly estimates the prevalence of the target item. This is because it does not rely on estimates from a separate control group, but only on within-participant information.

Predictive Checks and Information Criteria to Detect Invariance

Violating the assumption of response invariance will worsen performance of models CM4 (as well as CM2). Model CM3a is robust to invariance. However, model CM4a is typically more efficient than model CM3a. How can we check whether model assumptions are satisfied and select an appropriate model? With models CM1 and CM2, the response invariance assumption is untestable. However, p-values of posterior predictive checks (PPC) can be used to alert to violations of invariance with model CM4a. These p-values indicate whether the model predictions can replicate the observed percentage of “same/different” answers to the CM (Gelman et al., 1996). The reported p-values should not be extremely different from.5 when applying a model (e.g., less than.1 or more than.9). Subfigures ii (in Fig. 4) show the average PPC for model CM4a given changes of the parameter ${\gamma }_{\mathrm{bas}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _{bas}$$\end{document} in the DQ group. When response invariance is violated more severely (e.g., for a value of the x-axis close to $\pm .6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm .6$$\end{document} ) the PPC for model CM4a approaches 0 or 1. PPC can also be used to detect violations of the model assumptions beyond response invariance, as explored later.

Although model CM3a is more robust to the response invariance assumption, it is typically less efficient than model CM4a, as it does not leverage information from a separate DQ group. The deviance information criterion (DIC) can be used for model selection (Spiegelhalter et al., Reference Spiegelhalter, Best, Carlin and Van Der Linde2002). Subfigures iii show the DIC for models CM3a and CM4a. If the violation of response invariance is mild, the DIC favors using model CM4a, due to its higher efficiency. However, if the squared bias increases, the DIC favors using model CM3a, as it is robust to violations of response invariance.

Conditional Independence of Target and Baseline Items

Conditional independence of the target and baseline items simplifies identification of the prevalence of the target item. This section explores to what extent it is possible to test and account for dependence using IRT modeling, assuming that the baseline and target item are related as specified in Eq. 8. Violations of this “conditional independence” assumption can also be detected with PPC checks for model CM3a and CM4a (without modeling the relationship between the latent trait ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _i$$\end{document} and the latent target item $U_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_i$$\end{document} ). We vary the coefficient ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} in Eq. 8 between − .5 and .5, leading to more or less dependence between the target and baseline items. ${\beta }_{\theta }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }=0$$\end{document} implies statistical independence. Figure 5 shows the average PPC criterion across simulations for models CM3a and CM4a, using both single group (p = 1) and 2-groups design (p = 2). Hence, model CM4a is most sensitive to violations of the conditional independence assumptions when using 2 groups: for instance, the average PPC is more (less) than .95 (.05) when the coefficient ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} is larger (smaller) than.25 (− .25). The PPC of Model CM3a with a 2-group design is also sensitive to violations of conditional independence, but less so than model CM4a.

Figure 5 Bayesian posterior predictive check (PPC) when violating conditional independence (varying value of coefficient ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} in x-axis).

We next study whether each model is suitable to estimate dependency between the target and baseline item, assuming that the probit model of Eq. 8 is correctly specified. We assume a probit link with parameters: ${\beta }_0=-1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _0 = -1$$\end{document} , ${\beta }_{\theta }=.4$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta } =.4$$\end{document} . We then also include a single regressor $x_i\sim N(0,1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_i \sim \mathcal {N}(0,1)$$\end{document} with parameter ${\beta }_1=-.4$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1} = -.4$$\end{document} (subfigures c and d). We test models CM2, CM3a and CM4a in a single group design (p = 1), and models CM3a and CM4a with a 2-groups design (p = 2). We use discrimination parameters corresponding to reliability of.7 (shown in Fig. 6) and.8 (shown in Fig. 7).

Figure 6 Estimation of statistical dependence with reliability. 7.

Figure 7 Estimation of statistical dependence with reliability .8.

Subfigures a and c show to what extent we can recover the correct population prevalence of the target item. The subfigures reveal that when the two items (baseline and target) are correlated, model CM2 is unsuitable to recover the correct target item prevalence. In this scenario, the bias is upward because ${\beta }_{\theta }=.4>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta } =.4 > 0$$\end{document} . The bias becomes smaller when using models CM3a and CM4a, but it is negligible only when implemented in a 2-groups design. Subfigures b and d examine estimation of the coefficients in the probit regression model. The regression is not estimated for model CM2 in subfigure b since there is no regressor. When estimating models CM3a and CM4a there is a downward bias for the coefficient ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} , which is reduced when using scales with higher reliability (e.g., see subfigure 5d). This suggests that the coefficient ${\beta }_{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{\theta }$$\end{document} can also be used to test for (linear) dependence between the target and the baseline item under IRT modeling. However, some care must be taken with this test since a) it requires collecting data with a p-groups design and b) the simulations suggests that it may be biased toward zero, if using scales with lower reliability.

The simulation experiments provide the following insights. First, models CM4a and b improve efficiency over the more commonly used model CM1 and consequently CM2. Second, models CM3a and b can correctly estimate the prevalence of the target item, without relying on external information about the prevalence of the baseline item via a control group. However, their performance crucially depends on the reliability of the scale used or accurate prediction of the baseline item: we recommend using scales of reliability .7 – .8. Third, only model CM3 is robust to violations of response invariance. Fourth, and finally, IRT models CM3a and CM4a can detect and model potential association between the baseline and target item, although we have to acknowledge that their performance also depends on correct parametric specification.