Setup

The DLEs in Alvarez et al. (Reference Alvarez, Atkeson, Levin and Li2019) keep the order of baseline lists fixed and randomize the location of the sensitive item. This is not the only admissible version of the DLE. Researchers can choose whether to randomize the order of lists and the location of the sensitive item. The possible combinations are outlined in Table 2.

Table 2.

DLE variants

The fixed-fixed design is inadmissible since it does not include any experimental manipulation. The randomized-fixed design changes the order of baseline lists but fixes the sensitive item, usually in the second list. This design is not compatible with the proposed tests because the location of the sensitive item does not vary. One may consider this desirable since it prevents respondents from altering responses to the second list after seeing the sensitive item in the first list. The proposed tests only apply for designs that manipulate the location of the sensitive item, fixed-randomized and randomized-randomized. One may consider adopting these designs, at least at the piloting stage, to justify the choice of baseline lists through the proposed tests.

The purpose of the tests is to detect strategic misreporting that violates the assumptions of a DLE. The no design effects assumption states that the inclusion of a sensitive item does not alter the responses to the baseline items (see Blair and Imai Reference Blair and Imai2012 for details). Ceiling and floor effects are common violations to this assumption. If none or all of the baseline items apply to a respondent, then answering truthfully may entail revealing the sensitive item to the researcher unequivocally, which betrays anonymity. This may lead respondents with answers at the extremes to deflate or inflate their responses (Miller Reference Miller1984; Kuklinski, Cobb, and Gilens Reference Kuklinski, Cobb and Gilens1997). ^{Footnote 4} The conventional advice to avoid ceiling and floor effects is to prevent extreme answers while crafting baseline lists, for example, by inducing negative correlation between items (Glynn Reference Glynn2013).

This paper focuses on the strategic misreporting that happens when the inclusion of a sensitive item in the first list of a DLE leads respondents to alter responses in both lists. I term this carryover design effects. Ceiling or floor effects are unlikely to produce them, since they pertain to the distribution of items within a baseline list.

Instead, carryover design effects happen when the inclusion of a sensitive item leads respondents to interpret baseline items differently (Miller Reference Miller1984). In this context, response deflation happens when the sensitive item is a frowned-upon attitude, such as admitting to racial prejudice. Similarly, response inflation happens when the absence of the sensitive behavior is frowned upon, such as supporting the regime in a dictatorship. In either case, deflation/inflation happens because the sensitive item stands out to respondents, which cues them on the attitude or behavior of interest. This leads to altered responses in the baseline items that are associated with the sensitive item. These effects can happen regardless of the number of baseline items that apply to a respondent, and even among those who do not hold the sensitive trait.

Carryover design effects happen in DLEs because list experiment questions have a distinct format, tend to appear close to each other in survey flows, and the advice of using lists with paired items to induce positive correlation across lists for the sake of precision (Glynn Reference Glynn2013). These factors allow respondents to connect items across lists when the sensitive item stands out. For example, in Alvarez et al. (Reference Alvarez, Atkeson, Levin and Li2019), supporters of the American Family Association (list A) may also support the Tea Party Patriots (list B). The inclusion of an anti-immigration organization as the sensitive item in the first list may alert respondents about the researcher’s interest on support for conservative organizations, which may lead to response deflation in both lists if the sensitive item appears first.

Fixed-randomized and randomized-randomized DLE designs allow diagnosis of carryover design effects. Let ${Y_{i1}} = {z_i}{Y_{i1}}\left( 1 \right) + \left( {1 - {z_i}} \right){Y_{i1}}\left( 0 \right)$ be individual $i$ ’s observed response to the first list, and ${Y_{i2}} = \left( {1 - {z_i}} \right){Y_{i2}}\left( 1 \right) + {z_i}{Y_{i2}}\left( 0 \right)$ the observed response to the second list, with ${z_i}$ indicating whether a respondent sees the sensitive item first. At the individual level, the researcher only observes the paired responses $\left[ {{Y_{i1}}\left( 1 \right),{Y_{i2}}\left( 0 \right)} \right]$ or $\left[ {{Y_{i1}}\left( 0 \right),{Y_{i2}}\left( 1 \right)} \right]$ . These reflect treatment schedules with the sensitive item appearing first or second, respectively. Under the first schedule, respondents can react to the sensitive item in both questions. Under the second schedule, respondents can only react to the sensitive item in ${Y_{i2}}$ .

To illustrate carryover design effects. Table 3 shows the answers of a hypothetical respondent. The first two columns denote the observed responses under different scenarios. As a baseline, the respondent answers the same number in both questions. For now, assume that the sensitive item does not apply to this individual and that they do not engage in any kind of misreporting.

Table 3.

An illustration of strategic responses in a DLE

Now imagine that the respondent does not hold the sensitive trait but still seeks to avoid association with it by deflating (inflating) their response by one unit. In this case, the observed responses depend on the placement of the sensitive item. When ${z_i} = 1$ , the relationship between ${Y_{i1}}\left( 1 \right)$ and ${Y_{i2}}\left( 0 \right)$ stays the same because both shift in the same direction. However, under ${z_i} = 0$ , the respondent alters their response only in the second list. In this example, responses become further apart because the baseline counts were the same.

More generally, with similar baseline lists, carryover design effects under the ${z_i} = 1$ schedule lead to changes in responses in the same direction, although not necessarily in the same magnitude. However, the ${z_i} = 0$ schedule only allows for strategic misreporting in the second list. The goal of the tests is to detect this asymmetric shift across treatment schedules.

Difference in differences

This test compares whether mean responses vary across treatment schedules. The quantities

(1) $${\hat \tau _1} = {1 \over {{N_{11}}}}\sum _{i = 1}^N{z_i}{Y_{i1}} - {1 \over {{N_{01}}}}\sum _{i = 1}^N(1 - {z_i}){Y_{i1}}$$

and

(2) $${\hat \tau _2} = {1 \over {{N_{12}}}}\sum _{i = 1}^N(1 - {z_i}){Y_{i2}} - {1 \over {{N_{02}}}}\sum _{i = 1}^N{z_i}{Y_{i2}}$$

denote the difference in means between responses with and without the sensitive item for the first and second list, with ${N_{\rm{*}}}$ as the sample size for the treatment and control groups in each question. ^{Footnote 5}

The null hypothesis is that the two differences in means are equal, ${H_0}:{\hat \tau _1} - {\hat \tau _2} = 0$ . For a fixed-randomized DLE, ${\hat \tau _1}$ and ${\hat \tau _2}$ correspond to the single-list prevalence estimates, and the test is equivalent to the consistency test proposed by Chuang et al. (Reference Chuang, Dupas, Huillery and Seban2021). For the randomized-randomized design, this quantity is the difference in differences in means between responses to the first and second question instead.

Since the control group in the first question has not seen the sensitive item yet, the sign of the test statistic depends mainly on ${\hat \tau _1}$ . A negative test statistic suggests deflation, while a positive value suggests inflation. Calculating the difference in differences is straightforward, but the computation of p-values must consider the clustered structure of the data, since each participant has two responses. Both randomization inference and linear regression with clustered standard errors accommodate this structure.

Signed rank test

The alternative test evaluates whether one can attribute extreme differences in paired responses to the variation in treatment schedules. Rosenbaum (Reference Rosenbaum2007, Reference Rosenbaum2020) proposes Stephenson’s (Reference Stephenson1981) signed rank test to detect heterogeneous effects in pair-randomized experiments. ^{Footnote 6}

The test applies to DLEs since responses are paired by participant. The test statistic is

(3) $$\tilde T= \sum _{i = 1}^N{\mathop{\rm sgn}} \{ ({z_i} - (1 - {z_i}))({Y_{i1}} - {Y_{i2}})\} \times {\tilde q_i}$$

which is the sum of signed ranks ${\tilde q_i}$ , defined as

(4) $${\tilde q_i} = {q_i} - 1 \matrix{ {} \cr {m - 1} \cr } {\rm{for}}\ {q_i} \ge m; {\tilde q_i}= 0\ {\rm{for}} \ {q_i} \gt m$$

with ${q_i}$ denoting the rank of the absolute difference in paired responses $\left| {{Y_{i1}} - {Y_{i2}}} \right|$ . So ${\tilde q_i}$ records the number of possible subsets of size $m$ in the data in which $\left| {{Y_{i1}} - {Y_{i2}}} \right|$ is the largest.

The choice of $1 \le m \le N$ determines what counts as an extreme difference. For example, with $m = 2$ the test is equivalent to Wilcoxon’s signed rank, but with ranks ranging from $0$ to $n - 1$ . As $m$ increases, more ranks are considered zero and more weight goes to large differences. The choice of $m$ is arbitrary, but researchers can use simulations at the pre-analysis stage to find the value that maximizes the power of the test. ^{Footnote 7} In the application and simulations, I report only $m = 10$ to facilitate exposition. Section D of the appendix reports results under additional values and gives an example of how to calibrate $m$ .

Without ties in ranks, $\tilde T$ is a distribution-free statistic, meaning its p-values are known in advance. With ties, one can compute exact p-values in small samples, while the analytical derivation is a good approximation for experiments with large samples (Rosenbaum Reference Rosenbaum2020).

To illustrate the behavior of this test statistic, consider the last two columns of Table 3. Under the baseline, treatment assignment does not change responses, so the sign of ${\tilde q_i}$ flips randomly and $\tilde T$ is zero in expectation. The only way $\tilde T$ can be negative is in the presence of response deflation. In the example, the first treatment schedule does not contribute to the test statistic, but the second schedule adds negatively.

One limitation of this test is that $\tilde T$ can be positive in the presence of either response deflation or a non-zero prevalence rate, leading to false positives. In the example, response inflation exhibits the mirror pattern of response deflation, now contributing to a positive test statistic. However, as the last two rows show, a respondent who reports the sensitive item without strategic misreporting also contributes to a positive test statistic on either treatment schedule.

This means that the signed rank test is more appropriate to evaluate the alternative hypothesis of ${H_a}:\tilde T\lt 0$ . Addressing response inflation with this test requires a null hypothesis different than the sharp null, which involves making statements on the prevalence rate, sample size, distribution of outcomes, and $m$ . These are rarely known in advance.

Application

Table 4 applies both tests to the running example, treating each sensitive organization separately. Since Figure 1 suggests response deflation for Organization Y, I report two-sided p-values for the difference in differences and one-sided lower-tail p-values for the signed rank.

Table 4.

Testing for response deflation in Alvarez et al (Reference Alvarez, Atkeson, Levin and Li2019)

For Organization X, including the sensitive item in the first question leads to a difference in means about $0.08$ points larger than in the second question, the p-value of $0.62$ suggests little evidence against the null of equal differences in means. For Organization Y, the difference in differences is around $ - 0.26$ , which implies a smaller difference in means when the sensitive item goes first. The p-value of $0.08$ gives evidence against the null, although not sufficient to reject it under conventional standards.

The signed rank test statistic is positive for both sensitive items, and since both p-values are $1$ , one may conclude that the difference in differences test more appropriate. The simulations in the next section check if this intuition generalizes.

Setup

I simulate DLEs with a sample size of $1,000$ respondents and fixed list order. The potential outcome of responses to the first list is ${Y_{i1}}\left( 0 \right)\sim B\left( {4,0.5} \right)$ . This implies four baseline items, each applying to respondent $i$ with probability 0.5. This creates responses centered around middle values, which mimics an attempt to avoid floor and ceiling effects. The potential outcome for the second list, ${Y_{i2}}\left( 0 \right)$ , follows the same distribution and associates with ${Y_{i1}}\left( 0 \right)$ with rank correlation $\rho $ . I consider $\rho = \left\{ {0,0.4,0.8} \right\}$ to capture how inducing correlations between lists affects performance.

I assume 15% of the respondents hold the sensitive trait at random. Following the simulations in Blair, Coppock, and Moor (Reference Blair, Coppock and Moor2020), a standard list experiment is underpowered to detect this under conventional standards, but a DLE has over 80% power. This is a case in which opting for a DLE is consequential.

Also at random, a proportion $\gamma \in \left[ {0,1} \right]$ of the participants alter responses by $1$ or $2$ units when they see the sensitive item, doing so in both questions if they see it first. The magnitude is chosen at random and independently between questions. This reflects a setting with moderate but not symmetrical carryover design effects. To facilitate interpretation, I simulate response deflation and inflation separately. Figure D3 in the appendix shows how inflation and deflation introduce bias in DLE estimates.

For each combination of parameters, I simulate $1,000$ experiments and calculate power as the proportion of tests with p-values smaller or equal than $0.05$ . For the difference in means, the p-values are always two sided. For the signed rank, the p-values are left-tailed for deflation and right-tailed for inflation.

Results

Figure 2 shows the power of the proposed tests across parameter combinations for deflation and inflation. In general, power increases with the proportion of unintended responses in both tests. The exception is the signed rank test under inflation, which is sensitive to false positives as it captures the positive prevalence rate. Exception aside, both tests are well powered to detect a proportion of unintended responses that exceeds the true prevalence rate.

Figure 2.

Statistical power under response deflation and inflation.

Note: Each point is based on 1,000 simulations. The dotted vertical line denotes the true prevalence rate.

Everything else constant, the difference in differences has more power under response deflation than under inflation. One implication of this result is that, if possible, researchers should prefer sensitive items that are frowned upon over those one would pretend to have. For example, “I do not support the regime” over “I support the regime.” Yet this conversion is not always straightforward.

Finally, as the correlation between baseline lists increases, the difference in differences has less power under both deflation and inflation. Under deflation, the performance of the signed rank test improves with the correlation. The difference appears trivial in stylized simulations, but Figure D5 in the appendix shows that it becomes more pronounced as the magnitude of response deflation increases. Since previous work recommends inducing positive correlation between lists to increase the precision of the DLE estimator (Glynn Reference Glynn2013), researchers should consider reporting both tests if response deflation is a concern.