4.1. The MTME framework

The MTME is a generalization of within-person experimental designs, such as test–retest or the MTMM (Cernat and Oberski, Reference Cernat, Oberski, Lavrakas, Traugott, Kennedy, Holbrook and de Leeuw2019, Reference Cernat and Oberski2022). A within-person experimental design implies asking respondents the same questions multiple times. The design makes estimating reliability possible by examining how consistently respondents answer the same question. The simple design can also be expanded to include correlated sources of bias. For example, MTMM designs, in addition to asking the same questions, also manipulate the response scale of the questions, with respondents receiving different answer options in the follow-up (Campbell and Fiske, Reference Campbell and Fiske1959; Saris and Andrews, Reference Saris, Andrews, Biemer, Groves, Lyberg, Mathiowetz and Sudman1991). This makes it possible to estimate “methods effects,” which is a source of correlated variance due to the response scale and not the content of the question.

The MTME framework generalizes this approach and implies the development of experimental manipulations that can lead to different sources of measurement error. Furthermore, the MTME framework is theory-driven and allows researchers to study relevant errors and estimate their relative impact. For example, Cernat and Oberski (Reference Cernat and Oberski2022) manipulated the response scale, the categories’ order, and the questions’ content to estimate method effects, acquiescence, and social desirability (in addition to random error). Such a complex design leads to multiple “forms” of the questions (way to word the question and response category). A split ballot factorial design is typically used to minimize the burden on the respondents (Saris et al., Reference Saris, Satorra and Coenders2004; Revilla et al., Reference Revilla, Bosch and Weber2019), in which respondents are randomly allocated to groups and receive the questions twice using different forms.

The MTME framework offers a flexible way to experimentally explore different measurement error sources. That being said, the approach is a within-person experimental design and thus inherits its limitations, such as memory effects (people remembering they were asked the questions already). Possible solutions to this issue are increasing the time between re-interviews (although too long of a time might lead to true change in the topic of interest), randomly changing the order of the forms, or accounting for memory effects (e.g., controlling for cognitive ability or period to reinterview). In the next sections, we will discuss how we developed an MTME design to estimate measurement error in responses to issue questions, how we modeled the data, and how we minimized and controlled for memory effects.

4.2. The MTME design and data

The MTME experiment design was included in waves five (n = 3,763; RR = 75.2%) and six (n = 3,885; RR = 76.9%) of the Finnish panel Citizens’ Opinion (Finnish: Kansalaismielipide) 2023 parliamentary election study (Grönlund and Strandberg, Reference Grönlund and Strandberg2023). The panel is recruited via nonprobability and probability sampling, and data are collected online using Qualtrics. The two waves used in this study were conducted in the period from April 4 to 24, 2023, after the parliamentary election on the 2nd of April. The entire election study consisted of 4,875 respondents, while this dataset only uses respondents who answered the relevant questions in both waves (n = 3,175).Footnote ¹ See Backström et al. (Reference Backström, Grönlund, Strandberg, Grönlund and Strandberg2023) for a more in-depth description of the panel and the election study. Data for the entire election study can be accessed at the Finnish Social Science Data Archive (FIRIPO and Strandberg, Reference FIRIPO and Strandberg2023), while data and replication files for the MTME experiment can be accessed at the PSRM Harvard Dataverse (Backström et al., Reference Backström, Cernat, Siren and Söderlund2025).

Ten survey items were included in the two waves using a bipolar matrix question that asked respondents: ‘What is your opinion on the following political proposals?’ The items are used to capture different issue dimensions prevalent in Finland, such as the traditional economic left–right dimension, and various aspects of the sociocultural dimension. We also include an item used to capture the historically relevant center-periphery dimension. Westinen (Reference Westinen2015) presents an overview of the political cleavage structures in Finland, and Söderlund (Reference Söderlund, Grönlund and Strandberg2023) discusses a more in-depth analysis of the substantive results and the ability of the items to capture different dimensions in the election study. The matrix questions were included at the end of the respective waves, and the total wave response times were about 10 minutes (wave five median: 10.5 minutes; wave six median: 10.1 minutes). Table 1 presents the exact statement wordings and what political issue dimension the items attempt to capture.

Table 1.

Survey items used and their representative issue dimension

The experimental design is based on varying the number and direction of scale points that respondents can choose from when answering the questions. These variations are done to capture measurement errors resulting from method effects (scale points) and acquiescence (scale direction) while estimating random error. The survey attributes were manipulated in a 2 × 2 experimental design, where respondents could answer using either 5- or 11-point scales, and the scale direction was either from good to bad proposal (decremental) or bad to good proposal (incremental). All scale points had verbal labels in the 5-point scale forms,Footnote ² while only the polar scale points had verbal labels in the 11-point forms. Furthermore, all scale points in the 11-point forms had numeric labels ranging from 0 to 10. These variations resulted in four possible forms for the question (Table 2).

Table 2.

Question forms when varying scale points and directions (2 × 2)

However, the MTME approach is based on a within-person experimental design, meaning that respondents must answer two forms of the question at two different points to estimate measurement error. Using a split ballot design (Saris and Gallhofer, Reference Saris and Gallhofer2014), the four forms over two measurement points result in six combinations, or form pairs, that must be administered to achieve the design.Footnote ³ However, the form pairs’ order should be randomized to counteract potential carryover effects. Randomizing the order of the forms results in 12 treatment groups (2 × 6) (Table 3).

Table 3.

Randomized form order combinations across measurement points for the treatment groups (n = 3,175)Footnote ⁴

Memory effects are also a concern when carrying out within-person experimental designs. Respondents may remember they answered the same questions earlier, affecting their answers. For example, they may simply repeat the previously given answer due to satisficing (Rettig and Blom, Reference Rettig, Blom, Cernat and Sakshaug2021). Increasing the time between measurement points can decrease the risk of memory effects. Previous research has found that memory effects are rare after at least 20 minutes of interview time (Saris et al., Reference Saris, Revilla, Krosnick and Shaeffer2010; Saris, Reference Saris2013). However, new research found respondents to be able to reproduce their answers after 20 minutes across questions on beliefs, attitudes, and behavior (Rettig et al., Reference Rettig, Blom and Karem Höhne2023; Rettig and Struminskaya, Reference Rettig and Struminskaya2023), indicating a need for more time between measurement points.

However, when studying measurement error using within-person experimental designs, keeping time between measurement points too long can also be problematic since there is a risk of true change in the concepts of interest (Cernat and Oberski, Reference Cernat, Oberski, Lavrakas, Traugott, Kennedy, Holbrook and de Leeuw2019). This means that differences observed between forms of the questions for the same respondent could be caused by a change in the underlying trait and not by measurement error. However, preferences on sociocultural issues generally are more stable (Zaller, Reference Zaller2012), meaning that survey items used should not be prone to true change in traits.

The two waves that included the MTME experiment took place after the parliamentary election, meaning there should not be an election outcome effect that could have changed the true score between the two waves. However, the waves’ field periods overlapped each other in time. To ensure the correct form orders and decrease the impact of memory effects, we excluded respondents who answered the waves in the wrong order (n = 10) or both waves on the same day (n = 46). This resulted in 3,175 respondents, with a mean time between waves of 9.6 days (min: 1; max: 19; sd: 2.3). Ensuring there was at least 1 day between responses, randomizing the form order, the fact that both waves occurred after the election, and the relative stability of the topic of interest should balance well the possible confounders of memory effects and true change. We will also run sensitivity analyses to investigate how the time between measurements impacts error estimates.

Furthermore, from a data quality standpoint, probability samples are preferable to nonprobability samples since analyzing data from nonprobability samples increases requirements regarding transparency, design, and modeling assumptions (Baker et al., Reference Baker, Michael Brick, Bates, Battaglia, Couper, Dever, Gile and Tourangeau2013). The critique on nonprobability samples is based on the uncertainty regarding being able to make accurate inferences about a larger population. Lately, this critique has focused on online incentivized opt-in panels such as Amazon’s Mechanical Turk (Kennedy et al., Reference Kennedy, Clifford, Burleigh, Waggoner, Jewell and Winter2020; Ahler et al., Reference Ahler, Roush and Sood2021). However, the Citizens’ Opinion panel does not employ comparable incentives for panel participants, and almost all of the panelists recruited via nonprobability sampling have been in the panel for over 3 years, meaning they are active and willing participants even without incentives used in other online panels. Still, while there are exceptions (e.g., Einarsson et al., Reference Einarsson, Sakshaug, Cernat, Cornesse and Blom2022), based on the previous literature, nonprobability samples can be expected to generate more measurement errors. Because of this, we will decompose variance between panelists recruited via non-probability (n = 1,154) and probability (n = 2,021) sampling as a sensitivity analysis.

4.3. The MTME model

Based on the experimental design presented in the previous section, we develop a statistical model that separates three different sources of measurement error from the trait, or concept of interest (Cernat and Oberski, Reference Cernat, Oberski, Lavrakas, Traugott, Kennedy, Holbrook and de Leeuw2019, Reference Cernat and Oberski2022). First, we estimate method effects as the variance due to the response scale, 5 versus 11 points. Second, we estimate acquiescence effects, or the tendency to agree with statements regardless of their content, by comparing situations when positive response categories are presented first versus when presented last. This is based on the expectation that it is easier to agree with a statement if the positive response category is presented first. Finally, we also estimate random errors or noise.

To separate these different sources of variation concurrently, we develop an MTME model in the Structural Equation Modeling framework (Bollen, Reference Bollen1989):

\begin{equation*}y_{tma}^* = \lambda _{tma}^{{{\left( T \right)}^*}}{T_t} + \lambda _{tma}^{{{\left( M \right)}^*}}M + \lambda _{tma}^{{{\left( A \right)}^*}}A + {\varepsilon _{tma}}\end{equation*}

where $y_{tma}^*$ is the observed variable measuring a particular trait or topic, t, a method or response scale, m, and a scale direction, a. We decompose these observed variances in four sources of variation: T, measuring the trait, M, measuring the method effect, A, measuring acquiescence and an item specific random error, ${\varepsilon _{tma}}$. The trait variance represents the valid source of variation that measures the concept of interest. We reason that, even when working with items used to create indices, it is essential to detect measurement error at the individual-item level to achieve the best possible data quality. The Acquiescence and Method variances are correlated measurement errors as they represent consistent answering patterns due to the format of the response scale and not the content. The random error represents noise in the data that can bias confidence intervals and multivariate analyses. The visual representation of the model can be seen in Figure 1.

Figure 1.

Representation of the MTME model estimated in the SEM framework. Observed variables are represented by squares, each topic being measured using four different forms (F1–F4). Latent variables, or factors, are represented by circles. The “T” latent variables represent the concept of interest, while the “M” latent variables represent method effects due to the response scale, and “A” represents acquiescence caused by the direction of the scale. Only 3 out of the 10 topics are presented for ease of reading. Residual errors are not presented for the same reason.

In the MTME models, correlated measurement error (such as Method (M) or Acquiescence (A)) can be identified using either a latent variable for each condition or using a dummy coding approach (Cernat and Oberski, Reference Cernat, Oberski, Lavrakas, Traugott, Kennedy, Holbrook and de Leeuw2019, Reference Cernat and Oberski2022). Here, we use the former approach for the method effect, thus estimating two latent variables (M1 and M2), one for each method used, and the latter for acquiescence (A). As a result, the method effects can be interpreted as the amount of variation due to each method compared to a hypothetical answer without a method effect. The acquiescence latent variable variance can be interpreted as the amount of extra variation due to presenting a positive response category first compared to presenting it last. Using the dummy coding approach can help estimate the models (as used in the MTMM-1 models; see Eid et al., Reference Eid, Nussbeck, Geiser, Cole, Gollwitzer and Lischetzke2008) but can sometimes be harder to interpret.Footnote ⁵

In addition to running the MTME model for the entire sample, we also investigate if MTME estimates vary by key groups identified in the literature. We will do this by separating the MTME estimates by the degree of political interest, degree of internal political efficacy, and education level, which we view as indicators of political sophistication.Footnote ⁶

Political interest is indicative of general interest in answering the survey, considering the political theme. Groves (Reference Groves2004) found an association between survey topic interest and degree of non-attitude and acquiescence, meaning respondents with lower political interest can be expected to achieve lower data quality on questions used to measure political issue positions. Political interest is estimated through the question “Generally, how interested are you in politics?” with the response options “very interested,” “‘fairly interested,” “not particularly interested,” and “not interested at all.” However, to achieve better class balance, the response options “fairly interested,” “not particularly interested,” and “not interested at all” were combined in the analyses, creating two groups that either have “high political interest” (n = 1,275) or “low political interest” (n = 1,888).

Internal political efficacy is seen as an individual’s assessment of their ability to understand what is going on in politics (Niemi et al., Reference Niemi, Craig and Mattei1991). Considering that a lack of knowledge or comprehension of a subject indicates non-attitudes (Groves, Reference Groves2004), it is reasonable to assume that a low level of internal political efficacy is also associated with non-attitudes on political issues. A higher propensity to answer these surveys randomly will lead to a greater overall degree of error. We use three survey items and the four response options to create a sum variable (scale 0-9), where a higher value indicates a higher degree of internal political efficacy. See Table A2 in the appendix for a closer decomposition of the items and response options used to create the sum variable and differences within the sample regarding internal political efficacy. To achieve class balance, we recoded the sum variable into a dichotomous variable where respondents with a value under the mean (6.28) were coded as having a low degree of internal efficacy (n = 1,438), while those over the mean were coded as having a high degree (n = 1,238).

Finally, having an academic degree is also viewed as indicative of political sophistication and cognitive ability. Again, being less educated is associated with a higher chance of non-attitude and acquiescence (Groves, Reference Groves2004). Meisenberg and Williams (Reference Meisenberg and Williams2008) also found that cognitive ability or education level is negatively related to acquiescence and extreme response styles. Having a degree was measured using the question “What is your highest achieved education?” with nine response options. The responses were then recoded based on whether the respondent had achieved a university or university of applied sciences degree (n = 1,666) or not (n = 1,489).

We view these variables as indicators of political sophistication, meaning a respondent with low political interest, low internal political efficacy, and/or without an academic degree can be expected to have lower political sophistication compared to those with a higher formal sophistication concerning politics. Political interest, internal political efficacy, and education level are all closely related and can be used as proxies for political knowledge (Rapeli, Reference Rapeli2022). Furthermore, high political knowledge is related to correct voting (Lau et al., Reference Lau, Patel, Fahmy and Kaufman2013; Pierce and Lau, Reference Pierce and Lau2019), meaning those politically knowledgeable can better vote according to their preferences. Based on this, we expect respondents with low political sophistication to be more error-prone.

Data were cleaned in R 4.3.1 (R Core Team, 2023), and the SEM models were estimated in the lavaan package (Rosseel, Reference Rosseel2012). Missing dataFootnote ⁷ were dealt with using Full Information Maximum Likelihood, which assumes Missing At Random (MAR) given the model (Enders, Reference Enders2022).