Introduction
Voting-advice applications (VAAs) have become useful tools for helping citizens decide which party to vote for in an election, and are now commonly available and used ahead of elections in many democracies, in particular in multiparty settings. But do VAAs make a difference? For example, does completing a VAA increase the likelihood that someone will vote in an election, influence which party they vote for, or increase their political knowledge? Evidence from observational studies suggests that VAAs influence these outcomes. However, most of these studies are limited by powerful selection effects, as the only people being observed are those who opt in to completing a VAA. Recent experimental studies, randomly assigning a VAA encouragement to a particular treatment group, tend to find limited or no effects of VAA usage. However, almost all existing VAA experiments had small sample sizes, and so were underpowered to detect small effects.
To improve on existing research, we conducted the largest VAA encouragement field experiment to date (n = 6,501), in three countries – Germany, Italy, and France – ahead of the 2024 European Parliament (EP) election. European Parliament elections are a good laboratory for investigating the effects of a VAA on vote choice as the elections are held at the same time across European Union (EU) member states, with some form of (proportional) electoral system in every state. While different national parties stand in the elections in each member state, these national parties belong to European-wide political parties – so, in effect, voters in different countries choose between the same set of EU-level parties.Footnote 1
We combined the VAA encouragement treatment design with a pre-election/post-election panel survey, and so were able to observe whether completing the VAA increased the likelihood that a respondent voted in the election, changed which party they planned to vote for, or increased their knowledge of EU parties and policies. Consistent with some of the existing experimental research, but counter to much of the observational literature, we find that completing the VAA did not have any statistically significant effects on voter turnout, the frequency of vote switching, or political knowledge. However, applying a novel regression discontinuity design (RDD) around the proportional agreement scores for the top-recommended parties in the VAA (n = 10,535), we do find that people were significantly more likely to vote for their ideologically most aligned party.Footnote 2 These results suggest that VAAs influence behavior only in the way they are intended to: by helping people make a more informed vote choice for the party they are ideologically most aligned with.
VAA Usage and Voting Behavior
Existing research has analyzed the effects of VAA usage on political behavior, with research growing on this topic over the last fifteen years. The three most analyzed outcomes are voter turnout, vote choice, and political knowledge (Munzert and Ramirez-Ruiz Reference Munzert and Ramirez-Ruiz2021). Observational studies, which are abundant in this research, generally find that VAA usage has sizeable effects on turnout (Garzia et al. Reference Garzia, Trechsel and De Angelis2017; Germann and Gemenis Reference Germann and Gemenis2019; Gemenis and Rosema Reference Gemenis and Rosema2014), on vote choice and vote switching (Garzia and Marschall Reference Garzia and Marschall2014; Kleinnijenhuis et al. Reference Kleinnijenhuis, van de Pol, van Hoof and Krouwel2019), and a small but significant positive effect on political knowledge (Kamoen et al. Reference Kamoen, Holleman, Krouwel, van de Pol and de Vreese2015; Schultze Reference Schultze2014).
However, observational VAA studies suffer from an important limitation: due to self-selection into completing a VAA, the causal link between VAA usage and political behavior is likely to be confounded by a battery of observable and unobservable respondent characteristics, such as political interest, political attitudes, education, or peer influence (Pianzola Reference Pianzola2014). Experimental designs that encourage a random subsample of respondents to participate in a VAA before measuring the outcomes of interest can circumvent these limitations. However, the few published articles reporting such experimental results mostly rely on small samples, and do not come to clear-cut conclusions (Munzert and Ramirez-Ruiz Reference Munzert and Ramirez-Ruiz2021). Sizeable effects of randomized VAA treatments have been found on turnout (Garzia et al. Reference Garzia, Trechsel and De Angelis2017) and vote choice (Alvarez et al. Reference Alvarez, Levin, Mair and Trechsel2014; Germann et al. Reference Frese, Hix and Lachat2023; Pianzola et al. Reference Pianzola, Trechsel, Vassil, Schwerdt and Alvarez2019; Vassil Reference Vassil2012). Mahéo (Reference Mahéo2016,Reference Mahéo2017) and Munzert et al. (Reference Munzert, Barbera, Guess and Yang2020), however, did not find any such effects, and other studies find mixed results (Enyedu Reference Enyedi2016).
A recent meta-analysis concludes that the average effects of VAA usage on turnout and vote choice are far smaller in field experiments than in observational research, if they exist at all (Munzert and Ramirez-Ruiz Reference Munzert and Ramirez-Ruiz2021). The effect of VAA usage on political knowledge, on the other hand, is similarly small in size across all research designs, with the only experiment in this regard by Munzert et al. (Reference Munzert, Barbera, Guess and Yang2020). With small effect sizes on average, and past experimental studies relying on small samples,Footnote 3 it is reasonable to assume that most VAA field experiments have been too underpowered to adequately study the effects of VAA usage.
In response to these limitations and inconsistencies, we aim to test the most frequently analyzed and debated hypotheses in VAA research with a series of high-powered field- and quasi-experiments: that using a VAA makes people more likely to vote (H1); more likely to switch votes (away from their initial preference) (H2a); more likely to vote for the party that receives the top recommendation in the VAA (H2b); and more knowledgeable about politics (H3). All hypotheses tested in the main manuscript were pre-registered ahead of data collection on AsPredicted (see Appendix A1).
Data and Methods
To test these hypotheses, we rely on two data sources: (1) a two-wave panel study that we fielded through OpinionWay in Germany, Italy, and France before and after the EP election in June 2024; and (2) complete user-level data from the VAA provider EuroMPmatch, which allows us to link survey responses to VAA responses.Footnote 4 In Appendix A8, we document the demographic composition of our samples and provide additional context on our sampling procedure.
To test hypotheses 1, 2a, and 3, we embedded VAA encouragement experiments in the pre-election surveys, randomly exposing half of our respondents to the EuroMPmatch VAA, replicating previous experimental setups by Mahéo (Reference Mahéo2016), Mahéo (Reference Mahéo2017), Pianzola et al. (Reference Pianzola, Trechsel, Vassil, Schwerdt and Alvarez2019), and Munzert et al. (Reference Munzert, Barbera, Guess and Yang2020). Our outcomes of interest are self-reported measures at the individual level in the post-election survey. We measure voter turnout as 1 for respondents who voted and 0 for those who did not (H1), vote switching as 1 for respondents who switched away from their pre-election preference and 0 for those who did not (H2a), and political knowledge in the form of two brief quizzes where each correct answer adds 1 to a continuous knowledge indicator (H3). The knowledge questions are designed so that all the information required to answer them is contained within the VAAs. They thus represent a rather liberal test of the effects of VAA usage on political knowledge (compared to a more conservative test where knowledge outside the VAA is tested). Nonetheless, the questions are asked several weeks after taking the VAA, so a knowledge effect of VAA usage would require respondents to retain secondary information for a prolonged period of time. Looking at the distribution of responses, participants had a slightly harder time answering the test question about parties compared to the test question about policies.Footnote 5 The relevant survey items are presented in Appendix A5.
With a final sample of more than 6,400 participants, we can detect very small effect sizes for our VAA encouragements: down to 0.1 standard deviations at 90 per cent power and small effect sizes down to 0.2 standard deviations at 99.9 per cent power. Appendix A4 documents these calculations and also shows that the aforementioned previous experiments with 200 to 2,000 participants were underpowered to detect small effects of VAA usage.
Access to the VAA responses of our survey respondents allows estimating the causal effects of compliance with the treatment assignment. We were also able to leverage this VAA access to test H2b with a novel RDD around the VAA result threshold needed for a party to become the top recommendation for a user. The VAA results gave respondents a proportional agreement score for each party, and ranked parties in decreasing order of these scores. We estimate the effect of a party being ranked first on citizens’ votes through an RDD at the user–party level, with n = user*party observations (> 10,000 effective observations). The agreement score is our running variable for each user–party observation, centered at 0 around the score needed to be ranked first. It is always 0 or positive for the top-ranked party and negative for all lower ranked parties. Our outcome variable in the RDD is a dummy that indicates whether a given user voted for a given party in the EP election.
The important assumption behind this RDD is that there is no substantial (endogenous) increase in the probability to vote for a party based on one additional percentage point of agreement, except when that additional percentage point moves the party to the top of the recommendation list. We demonstrate that this assumption holds with a serious of robustness checks and placebo tests as discussed in the next section.
More detailed elaborations on our data and methods are documented in Appendix A2, which also reports the formal notation for our three estimation strategies: linear regressions to estimate intent-to-treat effects (ITTs) of VAA encouragements in the field experiments; instrumental variable regressions to estimate complier average causal effects (CACEs) of VAA usage in the field experiments; and the RDD to estimate local average treatment effects (LATEs) on voting for parties around the top recommendation threshold. The data sources and estimations behind all results reported in the main manuscript were pre-registered ahead of data collection on AsPredicted (see Appendix A1).
Results
Field Experiment
Figure 1 presents the main results of the field experiment when pooling data from all three countries. For each hypothesis, we report estimates from a model with country fixed effects only, and estimates from models including the pre-registered control variables (age, gender, and education). For all models and dependent variables, the ITTs of VAA usage (that is, the coefficients of the effects of being encouraged to participate in the VAA) indicate null effects. We do not find a statistically significant effect of VAA encouragements on voter turnout, vote switching, knowledge about political parties, or knowledge about EU policies.Footnote 6
Intent-to-treat effects (ITTs) and complier average causal effects (CACEs) for all field experiments.
Note: Error bars are 95 per cent confidence intervals.
We further use treatment assignment as an instrumental variable for treatment intake to test our treatment effects only for those respondents who complied with treatment assignment. Based on the data from the VAA provider, ≈40 per cent of treated respondents complied with their treatment assignment and participated in the VAA, while ≈34 per cent completed the VAA in its entirety. Overall compliance across the treatment and control group was at ≈70 per cent, which is higher than for previous VAA experiments that reported compliance.Footnote 7 As reported in Figure 1, the soft and strict CACEs for all four outcomes are also statistically indistinguishable from zero.
Appendix A10 reports a set of pre-registered heterogeneity analyses: we investigate between-country differences and the moderating effects of education, EU support levels, and survey timing. None of these factors substantially moderate the relationship between the treatment and voting behavior, with almost all subgroup and interaction results remaining non-significant. Appendix A11 shows that there was no differential attrition (from the pre-election panel wave to the post-election wave) between the treatment and the control group. Appendix A12 further demonstrates that VAAs are not ideologically manipulative, as they do not increase voting for any particular political camp.
These null results in the field experiments are highly informative as they precisely demonstrate the limits of VAA effects. In Table 1, we go through the checklists for alternative explanations of null results by Alrababa’h et al. (Reference Alrababa’h, Williamson, Dillon, Hainmueller, Hangartner, Hotard, Laitin, Lawrence and Weinstein2023) and Kane (Reference Kane2025) to demonstrate that our design choices reduce the possibility for false negatives as much as possible. As discussed in Table 1, the only factor that is not fully accounted for – and that no VAA experiment can fully account for by design – is the possibility of a pre-treatment effect. While we minimize the possibility of contamination from pre-treatment use of our VAA by opting to use the less-publicized EuroMPmatch VAA instead of more-publicized alternatives such as the Wahl-O Mat, it is possible that our experimental subjects have been pre-treated with other VAAs, which could still water down any potential treatment effects (albeit to a lesser degree than an identical pre-treatment VAA). However, if pre-treatment VAA exposure would have contaminated our treatment effects, we should expect to see larger effects in France and Italy compared to Germany, given that Germany is home to the most commonly used VAA in Europe, the Wahl-O Mat. While the Wahl-O Mat is regularly used by more than a quarter of the entire German electorate ahead of elections, the largest VAAs in France and Italy merely reach audiences in the low single digit millions. The fact that our coefficients are similarly small across countries, as we show in the heterogeneity analyses in Appendix A10, suggests that this stronger potential contamination in Germany was not the reason for these null results either.
Alrababa’h et al., (Reference Alrababa’h, Williamson, Dillon, Hainmueller, Hangartner, Hotard, Laitin, Lawrence and Weinstein2023) and Kane (Reference Kane2025) checklist of alternative explanations for null results
RDD
The results of the RDDs are reported in Figure 2. Around the discontinuity (the additional percentage point in policy agreement needed to become the top-recommended party in the VAA), VAA users are significantly more likely to vote for the top-recommended party than the second-most-recommended party. This holds both for recommendations of EU parties and for recommendations of national parties.
Regression discontinuity design (RDD) vote choice plots for EU parties (left) and national parties (right).
Note: dotted lines are 95 per cent confidence intervals. The Y-axis shows the probability of voting for a given party. The X-axis shows the point differential from the threshold for the top recommendation in the VAA. Note that the graph depicts a non-parametric fit, yet the slopes are almost perfectly linear. The models in Table 2 are estimated with linear slopes.
Regression discontinuity design (RDD) results across different specifications
Note: A detailed explanation of the different specifications can be found in Appendix A14.
Standard errors are clustered at the user level.
*p < 0.1; **p < 0.05; ***p < 0.01.
As we show in Table 2 and Appendices A14–A15, the RDD results are robust across different specifications of the outcome variable and different bandwidths.Footnote 8 In all specifications, the probability of receiving a user’s vote increases by 2–6 percentage points for top-recommended parties at the discontinuity. In Appendix A13, we again report the results for the three separate country samples. In Appendix A18, we report results for two separate split samples: by whether or not the top- and the second-most-recommended party in the VAA are ideologically aligned. These supplementary results suggest that persuasion effects are particularly strong when there is a clear ideological difference between the top two recommendations.
In Appendix A17, we show with a truncated sample that the differential results between the encouragement experiments and the RDD are not related to differences in sample composition. Rather, the results suggest that VAA usage, while not increasing the frequency of vote switching, does improve the quality of vote switching, meaning that VAA users are more likely to switch their vote to ideologically close parties, while non-users switch their vote similarly often but in a less predictable manner. This interpretation is further supported by descriptive statistics that show a high frequency of vote switching of around 18 per cent in both the treatment and the control group, which allows for large differences in ideological sorting between both groups.
As outlined in the methods section, this RDD rests on the assumption that there is no substantial (endogenous) increase in the probability to vote for a party based on one additional percentage point of agreement, except when that additional percentage point moves the party to the top of the recommendation list. For example, if there were three parties, and after taking the VAA a respondent was informed that they have a 70 per cent agreement with party A, a 71 per cent agreement with party B, and a 90 per cent agreement with party C, we would not expect that respondent to become more likely to vote for party B over party A, based on those agreement percentages. However, if party C were to have only a 60 per cent agreement, which would mean that party B was then the top-recommended party, we would expect that respondent to become more likely to vote for party B.
To test whether this assumption holds, we conducted placebo tests at fake cut-off points in the control group. If the placebo RDDs would also produce significant increases in voting probabilities, this would indicate that the assumption does not hold. However, as we demonstrate in Appendix A16, none of the placebo cut-offs produce significant discontinuities. This suggests that the main RDD indeed captures significant increases in voting probabilities beyond their endogenous correlation with policy agreements.
Conclusion
Previous research on the effect of VAAs on political behavior suffers from several limitations that might partially explain the inconsistent results. These limitations include self-selection into the treatment in observational studies, and low power and inaccurate measures of compliance and outcomes in experimental studies.
We replicated the most promising experimental design in three large-scale encouragement experiments in Germany, France, and Italy, during the 2024 European Parliament election. Our design included two-wave panel studies powered to detect very small effects, combined with full access to data from a VAA provider and measures of self-reported voting behavior. We found that VAA usage had no statistically significant effects on voter turnout, the frequency of vote switching, or political knowledge. Because our design choices were deliberately optimized to reduce the risk of false negatives, these experiments provide a strong refutation of much of the observational – and some of the experimental – literature in this vein of research, which found strong VAA effects on these outcomes.
However, to further distinguish between effects on the frequency of vote switching and the quality of vote switching, we also analyzed the voting behavior of VAA users in a novel RDD designed to causally identify vote switching to the top-recommended party in the VAA – in other words, the ideologically most aligned party. Here, we did indeed find that the same respondents were up to 6 percentage points more likely to vote for the top-recommended party by the VAA. This suggests, in line with a few previous studies (Alvarez et al. Reference Alvarez, Levin, Mair and Trechsel2014; Germann et al. Reference Frese, Hix and Lachat2023; Pianzola et al. Reference Pianzola, Trechsel, Vassil, Schwerdt and Alvarez2019), that VAAs can have a positive effect on the quality of vote switching.
VAA users are more likely to make informed decisions – to make vote choices based on ideological alignment. These findings indicate that VAAs can work exactly as intended: beyond their declared purpose of creating a more informed electorate, they do not influence electoral behavior in any inadvertent ways. For example, we confirm that our VAA did not increase vote switching at random and did not manipulate respondents into supporting particular political parties. We consider this combination of null results in our field experiments and positive results in the quasi-experiment maximally informative, as we demonstrate precisely what VAAs can and cannot do prior to elections. Our findings are confirmed and strengthened through a rich battery of supplementary analyses.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S000712342510118X.
Data availability statement
Replication data for this article can be found in Harvard Dataverse at: https://doi.org/10.7910/DVN/TEDDPX.
Acknowledgments
We thank the EuroMPmatch team and especially Stefano Brocchi for their fruitful collaboration. We thank Vin Arceneaux, Elias Dinas, Andres Reiljan, Álvaro Canalejo-Molero, Sven Hegewald, and audience members at the EUI PhD colloquium, MPSA, ESRA, and APSA for engaging with previous versions of this paper and providing helpful feedback.
Financial support
This project relies on data from the project ‘Europe’s Future: Youth and the 2024 European Parliamentary Elections in Italy, France, and Germany’, funded by CIVICA and Bocconi University.
Competing interests
Simon Hix was involved in the development of the EuroMPmatch VAA. This did not affect any of the results reported in this paper. All results reported in the main manuscript are strictly based on the hypothesis tests and model specifications that were pre-registered ahead of data collection.
Ethics declaration
This project has received full ethics approval from the EUI ethics committee and fully adheres to APSA’s Principles and Guidance for Human Subjects Research. Appendix A3 discusses ethical considerations in more detail.
Open access
