Exit polls have a long history in the field of electoral behaviour (Scheuren and Alvey, Reference Scheuren and Alvey2008: 5). After a timid start in the 1940s, they appear on the radar of American media and academic research in the late 1960s (Hilmer, Reference Hilmer2008: 94). It took another fifteen years before the CBS Network made it a regular feature of election night coverage. Canada has “no history of exit polling as part of its election coverage and analysis’’ (Brown et al., Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006: 919). Examples of exit polls in the Canadian context are thus scarce. Moreover, logistical obstacles are important in Canada because of the geographic dispersion, strong diversity of its electorate, and the difficulty in finding the proverbial barometer voting polls. Still, there are only a few instances of academic research using exit polls in Canada (Brown et al., Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006; Brie and Ouellet, Reference Brie and Ouellet2020).
In public opinion research, exit polls are considered a garden variety of survey analysis, with a heavy focus on getting quick and reliable vote intention data. Moreover, exit polls can be an enriching, hands-on learning experience for participating students (see, for example, Berry and Robinson, Reference Berry and Robinson2012). Exit polls are usually done through one-page questionnaires or short in-person interviews where voters are intercepted at the exit of a polling location (Best and Krueger, Reference Best and Krueger2012). The questions generally revolve around the vote choice and its covariates. Exit polls differ from other types of surveys in two ways: (1) the target group is actual voters—or at least individuals who cast a ballot—not only potential or eligible ones, and (2) the interviews are right after the vote is cast. In general, nonresponse rates are lower because of the direct contact with interviewers (see, for general discussions, Hilmer, Reference Hilmer2008, and Grimshaw et al., Reference Grimshaw, Christensen, Magleby and Patterson2004). Exit polls can also be cheaper than online surveys with commercial partners, especially when students participate in the process.
In this research note, we provide two methodological contributions to the study of exit polls in the Canadian context. First, it is to our knowledge the first series of academic exit polls conducted simultaneously in multiple electoral districts in Canada, and the first to include exit-poll data from federal and provincial elections conducted within the same geographical area. Second, we conducted small-scale efforts to see whether data collected in these contexts are reliable. We show that the precision of exit-poll estimates indeed requires only a small sample of respondents. More precisely, we test for chance error and bias error, and show that voters supporting the right-of-the-centre parties are harder to reach, leading to a systematic underestimation of the actual support for these parties, in the Quebec context at least.
Methodological Issues Regarding Exit Polls
Sampling in the context of exit polls face a series of methodological issues that range from the consequences of interactions between interviewers and interviewees (Traugott and Price, Reference Traugott and Price1992; Converse, 1971; Bishop and Fisher, Reference Bishop and Fisher1995) to ethical concerns (Sønderholm, Reference Sønderholm2016; Milavsky et al., Reference Milavsky, Swift, Roper, Salant and Abrams1985) to the representativeness of the data collected. Other minor concerns include potential respondents’ misinterpretation of questions, reluctance to collaborate with interviewers, and errors during transmission or data entry (Scheuren and Alvey, Reference Scheuren and Alvey2008: 12).
Nonresponse bias, one of the main causes of inaccurate estimates of exit polls, occurs when certain types of voters refuse to participate or are missed by interviewers (Merkle and Edelman, Reference Merkle, Edelman, Groves, Dillman, Eltinge and Little2002; Clinton et al., Reference Clinton, Lapinski and Trussler2022). It becomes concerning when the odds of voters from different parties responding to the exit poll are unbalanced (Pavía, Reference Pavía2010: 69; Pavía et al., Reference Pavía, Badal and García-Cárceles2016). In other words, the voters who decline the interview may present different voting behaviours than those who choose to participate, which could decrease the precision of later estimates (Bautista, Callegaro, Vera, and Abundis, Reference Bautista, Callegaro, Vera and Abundis2007: 493; Best and Krueger, Reference Best and Krueger2012). It is specified that although lower response rates do not automatically lead to systematic errors, it is sometimes preferable to have a larger sample size to reduce sampling error (Reference Best and Krueger2012: 500).
Nonresponse is typically associated with sociodemographic and ideological characteristics (Bautista et al., Reference Bautista, Callegaro, Vera and Abundis2007: 500; Panagopoulos, Reference Panagopoulos2013; Abramson, Aldrich, and Rohde, Reference Abramson, Aldrich and Rhode2012). Timing is crucial to minimizing such bias. Indeed, different groups vote at different moments of the day; it is therefore recommended to schedule interviews at different moments during the day to minimize sampling errors (Mitofsky, Reference Mitofsky, Lavrakas and Holley1991). Brown and colleagues (Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006: 925) also find an increase in cooperation after 4:00 pm. More importantly for our purpose, recent research on partisan nonresponse bias in the United States (Clinton et al., Reference Clinton, Lapinski and Trussler2022) has shown more reluctance among Republican voters to participate in phone surveys. Partisan bias on passive sampling platform has also been measured (Coppock and McClellan, Reference Coppock and McClellan2019).
It remains evident that the most important concern of researchers interested in survey methods that necessitates sampling of a population is the validity of data. The best tool available remains a systematic randomization protocol. In the case of exit polls, this can be achieved with a predetermined skipping interval by interviewers. Another method is to pick the sample purposely to fit a list of social and political characteristics (Barreto et al., Reference Barreto, Guerra, Marks, Nuno and Woods2006: 479). This is a way to ensure accurate representation of socioeconomic and ethnic diversity. The size of a sample can vary according to the context of every precinct or voting system. Hilmer (Reference Hilmer2008: 98) suggests that majority-voting systems require larger samples because projections are built using local or regional geographical units instead of the national electorate. A sampling frame can be built based on past race data, in which the selection chances of a precinct is proportional to its number of voters (Mitofsky, Reference Mitofsky, Lavrakas and Holley1991: 95).
In the case of small-scale exit polls, a fundamental question is how little is enough. Recent research suggests that data saturation can be achieved rapidly (Klima et al., Reference Klima, Schlesinger, Thurner and Küchenhoff2019), though it might be necessary to consider additional sources of information to improve estimation. Are small-scale exit polls too sensitive to chance error? And do these data gathering efforts exacerbate bias error? These are the two questions of particular interest in this research note.
Exit Polls in Two General Elections in Quebec
As stated in the introduction, there exist very few examples of academic exit polls in Canada. The largest study to date is certainly the Brown et al. (Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006) 2003 exit poll conducted during the Ontario provincial election in the district of Kitchener Centre. This exit poll was conducted in more than a dozen representative voting locations, with the help of fifty students. In total, 653 interviews were completed. The authors explain the numerous logistical challenges they had to face (weather, bureaucratic resistance, and so forth) and still obtain quite impressive levels of precision at the district level for the three major parties. Surprisingly, this research project did not lead to a wave of replications in other provinces, not at the federal level. A more recent exercise was conducted during the 2018 provincial Quebec election (Brie and Ouellet, Reference Brie and Ouellet2020) and included an innovative wording experiment about exposure to bilingualism, a method that had never been used in a Canadian context previously.
We make use of two pilot projectsFootnote 1 over two elections to bring a fresh look at the methodological challenges faced in the context of exit polls, and in particular in small-scale exit polls. In both cases, we have chosen federal and provincial districts that overlap to minimize potential sources of discrepancies. The first exit poll was conducted during the 2018 Quebec provincial election in four voting locations situated in four different districts. Two districts were situated in urban areas of Québec City, in the electoral districts of Taschereau (34 respondents) and Jean-Talon (49 respondents). The two other districts were situated in Montréal, in the urban districts of Mont-Royal-Outremont (29 respondents) and Laurier-Dorion (45 respondents).Footnote 2 Voting locations were selected because they were easy to reach by public transport and because it was possible to stay close enough to the building to intercept voters without violating electoral law. Voting locations regroup more than one voting section but always in the same electoral district. It is impossible to know with which section voters were associated, but we know which sections are associated with each voting location. The survey questionnaire was administrated by two female graduate students, first in Montréal in the morning and then in Québec City in the late afternoon and evening. Respondents were asked to answer anonymously (they would write their own answer on the paper questionnaire without supervision) a series of questions about the election and related political issues as well as their sociodemographic background.Footnote 3 The main question of interest here is vote choice. For all questions, respondents could choose to answer or decline to answer. In total, 17 respondents did not to report their vote choice or reported they had not voted. Table 1 summarizes the raw numbers of votes for each of the main parties in the exit polls and in the official results in 2018, and the corresponding percentages of total votes. Advanced and special voting is excluded from the computation of official results to ensure appropriate comparison with exit-poll data.
Table 1. Summary statistics for the 2018 provincial election exit poll
Following the experience of 2018, a second series of exit polls was conducted during the 2019 Canadian federal election. Again, polling was conducted on election day, in two voting locations in the federal electoral districts of Québec City (265 respondents) and Louis-Hébert (118 respondents).Footnote 4 Both districts are situated in urban areas of Québec City and recoup parts of Jean-Talon and Taschereau districts at the provincial level. Again, this allows for a powerful comparison as the federal and provincial districts under study overlap. In the district of Québec City, five male and two female undergraduate students were mobilized at different moments during the day. In Louis-Hébert, two male and five female undergraduate and graduate students conducted interviews, again, at different times of the day. This larger workforce allowed us to roughly triple the number of respondents compared to 2018. Detailed results are presented in Table 2. Respondents were asked to fill out a questionnaire about the election themselves, and anonymously some additional questions on social and political issues as well as their sociodemographic background. The main question of interest here is again vote choice. This time, 42 voters refused to disclose their vote choice and are thus excluded from our analysis. The number of respondents is much smaller than what Brown et al. (Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006) achieved in 2003 but we compensate that shortcoming with more variety in terms of districts visited and by combining provincial and federal data. The total cost for these two small-scale exit polls was a few hundred Canadian dollars.
Table 2. Summary statistics for the 2019 federal election exit poll
Assessing the Accuracy of Exit Polls
We now want to assess the accuracy of exit polls in the six districts where data were collected to identify sources of discrepancies based on sample size and the nature of the parties at play. We have two measures of accuracy here. First, we simply compute the difference—or the error (E) —by party between the vote share in the exit poll and in the official results in percentage points. It is a simple measure of bias. Second, we compute the absolute error (AE), which is the absolute transformation of E to only measure inaccuracy without regard to the direction of bias.
We are first interested in the partisan bias in our sample. We expect that our exit polls will underestimate the support for parties on the right of the political spectrum. We have included in that group the Conservative Party of Canada, the Liberal Party of Quebec, and the Coalition Avenir Québec.Footnote 5 The reference category brings together a heteroclite group of parties of the centre or the Left (the New Democratic Party of Canada, the Liberal Party of Canada, the Bloc Québécois, the Parti Québécois, and Québec Solidaire). Published research finds a partisan bias in nonresponse that would yield an underrepresentation of Republican voters in the United States in phone surveys (see Clinton et al., Reference Clinton, Lapinski and Trussler2022) and in exit polls (Best and Krueger, Reference Best and Krueger2012: 9). It is also the case in parliamentary systems such as Spain where supporters of the People's Party (Conservative) tend to respond less to exit polls (Pavía et al., Reference Pavía, Badal and García-Cárceles2016). There are no—to our knowledge—empirical studies in Canada that would support such hypothesis.
We are also interested in the empirical relationship between sample size (that is, the number of supporters for a given party in the exit poll sample) and accuracy (AE). We expect, as the law of large numbers would predict, more accuracy when the sample is bigger, but we want more specifically to see if exit polls can perform well when the number of respondents is much smaller than the expected sampling proportion. Interestingly, the American research on exit polls have rarely looked at small-scale samples. Klofstad and Bishin (Reference Klofstad and Bishin2012) did run an experiment in the American presidential election of 2008 with small samples (470 completed surveys in total) but their attention was focused on the differences between interviews completed before and after the vote was cast.
In Figure 1, we look at the relationship between the number of respondents in the exit poll and the difference between survey results and official results (E), in percentage points. Each circle represents this measure for each party in each district, with full black dots representing parties on the right of the political spectrum and hollow dots other parties. The 2018 and 2019 exit polls are lumped together. The left window displays the simple raw quantities (number of respondents who reported voting for a party) while the right window looks at the proportion of actual votes for the party in the exit-poll sample. We can see that parties on the right of the political spectrum seem to be more frequently underestimated within our samples, compared to other parties. The right window seems to suggest that when a bigger proportion of total votes are captured in the exit poll, there is less risk of serious over- and under-estimation.
Figure 1. Over- and under-estimation (E) of parties’ vote share.
Figure 2 displays the AE between actual votes and our estimations, showcasing the absolute level of inaccuracy of our samples without taking direction into account. This series of box plots gives a general idea of the variation in imprecision, with increases in respondents having a substantial impact on both the variation of AE and the median AE. The numbers in parentheses represent the total amount of questionnaires filled in each district. Taschereau is especially imprecise due to the overrepresentation of Québec Solidaire voters in our exit-poll sample.
Figure 2. Distribution of Absolute Error (AE) per district.
We now move to a multivariate framework to estimate the determinants of E and of AE. We estimate four models in Table 3 using ordinary least square (OLS) regressions with standard errors clustered around our six voting locations. The unit of analysis is a party, at a given location. There are thus twenty-four observations. Models 1 and 2 use the outcome variable as the difference (in percentage points) between vote share in exit-poll sampling and official results. This is referred to as the error (E). Models 3 and 4 rather display the absolute error (AE) as the outcome variable. In Models 1 and 3, we mobilize the raw number of respondents who said they would support the party while Models 2 and 4 use of the share of total official votes in our sample as the outcome variable. We also include a dichotomous variable that takes the value 1 for parties on the right of the political spectrum and 0 otherwise. We also control for the official vote share of the party in the district since E and AE can both be affected by electoral strengths as more marginal parties provide less space for underestimation as their actual supports get close to zero percentage points.
Table 3. Regression analysis (OLS) to identify the sources of errors
* p<0.05; Clustered standard errors
In Model 1, we want to test whether parties on the right of the political spectrum are disproportionately underestimated in exit polls. The model is performing rather well with a R2 of 18.65 per cent and a RMSE of 10.34. Controlling for two covariates, we find that support for parties on the right is underestimated in our exit-poll samples by an average of 9.64 percentage points (almost the size of the RMSE). Model 2 provides a better fit and a comparable substantive result, with an R2 that reaches 21.29 per cent and a RMSE of 10.17. Here, controlling for two covariates, we find that parties on the right are underestimated in our exit-poll samples by an average of 9.82 percentage points. In both models, the underestimation reached statistical significance at a 95 per cent level of confidence.
In Model 3, we estimate the impact of sample size on absolute sample precision. We thus use the AE as our outcome variable, the raw number of respondents stating they voted for a given party as our main predictor of interest, and the ideological inclination of the party and its official vote share as controls. The model explains 27.38 per cent of the variance and has an RMSE of 6.58. Results suggest that, on average, each additional vote for a given party captured in our exit poll increases the precision of our vote share estimation by 0.11 percentage points. In other words, we need roughly an extra nine respondents that say they support a given party to improve the precision of our estimation by 1 percentage point, regardless of the party's total proportion of the vote on election day. In Model 4, we use the proportion of the total votes for a party captured in our exit poll instead of the raw number of votes as a predictor. The model here is performing less with an R2 of 14.82 per cent and an RMSE of 7.13. The relationship between the proportion of the votes and the AE is not statistically different from 0. These results suggest that what matters here is not the proportion of actual votes, but rather the raw number of votes intercepted by interviewers.
Discussion
Exit polls, like all survey research, require constant attention in the selection of samples, questionnaire design, methods of field work, and data collection and analysis (Levy, Reference Levy1983: 56). They are an effective tool for election-day coverage as well as a “useful complement to data collected through more conventional designs” in the electoral behaviour research (Brown et al., Reference Brown, Docherty, Henderson, Kay and Ellis-Hale2006: 920). They can mobilize a variety of research designs such as experiments (Brie and Ouellet, Reference Brie and Ouellet2020) and can be an interesting teaching opportunity (Berry and Robinson, Reference Berry and Robinson2012). An ideal exit poll is one that uses a random selection of voters in a sample of representative precincts. A self-administered, short questionnaire to preserve the anonymity of respondents leads to higher response rates (Mitofsky, Reference Mitofsky, Lavrakas and Holley1991; Traugott and Price, Reference Traugott and Price1992; Bishop and Fisher, Reference Bishop and Fisher1995). A high-quality poll is conducted throughout an entire voting day, aiming for around a hundred respondents per precinct.
These two pilot studies partially meet most of these criteria. The questionnaires were short and self-administered, and in the case of the federal poll, conducted during the whole day. Yet, due to limited resources, we only managed to gather small samples going from a few dozens to a few hundred. This provided us with an opportunity to test the accuracy—both in terms of chance and bias errors—of small-scale exit polls in the Canadian context.
Compared to official results, we conclude that out of six voting locations, only one (Taschereau) was substantially imprecise. This is good news. More respondents meant more precise results, but the exact size needed is not that big; a few hundred per voting location were sufficient to produce good results. We also conclude that, on average, vote shares for parties on the right of the political spectrum were underestimated by a significant margin. The underrepresentation in our samples of right-of-centre voters is in line with what has been observed outside Canada, notably in the United States and Spain. Still the magnitude of this underrepresentation (around 9 points) is surprising and similar to dramatic cases in the United States (Best and Krueger, Reference Best and Krueger2012). This is unexpected since electoral politics is generally considered less polarized in Quebec than in the American context.
We count fifteen years between the 2003 Ontario exit poll and the 2018 and 2019 pilot studies mobilized—this gap was much too long. Canadian scholars interested in voting behaviour should invest more time and energy in organizing exit polls. Despite their shortcomings, they are simple and powerful survey design that complement other forms of data gathering, are relatively inexpensive to organize, and have strong potential to reach voters less likely to fill out online surveys or participate in phone surveys. Canadian political science has ignored a powerful research tool in the past decades. We hope this research note will be a step forward in promoting this method.
Open access
