Political scientists have become increasingly interested in naturally occurring randomized experiments stemming from government-conducted lotteries. For example, scholars have used the military draft lottery during the Vietnam War to assess its effects on political attitudes and behavior (Erikson and Stoker, Reference Erikson and Stoker2011; Green et al., Reference Green, Davenport and Hanson2018), the Georgia land lottery to investigate the effects of slave ownership on military service on behalf of the Confederacy (Hall et al., Reference Hall, Huff and Kuriwaki2019), the random assignment of judges to cases to understand how their rulings are affected by their partisan or ethnic backgrounds (Hall, Reference Hall2009; Gazal-Ayal and Sulitzeanu-Kenan, Reference Gazal-Ayal and Sulitzeanu-Kenan2010), random term lengths of state senators to assess how election proximity affects legislative effort and position taking (Titiunik and Feher, Reference Titiunik and Feher2018), random reservation of seats for women candidates to assess how reservations affect women's representation in subsequent elections (Bhavnani, Reference Bhavnani2009; Clayton and Tang, Reference Clayton and Tang2018), and an assortment of legislative procedures to gauge their effects on legislators' behavior or career paths (Kellermann and Shepsle, Reference Kellermann and Shepsle2009; Broockman and Butler, Reference Broockman and Butler2015; Cirone and Van Coppenolle, Reference Cirone and Van Coppenolle2018). The appeal of this research design is threefold. Random assignment allows for unbiased causal inference; government-run interventions often go well beyond what researchers could feasibly or ethically implement on their own; and ongoing lotteries represent an opportunity for researchers to obtain increasingly precise estimates of causal effects.
In this essay, we study the extent to which legislators who propose a bill or motion enhance their prospects for re-election. Since 2004, the Canadian House of Commons has conducted a lottery to allow Members of Parliament (MPs) to propose bills or motions. Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) were the first to make use of this lottery assignment to investigate the causal relationship between proposal power and electoral success. To the two parliaments (n = 404 legislators) they studied in their pioneering work, we add four subsequent parliaments (comprising 837 legislators). The cumulative dataset generates much more precise estimates than those originally reported. Like Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014), we find a weak overall main effect, but, unlike Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014), we find no evidence that effects are larger for those legislators in the government party. If anything, the pooled regression results show that significant positive interaction that Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) find between the treatment and legislators in the governing party has become weakly negative. These updated results change the theoretical implications that Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) drew from their initial findings. The present paper retraces their steps, presents updated estimates, and offers a fresh theoretical interpretation.
This paper is structured as follows. We begin by briefly describing key features of this legislative experiment—the random assignment process, the population of eligible subjects, and the coding of the treatment and outcome variables. Our version of the data from the first two parliaments closely matches the data originally analyzed by Loewen et al. Extending the dataset to include additional parliaments, we verify that the pattern of assignments creates treatment and control groups that, in each parliament, have covariate profiles that differ no more than would be expected by chance. After briefly describing the statistical models used to estimate the average treatment effect of proposal power as well as its interaction with belonging to the governing party, we present results for each of the six parliaments. Pooling the results over time, we find no evidence of a sizable treatment effect, whether among all eligible legislators or just those in the governing party. The paper concludes by reflecting on what the updated results suggest about the electoral value of proposal power in the context of strong party systems.
1. The Canadian proposal lottery
In Canada's Westminster-style system, MPs in the House of Commons are elected via a first-past-the-post electoral system. The party with the most seats forms the government and structures the legislative agenda. Prior to 2004, this system gave backbenchers little opportunity to shape the legislative agenda, but the rules were changed to allow certain MPs to make proposals.
At the start of each parliament, the names of all eligible members (i.e., those who are neither cabinet members nor parliamentary secretaries) are randomly chosen to determine who has the right to introduce bills or motions. The names of MPs and the order in which they are drawn form the Order of Precedence. Over the course of a parliamentary session, MPs are recognized in this random order until the session ends, which may occur before all eligible MPs have had a chance to make a proposal.
Because names are ordered randomly, researchers are well-positioned to compare the subsequent vote share of MPs who won the right to introduce a bill or motion with those who were not afforded this right. As Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) point out, this research design has clear advantages over non-experimental approaches: “Without being able to separate the independent effect of legislative action from other possible explanations of electoral viability, we cannot be certain of a causal relationship between legislating and winning elections” (p.189). Loewen et al.'s research design arguably improves on similar studies of other legislatures. For example, in his study of the British House of Parliament, Bowler (Reference Bowler2010) makes no use of its lottery and instead focuses on the correlation between the number of proposals that MPs make and the vote share in the next election.
Another attractive feature of the Canadian lottery is that a large number of MPs seize the opportunity to make proposals. Of the 599 legislators who, between 2004 and 2019, received a lottery number low enough to allow them to make a proposal in time for it to receive a second reading, 573 did so.Footnote 1 Conversely, none of the 667 MPs with “losing” lottery numbers made proposals. Thus, the “first-stage” relationship between random assignment and uptake is very strong.
2. Recreating Loewen et al.'s dataset
We begin by revisiting the data gathered by Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) from the first two legislative sessions in which proposal power was allocated randomly. The data for the present study come from two sources. At the beginning of each Parliament, the names of all eligible MPs (excluding the cabinet or parliamentary secretaries) are placed on a randomly ordered priority list for the consideration of Private Members' Business. Only MPs in the Order of Precedence can introduce a bill or motion during Private Members' Hour, which is time dedicated to debating and voting on bills and motions. We assembled this list by going through each Parliament's Order Paper and Notice Paper. We then went through each session's Status of House Business to determine which members appeared in the Order of Precedence and, if so, whether they proposed a bill or motion that was debated at second reading.Footnote 2 Second, we merged the list of all eligible experimental subjects with Sevi's (Reference Sevi2021) data on federal electoral results and candidates' sociodemographic attributes.
Coding the data freshly ourselves revealed only a few innocuous errors in the original dataset.Footnote 3 But the exercise called our attention to some subtle statistical assumptions. The first is that Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) code MPs in the 2004 session as treated (i.e., having the right to propose a bill or motion) if their random position on the list for consideration of private members' business was sufficient to allow their bill or motion to be discussed, debated, and voted on at second reading. This threshold was inferred based on the last MP who appeared in the Order of Precedence. All MPs whose lottery number placed them before this threshold are coded as treated; remaining MPs are considered untreated. This coding rule makes treatment somewhat dependent on legislators' choices, which potentially jeopardizes the key assumption that treatment and control subjects have the same expected potential outcomes. Fortunately, our robustness checks suggest that this coding has no material bearing on the results. Tables A7.1 and A7.2 show that results are unaffected if we exclude MPs whose lottery number put them at the threshold, and Tables A8.3 and A8.4 show that results are unchanged if we instead use the raw lottery number as an instrumental variable for making a proposal.
Another statistical assumption concerns attrition. The outcome measure is the incumbent's vote share in the ensuing election. However, the decision to run for election is post-treatment, which raises the possibility of differential attrition among treatment and control groups. Again, our robustness checks uncovered no evidence of bias stemming from attrition. Of the MPs eligible for the lottery in these two sessions, 87 percent ran for reelection. Inspection of session-by-session rates of attrition, each subdivided by treatment assignment (see Table 1), reveals no apparent asymmetries across experimental groups.Footnote 4
Session by session tabulation of attrition due to retirement, by random assignment
Figure 1 confirms that our own coding produces results that are quite close to those obtained using replication data from Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014). The graph shows the average vote share received by treatment and control incumbents after each of the two legislative sessions. Following Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014: 193), these averages are broken down according to whether the MP belonged to the governing party, to illustrate their principal conclusion: “the power to propose imparts a significant electoral advantage to members of the governing party” (p. 189). Because the Liberal Party was in government during the first session and the Conservative Party was in government during the second session, the implication is that incumbents benefit electorally only when they are in a position to propose motions that go on to pass (p. 192).
Incumbent vote shares by session, membership in the governing party, and data source.
Notes: Figure 1 shows box plots for vote share in the next election by session, government, and treatment status. The line in the middle of each box marks the median. The top of the box marks the upper quartile, and the bottom of the box marks the bottom quartile. Whiskers mark the range of the data apart from outliers, which are defined as data points lying more than 1.5 times the interquartile range away from the nearest quartile. Circles indicate outliers. The Ns are smaller for government than they are for opposition because the two government parties in this period were minority governments.
3. Out of sample replication and extension
In this section, we extend Loewen et al.'s (Reference Loewen, Koop, Settle and Fowler2014) original findings from the 38th (2004–2006) and 39th (2006–2008) Canadian parliaments to the 40th (2008–2011), 41st (2011–2015), 42nd (2015–2019), and 43rd (2019–2021). The resulting database contains a total of 1266 incumbents across the six parliaments from 2004 to 2019.
Table 2 shows the number of eligible MPs assigned to treatment or control for each of the six legislative sessions, restricting attention to those who stood for reelection. Table 3 shows their average demographic and party profiles, using the covariate information found in the Sevi (Reference Sevi2021) database. Tables A3.3–A3.5 confirm that these covariates are weakly related to MPs' randomly assigned treatments. F-tests that assess the joint significance with which these covariates predict treatment assignment are, as expected, insignificant at the 0.05 level in every session and for all six sessions pooled.
Assigned treatment by legislative session
Note: The Liberal Party governed in the 2004, 2015, and 2019 sessions. The Conservative Party governed in the 2006, 2008, and 2011 sessions. Only the 2011 and 2015 parliaments were majority governments.
Descriptive statistics
4. Statistical model
The primary estimand is the average causal effect of proposal power on an incumbent's vote share in the next election. This estimand is an intent-to-treat effect insofar as it focuses on the effect of the random assignment, regardless of whether legislators go on to actually propose a bill or motion. For a given legislative session, the regression model may be written
where Y i denotes the vote share won by legislator i, T i denotes the legislator's assigned treatment (1 = power to propose, 0 otherwise), G i indicates whether the legislator belongs to the governing party, F i indicates whether the legislator is female, V i represents the legislator's vote share in the previous election, N i refers to the number of terms that a legislator has served, and u i constitutes the unobserved disturbance term. The key parameter of interest is γ 1, which represents the average intent-to-treat effect. Covariates are included to dampen disturbance variance, thereby improving the precision with which γ 1 is estimated. These covariates are specified in our pre-analysis plan (https://osf.io/q567c). Robust standard errors are used to assess sampling variability.Footnote 5
In order to gauge whether treatment effects vary systematically according to whether each legislator belongs to the governing party, we follow Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) in adding an interaction (T iG i) term to the regression model:
When G i = 0 (i.e., the legislator is not a member of the governing party), the conditional average intent-to-treat effect is β 1. When G i = 1 (i.e., the legislator belongs to the governing party), the conditional average intent-to-treat effect becomes β 1 + β 6. In other words, the interaction effect β 6 represents the extent to which the average intent-to-treat effect changes as we shift our attention from legislators not in the governing party to those who are part of the governing party. Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) find this interaction effect to be positive, implying that legislators in the governing party benefit especially from proposal power. Intuition suggests that MPs in the governing party may benefit from proposal power because they are better positioned to have their proposals adopted. In our dataset their adoption rate is 63 percent, compared to 23 percent for MPs in the opposition.
A final concern when analyzing all six legislative sessions jointly is that the probabilities of assignment to treatment vary somewhat from session to session (see Table 1). For this reason, the pooled regressions include dummy variables for each of the sessions (less one), which reflects the fact that experimental design is essentially blocked by session. The use of session indicators follows Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014: 193).Footnote 6
5. Results
We begin by estimating the average intent-to-treat effect for each session. The first two columns of Table 4 estimate the average effects in the two legislative sessions studied by Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014). The estimate for 2004 is −0.511 (SE = 0.907), while that for 2006 is 0.217 (SE = 0.851). Taken together, these estimates imply that the electoral gains from winning the lottery are modest. A similar pattern holds for our out-of-sample replication. One of the updated estimates is negative, and three are positive. Only one of the positive estimates is larger than one percentage point.
Regression estimates of average treatment effects by session, controlling for covariates
Robust standard errors in parentheses.
How do the results change when we interact treatment with governing party? The first row in Table 5 reports the estimated intent-to-treat effect among legislators who are not part of the governing party. Three of the six legislative sessions, including both sessions studied by Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014), generate a negative estimate for this subgroup. The third row reports the estimated interaction effect, or the additional electoral benefit conferred on members of the governing party. In the first two legislative sessions, both estimates are substantial and positive: 3.899 (SE = 1.816) in 2004 and 3.355 (SE = 1.629) in 2006. But the out-of-sample replications generate a negative interaction in all four sessions. It no longer appears that proposal power confers special electoral benefits on legislators from the governing party.
Interaction of treatment assignment with membership in the governing party, by session
Robust standard errors in parentheses.
Summarizing the results across sessions, Table 6 reports the pooled regression and shows that the estimated treatment effects are scarcely affected by the modeling decision to include covariates in the regression models. Column (2) reports an average intent-to-treat effect of 0.598 (SE = 0.520) when all the sessions are pooled and the full set of covariates is included. Although this estimate is not statistically distinguishable from zero at conventional significance levels, its substantive interpretation is that, on average, legislators who win the lottery receive a bonus of six-tenths of a percentage point in the next election. Column (4) reports the pooled regression results when an interaction term is included, as in equation (2). The estimated interaction is negative, though statistically indistinguishable from zero (−0.806, SE = 1.047). It appears that legislators in the governing party receive no special benefit from winning the lottery.
Pooled regression results, with and without interactions between treatment and membership in the governing party
Robust standard errors in parentheses.
6. Conclusion
The present study offers an out-of-sample replication of the pioneering Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) study and comes to rather different conclusions. The original study found evidence of a statistically significant positive interaction between the proposal lottery and membership in the governing party, implying that “members of the government who are randomly granted the right to propose legislation on average earn 2.73 percentage points more vote share than those government members who are not” (p. 193). Adding data from four legislative sessions, our updated assessment of proposal power suggests that it confers a 0.869–0.806 = 0.063 percentage points boost for government members, which lies outside the 95 percent confidence interval that the authors originally reported (p. 192). The interaction effect, too, disappears. The point estimate for the interaction becomes weakly negative and again lies outside the originally reported 95 percent confidence interval. In sum, we no longer see evidence that government MPs benefit more from the lottery than non-government MPs. Because government MPs are much more likely than MPs from opposition parties to make proposals that pass, the apparent lack of interaction implies that voters are not especially prone to reward MPs whose proposals are adopted.
What is the average treatment effect of proposal power for all MPs across all sessions? Our best guess from the pooled dataset is that winning the lottery confers an average benefit of just under six-tenths of a percentage point for all MPs. Although tripling the size of the original dataset has reduced the margin of error considerably, our pooled estimate (see column 2 in Table 6) of the intent-to-treat effect still implies a 95 percent confidence interval (−0.421, 1.617) that includes zero. At the top of this confidence interval, effects are large enough to be substantively meaningful (and would have changed election outcomes in 5 percent of the races in our sample). For now, neither the intent-to-treat effect nor complier average causal effect is convincingly greater than zero.
What are the substantive and methodological implications of relatively small electoral gains from proposal power? Substantively, the updated results resolve an anomaly pointed out by Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014). The large magnitude of the effect they observed is surprising given the Canadian context. “Strong evidence of a ‘personal vote’ is unexpected in a parliamentary system characterized by strong party discipline” (p.194). “Canadian elections,” they point out, “are strongly party-centered rather than candidate-centered” (p.190). Party discipline has long been and remains a prominent theme in scholarship on Canadian party politics and legislative elections. MPs who deviate from the party line do so to the detriment of their political careers (Godbout and Hoyland, Reference Godbout and Hojland2011, Reference Godbout and Hojland2017). Other work suggests there is no personal incumbency effect in recent legislative elections (Sevi, Reference Sevi2022), although there appears to be a party incumbency effect (Kendall and Rekkas, Reference Kendall and Rekkas2015). Our updated results offer much more tentative support for the claim “that politicians take advantage of legislative opportunities and that voters reward them for doing so, even in parliamentary systems that deemphasize the role of the individual legislator and emphasize party loyalty over constituency service” (p. 190). Yes, legislators do make use of proposal power, but few voters seem to reward them for doing so.
The methodological implications of the current exercise are also noteworthy. First and foremost, our results underscore the importance of ongoing out-of-sample replication. Loewen et al. (Reference Loewen, Koop, Settle and Fowler2014) deserve credit for directing researchers to an important lottery, for developing sound procedures for reconstructing the treatment assignments from archival data, and for making their data and procedures transparent and publicly available. They made it easy for subsequent researchers to follow in their footsteps. Loewen et al. should also be credited with exploring their data in ways that generated an important and testable hypothesis about the interaction between the power to propose and being part of the governing party. But as sometimes happens when pathbreaking studies report an intriguing and statistically significant treatment-by-covariate interaction, the interaction that is the focus of the Loewen et al. article did not hold up when tested out-of-sample. This replication failure does not stem from questionable research practices. The fact that this particular hypothesis did not pan out serves as a reminder that propositions generated inductively remain provisional pending out-of-sample replication.
The broader message of this essay is to underscore the importance of periodically revisiting ongoing naturally occurring random assignments, for two reasons. The first is to assess whether results change as conditions change. Other replications have discovered important evidence of structural changes over time. For example, when replicating Washington's (Reference Washington2008) finding that members of the US House of Representatives with daughters were more likely to cast “feminist” roll call votes, Costa et al. (Reference Costa, Greenlee, Nteta, Rhodes and Sharrow2019) find that this relationship subsided over time, perhaps reflecting deepening partisan divides. The second reason is that few, if any, pioneering experiments are analyzed based on a pre-registered plan. In the course of exploring an intriguing natural experiment, researchers may sift through main effects and interactions in search of substantively interesting results. Exploration plays a crucial role in social science research, but it may create a multiple comparisons problem (Humphreys et al., Reference Humphreys, de la Sierra and van der Windt2013) that exaggerates the statistical significance of the findings that ultimately make it into print. In some sense, sentinel studies serve as pre-analysis plans for out-of-sample replications, such as ours.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2022.60.
Acknowledgements
The authors thank Peter Loewen, André Blais, and Marius Sältzer for their helpful comments on an earlier draft and Maxime Ricard, Procedural Clerk for the House of Commons, who patiently answered our questions and guided us through archived documents.
Conflict of interest
None.
Open access
