TWFE Estimates of Shootings’ Effect on Vote Shares

We start by showing that different conclusions across studies on the electoral effects of shootings are not driven by data choices. Figure 1 removes differences in methodological approaches in previous studies and shows the effects of TWFE models (GMAL’s and Yousaf’s approaches) using the data from all of the studies. Figure 1 also splits the results by treatment coding approaches outlined above.Footnote ¹⁵

As shown in Figure 1, TWFE specifications consistently produce substantive positive statistically significant effects regardless of the time frame, treatment codings—be they “rampage-style” school shootings (GMAL), school shootings (HHB), or all mass shootings (Yousaf)—or how long treatments apply. With GMAL’s data, we find—like GMAL did—that “rampage-style” shootings correlate with increases in Democratic vote share. This estimate (while varying by specifications) is 5.5 percentage points in the TWFE estimator for the first treatment coding (i.e., panel a) and is 8.7 percentage points in the second treatment coding (i.e., panel b). Both estimates are highly significant ( $ p<0.01 $ ). The effects for HHB and Yousaf are very similar. Simply, when using the same model choices, the effects of shootings of different types are consistently sizeable. In other words, previous differences in conclusions across studies of the electoral effects of gun violence are not due to choices about which shootings count as treatment. In short, TWFE estimates, regardless of the coding of shootings, indicate significant and substantively meaningful positive effects of shootings on Democratic vote share.

We pause to discuss effect magnitude. Upwards of an 8.7 percentage point shift in Democratic vote share is large—as are many of the other estimates. As GMAL note, these effects represent “a remarkable shift in an age of partisan polarization and close presidential elections” (GMAL, 809). We see how large these effects are by benchmarking them to other studies using county-level vote shares and difference-in-differences designs. For example, Sides, Vavreck, and Warshaw (Reference Sides, Vavreck and Warshaw2022, 709) estimate a six-standard-deviation shift in relative television advertising produces a 0.5-point change in two-party vote share. Hence, if we believe these results travel, GMAL’s simple TWFE estimates indicate one school shooting has an effect on Democratic vote share equivalent to a shift of approximately 66–104 standard deviations in relative advertising. Using an economic comparison—the most common of retrospective voting treatments—Healy and Lenz (Reference Healy and Lenz2017, 1423) show a “1 percentage point increase in mortgage delinquencies increases Democratic vote share by 0.33 percentage points.” Thus, the effect of “rampage-style” shootings is roughly the equivalent to a 16.7–26.4 percentage point increase in mortgage delinquencies; or moving from a world with no delinquencies to one where about one-fifth of residents are at risk of losing their homes.

In short, TWFE models suggests gun violence—regardless of how shootings are coded—fundamentally reshapes electoral results in the local communities in which they occur. Is this sizable relationship causal and robust? Recent methodological developments provide us a guide to answer this question.

Checking for Visual Evidence of Differential Pretreatment Trends

Because of the fundamental problem of causal inference, we do not observe counter-factual worlds where the treated and untreated groups are exposed to opposite conditions. Hence, no singular test can prove parallel trends is satisfied; however, treated and untreated units not moving together before treatment exposure, indicates potential issues (De Chaisemartin and D’Haultfoeuille Reference De Chaisemartin and D’Haultfoeuille2023; Marcus and Sant’Anna Reference Marcus and Sant’Anna2021).Footnote ¹⁶ An appropriate first step is to visually inspect patterns in aggregate-level data to see whether, prior to treatment, treatment and control areas are trending in different directions.Footnote ¹⁷

Figure 2 examines differential pretreatment trends separating counties into two bins; panel (a) contains counties with a shooting excluding all post-treatment observations, and panel (b) contains all counties without shootings.Footnote ^{18 ,} Footnote ¹⁹ Figure 2 illuminates what TWFE models absorb and do not absorb. County fixed effects adjust for differences in Democratic vote share across counties. Year fixed effects account for differences across years. However, TWFE models do not account for the possibility counties’ Democratic vote shares change at different rates over time, a significant problem in the context of school shootings.

As Figure 2 shows, shootings happen in communities—proceeding and unrelated to shootings themselves—trending more Democratic relative to other locales. This is likely because mass shootings occur disproportionately in growing populations and have increased over time (Musu-Gillette et al. Reference Musu-Gillette, Zhang, Wang, Kemp, Diliberti and Oudekerk2018; U.S. Government Accountability Office 2020) at the same time American politics has realigned with these same more populated areas becoming more Democratic (DeSilver Reference DeSilver2016). In general, researchers should take care when demographic/political changes predating treatment aligns with short-term treatment exposure.

This (coincidental) pretreatment trend separation becomes especially prevalent after 2004. This is particularly problematic because GMAL explicitly note shootings effects before 2004 are essentially null (or negative), but in 2004 the positive effects on Democratic vote share increase (GMAL, Figure 7). Figure 2 indicates this is the exact time when parallel trends assumptions become particularly tenuous. Two-way fixed effects models do not absorb these trends and, as such, are likely biased estimates.

Checking for Pretreatment Effects with the Model Specifications Used

While Figure 2 provides visual evidence of differences in pretreatment trends, it is not dispositive. Graphical representations can differ and changes in formatting can minimize or exacerbate the appearance of differential trends leading researchers to draw different conclusions.Footnote ²⁰ Hence, the next check to assess TWFE design validity which should be standard practice is whether this model suggests impacts prior to treatment (Grimmer et al. Reference Grimmer, Hersh, Meredith, Mummolo and Nall2018). In our case, this placebo test is informative as shootings—something that people cannot precisely anticipate—should not affect vote shares prior to their occurrence. If there are effects, it suggests TWFE estimates are likely not causal (Angrist and Pischke Reference Angrist and Pischke2008; Hansen and Bowers Reference Hansen and Bowers2008).

Specifically, we run the specification listed in Equation 2. Equation 2 is the same as Equation 1, except for the outcome variable. Instead of estimating shootings’ effects ( $ {D}_{ct} $ ) in subsequent elections ( $ {Y}_{ct} $ ), we substitute a lagged outcome variable ( $ {Y}_{ct-k} $ ). Here, k corresponds to the number of lagged periods included. We include seven lagged periods in our models as the GMAL panel is sufficiently long. However, power considerations may influence the number of lags used. We recommend initially looking for effects in one period lag, then looking how far back one can estimate precise-enough specifications:Footnote ²¹

(2) $$ \begin{array}{rl}{Y}_{ct-k}={\phi}_c+{\lambda}_t+\beta *{D}_{ct}+{\epsilon}_{ct}.& \end{array} $$

Panels a and b (the top section) of Figure 3 (we discuss panels c–f later) show the TWFE models for various shootings on lagged measures of Democratic vote share and show there is substantial imbalance in lagged outcomes.Footnote ^{22 ,} Footnote ²³ We start on the left of each panel, with the presidential election prior to the shooting and work up to seven presidential elections (28 years) before shootings occurred.Footnote ²⁴ Effects vary by specification, but range between 2 and 7 percentage points, with most highly significant. This analysis indicates mass shootings have a significant and substantive effect on Democratic vote shares up to 20 years prior to a shooting. Simply, the TWFE does not recover balance prior to shootings, regardless of data used.

There is little reason—theoretically or empirically documented—to suspect shootings should have anticipatory effects on vote shares, given these events are unexpected where they occur. There is, however, the possibility pretreatment effects show up where there is not bias if treatment in one period ( $ {D}_{ct} $ ) is highly correlated with treatment in prior periods ( $ {D}_{c,t-k} $ ). In that case, the coefficient on $ {D}_{ct} $ may show an effect if there is an effect of $ {D}_{c,t-k} $ on $ {Y}_{c,t-k} $ . In such cases, pretreatment effects could emerge even without errors in the research design. Therefore, we recommend also modeling effects of lagged and leaded treatment using an event-study design.

Checking for Pretreatment Trends with Event-Study Designs

An event-study design traces effects before and after treatment and provides another way to see pretreatment imbalances (Armitage Reference Armitage1995; Binder Reference Binder1998). An event-study is an increasingly common difference-in-differences model, given its less restrictive and more transparent modeling assumptions, but still relatively rare in political science. An event-study (usually) uses TWFE but also includes lagged and lead treatment variables as shown in Equation 3 below. We list treatment in a given county (c) and year (t), lagged or leaded by the corresponding periods since treatment (k). For simplicity, Equation 3 shows the event-study model for only one pretreatment period ( $ k-2 $ ), the period when treatment occurs (k), and one period after treatment occurs ( $ k+1 $ ). The baseline is the period before treatment occurs ( $ k-1 $ ) (Armitage Reference Armitage1995; Binder Reference Binder1998):

(3) $$ \begin{array}{rl}{Y}_{ct}={\phi}_c+{\lambda}_t+{\beta}_{-2}*{D}_{ct,k-2}+{\beta}_0*{D}_{ct,k}+{\beta}_1*{D}_{ct,k+1}+{\epsilon}_{ct}.& \end{array} $$

Figure 4 shows nine preelection treatments and eight posttreatment periods included using the GMAL data. The right of Figure 4 (right of the gray vertical line) shows an immediate, significant, and substantive jump in Democratic vote share the election year following a shooting and grows after the event occurs, having an increasingly larger effect on Democratic vote share (10–13 points), and allows us to rule out smaller effects using equivalence testing. While not completely impossible, the long-lasting and growing effect remains theoretically unexplained.

However, by looking to the left of the baseline period (left of the gray dotted vertical line), Figure 4 also illuminates the “effect” is unlikely to be causal. If the TWFE models were causal, these coefficients should not be significantly/substantively distinct from zero. Instead, Figure 4 shows that relative to one election prior to a shooting, prior election years see less Democratic support and this underperformance increases further back in time. Simply, vote shares trend more Democratic before shootings in counties where shootings occur (relative to counties without shootings). Elections after a shooting are just a continuation—indeed, the points represent an almost perfect linear function—further evidence that TWFE estimators are biased in this case. We recommend parameterizing models as an event-study become standard in difference-in-differences applications.

Controlling for Any Differential Unit-Specific Time Trends

Facing potential parallel-trends violations, one potential remedy is adjusting for factors—observed or unobserved—leading to pretreatment imbalances. In this case, visual inspection (see Figure 2) reveals treated and untreated units have different pre-trends. A solution is to include unit-specific time trends (Angrist and Pischke Reference Angrist and Pischke2008; Reference Angrist and Pischke2010; Wing, Simon, and Bello-Gomez Reference Wing, Simon and Bello-Gomez2018), making the identifying assumption deviation from county-year trends. Identification comes from sharp deviations from otherwise smooth unit-specific trends corresponding to the following equation:

(4) $$ \begin{array}{rl}{Y}_{ct}={\phi}_c+{\lambda}_t+{\gamma}_c*t+\beta *{D}_{ct}+{\epsilon}_{ct}.& \end{array} $$

While Equation 4 includes linear county-specific time trends ( $ {\gamma}_c*t $ ), the functional form is a potentially influential decision in the hands of a researcher.Footnote ²⁵ As a result, we run many model specifications—all taking slightly different tacts to adjusting for differential pre-trends. The next section shows various other approaches, including methods recently developed by Liu, Wang, and Xu (Reference Liu, Wang and Xu2024), Freyaldenhoven et al. (Reference Freyaldenhoven, Hansen, Pérez and Shapiro2021), and Rambachan and Roth (Reference Rambachan and Roth2021). We recommend scholars run robustness checks across several parameterizations of the unit-specific trends, acknowledging higher-order unit-specific trends could face a bias-variance tradeoff, especially in smaller datasets.

Figure 3c–f shows the pretreatment effect models with linear and quadratic trends (adding $ {\gamma}_c*{t}^2 $ to Equation 4). For cubic and quartic county-specific trends, see Supplementary Figures S9 and S10. In contrast to the TWFE (Figure 3a,b), all models with county-specific trends are balanced pre-treatment. Importantly, these null effects (especially in proximate periods) are not driven by standard error inflation ruling out even modest pretreatment differences using equivalence testing (effects outside the $ -0.99 $ to 0.57 percentage point range).

Figure 5 shows estimates for models including linear and quadratic county-specific time trends on the posttreatment outcomes with only counties with shootings in that year coded as treated.Footnote ²⁶ As Figure 5 shows, once we make this necessary adjustment to correct for the violation of the parallel trends assumption, the effects of mass shootings of all types attenuate heavily. All effect estimates are much smaller and virtually all are not significant.

Specifically, Figure 5a,b—the estimates for “rampage-style” school shootings (i.e., GMAL’s treatment)—indicates a 0.7 percentage point increase in Democratic vote share, an effect that is not statistically significant and 7.9 times (i.e., 790%) smaller than the original TWFE effectively ruling out effects as large as the TWFE with a high degree of confidence.Footnote ^{27 ,} Footnote ²⁸ We can effectively rule out effects of “rampage-style” shootings (GMAL’s treatment) larger than 1.5 percentage points and smaller than $ -0.1 $ percentage points, effects of all mass shootings (Yousaf’s treatment) larger than 1.2 percentage points and smaller than $ -1.6 $ percentage points.Footnote ²⁹ Effects of all school shootings (HHB’s treatment) are also much smaller and not significant. Simply, there is no evidence of large effects documented in previous work finding an effect, and no consistent evidence for positive (or negative) effects statistically distinguishable from zero regardless of the data used.Footnote ³⁰ While statistically significant effects infrequently show up in Figure 5, they are not robust. Notably, if we add higher-order polynomials—as in Supplementary Figure S9—no effects are significant. This—along with further checks below—emphasizes the lack of support for the conclusion that shootings have significant, systematic, or large effects on Democratic vote shares.Footnote ³¹

Using an event-study design that accounts for the differential pre-treatment trends by adding leads and lags of the treatment values bolsters conclusions as effects are even smaller and more precise. Figure 6 uses Freyaldenhoven et al. (Reference Freyaldenhoven, Hansen, Pérez and Shapiro2021) methods displaying event-study designs and accounting for pre-trends in event-study designs.Footnote ^{32 ,} Footnote ³³ Figure 6 uses the same y-axis as Figure 4 for ease in comparison. Adding trends attenuates pre-treatment imbalances. So too, however, are any large post-treatment differences as the immediate effect of “rampage-style” school shootings is a mere 0.8 percentage point bump for Democrats. This effect is barely statistically significant using the linear trends models (p=0.048) but still allows us to confidently rule out large effects; we can rule out effects larger than 1.67 percentage points using equivalence testing, nowhere near the size of GMAL’s 5 percentage point estimate emphasized throughout their text. Using the quadratic trends model, the effect is only marginally significant at the 10% level (p<0.066) and we can rule out effects larger than 1.71 percentage points.Footnote ³⁴ These effects appear to be the upper bound produced from this method.

If we use Clarke and Tapia-Schythe’s (Reference Clarke and Tapia-Schythe2021) approach to estimating the event-study with trends and the corresponding eventdd command in STATA, we get negative effects on Democratic vote shares (albeit statistically indistinguishable from zero). With this slightly different approach, the effects in elections immediately following shootings are $ -0.13 $ percentage points ( $ p=0.898 $ ; 95% CI: [−2.05, 1.80]).Footnote ³⁵ Other shooting codings are similarly very small and insignificant. Moreover, testing the robustness of these effects to other pretreatment periods—as the approach designed by Freyaldenhoven et al. (Reference Freyaldenhoven, Hansen, Pérez and Shapiro2021) allows—effects are even smaller and even less suggestive of an effect (see Supplementary Figures S2–S5). Benchmarked to the two-period lag trend, the estimate for elections immediately after a shooting using linear county trends is 0.47 percentage points ( $ p=0.324 $ ; 95% CI: [−0.5, 1.4]). Estimates for elections after a shooting for the quadratic county trends model is 0.5 percentage points ( $ p=0.359 $ ; 95% CI: [−0.5, 1.5]). Moreover, none of the large longer-term effects remain.

In short, in event-studies adjusting for unit-specific trends, there is little evidence for the sizeable effects previously suggested. In fact, there is little evidence for any significant effect. The occasional effect crossing the $ p<0.05 $ threshold are not robust to reasonable model variations, such as the baseline comparison points one uses. Moreover, while some specifications cannot fully exclude some much smaller but non-negligible positive effects, most 95% confidence intervals also cannot rule out non-negligible negative effects.

Additional Checks Addressing Violations of Parallel Trends Assumptions

Including unit-specific time trends (see above) is not the only solution to parallel-trends violations; indeed, scholars may desire more flexible solutions. Recent advances have suggested many alternative solutions to potential parallel-trends violations or unobserved time-varying confounders. Unfortunately, these approaches have not yet been benchmarked to one another, let alone compared in different data contexts. The standard approach is to develop a new estimator and assert it applies in all contexts. We are currently unaware of work testing these methods head-to-head, let alone providing recommendations regarding which methods to apply in different contexts. As such, we recommend scholars run a variety of specifications, as we do below, identifying general patterns. We recommend that scholars implement, at minimum, checks suggested by Liu, Wang, and Xu (Reference Liu, Wang and Xu2024) and Rambachan and Roth (Reference Rambachan and Roth2021) outlined below.

Building on research exploring factor-augmented models for causal identification (e.g., Bai and Ng Reference Bai and Ng2021; Xu Reference Xu2023), Liu, Wang, and Xu (Reference Liu, Wang and Xu2024) develop procedures—including what they call the fixed effects counterfactual estimator, the interactive fixed effects counterfactual estimator, and the matrix completion estimator—to “estimate the average treatment effect on the treated by directly imputing counterfactual outcomes for treated observations” (1).Footnote ³⁶ Using simulations, Liu, Wang, and Xu (Reference Liu, Wang and Xu2024) show that the interactive fixed effects counterfactual estimator provides more reliable causal estimates than conventional TWFE models when unobserved time-varying confounders exist. The interactive fixed effects counterfactual estimator can be applied with the package panelView, which is available in both Stata and R (Mou, Liu, and Yiqing Reference Mou, Liu and Yiqing2022b).

Figure 7 applies Liu, Wang, and Xu’s (Reference Liu, Wang and Xu2024) approach using the GMAL data.Footnote ³⁷ Panel a shows the TWFE and panels b–d show interactive fixed effects counterfactual estimators. In the TWFE model, there are pretreatment imbalances and a general overall upward trend—as Figure 5 indicated previously. Again, this suggests the TWFE picks up on a general pro-Democratic trend in pre-treatment periods. However, after adjusting for pretreatment differences using Liu, Wang, and Xu’s (Reference Liu, Wang and Xu2024) approach to address pretreatment imbalances, there is virtually no evidence shootings substantially or significantly affect vote shares in subsequent elections. Nor is this for lack of statistical power and we can rule out larger effects using equivalence testing. Moreover, any (much smaller) effects that appear intermittently are not robust to reasonable model variations in the realm of researcher decision-making.

Figure 7.

Liu, Wang, and Xu (Reference Liu, Wang and Xu2024) Interactive Fixed Effects Counterfactual Estimator

Note: The interactive fixed effects counterfactual estimator developed by Liu, Wang, and Xu (Reference Liu, Wang and Xu2024) using GMAL’s data. Panel a shows the TWFE estimated by Liu, Wang, and Xu’s (Reference Liu, Wang and Xu2024) FECT package; it is analogous to Figure 5, but their procedure estimates slight differences—for example, in the number of pre- and posttreatment periods. In panel b, the number of factors (r) is set to 3—that chosen by cross-validation and the degree of the polynomial is set to 4. In the bottom row, r is set to 1 in both panels and degree 2 in panel c and 4 in panel d. For other variations, see the Supplementary Material. Takeaway: The upward trend in the TWFE model (i.e., panel a) is indicative of violation of the parallel trends assumption. In the interactive fixed effects models, there is no evidence of the substantial effects shown in more simplistic model specifications that do not account for potential violations of the parallel trends assumption.

Rambachan and Roth (Reference Rambachan and Roth2021) propose another solution using a sensitivity analysis approach for potential violations of the parallel trends assumption allowing researchers to avoid arbitrarily choosing a parametric model.Footnote ³⁸ This approach is particularly useful as researchers often struggle to know the functional form of the underlying system.

This sensitivity analysis can be formalized in several ways. For example, researchers can see how robust their effects are to varying departures from differential trends evolving smoothly over time. This may be especially useful when concerned “about confounding from secular trends … evolv(ing) smoothly over time” (Rambachan and Roth Reference Rambachan and Roth2021, 13)—as we are in this case. This sensitivity test is “done by bounding the extent to which the slope may change across consecutive periods” (Rambachan and Roth Reference Rambachan and Roth2021, 12).Footnote ³⁹ They call this the $ SD $ or “second derivative” or “second differences” approach.Footnote ⁴⁰ Using Rambachan and Roth’s (Reference Rambachan and Roth2021) general approach, conclusions do not depend on arbitrary model specification choices. In essence, this approach “show[s] what causal conclusions can be drawn under various restrictions on the possible violations of the parallel-trends assumption” (Rambachan and Roth Reference Rambachan and Roth2021, 1). This approach is implementable through the HonestDID package in R and STATA (Rambachan and Roth Reference Rambachan and Roth2021; Rambachan, Roth, and Bravo Reference Rambachan, Roth and Bravo2021).

Rambachan and Roth (Reference Rambachan and Roth2021, 28) note “it is natural to report both the sensitivity of the researcher’s causal conclusion to the choice of this parameter and the ‘breakdown’ parameter value at which particular hypotheses of interest can no longer be rejected.” Figure 8 shows Rambachan and Roth’s sensitivity approach using GMAL’s and HHB’s data. Figure 8 uses the $ SD $ approach, plotting robust confidence sets for the treatment effect in the mass shooting case for different values of the parameter M. The confidence sets show that the effect of mass shootings on Democratic vote share is only positive and significant in the coefficient on the far left—the original estimate not allowing for any violations of parallel trends—indicating effects of mass shootings are highly sensitive to any minor departures from parallel trends. The robust confidence intervals include zero when allowing for linear violations of parallel trends ( $ M=0 $ ), and become even wider allowing for nonlinear violations of parallel trends ( $ M>0 $ ). Such a low breakdown suggests any meaningful departure of the slope changing between consecutive periods, would cause the observed effects in GMAL’s data to not be significant. Overall, these results indicate effects of shootings (using the GMAL data) are highly sensitive to even minor parallel-trends violations.

Figure 8.

Implementing Rambachan and Roth’s Sensitivity Analysis in the Shooting Example

Note: Results from the sensitivity analysis suggested by Rambachan and Roth (Reference Rambachan and Roth2021) using GMAL’s data—that is, testing for effect sensitivity across $ {\triangle}^{SD}(M) $ . The models incorporate information from three elections prior to treatment and five post-treatment periods. Takeaway: The results show the effects of shootings on vote shares are highly sensitive and do not hold with even minor deviations from parallel trends.

Summarizing Tests for Violations of Parallel Trends

Testing for potential violations of parallel trends is often not easy or straightforward because of the fundamental problem of causal inference—that is, the inability to observe what would have happened to treated groups without treatment. But even when considering only tests of pre-trends, there are challenges. For example, in choosing methods to address this core issue, one must consider the nature of the data available, the statistical power, and numerical degrees of freedom (Bilinski and Hatfield Reference Bilinski and Hatfield2018; Roth Reference Roth2022). We are not arguing every case employ unit-specific trends. What we are arguing is that all researchers should diagnose and address potential violations of this core assumption. How they do so—with the many tools at their disposal that we have outlined above—is less important than that they do so in a way that is transparent, thorough, and appropriate to the applied case.

In the case of mass shootings, this much is clear: once we make adjustments for clearly differential pre-trends, the evidence for any effect gets much more muddled than previous studies have suggested. Effects are substantively smaller than what simple TWFE models suggest. There is no clear evidence of durable lasting effects. Moreover, depending on the dataset one uses and the way treatment is coded, effects can be marginally positive, negative, or (approximately) zero, are very rarely significant, and highly sensitive to any meaningful departures from strictly parallel trends.