RISKS OF OVERFITTING FUNDAMENTALS FORECASTS OF GUBERNATORIAL RACES

Forecasting gubernatorial elections from fundamentals requires grappling with a major methodological challenge: a high predictor-to-observation ratio risks overfitting. Overfitted models nicely predict in-sample elections but poorly predict out-of-sample elections. This looms large for US presidential elections, which offer relatively few races on which to train and evaluate model performance. Subnational races and subnational data potentially increase the number of cases, although the degree to which these cases are independent is unclear.Footnote ²

Forecasting gubernatorial races has the potential to avoid this “small-n problem” (Turgeon and Rennó Reference Turgeon and Rennó2012) by pooling elections in more than 50 jurisdictions and many historical elections.Footnote ³ Realistically, however, scholars often lack data for key fundamentals— governor approval, state-level presidential approval, and economic conditions—from state-specific sources.Footnote ⁴ Thus, without a strategy to pool data from different sources, previous gubernatorial election forecasts have used national-level presidential approval as the key political fundamental, which does not vary across states (Hummel and Rothschild Reference Hummel and Rothschild2014; Klarner Reference Klarner2012). Novel data described later partially alleviate this issue, but data scarcity continues to depress the prediction-to-observation ratio’s denominator by restricting certain years and states from the training set. A particular challenge is low-population states in which gubernatorial and presidential approval are infrequently polled (Singer Reference Singer2023). More importantly, extant fundamentals-based forecasts of governors’ races raise the specter of overfitting by including numerous predictors and temporal lags. Among the 30 variables in Klarner’s forecast (2012, 689), for example, are a lagged dependent variable and the lags of 16 other predictors; only 3 reach conventional levels of statistical significance. Although this approach may have some advantages in exploratory settings, it exacerbates problems of overfitting and collinearity that hamper the model’s interpretability and parsimony. Absent micro-foundations to guide variable selection, effective and efficient models can prove elusive. Yet, we argue, a fundamentals-based forecast of subnational elections is possible if forecasters have a systematic strategy to address limited data and manage degrees of freedom.

THE LASSO-POPULARITY MODEL: REDUCING THE FORECAST’S PREDICTOR-TO-OBSERVATION RATIO

To avoid overfitting a fundamentals forecast of gubernatorial elections, we seek to reduce the numerator of the predictor-to-observation ratio via regularized regression using LASSO (least absolute shrinkage and selection operator). This machine-learning approach explores many variables but only selects those with significant predictive power. Initially developed in geophysics, LASSO has been adopted in economics and finance for forecasting (Santosa and Symes Reference Santosa and Symes1986; Tian, Yu, and Guo Reference Tian, Yu and Guo2015). Variable selection proceeds as follows. Variables are introduced into the model and weighted with a penalty that attempts to shrink their coefficients to zero without an increase to a loss function. When a coefficient cannot be shrunken to zero without increasing the loss function, then the variable is selected. Through this recursive routine, LASSO selects a parsimonious model that is a “good fit” for the training data.Footnote ⁵ And because it is not overfitted, the model should generate more robust out-of-sample forecasts. Overall, we expect LASSO to perform well in forecast model development and selection for our fundamentals forecast of governors’ races and for any forecast with a high predictor-to-observation ratio, including the other forecasts in this symposium.

We also raise the denominator of our predictor-to-observation ratio with new data. Whereas presidential approval is the dominant political fundamental in presidential election forecasts, gubernatorial popularity, as well as state-level presidential approval, likely shapes gubernatorial elections. Lacking data on both, previous gubernatorial forecasts relied on national-level presidential approval as a proxy (Hummel and Rothschild Reference Hummel and Rothschild2014; Klarner Reference Klarner2012). Our model, however, uses governor popularity from the State Executive Approval Dataset (SEAD; Singer Reference Singer2023).Footnote ⁶ This resource updates Beyle, Niemi, and Sigelman’s (Reference Beyle, Niemi and Sigelman2002) database of governor approval and then uses Stimson’s (Reference Stimson1991) dyads-ratio algorithmFootnote ⁷ to combine all available polls on citizens’ evaluations of governor performance, extract their latent common variance, account for “house effects,”Footnote ⁸ and produce quarterly estimates of governor approval ratings. We complement SEAD data with state-level measures of presidential approval generated using the same methodology.Footnote ⁹

In selecting input variables for the LASSO routine, we build from the substantial fundamentals forecast literature for presidential races and the small literature on state-level races. Specifically, we incorporate governor and state-level presidential approval estimates alongside a range of other political and economic fundamentals described later and measured two quartersFootnote ¹⁰ or more before Election Day into forecasts of (1) vote share and (2) incumbent-party victory.

TRAINING THE LASSO-POPULARITY FUNDAMENTALS FORECAST MODEL

Table 1 displays the results of an OLS training model using the predictors selected by LASSO on incumbent-party vote share. We present OLS coefficients instead of the direct LASSO estimates because the coefficients between the two estimation approaches are nearly identical (appendix 3), and LASSO’s penalized regression approach does not produce estimates of uncertainty or model fit (standard errors, R², etc.).

Table 1

Training Models: LASSO-Popularity Forecast of U.S. Governors’ Races

Notes: OLS estimates based on model selected by linear LASSO.

Standard errors in parenthesis. **p < 0.01, * p < 0.05 (two-tailed).

Overall, the LASSO-Popularity model performs well. Most coefficients are precisely estimated, and model fit is good but not overly good (Adj. R² = 0.44). The model also exhibits substantial out-of-sample accuracy over the dataset’s range. Using leave-one-out validation (LOOV),Footnote ¹⁶ its mean absolute error (MAE) is 5.6 percentage points, a 37% improvement in accuracy over a naïve, intercept-only model (MAE = 7.7 percentage points).

We can also assess the LASSO-Popularity model’s validity by using only “before-the-fact” training data for each election since 1990. To do this, we forecast each gubernatorial race using only data that occurred before that date; that is, 1990 predictions are based only on pre-1990 elections. Given diminished training data, the MAE for this approach is unsurprisingly larger but only marginally so (5.9).

The substantive results are consistent with a fundamentals-based model. Gubernatorial elections significantly reflect the incumbent governor’s popularity, especially if the incumbent is running for reelection, but even when not.Footnote ¹⁷ National politics also influence state politics. Presidential approval matters for gubernatorial races, particularly when the governor and the president are from different parties. These factors are predictive apart from the effects of state-level tendencies to vote for either the Democrats or the GOP.

Visually, we observe a strong correlation between gubernatorial approval two quarters from Election Day and incumbent-party candidate vote share (figure 1A). The “before-the-fact” MAE for this bivariate model is 6.9 percentage points.Footnote ¹⁸ But, as figure 1B shows, “before-the-fact” predictions from our LASSO-Popularity fundamentals model substantially improve model fit.

Figure 1

Model Fit: Bivariate vs. Full LASSO-Popularity Forecast (“Before-the-Fact”)

A logit implementation of LASSO selects the same predictors for election winner as for vote share. Though typically the main focus of pundits and the public, predicting the winner is often a less rigorous test of a forecast, because many races are uncompetitive. Again, our fundamentals-based forecast is quite accurate,Footnote ¹⁹ predicting the winner in 75% of races: a 10% improvement over assuming the incumbent party always wins and a 50% improvement over a coin toss. The LOOV accuracy, 73%, is nearly as high as the within-sample accuracy. Our “before-the-fact” forecast for 2012 performs substantially better than the 2012 out-of-sample forecasts of Klarner (Reference Klarner2012) and Hummel and Rothschild (Reference Hummel and Rothschild2014). It correctly predicted 91% (10 of 11) races in 2012, compared to Klarner’s and Hummel and Rothschild’s success rates in 2012 of 70% and 79%, respectively. Overall, our model correctly predicts winners more reliably than extant models but with greater parsimony.

VALIDATING THE LASSO-POPULARITY FUNDAMENTALS FORECAST: PREDICTING THE 2022 GUBERNATORIAL ELECTIONS

Thus far, the OLS results in table 1 and the patterns in figure 1 suggest our LASSO-Popularity forecast is substantially accurate. The model’s low LOOV MAE and low “before-the-fact” MAE, especially compared to a naïve model, indicate high out-of-sample predictive power. To further gauge out-of-sample robustness, we registered a preelection forecast for the governors’ races in 2022, an election cycle not included in our training data.Footnote ²⁰ Because the SEAD dataset stops in 2020, we use Morning Consult’s state-level governor and presidential approval estimates to generate predictions.

The 2022 race included three races that, a priori, were considered highly likely to defy a fundamentals forecast: Massachusetts, Maryland, and Hawaii. Massachusetts and Maryland both had very popular, term-limited Republican governors. Given these states’ strong Democratic partisan makeup, any Republican other than the outgoing governor would have low odds of retaining the Governor’s Mansion. Hawaii’s outgoing incumbent hailed from the state’s dominant Democratic Party but was among the nation’s least-liked governors. Thus, long before Election Day, we knew our model was likely to substantially overestimate Republican performance in these three states.

Figure 2 graphs our forecast for the incumbent-party candidate in each 2022 governor’s race. Most were not expected to be competitive, and most were not. The median vote percentage won by the incumbent-party candidates was 56%. The combination of partisan sorting, polarization, and incumbent advantage makes it straightforward to forecast the incumbent party’s likelihood of winning. A higher hurdle is accurately predicting the percentage of votes won by the incumbent-party candidate. To assess the LASSO-Popularity model’s performance in 2022, we look at its error, both directionally and in absolute terms, and compare it to two other forecasts that covered all races in 2022.

Figure 2

LASSO-Popularity Forecast of 2022 Governors’ Races and Actual Vote Share

Forecast of governors’ races with 95% confidence intervals; actual vote share shown with gray diamonds.

The first alternative is 538’s Election Day prediction. Updated daily throughout the race, as Election Day drew near this model is dominated by horse-race polls (replacing fundamentals). A second alternative is the baseline prediction from Cook Political Report’s Partisan Voting Index (PVI). The 2022 PVI boils down to a very simple fundamentals model: the average of the past two presidential elections. Comparing our model to the PVI shows if, and how much, governor approval, state-level presidential approval, and incumbency improve accuracy over the heuristic of presidential election history. Comparing 538’s approach, with its final prediction mere hours before voting began, against our strict fundamentals approach based on data from two quarters before the election, tells us how much fundamentals anchor these races.

Figure 3A displays the forecast errors by state for 538’s Election Day model, our LASSO-Popularity model, and Cook’s PVI model. The largest errors of LASSO-Popularity were the expected cases—Hawaii, Maryland, and Massachusetts—and in the anticipated direction. PVI produced the largest errors, highlighted by Vermont’s popular Republican governor coasting to reelection despite running in a deeply Democratic state.

Figure 3

Forecast Errors across 538, LASSO-Popularity, and Cook’s PVI Models

Although the lack of a consistent error (bias) lends these models validity, their absolute error rates help assess their accuracy and reliability. Figure 3B shows the absolute error by model and state. To examine any potential partisan bias in the forecasts, we split the states by the party of the incumbent governor. With few exceptions, both 538 and our LASSO-Popularity forecast appear highly accurate, often with errors of less than five percentage points. Cook’s PVI is notably less accurate. We observe no evidence that any of these models performed better or worse in races where Republicans held the governor’s office.

For a more quantitative assessment of our model’s relative accuracy, we can examine the forecasts’ root mean squared error (RMSE) and MAE. RMSE weights larger errors as more influential, whereas the MAE weights all errors equally. Table 2 shows the RMSE and MAE for the three models with and without Hawaii, Maryland, and Massachusetts. Not surprisingly, the forecast made closest to the election, 538’s, performed best, and the forecast with the least data, Cook’s PVI, performed worst. Our LASSO-Popularity model performed well in this full out-of-sample forecast with an MAE of 4.16, substantially better than the LOOV MAE from the training data (5.06). If we remove the three a priori outlier cases, then the LASSO-Popularity’s performance, based on data collected two quarters or more before the election, equaled that of 538’s Election Day model. This substantiates the notion that, in most cases, the myriad of vote intention polls released during the campaign contribute little beyond the fundamentals.

Table 2

Forecast Accuracy of 2022 Gubernatorial Races by Model

Note: 538 forecast was accessed here: https://projects.fivethirtyeight.com/2022-election-forecast/governor/. Cook PVI was accessed here: https://www.cookpolitical.com/cook-pvi/2022-partisan-voting-index

LASSO-POPULARITY FUNDAMENTALS FORECASTS OF 2024 GUBERNATORIAL ELECTIONS

Our LASSO-Popularity fundamentals forecast has thus far performed well, both within- and out-of-sample, demonstrating the technique’s utility and range of application. How robustly will it perform in 2024? Unlike in 2022, we have no a priori expectations that our model will perform poorly in any specific 2024 race. Leveraging the same model from the training dataset used to forecast the 2022 governors’ races, we make predictions for 11 gubernatorial races in 2024, again based on approval estimates from Morning Consult (table 3).

Table 3

LASSO-Popularity Forecasts for 2024 Gubernatorial Elections: Incumbent-Party Vote Share and Win Probability

* Incumbent running for reelection.

In all, the LASSO-Popularity model predicts the incumbent-party candidate, including governors seeking reelection, will win in 10 of the 11 races based on the climate in that state. North Carolina is the exception: the incumbent governor is not particularly popular (53% approval), is not running for reelection, President Biden is unpopular in the state (40% approval), and the two-party vote has favored the GOP, and so our model predicts the governorship will change party hands. However, the model predicts races in Washington and Delaware, two traditionally Democratic states, will be closer than in previous elections. Moreover, it forecasts that New Hampshire Republicans, buoyed by outgoing Governor Sununu’s high popularity (66%) and Biden’s unpopularity (43%), will retain the governorship. In sum, our LASSO-Popularity model presages a tough climate for Democrats in gubernatorial contests across most states in 2024. Deviations from model predictions will likely illustrate candidate-specific factors, including candidate quality, that sharply diverge from the “average” candidacies on which this fundamentals-based model rests.