What is the concept to be measured?

Our first task is to reflect on the concept these indices purport to measure and to suggest some implications of the very different operationalizations these researchers use. The concept of interest is state policy mood, which is the predisposition of state electorates for more or less government activity on an array of economic policy areas. Both the BRFH and the LJKE measures seek to replicate Stimson’s (Reference Stimson1999) measure of national mood at the state level. BFHC’s effort in this issue offers correlations of their measure with a variety of combinations of items drawn from the General Social Survey (GSS) to make the case for its validity. In LJKE’s update of the Enns and Koch (Reference Enns and Koch2013) work on state policy mood, the primary data used to construct the measure are the same GSS items Stimson uses to measure national policy mood. These items are restricted to a limited set of economic and related policies, including health, education, and welfare. They explicitly exclude cultural issues such as race, abortion, and guns, which means that these mood measures do not capture that underlying dimension. This measurement strategy explicitly excludes state opinions toward cultural issues. If that is the case, then state policy mood as a primarily economic indicator may not be appropriate for analyses involving morality policies.

Another potential problem with the validity of both the BRFH and LJKE measures is that they have, at best, an indirect connection to factors that scholars often want to explain, such as patterns of state policy or mass attitudes toward state policy. Stimson’s measure clearly assumes that survey respondents are thinking about the policy efforts of the federal government when they answer GSS questions. So if either index is able to replicate Stimson’s measure at the state level, we still have the problem of using a state-level measure of national policy mood to explain changes in state policies and election outcomes.

One way to theoretically address this problem is to assume that citizens do not differentiate between levels of government in their policy preferences. If so, a national policy mood would be roughly the same as a state mood measure and the problem is mitigated. However, just a bit of reflection suggests that this is not a defensible way around the problem. It is only too easy to imagine sets of citizens who would have quite different preferences for policy changes at the state versus the national level due to variations in the status quo at each level. Take, for example, the group of respondents who might have moderate (“spending enough”) responses to national welfare or education expenditures. Those living in Alabama or Mississippi might see the stinginess of their state governments’ efforts as “too little,” whereas their counterparts living in higher-spending states like New York or California could feel that “too much” is being spent by their state governments. We can imagine parallel scenarios in other policy areas.

If either of the measures seems not to “work” in an analysis, we would have to question whether state governments do indeed ignore the preferences of their electorates, or whether we actually have measures of something else (national policy mood) that is not systematically related to the key concept of state policy mood. In fact, it would be surprising if public reactions to national economic policies – a constant at any time point – could capture the variety of citizen reactions to the increasingly divergent patterns of a subset of state policies. If the patterns of “leapfrog representation” identified by Bafumi and Herron (Reference Bafumi and Herron2010) hold for state governments, then we might expect wildly divergent reactions of state electorates as the blue states race toward more liberal policy regimes and the red states adopt increasingly conservative policies. Our simple, but for state politics researchers quite important, point is that we have lots of reasons to suspect that measures of national policy mood, even measured at the state level, do not consistently measure citizens’ evaluations of the policies in their states.

How does the concept fit with the research question?

Decisions on how to measure state public preferences depend on how central the concept is to answer the research question. For some lines of inquiry, state preferences may be a control variable included to capture a state’s tendency toward a particular policy or political outcome. Take the vast literature on policy diffusion as an example. Here, scholars are primarily interested in understanding why a state adopts a policy at a particular time and how policies spread across the states. While public opinion may matter for these processes (Pacheco Reference Pacheco2012), it is often secondary to the role of elites (Mintrom Reference Mintrom1997), pressures within and across states (e.g., Berry and Berry Reference Berry and Berry1990; Desmarais, Harden, and Boehmke Reference Desmarais, Harden and Boehmke2015), and influence from local governments and the federal government (e.g., Karch and Rose Reference Karch and Rose2019; Shipan and Volden Reference Shipan and Volden2006).

For research questions where public opinion is a secondary concern, researchers might do well to follow the lead of early state opinion scholars (e.g., Brace et al. Reference Brace, Butler, Arceneaux and Johnson2002; Erikson, Wright, and McIver Reference Erikson, Wright and McIver1993) and include direct aggregate measures of partisanship, ideology, or vote choice. These measures may be drawn from single national surveys with large sample sizes, such as the Cooperative Election Study (CES) or the National Annenberg Election Surveys (NAES), pooled cross-sectional surveys, such as the American National Election Survey (ANES), or state exit polls. All of these options are simplified ways to categorize the broad political leanings of state residents. Regional measures of geography can sufficiently capture meaningful variations in state culture. Of course, the indicators discussed here could also be used, but researchers should be careful to accurately describe what and why they include particular control variables over others.

Scholars must be more thoughtful when state opinion is the dependent variable. A scholar might ask whether state opinion is changing – is state opinion stable or dynamic? If state opinion changes over time, does it change gradually or quickly? Related, are states generally trending in similar or divergent ways? Are there parallel publics across the states (e.g., Page and Shapiro Reference Page and Shapiro1992)? These are important questions to ask because the combination of the rate of change (stable, gradual, or abrupt) and the patterns of change (homogeneous or heterogeneous) provide insight into the determinants of aggregate state opinion. Thus, these types of descriptive analyses are useful for understanding theories in political science. They are also informative to journalists, electoral strategists, and policymakers, all of whom are likely especially interested in the practical implications of what determines general state opinions and, specifically, state policy mood.

A guiding principle for these research questions is whether we are interested in describing broad political patterns or narrowly defined issue attitudes. For the former, scholars essentially have three choices: partisanship, ideology, or policy mood. State macropartisanship and macroideology move slowly over time where the relative rankings of the states are fairly consistent (Enns and Koch Reference Enns and Koch2013; Pacheco Reference Pacheco2012; Wright and Birkhead Reference Wright and Birkhead2014). State policy mood, however, is more dynamic, regardless of whether one uses the measure from LJKE or BFHC’s indicator. The implication is that shifts in the partisan or ideological leanings of states are likely caused by gradual population changes such as generational replacement, migration, immigration, and differential birth and death rates (Brace et al. Reference Brace, Arceneaux, Johnson and Ulbig2004; Carmines and Stimson Reference Carmines and Stimson1989). Abrupt movements in opinion, such as is the case with state policy mood, are likely caused by reactions to current events, the economy, or other transient outcomes (Erikson, MacKuen, and Stimson Reference Erikson, MacKuen and Stimson2002; Stimson Reference Stimson2004).

The rate and patterns of opinion change across states on specific issues vary depending on the issue. In the most comprehensive analysis of the dynamic properties of specific state opinions, Pacheco (Reference Pacheco2014) finds that state attitudes on consumer sentiment and presidential approval are highly dynamic and exhibit homogenous trends. State attitudes toward education spending, welfare spending, and the death penalty exhibit more change over time with heterogeneous trends (Brace et al. Reference Brace, Butler, Arceneaux and Johnson2002; Pacheco Reference Pacheco2014). State attitudes toward abortion are essentially stable. More recent studies find that state attitudes toward the Affordable Care Act (ACA) exhibit various patterns, depending on whether residents responded to a general question about ACA favorability, the impact of the ACA, or the future impact of the ACA (Pacheco and Maltby Reference Pacheco and Maltby2019; see also Brodie, Deane, and Cho Reference Brodie, Deane and Cho2011). Finally, there is evidence that state racial resentment changes slowly over time with heterogenous trends (Smith, Kreitzer, and Suo Reference Smith, Kreitzer and Suo2020). Continuing this type of research to understand the descriptive characteristics of state public opinion is a fruitful avenue for the future.

Thoughtfully considering how to measure state preferences is equally important when it is the main independent variable of interest, which is the case for studies asking how closely state policies align with the mass public’s policy attitudes. For these studies, a scholar must be purposeful in making sure that the measure of state preferences reasonably matches the dependent variable. If the dependent variable is capturing broad policy tendencies of the states, for instance, state policy liberalism (e.g., Caughey and Warshaw Reference Caughey and Warshaw2018; Erikson, Wright, and McIver Reference Erikson, Wright and McIver1993), then it is reasonable to measure state opinion along a similar ideological dimension (see Caughey and Warshaw Reference Caughey and Warshaw2016). When the dependent variable is a specific policy outcome; for instance, on gay rights policies (Lax and Phillips Reference Lax and Phillips2009), the ACA (Pacheco and Maltby Reference Pacheco and Maltby2019), or antismoking legislation (Pacheco Reference Pacheco2012), it makes more sense for scholars to measure specific state attitudes. Linking concepts to data is a crucial aspect of any research design, especially in cases such as this one where there are multiple options from which scholars can choose.