This article proposes a new approach for measuring the quality of answers in political question-and-answer sessions. We assess the quality of an answer based on how easily and accurately it can be recognized among a random set of candidate answers given the question’s text. This measure reflects the answer’s relevance and depth of engagement with the question. Drawing a parallel with semantic search, we can implement this approach by training a language model on the corpus of observed questions and answers without additional human-labeled data. We showcase and validate our methodology within the context of the Question Period in the Canadian House of Commons. Our analysis reveals that while some answers only have a weak semantic connection to questions, suggesting some evasion or obfuscation, they are generally at least moderately relevant, far exceeding what we would expect from random replies. We also find meaningful correlations between the quality of answers and the party affiliation of the members of Parliament asking the questions.

Control over the legislative messaging agenda has important political, electoral and policy consequences. Existing models of congressional agenda-setting suggest that national polarization drives the agenda. At the same time, models of home style and formal models of leadership hypothesize that legislators shift their messaging as they balance coordination and information problems. We say the coordination problem dominates when conditions incentivize legislators to agree on the same message rather than fail to reach consensus. Conversely, the information problem is said to dominate in circumstances where legislators prefer to say nothing at all rather than reach consensus on the wrong political message. Formal theories predict that when coordination problems are pressing, legislative members follow the policy positions of party leaders. When their party’s information problem is acute, party members instead rely on the wisdom of the caucus to set the party’s agenda. To test these theories, we analyze the Twitter accounts of U.S. House members with a Joint Sentiment Topic model, generating a new understanding of House leadership power. Our analyses reveal complex leader-follower relationships. Party leaders possess the power to substantially affect the propensity of rank-and-file members to discuss topics, especially when the coordination problem dominates; these effects are pronounced even when coordination problems are pressing. That said, when the underlying politics are unclear, rank-and-file members exert influence on the discussion of a topic because the information problem is more acute. At the same time and for these uncertain topics, leadership influence decreases, consistent with theory. We show these results are robust to the underlying dynamics of contemporary political discussion and context, including leading explanations for party leadership power, such as national polarization.

In recent years, scholars, journals, and professional organizations in political science have been working to improve research transparency. Although better transparency is a laudable goal, the implementation of standards for reproducibility still leaves much to be desired. This article identifies two practices that political science should adopt to improve research transparency: (1) journals must provide detailed replication guidance and run provided material; and (2) authors must begin their work with replication in mind. We focus on problems that occur when scholars provide research materials to journals for replication, and we outline best practices regarding documentation and code structure for researchers to use.

Building on past research, we implement a hierarchical latent class model to analyze political participation from a comparative perspective. Our methodology allows simultaneously: (i) estimating citizens’ propensity to engage in conventional and unconventional modes of participation; (ii) classifying individuals into underlying “types” capturing within- and cross-country variations in participation; and (iii) assessing how this classification varies with micro- and macro-level factors. We apply our model to Latin American survey data. We show that our method outperforms alternative approaches used to study participation and derive typologies of political engagement. Substantively, we find that the distribution of participatory types is similar throughout the continent, and that it correlates strongly with respondents’ socio-demographic characteristics and crime victimization.

Does attentiveness matter in survey responses? Do more attentive survey participants give higher quality responses? Using data from a recent online survey that identified inattentive respondents using instructed-response items, we demonstrate that ignoring attentiveness provides a biased portrait of the distribution of critical political attitudes and behavior. We show that this bias occurs in the context of both typical closed-ended questions and in list experiments. Inattentive respondents are common and are more prevalent among the young and less educated. Those who do not pass the trap questions interact with the survey instrument in distinctive ways: they take less time to respond; are more likely to report nonattitudes; and display lower consistency in their reported choices. Inattentiveness does not occur completely at random and failing to properly account for it may lead to inaccurate estimates of the prevalence of key political attitudes and behaviors, of both sensitive and more prosaic nature.

With the discipline’s push toward data access and research transparency (DA-RT), journal replication archives are becoming increasingly common. As researchers work to ensure that replication materials are provided, they also should pay attention to the content—rather than simply the provision—of journal archives. Based on our experience in analyzing and handling journal replication materials, we present a series of recommendations that can make them easier to understand and use. The provision of clear, functional, and well-documented replication materials is key for achieving the goals of transparent and replicable research. Furthermore, good replication materials enhance the development of extensions and related research by making state-of-the-art methodologies and analyses more accessible.

We examine the quality of two probability-based polls, one interviewer administered (telephone) and one self-administered (Internet and mail mixed mode survey). The polls use the same sampling frame (registered voters) and the same questions. First, we examine the representativeness of both surveys using information known about the population, and although we find important differences between the two in terms of sampling and nonresponse bias, we also find that both surveys represent the underlying population despite low response rates. We also test for mode effects between surveys due to social desirability and how it influences nondifferentiation or satisficing. Using a variety of methods (t-tests, multivariate regression, and genetic propensity matching), we find evidence that the presence of an interviewer alters response patterns on ego-driven questions. The implications of our work are important, due to the increasing popularity of mixed mode surveys. Researchers need to be methodologically sensitive to these differences when analyzing surveys that allow for different response modes.

An important property of any party system is the set of choices it presents to the electorate. In this paper we analyze the distribution of parties relative to voters in the multidimensional issue space and introduce two measures of the dispersion of the parties in the issue space relative to the voters, which we call measures of the compactness of the parties in the issue space. We show how compactness is easily computed using standard survey items found on national election surveys. Because we study the spacing of the parties relative to the distribution of the voters, we produce metric-free measures of compactness of the party system. The measures can be used to compare party systems across issues, over time within countries, and across countries. Comparing the compactness of party systems across countries allows us to determine the relative amount of issue choice afforded voters in different polities. We examine the compactness of the issue space and test the impact it has on voter choice in four countries: the United States, the Netherlands, Canada, and Great Britain. We demonstrate that the more compact the distribution of the parties in the issue space on any given issue, the less voters weight that issue in their vote decision. Thus we provide evidence supporting theories suggesting that the greater the choice offered by the parties in an election, the more likely it is that issue voting will play a major role in that election.

This article provides a basic report about subject recruitment processes for Web-based surveys. Using data from our ongoing Internet Survey of American Opinion project, two different recruitment techniques (banner advertisement and subscription campaigns) are compared. This comparison, together with a typology of Web-based surveys, provides insight into the validity and generalizability of Internet survey data. The results from this analysis show that, although Internet survey respondents differ demographically from the American population, the relationships among variables are similar across recruitment methods and match those implied by substantive theory. Thus, our research documents the basic methodology of subject acquisition for Web-based surveys, which, as we argue in our conclusion, may soon become the survey interview mode of choice for social scientists.

Questions of causation are important issues in empirical research on political behavior. Most of the discussion of the econometric problems associated with multiequation models with reciprocal causation has focused on models with continuous dependent variables (e.g., Markus and Converse 1979; Page and Jones 1979). Yet, many models of political behavior involve discrete or dichotomous dependent variables; this paper describes two techniques which can consistently estimate reciprocal relationships between dichotomous and continuous dependent variables. The first, two-stage probit least squares (2SPLS), is very similar to two-stage instrumental variable techniques. The second, two-stage conditional maximum likelihood (2SCML), may overcome problems associated with 2SPLS, but has not been used in the political science literature. We demonstrate the potential pitfalls of ignoring the problems of reciprocal causation in nonrecursive choice models and examine the properties of both techniques using Monte Carlo simulations: we find that 2SPLS slightly outperforms 2SCML in terms of bias but that 2SCML produces more accurate standard errors. However, the 2SCML model offers an explicit statistical test for endogeneity. The results from our simulations, and the statistical test for exogeneity, lead us to advocate the use of 2SCML for estimation of this class of causal models. We then apply both of these techniques to an empirical example focusing on the relationship between voter preferences in a presidential election and the voter's uncertainty about the policy positions taken by the candidates. This example demonstrates the importance of these techniques for political science research.

Empirical researchers studying party systems often struggle with the question of how to count parties. Indexes of party system fragmentation used to address this problem (e.g., the effective number of parties) have a fundamental shortcoming: since the same index value may represent very different party systems, they are impossible to interpret and may lead to erroneous inference. We offer a novel approach to this problem: instead of focusing on index measures, we develop a model that predicts the entire distribution of party vote-shares and, thus, does not require any index measure. First, a model of party counts predicts the number of parties. Second, a set of multivariate t models predicts party vote-shares. Compared to the standard index-based approach, our approach helps to avoid inferential errors and, in addition, yields a much richer set of insights into the variation of party systems. For illustration, we apply the model on two data sets. Our analyses call into question the conclusions one would arrive at by the index-based approach. Software is provided to implement the proposed model.

Ordinal variables—categorical variables with a defined order to the categories, but without equal spacing between them—are frequently used in social science applications. Although a good deal of research exists on the proper modeling of ordinal response variables, there is not a clear directive as to how to model ordinal treatment variables. The usual approaches found in the literature for using ordinal treatment variables are either to use fully unconstrained, though additive, ordinal group indicators or to use a numeric predictor constrained to be continuous. Generalized additive models are a useful exception to these assumptions. In contrast to the generalized additive modeling approach, we propose the use of a Bayesian shrinkage estimator to model ordinal treatment variables. The estimator we discuss in this paper allows the model to contain both individual group—level indicators and a continuous predictor. In contrast to traditionally used shrinkage models that pull the data toward a common mean, we use a linear model as the basis. Thus, each individual effect can be arbitrary, but the model “shrinks” the estimates toward a linear ordinal framework according to the data. We demonstrate the estimator on two political science examples: the impact of voter identification requirements on turnout and the impact of the frequency of religious service attendance on the liberality of abortion attitudes.