This chapter provides an overview of the arguments for the stability of public opinion as well as the arguments of those who believe public opinion is unstable. We then explore conditionality, or the conditions under which public opinion appears to be more unstable. Concepts such as low information rationality, issue salience, and question wording are covered.

Forecasting elections is a high-risk, high-reward endeavor. Today’s polling rock star is tomorrow’s has-been. It is a high-pressure gig. Public opinion polls have been a staple of election forecasting for almost ninety years. But single source predictions are an imperfect means of forecasting, as we detailed in the preceding chapter. One of the most telling examples of this in recent years is the 2016 US presidential election. In this chapter, we will examine public opinion as an election forecast input. We organize election prediction into three broad buckets: (1) heuristics models, (2) poll-based models, and (3) fundamentals models.

To wit, we have three specific goals here. First, we want to review the activities of the three-hatted pollster. We do this to provide greater context for each type of pollster. Some of us are all three; others are some combination of these. Any pollster worth their salt must at least be a data scientist, or they risk losing credibility.

Second, we explore the role of the pollster in society. Ultimately, what is the purpose of the pollster? In our view, pollsters are critically important in any democracy. We believe this is often overlooked due to the ranking frenzy after every electoral cycle. Here, we put the profession into proper perspective.

And third, we discuss the use of non-survey, or alternative data, inputs as proxy measures for public opinion. We provide a framework for pollsters to think through them in a critical manner. Validation is a key concept which we introduce here – one more tool for the data scientist.

This chapter provides an overview of the purpose of the book, namely to help the user of public opinion data develop a systematic analytical approach for understanding, predicting, and engaging public opinion. This includes helping the reader understand how public opinion can be employed as a decision-making input, meaning a factor, or variable, to assess, predict, or influence an outcome. The chapter outlines how information from different disciplines, including cognitive psychology, behavioral economics, and political science, come together to inform the pollster’s work.

This chapter tackles two additional activities of the pollster as fortune teller. The first is the assessment and prediction of government approval ratings. As we have already seen in Chapter 8, approval ratings are extremely important in predicting elections. There is both an art and science to the analysis of such measures. Here, we want to lay out an analytical framework which will allow pollsters to assess both structural and policy factors related to approval ratings and then how to utilize multiple methods to triangulate future outcomes. We will focus on the Biden administration circa August 2022. Ultimately, a fairly large component of a pollster’s workload is the continual assessment of government initiatives and their convergence (or not) with what people want.

The second is a discussion of more context-based analysis. The pollster has an important role in helping decision-makers understand the bigger picture. Here, broader demographic and social trends help gird such analysis.

In this chapter, we discuss both the structural and the packaging perspectives in conceptual terms. It is worth noting that the communications literature is diffuse and poorly integrated. Some of it reads more like self-help books. To be fair, it does draw on many different disciplines – some more rigorous; others less so. As such, our purpose here is to provide a clear framework for the pollster and practitioner. There is considerable art and creativity to effective communications. Look at Cannes Lion every year- the Oscars of the PR and Marketing world. There is incredible creativity in the crafting of impactful messages. But public opinion is public opinion – with a few basic compositional truths. By nailing them down, the pollster is able to provide structure to the communications process.

Predictions often falter because of human error. Most misses have much more to do with our own human shortcomings than with the technical sophistication of the method at hand. In our experience, forecasting errors occur when we discard or misinterpret evidence right in front of us. The clues are there, but we are blinded by our own filters. This is why it is essential to tackle such biases and discuss corresponding solutions. In this chapter, we’ll look at studies on the forecasting prowess of experts. Then, we’ll focus on cognitive biases that skew predictions. Finally, we’ll present an applied approach to minimize such biases.

Chapter 8 presents random forests for regression, which – at least in some situations – may outperform the least-squares-based regression methods. The chapter discusses bagging in the context of regression applications of random forests, the algorithm for splitting nodes in regression trees, and the variable importance metrics applicable to regression.

Chapter 7 is dedicated to regularized regression methods, which – by penalizing models that are too complex – are capable of providing a reasonable tradeoff between bias and variance. Ridge regression implements L2 regularization, which results in more generalizable models, but does not perform any feature selection. L1 penalty used by the lasso allows, however, for simultaneous regularization and feature selection. The elastic net algorithm combines the two approaches by applying both L1 and L2 penalties, which allows for solutions combining the advantages of both ridge regression and the lasso. The chapter concludes by discussing a general class of Lq-regularized least squares optimization problems.

Chapter 15 starts with the identification of essential patterns by analyzing distributions of groups of variables (with similar patterns) among a large number of optimal-size biomarkers generated by parallel feature selection experiments. A similar approach leads to the identification of the essential variables of those essential patterns. As a result, the final multivariate biomarker identified via this method is most likely to represent a real population pattern associated with biological processes underlying changes in the investigated response variable. Furthermore, having the variables of the final biomarker associated with their respective essential patterns facilitates biological interpretation of the biomarker.

Chapter 2, while continuing with the concepts of importance to multivariate analysis of high-dimensional data, adds considerations related to the curse of dimensionality and descriptions of common misconceptions. Discussed is the fallacy of applying to high-dimensional data the same methods that were successfully used in low-dimensional settings. Explained are also misconceptions based on driving biomarker discovery studies by univariate and unsupervised approaches.