This chapter illustrates how to use randomized response treatments to assess possible nonresponse bias. It focuses on a 2019 survey and shows how nonignorable nonresponse may have deflated Trump support in the Midwest and among Democrats even as nonignorable nonresponse inflated Trump support among Republicans. We also show that Democrats who responded to the poll were much more liberal on race than Democrats who did not respond, a pattern that was particularly strong among White Democrats and absent among non-White Democrats. Section 12.1 describes a survey design with a randomized response instrument. Section 12.2 discusses nonignorable nonresponse bias for turnout questions. Section 12.3 looks at presidential support, revealing regional and partisan differences in nonignorable nonresponse. Section 12.4 looks at race, focusing on partisan and racial differences in nonignorable nonresponse. Section 12.5 assesses nonignorable nonresponse on climate, taxes, and tariffs.

This chapter presents the intuition behind why nonignorable nonresponse can be a problem and how it can arise in many contexts. With a foundation that explicitly centers this possibility, we can better reason through when the problem may be larger, how to diagnose it, and how to fix or at least ameliorate it. Section 5.1 describes qualitatively when nonignorable nonresponse may be likely. Section 5.2 works through the intuition about how and why nonignorable nonresponse undermines polling accuracy. Section 5.3 presents a framework for modeling nonignorable nonresponse and culminates by describing Meng’s (2018) model of sampling error. Section 5.4 raises the possibility that nonignorability varies across groups, over time, and even across questions.

Nonresponse is a challenge in many fields, including demography, economics, public health, sociology, and business. This chapter explores nonpolitical manifestations of nonignorable nonresponse by focusing on population health. For many conditions, the decision to get tested or the willingness to allow a test is deeply wrapped up in the likelihood of having the condition. During Covid, for example, people who thought they might have been exposed to the virus were almost certainly more likely to get tested meaning that nonignorable nonresponse complicated our ability to understand the Covid outbreak. Section 13.1 discusses the challenge of estimating public health variables in terms of a nonignorable missing data problem. Section 13.2 explores how first-stage instruments can improve the efficiency and accuracy of efforts to assess prevalence. Section 13.3 presents a framework for comparing Covid positivity rates across regions even when testing rates differ.

This chapter explains weighting in a manner that allows us to appreciate both the power and vulnerability of the technique and, by extension, other techniques that rely on similar assumptions. Once we understand how weighting works, we will better understand when it works. This chapter opens by discussing weighting in general terms. The subsequent sections get more granular. Sections 3.2 and 3.3 cover widely used weighting techniques: cell-weighting and raking. Section 3.4 covers variable selection, a topic that may well be more important than weighting technique. Section 3.5 covers the effect of weighting on precision, a topic that frequently gets lost in polling reporting. This chapter mixes intuitive and somewhat technical descriptions of weighting. The technical details in Sections 3.2 and3.3 can be skimmed by readers focused on the big picture how weighting works.

This chapter introduces selection models in a way that highlights important intuition about how they work. Section 8.1 formalizes the model we’ve been working with already. Section 8.2 uses the model to highlight a bad news, good news story. The bad news is that statistical estimation of a two-equation model like this will be challenging. The good news is that the model helps us recognize the traces nonignorable nonresponse leaves in observable data. Section 8.3 introduces the Heckman selection model. Section 8.4 uses the Heckman model to highlight the starkly different way that selection and weighting approaches use information. The Heckman model is far from perfect, however, as Section 8.5 explains.

This chapter explores the challenges of polling in light caused by nonignorable nonresponse. Nonprobability polling approach comes off poorly for reasons that harken back to the Literary Digest fiasco. The random sampling is far from perfect, but here we rename it the random contact approach – because what is random is who they contact, not who responds once contacted – and show that using random contact shifts error from being proportional to the population size – which can be catastrophic – to being proportional to response rates – which is not great, but much better. Section 6.1 assesses the big data approach by introducing the idea of effective sample size, a concept that allows us to compare potentially large nonrandom samples to their random sampling equivalents. Section 6.2 assesses the random contact approach that has become the last refuge of those clinging to the random sampling paradigm. Section 6.3 decomposes sampling error into elements associated with the choosing whom to contact and elements associated with individual choices given that they are contacted. This section helps clarify where the biggest threats are throughout the survey process.

Polling has become very difficult. People do not respond, and pollsters use methods that are far removed from the random sampling tools that built the field. This chapter introduces the book by outlining the main challenges facing polling today, how conventional tools fail to fully meet these challenges and how a new paradigm and new methods can more directly take on the full spectrum of nonresponse bias given contemporary polling practices.

This chapter highlights the critical importance of having the right kind of data for selection models that address nonignorable nonresponse. In general, we need a variable that is included in our response model and excluded from our outcome model. The best approach is creating a randomized response instrument that affects whether someone responds, but does not affect the content of their response. In many polling contexts, it is easy to create randomized response instruments. The pollster simply needs to figure out some protocol that affects response rate and then randomize it. Section 10.1 makes it clear that knowing the correct functional form is not enough to save a selection model. Section 10.2 highlights the difficulty of using observational response instruments. Section 10.3 discusses how and why to create randomized response instruments. Section 10.4 shows how to use randomized response instruments in a simple test for diagnosing nonignorable nonresponse. Section 10.5 shows how randomized response instruments enable us to use the full suite of selection models even when we do not observe data for nonrespondents.