Skip to main content Accessibility help
×
Home
Hostname: page-component-684899dbb8-ct24h Total loading time: 1.206 Render date: 2022-05-27T01:38:00.048Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true }

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias

Published online by Cambridge University Press:  29 September 2020

Devin Caughey
Affiliation:
Massachusetts Institute of Technology
Adam J. Berinsky
Affiliation:
Massachusetts Institute of Technology
Sara Chatfield
Affiliation:
University of Denver
Erin Hartman
Affiliation:
University of California, Los Angeles
Eric Schickler
Affiliation:
University of California, Berkeley
Jasjeet S. Sekhon
Affiliation:
University of California, Berkeley

Summary

We elaborate a general workflow of weighting-based survey inference, decomposing it into two main tasks. The first is the estimation of population targets from one or more sources of auxiliary information. The second is the construction of weights that calibrate the survey sample to the population targets. We emphasize that these tasks are predicated on models of the measurement, sampling, and nonresponse process whose assumptions cannot be fully tested. After describing this workflow in abstract terms, we then describe in detail how it can be applied to the analysis of historical and contemporary opinion polls. We also discuss extensions of the basic workflow, particularly inference for causal quantities and multilevel regression and poststratification.
Get access
Type
Element
Information
Online ISBN: 9781108879217
Publisher: Cambridge University Press
Print publication: 22 October 2020
Copyright
© Devin Caughey, Adam J. Berinsky, Sara Chatfield, Erin Hartman, Eric Schickler, and Jasjeet S. Sekhon 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Andridge, Rebecca R., and Little, Roderick J. A.. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics 27 (2): 153180.Google Scholar
Ansolabehere, Stephen, and Hersh, Eitan. 2012. “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate.” Political Analysis 20 (4): 437459.CrossRefGoogle Scholar
Ansolabehere, Stephen, and Rivers, Douglas. 2013. “Cooperative Survey Research.” Annual Review of Political Science 16: 307329.CrossRefGoogle Scholar
Aronow, Peter M., and Miller, Benjamin T.. 2019. Foundations of Agnostic Statistics. New York: Cambridge University Press.CrossRefGoogle Scholar
Baum, Matthew A., and Kernell, Samuel. 2001. “Economic Class and Popular Support for Franklin Roosevelt in War and Peace.” Public Opinion Quarterly 65 (2): 198229.CrossRefGoogle ScholarPubMed
Berinsky, Adam J. 2006. “American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data.” Public Opinion Quarterly 70 (4): 499529.CrossRefGoogle Scholar
Berinsky, Adam J., Neff Powell, Eleanor, Schickler, Eric, and Brett Yohai, Ian. 2011. “Revisiting Public Opinion in the 1930s and 1940s.” PS: Political Science & Politics 44 (3): 515520.Google Scholar
Berinsky, Adam J., and Eric, Schickler. 2011. The American Mass Public in the 1930s and 1940s [Computer file]. Individual surveys conducted by the Gallup Organization [producers], 1936–1945: Roper Center for Public Opinion Research, University of Connecticut [distributor].Google Scholar
Bethlehem, Jelke G. 1988. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 4 (3): 251260.Google Scholar
Bethlehem, Jelke, Cobben, Fannie, and Schouten, Barry. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Binder, D. A., and Theberge, A.. 1988. “Estimating the Variance of Raking- Ratio Estimators.” Canadian Journal of Statistics 16: 4755.CrossRefGoogle Scholar
Brick, J. Michael, and Montaquila, Jill M.. 2009. “Nonresponse and Weighting.” In Sample Surveys: Design, Methods and Applications, edited by Pfeffermann, Danny and Rao, C. R., vol. 29A, 163185. Handbook of Statistics. Elsevier.CrossRefGoogle Scholar
British Polling Council. 2016. “Performance of the Polls in the EU Referendum.” June 24. Accessed October 26, 2019. www.britishpollingcouncil.org/performance-of-the-polls-in-the-eu-referendum/.Google Scholar
Bunche, Ralph J. 1973. The Political Status of the Negro in the Age of FDR. Edited by Grantham, Dewey W.. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Caldeira, Gregory A. 1987. “Public Opinion and the U.S. Supreme Court: FDR’s Court-Packing Plan.” American Political Science Review 81 (4): 11391153.CrossRefGoogle Scholar
Caughey, Devin, and Hartman, Erin. 2017. “Target Selection as Variable Selection: Using the Lasso to Select Auxiliary Vectors for the Construction of Survey Weights.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Wisconsin-Madison, Madison, WI, July 13. https://ssrn.com/abstract=3494436.Google Scholar
Caughey, Devin, and Wang, Mallory. 2014. “Bayesian Population Interpolation and Lasso-Based Target Selection in Survey Weighting.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Georgia, Athens, GA, July 24. https://ssrn.com/abstract=3494430.Google Scholar
Caughey, Devin, and Wang, Mallory. 2019. “Dynamic Ecological Inference for Time-Varying Population Distributions Based on Sparse, Irregular, and Noisy Marginal Data.” Political Analysis. 27 (3): 388396. https://doi.Org/10.1017/pan.2019.4.CrossRefGoogle Scholar
Caughey, Devin, and Warshaw, Christopher. 2019. “Public Opinion in Subnational Politics.” Journal of Politics 81 (Symposium on Subnational Policymaking): 352363.CrossRefGoogle Scholar
Cochran, William G. 1977. Sampling Techniques. 3rd ed. New York: Wiley.Google Scholar
Converse, Jean M. 1987. Survey Research in the United States: Roots and Emergence. Berkeley: University of California Press.Google Scholar
Davison, A. C., and Hinkley, David V.. 1997. Bootstrap Methods and Their Application. New York: Cambridge University Press.CrossRefGoogle Scholar
Deming, W. Edwards, and Stephan, F. Frederick. 1940. “On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals Are Known.” Annals of Mathematical Statistics 11 (4): 427444.CrossRefGoogle Scholar
Dever, Jill A., and Valliant, Richard. 2010. “A Comparison of Variance Estimators for Poststratification to Estimated Control Totals.” Survey Methodology 36 (1): 4556.Google Scholar
Dever, Jill A., and Valliant, Richard. 2016. “General Regression Estimation Adjusted for Undercoverage and Estimated Control Totals.” Journal of Survey Statistics and Methodology 4 (3): 289318.CrossRefGoogle Scholar
Deville, Jean-Claude. 1991. “A Theory of Quota Surveys.” Survey Methodology 17 (2): 163181.Google Scholar
Deville, Jean-Claude. 2000. “Simultaneous Calibration of Several Surveys.” In Combining Data from Different Sources, Proceedings of Statistics Canada Symposium 99, 207212. Ottawa: Statistics Canada.Google Scholar
Deville, Jean-Claude, and Sarndal, Carl-Erik. 1992. “Calibration Estimators in Survey Sampling.” Journal of the American Statistical Association 87 (418): 376382.CrossRefGoogle Scholar
Dixon, John, and Tucker, Clyde. 2010. “Survey Nonresponse.” In Handbook of Survey Research, 2nd ed., edited by Marsden, Peter V. and Wright, James D., 593630. Bingley, UK: Emerald.Google Scholar
Economic Behavior Program, Survey Research Center, University of Michigan. 1948. Survey of Consumer Finances. Ann Arbor, MI: Institute for Social Research, Social Science Archive [producer], 1973. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2002. http://doi.org/10.3886/ICPSR03601.v1.Google Scholar
Elliott, Michael R., and Roderick, J. A. Little. 2000. “Model-Based Alternatives to Trimming Survey Weights.” Journal of Official Statistics 16 (3): 191209.Google Scholar
Enns, Peter K., and Koch, Julianna. 2013. “Public Opinion in the U.S. States: 1956 to 2010.” State Politics & Policy Quarterly 13 (3): 349372.CrossRefGoogle Scholar
Field, Alexander J. 2006. “Table Dg34–45: Telephone industry - telephones, access lines, wire, employees, and plant: 1876–2000.” In Historical Statistics of the United States, Earliest Times to the Present: Millennial Edition, edited by Carter, Susan B., Gartner, Scott Sigmund, Haines, Michael R., Olmstead, Alan L., Sutch, Richard, and Wright, Gavin. New York: Cambridge University Press. http://hsus.cambridge.org/SeriesDg8–116.Google Scholar
Folger, John K., and Charles, B. Nam. 1964. “Educational Trends from Census Data.” Demography 1 (1): 247257.CrossRefGoogle Scholar
Freedman, David A. 2001. “Ecological Inference and the Ecological Fallacy.” In International Encyclopaedia of the Social and Behavioural Sciences, edited by Smelser, N. J., and Baltes, P. B., 6:40274030. New York: Elsevier.CrossRefGoogle Scholar
Gelman, Andrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 153164.CrossRefGoogle Scholar
Gelman, Andrew, and John, B. Carlin. 2002. “Poststratification and Weighting Adjustments.” In Survey Nonresponse, edited by Graves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A., 289302. New York: Wiley.Google Scholar
Gelman, Andrew, and Thomas, C. Little. 1997. “Poststratification into Many Categories Using Hierarchical Logistic Regression.” Survey Methodology 23 (2): 127135.Google Scholar
Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70 (5): 646675.CrossRefGoogle Scholar
Guandalini, Alessio, and Yves, Tillé. 2017. “Design-Based Estimators Calibrated on Estimated Totals from Multiple Surveys.” International Statistical Review 85 (2): 250269.CrossRefGoogle Scholar
Hainmueller, Jens. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20 (1): 2546.CrossRefGoogle Scholar
Hájek, Jaroslav. 1958. “On the Theory of Ratio Estimates.” Applied Mathematics 3 (5): 384398.CrossRefGoogle Scholar
Hartman, Erin, Grieve, Richard, Ramsahai, Roland, and Sekhon, Jasjeet S.. 2015. “From Sample Average Treatment Effect to Population Average Treatment Effect on the Treated: Combining Experimental with Observational Studies to Estimate Population Treatment Effects.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178 (3): 757778.CrossRefGoogle Scholar
Hillygus, Sunshine. 2016. “The Practice of Survey Research: Changes and Challenges.” In New Directions in Public Opinion, 2nd ed., edited by Berinsky, Adam J., 3453. New York: Routledge.Google Scholar
Horvitz, D. G., and Thompson, D. J.. 1952. “A Generalization of Sampling without Replacement from a Finite Universe.” Journal of the American Statistical Association 47: 663685.CrossRefGoogle Scholar
Hur, Aram, and Achen, Christopher H.. 2013. “Coding Voter Turnout Responses in the Current Population Survey.” Public Opinion Quarterly 77 (4): 985993.CrossRefGoogle Scholar
Ireland, C. T., and Kullback, S.. 1968. “Contingency Tables with Given Marginals.” Biometrika 55 (1): 179188.CrossRefGoogle ScholarPubMed
Kalton, Graham, and Flores-Cervantes, Ismael. 2003. “Weighting Methods.” Journal of Official Statistics 19 (2): 8197.Google Scholar
Kastellec, Jonathan P, Lax, Jeffrey R., Malecki, Michael, and Phillips, Justin H.. 2015. “Polarizing the Electoral Connection: Partisan Representation in Supreme Court Confirmation Politics.” Journal of Politics 77 (3): 787804.CrossRefGoogle Scholar
Kennedy, Courtney, Blumenthal, Mark, Clement, Scott et al. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 133.CrossRefGoogle Scholar
Key, V O. Jr. 1949. Southern Politics in State and Nation. New York: Knopf.Google Scholar
King, Gary, Rosen, Ori, and Tanner, Martin A.. 2004. “Information in Ecological Inference: An Introduction.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 112. New York: Cambridge University Press.CrossRefGoogle Scholar
Kish, Leslie. 1965. Survey Sampling. New York: Wiley.Google Scholar
Kott, Phillip S. 2006. “Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors.” Survey Methodology 32 (2): 133142.Google Scholar
Lax, Jeffrey R., and Justin, H. Phillips. 2009. “How Should We Estimate Public Opinion in The States?American Journal of Political Science 53 (1): 107121.CrossRefGoogle Scholar
Leeman, Lucas, and Wasserfallen, Fabio. 2017. “Extending the Use and Prediction Precision of Subnational Public Opinion Estimation.” American Journal of Political Science 61 (4): 10031022.CrossRefGoogle Scholar
Leeuw, E. de, and Heer, W. de. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey Nonresponse, edited by Groves, R. M., Dillman, D. A., Eltinge, J. L., and Little, R. J. A., 4154. New York: Wiley.Google Scholar
Little, R. J. A. 1993. “Post-Stratification: A Modeler’s Perspective.” Journal of the American Statistical Association 88 (423): 10011012.CrossRefGoogle Scholar
Little, Roderick J. A., and Wu, Mei-Miau. 1991. “Models for Contingency Tables With Known Margins When Target and Sampled Populations Differ.” Journal of the American Statistical Association 86 (413): 8795.CrossRefGoogle Scholar
Little, Roderick J., and Vartivarian, Sonya. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?Survey Methodology 31 (2): 161168.Google Scholar
Luevano, Patricia. 1994. Response Rates in the National Election Studies, 1948–1992. Technical report 44. March. www.electionstudies.org/Library/papers/documents/nes010162.pdf.Google Scholar
Lumley, Thomas S. 2010. Complex Surveys: A Guide to Analysis Using R. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Mackuen, Michael B., Erikson, Robert S., and Stimson, James A.. 1989. “Macropartisanship.” American Political Science Review 83 (4): 11251142.CrossRefGoogle Scholar
Marken, Stephanie. 2018. “Still Listening: The State of Telephone Surveys.” Gallup, Methodology Blog, January 18. http://news.gallup.com/opinion/methodology/225143/listening–state–telephone–surveys.aspx.Google Scholar
McDonald, Michael. 2019. “CPS Vote Over-Report and Non-Response Bias Correction.” Accessed October 2. http://www.electproject.org/home/voter-turnout/cps-methodology.Google Scholar
Mickey, Robert W. 2015. Paths Out of Dixie: The Democratization of Authoritarian Enclaves in America’s Deep South. Princeton, NJ: Princeton University Press.Google Scholar
Miratrix, Luke, Sekhon, Jasjeet S., Theodoridis, Alexander G., and Campos, Luis F.. 2018. “Worth Weighting? Howto Think About and Use Weights in Survey Experiments.” Political Analysis 26: 275291.CrossRefGoogle Scholar
Mosteller, Frederick, Hyman, Herbert, McCarthy, Philip J., Marks, Eli S., and Truman, David B., eds. 1949. The Pre-Election Polls of 1948. New York: Social Science Research Council.Google Scholar
Moy, Corrine. 2015. “Fit-for-purpose sampling.” International Journal of Market Research 57 (3): 491494.Google Scholar
Norpoth, Helmut, Sidman, Andrew H., and Suong, Clara H.. 2013. “Polls and Elections: The New Deal Realignment in Real Time.” Presidential Studies Quarterly 43 (1): 146166.CrossRefGoogle Scholar
Park, David K., Gelman, Andrew, and Bafumi, Joseph. 2004. “Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls.” Political Analysis 12 (4): 375385.CrossRefGoogle Scholar
Pew, Research Center. 2016. As Election Nears, Voters Divided Over Democracy and “Respect”. Technical report. October. www.people–press.org/2016/10/27/as–election–nears–voters–divided–over–democracy–and–respect/.Google Scholar
Peytchev, Andy. 2012. “Multiple Imputation for Unit Nonresponse and Measurement Error.” Public Opinion Quarterly 76 (2): 214237.CrossRefGoogle Scholar
Quinn, Kevin M. 2004. “Ecological Inference in the Presence of Temporal Dependence.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 207233. New York: Cambridge University Press.CrossRefGoogle Scholar
Core Team, R. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. www.R–project.org/.Google Scholar
Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Non-randomized Studies.” Journal of Educational Psychology 66 (5): 688701.CrossRefGoogle Scholar
Ruggles, Steven, Alexander, J. Trent, Genadek, Katie, Goeken, Ronald, Schroeder, Matthew B., and Sobek, Matthew. 2010. Integrated Public Use Microdata Series: Version 5.0 [Machine-readable database]. Minneapolis: University of Minnesota.Google Scholar
Ruggles, Steven, Genadek, Katie, Goeken, Ronald, Grover, Josiah, and Sobek, Matthew. 2017. Integrated Public Use Microdata Series: Version 7.0 [dataset]. Minneapolis: University of Minnesota. https://doi.Org/10.18128/D010.V7..Google Scholar
Särndal, Carl-Erik, and Lundstrom, Sixten. 2005. Estimation in Surveys with Nonresponse. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Särndal, Carl-Erik, and Lundstrom, Sixten. 2008. “Assessing Auxiliary Vectors for Control of Nonresponse Bias in the Calibration Estimator.” Journal of Official Statistics 24 (2): 167191.Google Scholar
Särndal, Carl-Erik, and Traat, Imbi. 2011. “Domain Estimators Calibrated on Information from Another Survey.” Acta et Commentationes Universitatis Tartuensis de Mathematics 15 (2): 4360.Google Scholar
Schickler, Eric, and Caughey, Devin. 2011. “Public Opinion, Organized Labor, and the Limits of New Deal Liberalism, 1936–1945.” Studies in American Political Development 25 (2): 128.CrossRefGoogle Scholar
Splawa-Neyman, Jerzy. 1923. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.Roczniki Nauk Roiniczych, Tom X: 1–51. Reprinted in Statistical Science, 5 (4): 465472, 1990. Translated from Polish by D. M. Dabrowska and T. P. Speed.Google Scholar
Squire, Peverill. 1988. “Why the 1936 Literary Digest Poll Failed.” Public Opinion Quarterly 52: 125133.CrossRefGoogle Scholar
Stuart, Elizabeth A., Cole, Stephen R., Bradshaw, Catherine P., and Leaf, Philip J.. 2011. “The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials.” Journal of the Royal Statistical Society. Series A (Statistics in Society) 174 (2): 369386.CrossRefGoogle Scholar
Tanner, Martin A. 1996. Tools for Statistical Inference Methods for the Exploration of Posterior Distributions and Likelihood Functions. 3rd ed. New York: Springer-Verlag.CrossRefGoogle Scholar
Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B (Methodological) 58 (1): 267288.CrossRefGoogle Scholar
Treier, Shawn, and Jackman, Simon. 2008. “Democracy as a Latent Variable.” American Journal of Political Science 52 (1): 201217.CrossRefGoogle Scholar
Valliant, Richard, Dever, Jill A., and Kreuter, Frauke. 2018. Practical Tools for Designing and Weighting Survey Samples. Springer. PDF e-book.CrossRefGoogle Scholar
Verba, Sidney, and Lehman Schlozman, Kay. 1977. “Unemployment, Class Consciousness, and Radical Politics: What Didn’t Happen in the Thirties.” Journal of Politics 39 (2): 291323.CrossRefGoogle Scholar
Wagner, James. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76 (3): 555575.CrossRefGoogle ScholarPubMed
Wakefield, Jon. 2004. “Ecological Inference for 2 × 2 Tables.” Journal of the Royal Statistical Society. Series A (General) 167 (3): 385445.CrossRefGoogle Scholar
Warshaw, Christopher, and Rodden, Jonathan. 2012. “How Should We Measure District-Level Public Opinion on Individual Issues?The Journal of Politics 74 (1): 203219.CrossRefGoogle Scholar
Weatherford, M. Stephen, and Sergeyev, Boris. 2000. “Thinking about Economic Interests: Class and Recession in the New Deal.” Political Behavior 22 (4): 311339.CrossRefGoogle Scholar
Wiseman, Frederick, and McDonald, Philip. 1979. “Noncontact and Refusal Rates in Consumer Telephone Surveys.” Journal of Marketing Research 16 (4): 478484.CrossRefGoogle Scholar

Save element to Kindle

To save this element to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×

Save element to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×

Save element to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×