Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
  • Home
Element
Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Series: Elements in Quantitative and Computational Methods for the Social Sciences

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias

We elaborate a general workflow of weighting-based survey inference, decomposing it into two main tasks. The first is the estimation of population targets from one or more sources of auxiliary information. The second is the construction of weights that calibrate the survey sample to the population targets. We emphasize that these tasks are predicated on models of the measurement, sampling, and nonresponse process whose assumptions cannot be fully tested. After describing this workflow in abstract terms, we then describe in detail how it can be applied to the analysis of historical and contemporary opinion polls. We also discuss extensions of the basic workflow, particularly inference for causal quantities and multilevel regression and poststratification.

Related content

Powered by UNSILO
  • Export citation
  • Recommend to librarian
  • Buy the Element

  • Copyright

  • COPYRIGHT: © Devin Caughey, Adam J. Berinsky, Sara Chatfield, Erin Hartman, Eric Schickler, and Jasjeet S. Sekhon 2020

References

Hide all
Andridge, Rebecca R., and Little, Roderick J. A.. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics 27 (2): 153–180. Google Scholar
Caughey, Devin, and Hartman, Erin. 2017. “Target Selection as Variable Selection: Using the Lasso to Select Auxiliary Vectors for the Construction of Survey Weights.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Wisconsin-Madison, Madison, WI, July 13. https://ssrn.com/abstract=3494436. Google Scholar
Caughey, Devin, and Warshaw, Christopher. 2019. “Public Opinion in Subnational Politics.” Journal of Politics 81 (Symposium on Subnational Policymaking): 352–363. CrossRef | Google Scholar
Davison, A. C., and Hinkley, David V.. 1997. Bootstrap Methods and Their Application. New York: Cambridge University Press. CrossRef | Google Scholar
Hainmueller, Jens. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20 (1): 25–46. CrossRef | Google Scholar
Luevano, Patricia. 1994. Response Rates in the National Election Studies, 1948–1992. Technical report 44. March. www.electionstudies.org/Library/papers/documents/nes010162.pdf. Google Scholar
Warshaw, Christopher, and Rodden, Jonathan. 2012. “How Should We Measure District-Level Public Opinion on Individual Issues?” The Journal of Politics 74 (1): 203–219. CrossRef | Google Scholar
Laver, Michael Images as Data for Social Science Research Google Scholar
Hájek, Jaroslav. 1958. “On the Theory of Ratio Estimates.” Applied Mathematics 3 (5): 384–398. CrossRef | Google Scholar
Ruggles, Steven, Genadek, Katie, Goeken, Ronald, Grover, Josiah, and Sobek, Matthew. 2017. Integrated Public Use Microdata Series: Version 7.0 [dataset]. Minneapolis: University of Minnesota. https://doi.Org/10.18128/D010.V7.. Google Scholar
Valliant, Richard, Dever, Jill A., and Kreuter, Frauke. 2018. Practical Tools for Designing and Weighting Survey Samples. Springer. PDF e-book. CrossRef | Google Scholar
Ansolabehere, Stephen, and Hersh, Eitan. 2012. “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate.” Political Analysis 20 (4): 437–459. CrossRef | Google Scholar
Berinsky, Adam J. 2006. “American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data.” Public Opinion Quarterly 70 (4): 499–529. CrossRef | Google Scholar
Folger, John K., and Charles, B. Nam. 1964. “Educational Trends from Census Data.” Demography 1 (1): 247–257. CrossRef | Google Scholar
Gelman, Andrew, and John, B. Carlin. 2002. “Poststratification and Weighting Adjustments.” In Survey Nonresponse, edited by Graves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A., 289–302. New York: Wiley. Google Scholar
Leeuw, E. de, and Heer, W. de. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey Nonresponse, edited by Groves, R. M., Dillman, D. A., Eltinge, J. L., and Little, R. J. A., 41–54. New York: Wiley. Google Scholar
Quinn, Kevin M. 2004. “Ecological Inference in the Presence of Temporal Dependence.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 207–233. New York: Cambridge University Press. CrossRef | Google Scholar
Aronow, Peter M., and Miller, Benjamin T.. 2019. Foundations of Agnostic Statistics. New York: Cambridge University Press. CrossRef | Google Scholar
Brick, J. Michael, and Montaquila, Jill M.. 2009. “Nonresponse and Weighting.” In Sample Surveys: Design, Methods and Applications, edited by Pfeffermann, Danny and Rao, C. R., vol. 29A, 163–185. Handbook of Statistics. Elsevier. CrossRef | Google Scholar
Caughey, Devin, and Wang, Mallory. 2014. “Bayesian Population Interpolation and Lasso-Based Target Selection in Survey Weighting.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Georgia, Athens, GA, July 24. https://ssrn.com/abstract=3494430. Google Scholar
Converse, Jean M. 1987. Survey Research in the United States: Roots and Emergence. Berkeley: University of California Press. Google Scholar
Little, Roderick J. A., and Wu, Mei-Miau. 1991. “Models for Contingency Tables With Known Margins When Target and Sampled Populations Differ.” Journal of the American Statistical Association 86 (413): 87–95. CrossRef | Google Scholar
Schickler, Eric, and Caughey, Devin. 2011. “Public Opinion, Organized Labor, and the Limits of New Deal Liberalism, 1936–1945.” Studies in American Political Development 25 (2): 1–28. CrossRef | Google Scholar
Enns, Peter K., and Koch, Julianna. 2013. “Public Opinion in the U.S. States: 1956 to 2010.” State Politics & Policy Quarterly 13 (3): 349–372. CrossRef | Google Scholar
Key, V O. Jr. 1949. Southern Politics in State and Nation. New York: Knopf. Google Scholar
Kott, Phillip S. 2006. “Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors.” Survey Methodology 32 (2): 133–142. Google Scholar
Miratrix, Luke, Sekhon, Jasjeet S., Theodoridis, Alexander G., and Campos, Luis F.. 2018. “Worth Weighting? Howto Think About and Use Weights in Survey Experiments.” Political Analysis 26: 275–291. CrossRef | Google Scholar
Wagner, James. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76 (3): 555–575. CrossRef | Google Scholar
  • PubMed
    • Caughey, Devin, and Wang, Mallory. 2019. “Dynamic Ecological Inference for Time-Varying Population Distributions Based on Sparse, Irregular, and Noisy Marginal Data.” Political Analysis. 27 (3): 388–396. https://doi.Org/10.1017/pan.2019.4. CrossRef | Google Scholar
    Hartman, Erin, Grieve, Richard, Ramsahai, Roland, and Sekhon, Jasjeet S.. 2015. “From Sample Average Treatment Effect to Population Average Treatment Effect on the Treated: Combining Experimental with Observational Studies to Estimate Population Treatment Effects.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178 (3): 757–778. CrossRef | Google Scholar
    Hur, Aram, and Achen, Christopher H.. 2013. “Coding Voter Turnout Responses in the Current Population Survey.” Public Opinion Quarterly 77 (4): 985–993. CrossRef | Google Scholar
    Ireland, C. T., and Kullback, S.. 1968. “Contingency Tables with Given Marginals.” Biometrika 55 (1): 179–188. CrossRef | Google Scholar
  • PubMed
    • Berinsky, Adam J., and Eric, Schickler. 2011. The American Mass Public in the 1930s and 1940s [Computer file]. Individual surveys conducted by the Gallup Organization [producers], 1936–1945: Roper Center for Public Opinion Research, University of Connecticut [distributor]. Google Scholar
    Dever, Jill A., and Valliant, Richard. 2010. “A Comparison of Variance Estimators for Poststratification to Estimated Control Totals.” Survey Methodology 36 (1): 45–56. Google Scholar
    Hillygus, Sunshine. 2016. “The Practice of Survey Research: Changes and Challenges.” In New Directions in Public Opinion, 2nd ed., edited by Berinsky, Adam J., 34–53. New York: Routledge. Google Scholar
    McDonald, Michael. 2019. “CPS Vote Over-Report and Non-Response Bias Correction.” Accessed October 2. http://www.electproject.org/home/voter-turnout/cps-methodology. Google Scholar
    Ruggles, Steven, Alexander, J. Trent, Genadek, Katie, Goeken, Ronald, Schroeder, Matthew B., and Sobek, Matthew. 2010. Integrated Public Use Microdata Series: Version 5.0 [Machine-readable database]. Minneapolis: University of Minnesota. Google Scholar
    Twitter as Data Google Scholar
    Bethlehem, Jelke G. 1988. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 4 (3): 251–260. Google Scholar
    Kastellec, Jonathan P, Lax, Jeffrey R., Malecki, Michael, and Phillips, Justin H.. 2015. “Polarizing the Electoral Connection: Partisan Representation in Supreme Court Confirmation Politics.” Journal of Politics 77 (3): 787–804. CrossRef | Google Scholar
    Kennedy, Courtney, Blumenthal, Mark, Clement, Scott et al. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33. CrossRef | Google Scholar
    Little, R. J. A. 1993. “Post-Stratification: A Modeler’s Perspective.” Journal of the American Statistical Association 88 (423): 1001–1012. CrossRef | Google Scholar
    Mickey, Robert W. 2015. Paths Out of Dixie: The Democratization of Authoritarian Enclaves in America’s Deep South. Princeton, NJ: Princeton University Press. Google Scholar
    Mosteller, Frederick, Hyman, Herbert, McCarthy, Philip J., Marks, Eli S., and Truman, David B., eds. 1949. The Pre-Election Polls of 1948. New York: Social Science Research Council. Google Scholar
    Peytchev, Andy. 2012. “Multiple Imputation for Unit Nonresponse and Measurement Error.” Public Opinion Quarterly 76 (2): 214–237. CrossRef | Google Scholar
    Särndal, Carl-Erik, and Traat, Imbi. 2011. “Domain Estimators Calibrated on Information from Another Survey.” Acta et Commentationes Universitatis Tartuensis de Mathematics 15 (2): 43–60. Google Scholar
    Caughey, Devin, Berinsky, Adam J., Chatfield, Sara, Hartman, Erin, Schickler, Eric, and Sekhon, Jasjeet S. Google Scholar
    Deming, W. Edwards, and Stephan, F. Frederick. 1940. “On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals Are Known.” Annals of Mathematical Statistics 11 (4): 427–444. CrossRef | Google Scholar
    Horvitz, D. G., and Thompson, D. J.. 1952. “A Generalization of Sampling without Replacement from a Finite Universe.” Journal of the American Statistical Association 47: 663–685. CrossRef | Google Scholar
    Core Team, R. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. www.R–project.org/. Google Scholar
    Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Non-randomized Studies.” Journal of Educational Psychology 66 (5): 688–701. CrossRef | Google Scholar
    Binder, D. A., and Theberge, A.. 1988. “Estimating the Variance of Raking- Ratio Estimators.” Canadian Journal of Statistics 16: 47–55. CrossRef | Google Scholar
    British Polling Council. 2016. “Performance of the Polls in the EU Referendum.” June 24. Accessed October 26, 2019. www.britishpollingcouncil.org/performance-of-the-polls-in-the-eu-referendum/. Google Scholar
    Economic Behavior Program, Survey Research Center, University of Michigan. 1948. Survey of Consumer Finances. Ann Arbor, MI: Institute for Social Research, Social Science Archive [producer], 1973. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2002. http://doi.org/10.3886/ICPSR03601.v1. Google Scholar
    Freedman, David A. 2001. “Ecological Inference and the Ecological Fallacy.” In International Encyclopaedia of the Social and Behavioural Sciences, edited by Smelser, N. J., and Baltes, P. B., 6:4027–4030. New York: Elsevier. CrossRef | Google Scholar
    Marken, Stephanie. 2018. “Still Listening: The State of Telephone Surveys.” Gallup, Methodology Blog, January 18. http://news.gallup.com/opinion/methodology/225143/listening–state–telephone–surveys.aspx. Google Scholar
    Steinert-Threlkeld, Zachary A Practical Introduction to Regression Discontinuity Designs: Foundations Google Scholar
    Lax, Jeffrey R., and Justin, H. Phillips. 2009. “How Should We Estimate Public Opinion in The States?” American Journal of Political Science 53 (1): 107–121. CrossRef | Google Scholar
    Splawa-Neyman, Jerzy. 1923. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.” Roczniki Nauk Roiniczych, Tom X: 1–51. Reprinted in Statistical Science, 5 (4): 465–472, 1990. Translated from Polish by D. M. Dabrowska and T. P. Speed. Google Scholar
    Squire, Peverill. 1988. “Why the 1936 Literary Digest Poll Failed.” Public Opinion Quarterly 52: 125–133. CrossRef | Google Scholar
    Cochran, William G. 1977. Sampling Techniques. 3rd ed. New York: Wiley. Google Scholar
    Deville, Jean-Claude. 1991. “A Theory of Quota Surveys.” Survey Methodology 17 (2): 163–181. Google Scholar
    Deville, Jean-Claude. 2000. “Simultaneous Calibration of Several Surveys.” In Combining Data from Different Sources, Proceedings of Statistics Canada Symposium 99, 207–212. Ottawa: Statistics Canada. Google Scholar
    Dixon, John, and Tucker, Clyde. 2010. “Survey Nonresponse.” In Handbook of Survey Research, 2nd ed., edited by Marsden, Peter V. and Wright, James D., 593–630. Bingley, UK: Emerald. Google Scholar
    Särndal, Carl-Erik, and Lundstrom, Sixten. 2005. Estimation in Surveys with Nonresponse. Hoboken, NJ: Wiley. CrossRef | Google Scholar
    Tanner, Martin A. 1996. Tools for Statistical Inference Methods for the Exploration of Posterior Distributions and Likelihood Functions. 3rd ed. New York: Springer-Verlag. CrossRef | Google Scholar
    Verba, Sidney, and Lehman Schlozman, Kay. 1977. “Unemployment, Class Consciousness, and Radical Politics: What Didn’t Happen in the Thirties.” Journal of Politics 39 (2): 291–323. CrossRef | Google Scholar
    Laver, Michael Agent Based Models of Polarization and Ethnocentrism CrossRef | Google Scholar
    Williams, Nora Webb, Casas, Andreu, Wilkerson, John D. Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias Google Scholar
    Dever, Jill A., and Valliant, Richard. 2016. “General Regression Estimation Adjusted for Undercoverage and Estimated Control Totals.” Journal of Survey Statistics and Methodology 4 (3): 289–318. CrossRef | Google Scholar
    Elliott, Michael R., and Roderick, J. A. Little. 2000. “Model-Based Alternatives to Trimming Survey Weights.” Journal of Official Statistics 16 (3): 191–209. Google Scholar
    Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70 (5): 646–675. CrossRef | Google Scholar
    Leeman, Lucas, and Wasserfallen, Fabio. 2017. “Extending the Use and Prediction Precision of Subnational Public Opinion Estimation.” American Journal of Political Science 61 (4): 1003–1022. CrossRef | Google Scholar
    Mackuen, Michael B., Erikson, Robert S., and Stimson, James A.. 1989. “Macropartisanship.” American Political Science Review 83 (4): 1125–1142. CrossRef | Google Scholar
    Pew, Research Center. 2016. As Election Nears, Voters Divided Over Democracy and “Respect”. Technical report. October. www.people–press.org/2016/10/27/as–election–nears–voters–divided–over–democracy–and–respect/. Google Scholar
    Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B (Methodological) 58 (1): 267–288. CrossRef | Google Scholar
    Wiseman, Frederick, and McDonald, Philip. 1979. “Noncontact and Refusal Rates in Consumer Telephone Surveys.” Journal of Marketing Research 16 (4): 478–484. CrossRef | Google Scholar
    Bunche, Ralph J. 1973. The Political Status of the Negro in the Age of FDR. Edited by Grantham, Dewey W.. Chicago: University of Chicago Press. CrossRef | Google Scholar
    Deville, Jean-Claude, and Sarndal, Carl-Erik. 1992. “Calibration Estimators in Survey Sampling.” Journal of the American Statistical Association 87 (418): 376–382. CrossRef | Google Scholar
    Field, Alexander J. 2006. “Table Dg34–45: Telephone industry - telephones, access lines, wire, employees, and plant: 1876–2000.” In Historical Statistics of the United States, Earliest Times to the Present: Millennial Edition, edited by Carter, Susan B., Gartner, Scott Sigmund, Haines, Michael R., Olmstead, Alan L., Sutch, Richard, and Wright, Gavin. New York: Cambridge University Press. http://hsus.cambridge.org/SeriesDg8–116. Google Scholar
    Kalton, Graham, and Flores-Cervantes, Ismael. 2003. “Weighting Methods.” Journal of Official Statistics 19 (2): 81–97. Google Scholar
    King, Gary, Rosen, Ori, and Tanner, Martin A.. 2004. “Information in Ecological Inference: An Introduction.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 1–12. New York: Cambridge University Press. CrossRef | Google Scholar
    Little, Roderick J., and Vartivarian, Sonya. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?” Survey Methodology 31 (2): 161–168. Google Scholar
    Weatherford, M. Stephen, and Sergeyev, Boris. 2000. “Thinking about Economic Interests: Class and Recession in the New Deal.” Political Behavior 22 (4): 311–339. CrossRef | Google Scholar
    Ansolabehere, Stephen, and Rivers, Douglas. 2013. “Cooperative Survey Research.” Annual Review of Political Science 16: 307–329. CrossRef | Google Scholar
    Baum, Matthew A., and Kernell, Samuel. 2001. “Economic Class and Popular Support for Franklin Roosevelt in War and Peace.” Public Opinion Quarterly 65 (2): 198–229. CrossRef | Google Scholar
  • PubMed
    • Berinsky, Adam J., Neff Powell, Eleanor, Schickler, Eric, and Brett Yohai, Ian. 2011. “Revisiting Public Opinion in the 1930s and 1940s.” PS: Political Science & Politics 44 (3): 515–520. Google Scholar
    Bethlehem, Jelke, Cobben, Fannie, and Schouten, Barry. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley. CrossRef | Google Scholar
    Gelman, Andrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 153–164. CrossRef | Google Scholar
    Lumley, Thomas S. 2010. Complex Surveys: A Guide to Analysis Using R. Hoboken, NJ: Wiley. CrossRef | Google Scholar
    Moy, Corrine. 2015. “Fit-for-purpose sampling.” International Journal of Market Research 57 (3): 491–494. Google Scholar
    Norpoth, Helmut, Sidman, Andrew H., and Suong, Clara H.. 2013. “Polls and Elections: The New Deal Realignment in Real Time.” Presidential Studies Quarterly 43 (1): 146–166. CrossRef | Google Scholar
    Särndal, Carl-Erik, and Lundstrom, Sixten. 2008. “Assessing Auxiliary Vectors for Control of Nonresponse Bias in the Calibration Estimator.” Journal of Official Statistics 24 (2): 167–191. Google Scholar
    Cattaneo, Matias, Idrobo, Nicolás and Titiunik, Rocío Agent Based Models of Social Life: Fundamentals Google Scholar
    Caldeira, Gregory A. 1987. “Public Opinion and the U.S. Supreme Court: FDR’s Court-Packing Plan.” American Political Science Review 81 (4): 1139–1153. CrossRef | Google Scholar
    Gelman, Andrew, and Thomas, C. Little. 1997. “Poststratification into Many Categories Using Hierarchical Logistic Regression.” Survey Methodology 23 (2): 127–135. Google Scholar
    Guandalini, Alessio, and Yves, Tillé. 2017. “Design-Based Estimators Calibrated on Estimated Totals from Multiple Surveys.” International Statistical Review 85 (2): 250–269. CrossRef | Google Scholar
    Kish, Leslie. 1965. Survey Sampling. New York: Wiley. Google Scholar
    Park, David K., Gelman, Andrew, and Bafumi, Joseph. 2004. “Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls.” Political Analysis 12 (4): 375–385. CrossRef | Google Scholar
    Stuart, Elizabeth A., Cole, Stephen R., Bradshaw, Catherine P., and Leaf, Philip J.. 2011. “The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials.” Journal of the Royal Statistical Society. Series A (Statistics in Society) 174 (2): 369–386. CrossRef | Google Scholar
    Treier, Shawn, and Jackman, Simon. 2008. “Democracy as a Latent Variable.” American Journal of Political Science 52 (1): 201–217. CrossRef | Google Scholar
    Wakefield, Jon. 2004. “Ecological Inference for 2 × 2 Tables.” Journal of the Royal Statistical Society. Series A (General) 167 (3): 385–445. CrossRef | Google Scholar

    Metrics

    Altmetric attention score

    Full text views

    Total number of HTML views: 0
    Total number of PDF views: 0 *
    Loading metrics...

    Abstract views

    Total abstract views: 0 *
    Loading metrics...

    * Views captured on Cambridge Core between #date#. This data will be updated every 24 hours.

    Usage data cannot currently be displayed.