Skip to main content

Extracting Wisdom from Experts and Small Crowds: Strategies for Improving Informant-based Measures of Political Concepts

  • Cherie D. Maestas (a1), Matthew K. Buttice (a2) and Walter J. Stone (a3)

Social scientists have increasingly turned to expert judgments to generate data for difficult-to-measure concepts, but getting access to and response from highly expert informants can be costly and challenging. We examine how informant selection and post-survey response aggregation influence the validity and reliability of measures built from informant observations. We draw upon three surveys with parallel survey questions of candidate characteristics to examine the trade-off between expanding the size of the local informant pool and the pool's level of expertise. We find that a “wisdom-of-crowds” effect trumps the benefits associated with the expertise of individual informants when the size of the rater pool is modestly increased. We demonstrate that the benefits of expertise are best realized by prescreening potential informants for expertise rather than post-survey weighting by expertise.

Corresponding author
e-mail: (corresponding author)
Hide All

Authors' note: We would like to thank Lonna Rae Atkeson, Alex Adams, Ben Highton, Brad Jones, Chris Reenock, and the anonymous reviewers for helpful comments on previous drafts. Matthew Buttice began work on this project while at UC Davis and finished while at the California Research Bureau. The research results and conclusions expressed in this article do not necessarily reflect the views of the California Research Bureau or California State Library. Supplementary materials for this article are available on the Political Analysis Web site.

Hide All
Albright, Jeremy J., and Mair, Peter. 2011. Does the number of parties to place affect the placement of parties? Results from an expert survey experiment. Electoral Studies 30 (4): 858-64.
Andersson, Patric, Edman, Jan, and Ekman, Mattais. 2005. Predicting the World Cup 2002: Performance and confidence of experts and non-experts. International Journal of Forecasting 21 (3): 565-76.
Ansolabehere, Stephen. 2010. CCES Common Content, 2010. V3 [Version] (accessed January 18, 2014).
Benoit, Kenneth, and Laver, Michael. 2006. Party policy in modern democracies. London: Routledge.
Boyer, Kenneth K., and Verma, Rohit. 2000. Multiple raters in survey-based operations management research: A review and tutorial. Production and Operations Management 9 (2): 128-40.
Braouezec, Yann. 2010. Committee, expert advice, and the weighted majority algorithm: An application to the pricing decision of a monopolist. Computational Economics 35: 245-67.
Budge, Ian. 2000. Expert judgments of party policy positions: Uses and limitations in political research. European Journal of Political Research 37: 103-13.
Buttice, Matthew K., and Stone, Walter J. 2012. Candidates matter: Policy and quality differences in congressional elections. Journal of Politics 74 (3): 870-87.
Carroll, Royce, Lewis, Jeffrey B., Lo, James, Poole, Keith T., and Rosenthal, Howard. 2009. Measuring bias and uncertainty in DW-NOMINATE ideal point estimates via the parametric bootstrap. Political Analysis 17 (3): 261-75.
Castles, Francis G., and Mair, Peter. 1984. Left-right political scales, some “experts” judgments. European Journal of Political Research 12 (1): 7388.
Clinton, Joshua D., and Lewis, David E. 2008. Expert opinion, agency characteristics, and agency preferences. Political Analysis 16 (1): 320.
Cooke, Roger M., and Goossens, Louis H. J. 2004. Expert judgement elicitation for risk assessments of critical infrastructures. Journal of Risk Research 7 (6): 643-56.
Dancy, Logan, and Sheagley, Geoffrey. 2013. Heuristics behaving badly: Party cues and voter knowledge. American Journal of Political Science 57 (2): 312-25.
Feldman, Stanley, and Conover, Pamela. 1983. Candidates, issues and voters: The role of inference in political perception. Journal of Politics 45 (4): 810-39.
Gaissmaier, Wolfgang, and Marewski, Julian N. 2011. Forecasting elections with mere recognition from small lousy samples: A comparison of collective recognition, wisdom of crowds, and representative polls. Judgment and Decision Making 6 (1): 7388.
Hooghe, Liesbet, Bakker, Ryan, Brigevich, Anna, De Vries, Catherine, Edwards, Erica, Marks, Gary, Rovny, Jan, Steenbergen, Marco, and Vachudova, Milada. 2010. Reliability and validity of the 2002 and 2006 Chapel Hill Expert Surveys on party positioning. European Journal of Political Research 49: 687703.
Huber, John, and Inglehart, Richard. 1995. Expert interpretations of party space and party locations in 42 societies. Party Politics 1 (1): 73111.
Huckfeldt, Robert, Carmines, Edward G., Mondak, Jeffery J., and Zeemering, Eric. 2007. Information, activation, and electoral competition in the 2002 congressional elections. Journal of Politics 69 (3): 798812.
Javaras, Kristin N., Hill Godlsmith, H., and Laird, Nan M. 2011. Estimating the effect of a predictor measured by two informants on a continuous outcome: A comparison of methods. Epidemiology 22 (3): 390-99.
Jones, Bradford S., and Norrander, Barbara. 1996. The reliability of aggregated public opinion measures. American Journal of Political Science 40 (2): 295309.
Kitschelt, Herbert, and Kselman, Daniel M. 2012. Economic development, democratic experience, and political parties linkage strategies. Comparative Political Studies September:132.
Lorenz, Jan, Rauhut, Heiko, Schweitzer, Frank, and Helbing, Dirk. 2011. How social influence can undermine the wisdom of crowds effect. Proceedings of the National Acadamy of Science 108 (22): 9020-25.
Maestas, Cherie D., Buttice, Matthew K., and Stone, Walter J. 2013. Replication data for: Extracting wisdom from experts and small crowds: Strategies for improving informant-based measures of political concepts. IQSS Dataverse Network [Distributor] V1 [Version] (accessed January 18, 2014).
Martin, Tara G., Bergman, Mark A., Fidler, Fiona, Kuhnert, Petra M., Low-Choy, Samantha, McBride, Marissa, and Mengersen, Kerrie. 2012. Eliciting expert knowledge in conservation science. Conservation Biology 26 (1): 2938.
McCann, Stewart J. H. 1998. The extended American social, economic, and political threat index (1788–1992). Journal of Psychology 132 (4): 435-49.
Page Scott, E. 2007. The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton, NJ: Princeton University Press.
Phillips, Lynn W. 1981. Assessing measurement error in key informant reports: A methodological note on organizational analysis in marketing. Journal of Marketing Research 18 (4): 395415.
Poole, Keith T., and Rosenthal, Howard. 1997. Congress: A political-economic history of roll call voting. New York: Oxford University Press.
Powell, Lynda. 1989. Analyzing misinformation: Perceptions of congressional candidates’ ideologies. American Journal of Political Science 33: 272-93.
Saiegh, Sebastian M. 2009. Recovering a basic space from elite surveys: Evidence from Latin America. Legislative Studies Quarterly 34 (1): 117-45.
Sjoberg, Lennart. 2009. Are all crowds equally wise? A comparison of political election forecasts by experts and the public. Journal of Forecasting 28 (1): 118.
Steenbergen, Marco, and Marks, Gary. 2007. Evaluating expert judgments. European Journal of Political Research 46: 347-66.
Stone, Walter J., and Simas, Elizabeth N. 2010. Candidate valence and ideological positions in U.S. House elections. American Journal of Political Science 54 (2): 371-88.
Stone, Walter J., Fulton, Sarah A., Maestas, Cherie D., and Sandy Maisel, L. 2010. Incumbency reconsidered: Prospects, strategic retirement, and incumbent quality in U.S. House elections. Journal of Politics 72 (1): 178-90.
Stone, Walter J., Sandy Maisel, L., and Maestas, Cherie D. 2004. Quality counts: Extending the strategic politician model of incumbent deterrence. American Journal of Political Science 48 (3): 479-95.
Surowiecki, Jame. 2004. The wisdom of crowds. New York: Random House.
Van Bruggen, Gerrit H., Lilien, Gary L., and Kacker, Manish. 2002. Informants in organizational marketing research: Why use multiple informants and how to aggregate responses. Journal of Marketing Research 39 (4): 469-78.
Wagner, Stephan M., Rau, Christian, and Lindemann, Eckhard. 2010. Multiple informant methodology: A critical review and recommendations. Sociological Methods and Research 38 (4): 582618.
Whitefield, Stephen, Anna Vachudova, Matilda, Steenbergen, Marco R., Rohrschneider, Robert, Marks, Gary, Loveless, Matthew P., and Hooghe, Liesbet. 2007. Do expert surveys produce consistent estimates of party stances on European integration? Comparing expert surveys in the difficult case of Central and Eastern Europe. Electoral Studies 26 (1): 5061.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Maestas et al. supplementary material

 Word (43 KB)
43 KB


Altmetric attention score