Skip to main content
    • Aa
    • Aa

Extracting Wisdom from Experts and Small Crowds: Strategies for Improving Informant-based Measures of Political Concepts

  • Cherie D. Maestas (a1), Matthew K. Buttice (a2) and Walter J. Stone (a3)

Social scientists have increasingly turned to expert judgments to generate data for difficult-to-measure concepts, but getting access to and response from highly expert informants can be costly and challenging. We examine how informant selection and post-survey response aggregation influence the validity and reliability of measures built from informant observations. We draw upon three surveys with parallel survey questions of candidate characteristics to examine the trade-off between expanding the size of the local informant pool and the pool's level of expertise. We find that a “wisdom-of-crowds” effect trumps the benefits associated with the expertise of individual informants when the size of the rater pool is modestly increased. We demonstrate that the benefits of expertise are best realized by prescreening potential informants for expertise rather than post-survey weighting by expertise.

Corresponding author
e-mail: (corresponding author)
Hide All

Authors' note: We would like to thank Lonna Rae Atkeson, Alex Adams, Ben Highton, Brad Jones, Chris Reenock, and the anonymous reviewers for helpful comments on previous drafts. Matthew Buttice began work on this project while at UC Davis and finished while at the California Research Bureau. The research results and conclusions expressed in this article do not necessarily reflect the views of the California Research Bureau or California State Library. Supplementary materials for this article are available on the Political Analysis Web site.

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Jeremy J. Albright , and Peter Mair . 2011. Does the number of parties to place affect the placement of parties? Results from an expert survey experiment. Electoral Studies 30 (4): 858-64.

Patric Andersson , Jan Edman , and Mattais Ekman . 2005. Predicting the World Cup 2002: Performance and confidence of experts and non-experts. International Journal of Forecasting 21 (3): 565-76.

Kenneth K. Boyer , and Rohit Verma . 2000. Multiple raters in survey-based operations management research: A review and tutorial. Production and Operations Management 9 (2): 128-40.

Yann Braouezec . 2010. Committee, expert advice, and the weighted majority algorithm: An application to the pricing decision of a monopolist. Computational Economics 35: 245-67.

Ian Budge . 2000. Expert judgments of party policy positions: Uses and limitations in political research. European Journal of Political Research 37: 103-13.

Matthew K. Buttice , and Walter J. Stone 2012. Candidates matter: Policy and quality differences in congressional elections. Journal of Politics 74 (3): 870-87.

Royce Carroll , Jeffrey B. Lewis , James Lo , Keith T. Poole , and Howard Rosenthal . 2009. Measuring bias and uncertainty in DW-NOMINATE ideal point estimates via the parametric bootstrap. Political Analysis 17 (3): 261-75.

Francis G. Castles , and Peter Mair . 1984. Left-right political scales, some “experts” judgments. European Journal of Political Research 12 (1): 7388.

Joshua D. Clinton , and David E. Lewis 2008. Expert opinion, agency characteristics, and agency preferences. Political Analysis 16 (1): 320.

Roger M. Cooke , and Louis H. J. Goossens 2004. Expert judgement elicitation for risk assessments of critical infrastructures. Journal of Risk Research 7 (6): 643-56.

Stanley Feldman , and Pamela Conover . 1983. Candidates, issues and voters: The role of inference in political perception. Journal of Politics 45 (4): 810-39.

Liesbet Hooghe , Ryan Bakker , Anna Brigevich , Catherine De Vries , Erica Edwards , Gary Marks , Jan Rovny , Marco Steenbergen , and Milada Vachudova . 2010. Reliability and validity of the 2002 and 2006 Chapel Hill Expert Surveys on party positioning. European Journal of Political Research 49: 687703.

John Huber , and Richard Inglehart . 1995. Expert interpretations of party space and party locations in 42 societies. Party Politics 1 (1): 73111.

Robert Huckfeldt , Edward G. Carmines , Jeffery J. Mondak , and Eric Zeemering . 2007. Information, activation, and electoral competition in the 2002 congressional elections. Journal of Politics 69 (3): 798812.

Kristin N. Javaras , H. Hill Godlsmith , and Nan M. Laird 2011. Estimating the effect of a predictor measured by two informants on a continuous outcome: A comparison of methods. Epidemiology 22 (3): 390-99.

Bradford S. Jones , and Barbara Norrander . 1996. The reliability of aggregated public opinion measures. American Journal of Political Science 40 (2): 295309.

Tara G. Martin , Mark A. Bergman , Fiona Fidler , Petra M. Kuhnert , Samantha Low-Choy , Marissa McBride , and Kerrie Mengersen . 2012. Eliciting expert knowledge in conservation science. Conservation Biology 26 (1): 2938.

Stewart J. H. McCann 1998. The extended American social, economic, and political threat index (1788–1992). Journal of Psychology 132 (4): 435-49.

Lynn W. Phillips 1981. Assessing measurement error in key informant reports: A methodological note on organizational analysis in marketing. Journal of Marketing Research 18 (4): 395415.

Lynda Powell . 1989. Analyzing misinformation: Perceptions of congressional candidates’ ideologies. American Journal of Political Science 33: 272-93.

Sebastian M. Saiegh 2009. Recovering a basic space from elite surveys: Evidence from Latin America. Legislative Studies Quarterly 34 (1): 117-45.

Lennart Sjoberg . 2009. Are all crowds equally wise? A comparison of political election forecasts by experts and the public. Journal of Forecasting 28 (1): 118.

Marco Steenbergen , and Gary Marks . 2007. Evaluating expert judgments. European Journal of Political Research 46: 347-66.

Walter J. Stone , and Elizabeth N. Simas 2010. Candidate valence and ideological positions in U.S. House elections. American Journal of Political Science 54 (2): 371-88.

Walter J. Stone , Sarah A. Fulton , Cherie D. Maestas , and L. Sandy Maisel 2010. Incumbency reconsidered: Prospects, strategic retirement, and incumbent quality in U.S. House elections. Journal of Politics 72 (1): 178-90.

Walter J. Stone , L. Sandy Maisel , and Cherie D. Maestas 2004. Quality counts: Extending the strategic politician model of incumbent deterrence. American Journal of Political Science 48 (3): 479-95.

Gerrit H. Van Bruggen , Gary L. Lilien , and Manish Kacker . 2002. Informants in organizational marketing research: Why use multiple informants and how to aggregate responses. Journal of Marketing Research 39 (4): 469-78.

Stephan M. Wagner , Christian Rau , and Eckhard Lindemann . 2010. Multiple informant methodology: A critical review and recommendations. Sociological Methods and Research 38 (4): 582618.

Stephen Whitefield , Matilda Anna Vachudova , Marco R. Steenbergen , Robert Rohrschneider , Gary Marks , Matthew P. Loveless , and Liesbet Hooghe . 2007. Do expert surveys produce consistent estimates of party stances on European integration? Comparing expert surveys in the difficult case of Central and Eastern Europe. Electoral Studies 26 (1): 5061.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary Materials

Maestas et al. supplementary material

 Word (43 KB)
43 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 9 *
Loading metrics...

Abstract views

Total abstract views: 76 *
Loading metrics...

* Views captured on Cambridge Core between 4th January 2017 - 29th July 2017. This data will be updated every 24 hours.