Skip to main content

Cross-Validating Measures of Global Religious Intolerance: Comparing Coded State Department Reports with Survey Data and Expert Opinion

  • Brian J. Grim (a1) and Richard Wike (a2)

We address a pressing substantive issue as well as evaluate several methodologies in this article. Substantively, we ask whether the U.S. State Department has a clear understanding of the level of cross-national religious intolerance that triggers daily headlines around the globe. Methodologically, we ask whether data on social attitudes coded from systematic qualitative reports can reliably represent cross-national public opinion. We empirically address these questions by comparing cross-national levels of religious intolerance coded from the State Department's annual international religious freedom reports with relevant population survey data from the World Values Survey and the Pew Research Center, as well as with data from written interviews of country experts conducted by the Hudson Institute. The results indicate that the understanding of social religious intolerance embodied in the State Department reports is comparable with the results of population surveys and individual expert opinion. Methodologically, this suggests that cross-national public opinion survey data can be cross-validated with coded data from systematic qualitative analysis as well as with expert opinion.

Corresponding author
Address correspondence and reprint requests to: Brian J. Grim, Pew Research Center's Forum on Religion & Public Life. E-mail:
Richard Wike, Pew Research Center's Global Attitudes Project. E-mail:
Hide All
Bahr, Howard M., Caplow, Theodore, and Chadwick, Bruce A.. 1983. “Middletown III: Problems of Replication, Longitudinal Measurement, and Triangulation.” Annual Review of Sociology 9:243264.
Barro, Robert J., and McCleary, Rachel M.. 2003. “Religion and Economic Growth Across Nations.” American Sociological Review 68:760781.
Berger, Peter. 2006. “Religion in a Globalizing World.” (Accessed on January 7, 2007).
Bratton, Michael. 2002. “Wide but Shallow: Popular Support for Democracy in Africa.” Afrobarometer Paper No. 19.
Crittendon, Kathleen S., and Hill, Richard J.. 1971. “Coding Reliability and Validity of Interview Data.” American Sociological Review 36:10731080.
Curtin, Richard, Presser, Stanley, and Singer, Eleanor. 2000. “The Effect of Response Rate Changes on the Index of Consumer Sentiment.” Public Opinion Quarterly 64:413428.
Danan, Liora, Hunt, Alice, Barton, Rick, and von Hippel, Karin. 2007. “Mixed Blessings: U.S. Government Engagement with Religion in Conflict-prone Settings.” (Accessed on January 4, 2006).
Daves, Robert P., and Newport, Frank. 2005. “Pollsters under Attack: 2004 Election Incivility and Its Consequences.” Public Opinion Quarterly 69:670681.
Denzin, Norman K. 1970. The Research Act: A Theoretical Introduction to Sociological Methods. Chicago, IL: Aldine.
Dilulio, John J. Jr. 2007. “Spiritualpolitique: Religion Matters More than Ever in Global Affairs. But Don't Count on the Experts—or the State Department—to know that.”
Eisenstein, Marie A. 2006. “Rethinking the Relationship between Religion and Political Tolerance in the US.” Political Behavior 28:327348.
Farr, Thomas F. 2008. “Diplomacy in an Age of Faith: Religious Freedom and National Security.” Foreign Affairs 87:110124.
Fetzer, Joel S., and Soper, J. Christopher. 2003. “The Roots of Public Attitudes Toward State Accommodation of European Muslims' Religious Practices Before and After September 11.” Journal for the Scientific Study of Religion 42:247258.
Finke, Roger, and Stark, Rodney. 2005. The Churching of America 1776–2005: Winners and Losers in Our Religious Economy, 2d ed.New Brunswick, NJ: Rutgers University Press.
Fox, Jonathan. 2008. A World Survey of Religion and the State. New York, NY: Cambridge University Press.
Frankovic, Kathleen A. 2005. “Reporting ‘the Polls’ in 2004.” Public Opinion Quarterly 69:682697.
Funkhouser, , Ray, G., and Parker, Edwin B.. 1968. “Analyzing Coding Reliability: The Random Systematic-Error Coefficient.” The Public Opinion Quarterly 32:122128.
Grim, Brian J. 2004. “The Cities of God versus the Countries of Earth: The Regulation of Religious Freedom (RRF).” Presented at the annual meeting of the Association for Study of Religion, Economics, and Culture, October 22, Kansas City, KS.
Grim, Brian J. 2005. “Religious Regulation's Impact on Religious Persecution: The Effects of De Facto and De Jure Religious Regulation.” Ph.D. dissertation, Department of Sociology, The Pennsylvania State University, University Park, PA.
Grim, Brian J. 2008. “Dimensions of Restricted Religious Freedom.” In Religious Freedom in the World 2007, ed. Marshall, P.Lanham, MD: Rowman and Littlefield Publishers. 496498.
Grim, Brian J., and Finke, Roger. 2005. “Documenting Religion Worldwide: Decreasing the Data Deficit.” IASSIST Quarterly 29:1116.
Grim, Brian J., and Finke, Roger. 2006. “International Religion Indexes: Government Regulation, Government Favoritism, and Social Regulation of Religion.” (Accessed on September 3, 2006).
Grim, Brian J., and Finke, Roger. 2007. “Religious Persecution in Cross-National Context: Clashing Civilizations or Regulated Economies?American Sociological Review 72:633658.
Grim, Brian J., Finke, Roger, Harris, Jaime, Meyers, Catherine, and VanEerden, Julie. 2006. “Measuring International Socio-Religious Values and Conflict by Coding U.S. State Department Reports.” JMS Proceedings, AAPOR-Section on Survey Research Methods [CD-ROM]. Alexandria, VA: American Statistical Association: 4120–27.
Grim, Brian J., Harmon, Alison H., and Gromis, Judy C.. 2006. “Focused Group Interviews as an Innovative Quanti-Qualitative Methodology (QQM): Integrating Quantitative Elements into a Qualitative Methodology.” The Qualitative Report 11:516537.
Groves, Robert M., Cialdini, Robert B., and Couper, Mick P.. 1992. “Understanding the Decision to Participate in a Survey.” Public Opinion Quarterly 56:475495.
Groves, Robert M., Fowler, Floyd J. Jr.Couper, Mick P., Lepowski, James M., Singer, Eleanor, and Tourangeau, Roger. 2004. Survey Methodology. Hoboken, NJ: John Wiley & Sons.
Harkness, Janet A., van de Vijver, Fons J. R., and Mohler, Peter Ph.. 2002. Cross-Cultural Survey Methods. Hoboken, N.J: John Wiley-Interscience.
Health, Anthony, Fisher, Stephen, and Smith, Shawna. 2005. “The Globalization of Public Opinion.” Annual Review of Political Science 8:297333.
Holbrook, Allyson L., Green, Melanie C., and Krosnick, Jon A.. 2003. “Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias.” Public Opinion Quarterly 67:79125.
Huffington, Arianna. 1998. “Hang It Up.” (Accessed on December 13, 2007).
Hyman, H.H., and Sheatsley, P.B.. 1950. “The Current Status of American Public Opinion.” In The Teaching of Contemporary Affairs, Twenty-first Yearbook of the National Council of Social Studies, ed. Payne, J.C. Washington, DC: National Council of Social Studies. 1134.
Igo, Sahar. 2007. The Averaged American: Surveys, Citizens, and the Making of a Mass Public. Cambridge, MA: Harvard University Press.
Inglehart, Ronald. 2003. “How Solid is Mass Support for Democracy – And How Can We Measure It?PS: Political Science and Politics 36:5157.
Inglehart, Ronald, and Welzel, Christian. 2005. Modernization, Cultural Change, and Democracy: The Human Development Sequence. New York, NY: Cambridge University Press.
Jenkins, Philip. 2007. God's Continent: Christianity, Islam, and Europe's Religious Crisis. Oxford, UK: Oxford University Press.
Karpov, Vyacheslav. 2002. “Religiosity and Tolerance in the United States and Poland.” Journal for the Scientific Study of Religion 41:267288.
Keeter, Scott, Miller, Carolyn, Kohut, Andrew, Groves, Robert M., and Presser, Stanley. 2000. “Consequences of Reducing Nonresponse in a National Telephone Survey.” Public Opinion Quarterly 64:125148.
Keeter, Scott, Kennedy, Courtney, Clark, April, Tompson, Trevor, and Mokrzycki, Mike. 2007. “What's Missing from National Landline RDD Surveys? The Impact of The Growing Cell-Only Population.” Public Opinion Quarterly 71:772792.
Kohut, Andrew, and Stokes, Bruce. 2006. America Against the World: How We Are Different and Why We Are Disliked. New York, NY: Henry Holt and Co.
Lang, Gladys Engel, and Lang, Kurt. 1987. “Comments in: The Future Study of Public Opinion: A Symposium.” Public Opinion Quarterly 51:S181S182.
Lang, Kurt. 2008. “What Polls Can and Cannot Tell us About Public Opinion: Keynote Speech at the 60th Annual Conference of WAPOR.” International Journal of Public Opinion Research 20:322.
Lazarsfeld, Paul F., and Rosenberg, Morris. 1955. The Language of Social Research: A Reader in the Methodology of Social Research. New York, NY: The Free Press.
Lin, I-Fen, and Schaeffer, Nora Cate. 1995. “Using Survey Participants to Estimate the Impact of Nonparticipation.” Public Opinion Quarterly 59:236258.
Lynch, Marc. 2003. “Taking Arabs Seriously.” Foreign Affairs 82:8194.
Marshall, Paul. 2008. Religious Freedom in the World 2007. Lanham, MD: Rowman and Littlefield Publishers.
Maruyama, Geoffrey M. 1998. Basics of Structural Equation Modeling. Thousand Oaks, CA: Sage.
Moaddel, Mansoor. 2002. Jordanian Exceptionalism: A Comparative Analysis of Religion and State Relationships in Egypt, Iran, Jordan, and Syria. New York, NY: Palgrave.
Montgomery, Andrew, and Crittendon, Kathleen S.. 1977. “Improving Coding Reliability for Open-ended Questions.” The Public Opinion Quarterly 41:235243.
Netjes, Catherine E., and Binnema, Harmen A.. 2007. “The Salience of the European Integration Issue: Three Data Sources Compared.” Electoral Studies 26:3949.
Neuman, W. Russell. 1990. “The Threshold of Public Attention.” Public Opinion Quarterly 54: 159176.
Norris, Pipa. 1999. “Institutional Explanations for Political Support.” In Critical Citizens: Global Support for Democratic Governance, ed. Norris, Pipa.Oxford, UK: Oxford University Press. 217235.
Norris, Pippa, and Inglehart, Ronald. 2008. “Polls for the Public Good: Mass and Elite Evaluations of the Health of Democratic Governance.” Paper presented at the annual meeting of the World Assosication of Public Opinion Researchers, New Orleans, May 14.
North, Charles M., and Gwin, Carl R.. 2004. “Religious Freedom and the Unintended Consequences of the Establishment of Religion.” Southern Economic Journal, 71:103117.
Pew Forum. 2006. “Spirit and Power: A Ten-Nation Study of Pentecostals.” (Accessed on October 5, 2006).
Pew Global Attitudes Project. 2006. “The Great Divide: How Westerners and Muslims View Each Other.” (Accessed on September 3, 2006).
Putnam, Robert D., Leonardi, Robert, and Nanetti, Rffaella Y.. 1993. Making Democracy Work: Civic Traditions in Modern Italy. Princeton, NJ: Princeton University Press.
Riley, John W. Jr. 1962. “Reflections on Data Sources in Opinion Research.” Public Opinion Quarterly 26:313322.
Schuman, Howard, Kalton, Graham, and Ludwig, Jacob. 1983. “Context and Contiguity in Survey Questionnaires.” The Public Opinion Quarterly 47:112115.
Shah, Timothy Samuel, and Toft, Monica Duffy. 2006. “Why God is Winning.” Foreign Policy, July/August:3943.
Smith, Pamela Hyde. 2007. “Politics and Diplomacy: The Hard Road Back to Soft Diplomacy.” Georgetown Journal of International Affairs. (Accessed August 2008).
Teitler, Julien O., Reichman, Nancy E., and Sprachman, Susan. 2003. “Costs and Benefits of Improving Response Rates for a Hard-to-Reach Population.” Public Opinion Quarterly 67:126138.
Tetlock, Philip. 2006. Expert Political Judgment: How Good Is It? How Can We Know? Princeton, NJ: Princeton University Press.
Johan D, Van der Vyver. 2006. “Prospects for Converting the Declaration into a Binding Convention.” Presented at the U.S. State Department commemoration of the 25th anniversary of the United Nations' Declaration on the Elimination of All Forms of Intolerance and of Discrimination Based on Religion or Belief, October 30, Washington, DC.
Webb, Eugene J., Campbell, Donald T., Schwartz, Richard D., Sechrest, Lee, and Grove, Janet Belew. 1981. Nonreactive Measures in the Social Sciences, 2nd Ed.Boston, MA: Houghton Mifflin.
Webb, Eugene J., Campbell, Donald T., Schwartz, Richard D., and Sechrest, Lee. 1966. Nonreactive Measures in the Social Sciences. Chicago, IL: Rand McNally.
Wike, Richard, and Grim, Brian J.. 2007. “Levels of Negativity: How Muslim and Western Publics See One Another.” (Accessed on October 2007).
Wiktorowicz, Quintan. 2001. The Management of Islamic Activism: Salafis, the Muslim Brotherhood, and State Power in Jordan. Albany, NY: State University of New York.
Willis, Gordon B., Caspar, Rachel A., and Lessler, Judith T.. 1999. “Cognitive Interviewing: A ‘How to’ Guide,” (Accessed October 2003).
Yamagishi, T., and Yamagishi, M.. 1994. “Trust and Commitment in the United States and Japan.” Motivation and Emotion 18:129–66.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Politics and Religion
  • ISSN: 1755-0483
  • EISSN: 1755-0491
  • URL: /core/journals/politics-and-religion
Please enter your name
Please enter a valid email address
Who would you like to send this to? *