Skip to main content Accessibility help
×
Home

Measuring Emotional Response: Comparing Alternative Approaches to Measurement*

  • George E. Marcus, W. Russell Neuman and Michael B. MacKuen

Abstract

Scholarly interest in the role of emotion in accounting for how people react to political figures, events, and messages has escalated over the past two plus decades in political science and psychology. However, research on the validity of the measurement of subjective self-report of emotional responses is rather limited. We introduce here a new measurement approach, a “slider” format and compare it with the long used “radio button” item format. We assess the reliability and validity of these two approaches to the measurement of affect. The study examines self-report measures of emotion to three generated news stories about terrorist threats. We report that both measurement formats are able to extract the expected threefold affect structure from a ten affect word battery. The slider format is, however, modestly more reliable, and more efficient in time to complete, has the ability to limit missing data, and generates continuous data that is less truncated than data derived from the radio button format. Finally, we report on three tests of construct validity. Both approaches exhibit equivalent results on two of those tests. However, the radio button format does poorly on one test of construct validity, that on the anticipated relationship between anxiety and interest in novel information. We present an assessment of two methods for measuring emotional reactions to stimuli such as political issues, political figures, or events. Both methods are suitable for use in online surveys or computer-driven experiments. The traditional method utilizes labeled “radio buttons” that enable a participant in a study to select by clicking on one of an array of typically five response options, ranging from lower to higher of some identified affect term (e.g., how angry one might feel). Second, the slider method offers a participant the ability to move an “arrow” up or down to indicate how much (up) or little (down) they feel. The goal of both measures is to ascertain the level of a targeted emotion, i.e., how little or how much, say anger. The slider method has been specifically developed to be used with participants using a computer. The slider approach falls within the category of visual analog scales. This method for measuring affective responses to stimuli of whatever sort has not hitherto been examined to determine its reliability and validity. The literature on the reliability and validity of these measurement strategies is thin and we found no studies including an explicit comparison.

Copyright

Footnotes

Hide All
*

George E. Marcus, Professor of Political Science, Department of Political Science, Williams College, Williamstown, MA 01267 (gmarcus@williams.edu). W. Russell Neuman, Professor of Communications, New York University, MAGNET, 2 MetroTech Center, Suite 864, Brooklyn NY, 11201. Michael B. MacKuen, Professor of Political Science, Department of Political Science, University of North Carolina Chapel Hill, Chapel Hill NC 27599-3265. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2015.65

Footnotes

References

Hide All
Abelson, Robert P., Kinder, Donald R., Peters, Mark D., and Fiske, Susan T.. 1982. ‘Affective and Semantic Components in Political Personal Perception’. Journal of Personality and Social Psychology 42:619630.
Allport, Gordon W. 1954. The Nature of Prejudice. Cambridge, MA: Addison-Wesley.
Alwin, Duane F. 2010. ‘How Good is Survey Measurement? Assessing the Reliability and Validity of Survey Measures’. In Peter V. Wright Marsden and James D. Bingley (eds), Handbook of Survey Research, 405426. UK: Emerald Group.
Amodio, David M., Jost, John T., Master, Sarah L., and Yee, Cindy M. 2007. ‘Neurocognitive Correlates of Liberalism and Conservatism’. Nature Neuroscience 4:12461247.
Ansolabehere, Stephen D., and Iyengar, Shanto. 1995. Going Negative: How Political Advertisements Shrink and Polarize the Electorate. New York: The Free Press.
Bollen, Kenneth A., and Barb, Kenney. 1981. ‘Pearson’s R and Coarsely Categorized Measures’. American Sociological Review 46:232239.
Borsboom, Denny, Mellenbergh, Gideon J., and van Heerden, Jaap. 2003. ‘The Theoretical Status of Latent Variables’. Psychological Review 110:203219.
Brader, Ted. 2006. Campaigning for Hearts and Minds: How Emotional Appeals in Political Ads Work. Chicago, IL: University of Chicago Press.
Brader, Ted, Groenendyk, Eric W., and Valentino, Nicholas. 2010. ‘Fight or Flight? When Political Threats Arouse Public Anger and Fear’. Annual Meeting of the Midwest Political Science Association, April 22–25, 2010, Chicago, IL.
Brader, Ted, and Marcus, George E.. 2013. ‘Emotion and Political Psychology’. In Leonie Huddy, Jack S. Levy and David O. Sears (eds), Oxford Handbook of Political Psychology, 165204. New York: Oxford University Press.
Cacioppo, John T., and Petty, Richard W. 1982. ‘The Need for Cognition’. Journal of Personality and Social Psychology 42:116131.
Cacioppo, John T., Gardner, Wendi L., and Berntson, Gary G.. 1999. ‘The Affect System has Parallel and Integrative Processing Components: Form Follows Function’. Journal of Personality and Social Psychology 76:839855.
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald E.. 1960. The American Voter. New York: John Wiley and Sons.
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald E.. 1966. Elections and the Political Order. New York: John Wiley and Sons, Inc.
Campbell, Donald T., and Fiske, Donald W.. 1959. ‘Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix’. Psychological Bulletin 56:81105.
Chaiken, Shelly, and Trope, Yaacov (eds). 1999. Dual Process Models in Social Psychology. New York: Guilford Press.
Cornelius, Randolf R. 1996. The Science of Emotion: Research and Tradition in the Psychology of Emotions. Upper Saddle River, NJ: Prentice Hall.
Couper, Mick P., Tourangeau, Roger, Conrad, Frederick G., and Singer, Eleanor. 2006. ‘Evaluating the Effectiveness of Visual Analog Scales: A Web Experiment’. Social Science Computer Review 24:227245.
Feldt, Leonard S. 1969. ‘A Test of the Hypothesis That Cronbach’s Alpha or Kuder-Richardson Coefficient Twenty is the Same for Two Tests’. Psychometrika 34:363373.
Feldt, Leonard S., Woodruff, David J., and Salih, Fathi A.. 1987. ‘Statistical Inference for Coefficient Alpha’. Applied Psychological Measurement 11:93101.
Fiorina, Morris P., and Abrams, Samuel J.. 2008. ‘Political Polarization in the American Public’. Annual Review of Political Science 11:563588.
Fiske, Susan T., and Taylor, Shelley E.. 1991. Social Cognition. New York: McGraw-Hill.
Gadarian, Shana K., and Albertson, Bethany. 2014. ‘Anxiety, Immigration, and the Search for Information’. Political Psychology 35:133164.
Haidt, Jonathan. 2001. ‘The Emotional Dog and its Rational Tail: A Social Intuitionist Approach to Moral Judgment’. Psychological Review 108:814834.
Hayduk, Leslie A., and Littvay, Levente. 2012. ‘Should Researchers Use Single Indicators, Best Indicators, or Multiple Indicators in Structural Equation Models?’. BMC Medical Research Methodology 12:117.
Izard, Carroll E. 1971. Patterns of Emotions: A New Analysis of Anxiety and Depression. New York: Academic Press.
Izard, Carroll. E. 1977. Human Emotions. New York: Plenum Press.
Jost, John T., and Amodio, David M.. 2012. ‘Political Ideology as Motivated Social Cognition: Behavioral and Neuroscientific Evidence’. Motivation and Emotion 36:5564.
Jost, John T., Glaser, Jack, Kruglanski, Arie W., and Sulloway, Frank J.. 2003. ‘Political Conservatism as Motivated Social Cognition’. Psychological Bulletin 129:339375.
Just, Marion R., Crigler, Ann N., and Belt, Todd L.. 2007. ‘Don’t Give Up Hope: Emotions, Candidate Appraisals, and Votes’. In W. Russell Neuman, George E. Marcus, Ann N. Crigler and Michael B. MacKuen (eds), The Affect Effect, 231259. Chicago, IL: University of Chicago Press.
Kahneman, Daniel. 2011. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.
Krosnick, Jon A., and Presser, Stanley. 2010. ‘Question and Questionnaire Design’. In Peter V. Wright Marsden and James D. Bingley (eds), Handbook of Survey Research, 263314. UK: Emerald Group.
Lerner, Jennifer S., and Keltner, Dacher. 2000. ‘Beyond Valence: Toward a Model of Emotion-Specific Influences on Judgement and Choice’. Cognition and Emotion 14:473493.
Lerner, Jennifer S., and Keltner, Dacher. 2001. ‘Fear, Anger, and Risk’. Journal of Personality and Social Psychology 81:146159.
Lerner, Jennifer S., Gonzalez, Roxana M., Small, Deborah A., and Fischhoff, Baruch. 2003. ‘Effects of Fear and Anger on Perceived Risks of Terrorism: A National Field Experiment’. Psychological Science 14:144150.
Lodge, G. Milton, and Tursky, Bernard. 1979. ‘Comparisons Between Category and Magnitude Scaling of Political Opinion Employing SRC/CPS Items’. American Political Science Review 73:5066.
Lodge, G. Milton, and Tursky, Bernard. 1981. ‘On the Magnitude Scaling of Public Opinion in Survey Research’. American Journal of Political Science 25:376419.
Lord, Charles G., Ross, Lee, and Lepper, Mark R.. 1979. ‘Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence’. Journal of Personality and Social Psychology 37:20982109.
MacKuen, Michael B., Wolak, Jennifer, Keele, Luke, and Marcus, George E.. 2010. ‘Civic Engagements: Resolute Partisanship or Reflective Deliberation’. American Journal of Political Science 54:440458.
MacKuen, Michael B., Miller, Patrick R., Marcus, George E., and Neuman, W. Russell. 2013. ‘Affective Signatures and Attention: The Persistent Impact of Emotional Responses to the News’. Midwest Political Science Association Annual Meeting, April 11–13, 2013, Chicago, IL.
Marcus, George E. 2002. The Sentimental Citizen: Emotion in Democratic Politics. University Park, PA: Pennsylvania State University Press.
Marcus, George E. 2003. ‘The Psychology of Emotion and Politics’. In David O. Sears, Leonie Huddy and Robert Jervis (eds), Oxford Handbook of Political Psychology, 182221. Oxford: Oxford University Press.
Marcus, George E. 2008. ‘Different Situations, Different Responses: Threat, Partisanship, Risk, and Deliberation’. Critical Review 20:7589.
Marcus, George E., and MacKuen, Michael B.. 1993. ‘Anxiety, Enthusiasm and the Vote: The Emotional Underpinnings of Learning and Involvement During Presidential Campaigns’. American Political Science Review 87:688701.
Marcus, George E., MacKuen, Michael B., Wolak, Jennifer, and Keele, Luke. 2006. ‘The Measure and Mismeasure of Emotion’. In David Redlawsk (ed.), Feeling Politics: Emotion in Political Information Processing, 3145. New York: Palgrave Macmillan.
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2000. Affective Intelligence and Political Judgment. Chicago, IL: University of Chicago Press.
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2009a. ‘Assessing Subjective Emotional Responses: Two Approaches to Measurement’. Midwest Political Science Association Annual Meetings. Chicago, IL.
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2009b. ‘Measuring Subjective Emotional Response: New Evidence on an Alternative Method’. Annual Scientific Meeting of the International Society of Political Psychology, July 14–17, 2009. Dublin, Ireland.
Masters, Roger D. 2001. ‘Biology and Politics: Linking Nature and Nurture’. Annual Review of Political Science 4:345369.
Masters, Roger D., and Sullivan, Denis. 1993. ‘Nonverbal Behavior and Leadership: Emotion and Cognition in Political Attitudes’. In Shanto Iyengar and William McGuire (eds), Explorations in Political Psychology, 150182. Durham, NC: Duke University Press.
Masters, Roger D., and Sullivan, Denis G. 1989. ‘Facial Displays and Political Leadership in France’. Behavioral Processes 19:130.
Mauss, Iris B., and Robinson, Michael D.. 2009. ‘Measures of Emotion: A Review’. Cognition and Emotion 23:209237.
Mayer, John D., and Gaschke, Yvonne N.. 1988. ‘The Experience and Meta-Experience of Mood’. Journal of Personality and Social Psychology 55:102111.
McGuire, William J. 1969. ‘The Nature of Attitudes and Attitude Change’. In Gardner Lindzey and Elliot Aronson (eds), The Handbook of Social Psychology, 136314. Reading, MA: Addison-Wesley.
McHugo, Gregory J., Lanzetta, John T., Sullivan, Denis G., Masters, Roger D., and Englis, Basil. 1985. ‘Emotional Reactions to Expressive Displays of a Political Leader’. Journal of Personality and Social Psychology 49:15121529.
Mutz, Diana C., and Reeves, Byron. 2005. ‘The New Videomalaise: Effects of Televised Incivility on Political Trust’. American Political Science Review 99:115.
Neuman, W. Russell, Marcus, George E., and MacKuen, Michael B.. 2013. ‘Hardwired for News: Affective Intelligence and Political Attention’. Paper presented at the American Political Science Association Annual Meetings, Chicago, IL.
Nunnally, Jum C., and Bernstein, Ira H.. 1994. Psychometric Theory. New York: McGraw-Hill.
Ortony, Andrew, Clore, Gerald L., and Collins, Allan. 1989. The Cognitive Structure of Emotions. New York: Cambridge University Press.
Plutchik, Robert, and Conte, Hope R.. 1997. Circumplex Models of Personality and Emotions. Washington, DC: American Psychological Association.
Rammstedt, Beatrice, and John, Oliver P.. 2007. ‘Measuring Personality in One Minute or Less: A 10-Item Short Version of the Big Five Inventory in English and German’. Journal of Research in Personality 41:203212.
Redlawsk, David P. 2002. ‘Hot Cognition or Cool Consideration? Testing the Effects of Motivated Reasoning on Political Decision Making’. Journal of Politics 64:10211044.
Roseman, Ira. 1979. ‘Cognitive Aspects of Emotion and Emotional Behavior’. Paper presented at the 87th Annual Convention of the American Psychological Association, New York, September 1–5, 1979.
Roseman, Ira. 1984. ‘Cognitive Determinants of Emotions: A Structural Theory’. In Phillip R. Shaver (ed.), Review of Personality and Social Psychology, 1136. Beverly Hills, CA: Sage.
Ruef, Anna M., and Levenson, Robert W. 2007. ‘Continuous Measurement of Emotion: The Affect Rating Dial’. In James A. Coan and John J. B. Allen (eds), The Handbook of Emotion Elicitation and Assessment, 286297. New York: Oxford University Press.
Russell, James A., and Carroll, James M. 1999. ‘On the Bipolarity of Positive and Negative Affect’. Psychological Bulletin 125:330.
Sullivan, Dennis, and Masters, Roger. 1988. ‘Happy Warriors: Leaders’ Facial Displays, Viewers Emotions, and Political Support’. American Journal of Political Science 32:345368.
Valentino, Nicholas A., Brader, Ted, Groenendyk, Eric W., Gregorowicz, Krysha, and Hutchings, Vincent L.. 2011. ‘Election Night’s Alright for Fighting: The Role of Emotions in Political Participation’. Journal of Politics 73:156170.
Valentino, Nicholas A., Hutchings, Vincent L., Banks, Antoine J., and Davis, Anne K.. 2008. ‘Is a Worried Citizen a Good Citizen? Emotions, Political Information Seeking, and Learning Via the Internet’. Political Psychology 29:247273.
Watson, David. 1988aIntraindividual and Interindividual Analyses of Positive and Negative Affect: Their Relation to Health Complaints, Perceived Stress, and Daily Activities’. Journal of Personality and Social Psychology 54:10201030.
Watson, David. 1988bThe Vicissitudes of Mood Measurement: Effects of Varying Descriptors, Time Frames, and Response Formats on Measures of Positive and Negative Affect’. Journal of Personality and Social Psychology 55:128141.
Watson, David, and Tellegen, Auke. 1985. ‘Toward a Consensual Structure of Mood’. Psychological Bulletin 98:219235.
Watson, David, Wiese, David, Vaidya, Jatin, and Tellegen, Auke. 1999. ‘The Two General Activation Systems of Affect: Structural Findings, Evolutionary Considerations, and Psychobiological Evidence’. Journal of Personality and Social Psychology 76:820838.
Watson, David, and Clark, Lee Anna. 1994. The PANAS-X: Manual for the Positive and Negative Affect Schedule-Expanded Form, 35.
Watson, David, and Clark, Lee Anna. 1997. ‘Measurement and Mismeasurement of Mood: Recurrent and Emergent Issues’. Journal of Personality Assessment 68:267296.
Watson, David, Clark, Lee Anna, and Tellegen, Auke. 1988. ‘Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales’. Journal of Personality and Social Psychology 54:10631070.
Way, Baldwin M., and Masters, Roger D.. 1996. ‘Political Attitudes: Interactions of Cognition and Affect’. Motivation and Cognition 20:205236.
Zajonc, Robert B. 1980. ‘Feeling and Thinking: Preferences Need No Inferences’. American Psychologist 35:151175.
Zuckerman, Marvin. 1991. Psychobiology of Personality. Cambridge, England: Cambridge University Press.
Type Description Title
PDF
Supplementary materials

Marcus supplementary material
Appendix

 PDF (798 KB)
798 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed