Skip to main content Accessibility help
  • Print publication year: 2017
  • Online publication date: July 2017

12 - Automatic Analysis of Bodily Social Signals

from Part II - Machine Analysis of Social Signals


The human body plays an important role in face-to-face interactions (Knapp & Hall, 2010; McNeill, 1992). We use our bodies to regulate turns, to display attitudes and to signal attention (Scheflen, 1964). Unconsciously, the body also reflects our affective and mental states (Ekman & Friesen, 1969). There is a long history of research into the bodily behaviors that correlate with the social and affective state of a person, in particular in interaction with others (Argyle, 2010; Dittmann, 1987; Mehrabian, 1968). We will refer to these behaviors as bodily social signals. These social and affective cues can be detected and interpreted by observing the human body's posture and movement (Harrigan, 2008; Kleinsmith & Bianchi-Berthouze, 2013). Automatic observation and analysis has applications such as the detection of driver fatigue and deception, the analysis of interest and mood in interactions with robot companions, and in the interpretation of higher-level phenomena such as mimicry and turn-taking.

In this chapter, we will discuss various bodily social signals, and how to analyze and recognize them automatically. Human motion can be studied on many levels, from the physical level involving muscles and joints, to the level of interpreting a person's fullbody actions and intentions (Poppe, 2007, 2010; Jiang et al., 2013). We will focus on automatically analyzing movements with a relatively short time scale, such as a gesture or posture shift. In the first section, we will discuss the different ways of measurement and coding, both from motion capture data and images and video. The recorded data can subsequently be interpreted in terms of social signals. In the second section, we address the automatic recognition of several bodily social signals. We will conclude the chapter with a discussion of challenges and directions of future work

Measurement of Body Motion

Body movement can be observed and described quantitatively, for example, in terms of joint rotations or qualitatively with movement labels. While social signals are typically detected and identified as belonging to a certain category, body motion is typically described quantitatively. Therefore, the detection of bodily social signals is often based on a quantitative representation of the movement. From the perspective of computation, body motion is most conveniently recorded and measured using motion capture (mocap) devices. However, their obtrusive nature, cost, and the fact that they typically cannot be used outside the laboratory has limited their employment.

Argyle, Michael (2010). Bodily Communication (2nd rev. edn). New York: Routledge.
Baesler, E. James & Burgoon, Judee K. (1987). Measurement and reliability of nonverbal behavior. Journal of Nonverbal Behavior, 11(4), 205–233.
Bazzani, Loris, Cristani, Marco, Tosato, Diego, et al. (2013). Social interactions by visual focus of attention in a three-dimensional environment. Expert Systems, 30(2), 115–127.
Bente, Gary (1989). Facilities for the graphical computer simulation of head and body movements. Behavior Research Methods, Instruments, & Computers, 21(4), 455–462.
Bente, Gary, Petersen, Anita, Krämer, Nicole C., & De Ruiter, Jan Peter (2001). Transcript-based computer animation of movement: Evaluating a new tool for nonverbal behavior research. Behavior Research Methods, Instruments, & Computers, 33(3), 303–310.
Birdwhistell, Ray L. (1952). Introduction to Kinesics: An Annotation System for Analysis of Body Motion and Gesture. Louisville, KY: University of Louisville.
Bo, Liefeng & Sminchisescu, Cristian (2010). Twin Gaussian processes for structured prediction. International Journal of Computer Vision, 87(1–2), 28–52.
Bousmalis, Konstantinos, Mehu, Marc, & Pantic, Maja. (2013). Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and tools. Image and Vision Computing, 31(2), 203–221.
Bull, Peter E. (1987). Posture and Gesture. Oxford: Pergamon Press.
Burba, Nathan, Bolas, Mark, Krum, David M., & Suma, Evan A. (2012). Unobtrusive measurement of subtle nonverbal behaviors with the Microsoft Kinect. In Proceedings of IEEE Virtual Reality Short Papers and Posters March 4–8, 2012, Costa Mesa, CA.
Condon, William S. & Ogston, William D. (1966). Sound film analysis of normal and pathological behavior patterns. Journal of Nervous and Mental Disease, 143(4), 338–347.
Dael, Nele, Mortillaro, Marcello, & Scherer, Klaus R. (2012). The body action and posture coding system (BAP): Development and reliability. Journal of Nonverbal Behavior, 36(2), 97–121.
Deutscher, Jonathan, & Reid, Ian (2005). Articulated body motion capture by stochastic search. International Journal of Computer Vision, 61(2), 185–205.
Dittmann, Allen T. (1987). The role of body movement in communication. In A.W., Siegman & S., Feldstein (Eds), Nonverbal Behavior and Communication (pp. 37–64). Hillsdale, NJ: Lawrence Erlbaum.
Eichner, M., Marin-Jimenez, M., Zisserman, A., & Ferrari, V. (2012). 2D articulated human pose estimation and retrieval in (almost) unconstrained still images. International Journal of Computer Vision, 99(2), 190–214.
Eisler, Richard M., Hersen, Michel, & Agras, W. Stewart (1973). Videotape: A method for the controlled observation of nonverbal interpersonal behavior. Behavior Therapy, 4(3), 420–425.
Ekman, Paul (1965). Communication through nonverbal behavior: A source of information about an interpersonal relationship. In S. S., Tomkins & C. E., Izard (Eds), Affect, Cognition, and Personality (pp. 390–442). New York: Springer.
Ekman, Paul & Friesen, Wallace V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage and coding. Semiotica, 1(1), 49–98.
Felzenszwalb, Pedro F., Girshick, Ross B., McAllester, David, & Ramanan, Deva (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.
Fragkiadaki, Katerina, Hu, Han, & Shi, Jianbo (2013). Pose from flow and flow from pose. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2059–2066).
Frey, Siegfried & von Cranach, Mario (1973). A method for the assessment of body movement variability. In M. von, Cranach & I., Vine (Eds), Social Communication and Movement (pp. 389–418). New York: Academic Press.
Ganapathi, Varun, Plagemann, Christian, Koller, Daphne, & Thrun, Sebastian (2010). Real time motion capture using a single time-of-flight camera. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 755–762).
Griffin, Harry J., Aung, Min S. H., Romera-Paredes, Bernardino, et al. (2013). Laughter type recognition from whole body motion. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 349–355).
Groh, Georg, Lehmann, Alexander, Reimers, Jonas, Friess, Marc Rene, & Schwarz, Loren (2010). Detecting social situations from interaction geometry. In Proceedings of the International Conference on Social Computing (SocialCom). (pp. 1–8).
Guan, Peng, Weiss, Alexander, Blan, Alexandru O., & Black, Michael J. (2009). Estimating human shape and pose from a single image. In Proceedings of the International Conference On Computer Vision (ICCV).
Hall, Edward T. (1966). The Hidden Dimension. New York: Doubleday.
Harrigan, Jinni A. (2008). Proxemics, kinesics, and gaze. In J. A., Harrigan & R., Rosenthal (Eds), New Handbook of Methods in Nonverbal Behavior Research (pp. 137–198). Oxford: Oxford University Press.
Heylen, Dirk (2006). Head gestures, gaze and the principles of conversational structure. International Journal of Humanoid Robotics, 3(3), 241–267.
Hirsbrunner, Hans-Peter, Frey, Siegfried, & Crawford, Robert (1987). Movement in human interaction: Description, parameter formation, and analysis. In A.W., Siegman & S., Feldstein (Eds), Nonverbal Behavior and Communication (pp. 99–140). Hillsdale, NJ: Lawrence Erlbaum.
Hutchinson Guest, Ann (2005). Labanotation: The System of Analyzing and Recording Movement (4th edn). New York: Routledge.
Jiang, Yu-Gang, Bhattacharya, Subhabrata, Chang, Shih-Fu, & Shah, Mubarak (2013). High-level event recognition in unconstrained videos. International Journal of Multimedia Information Retrieval, 2(2), 73–101.
Kendon, Adam (1990). Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press.
Kleinsmith, Andrea, & Bianchi-Berthouze, Nadia (2013). Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing, 4(1), 15–33.
Kleinsmith, Andrea, Bianchi-Berthouze, Nadia, & Steed, Anthony (2011). Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 41(4), 1027–1038.
Klette, Reinhard, & Tee, Garry (2007). Understanding human motion: A historic review. In B., Rosenhahn, R., Klette, & D., Metaxas (Eds), Human Motion: Understanding, Modelling, Capture and Animation (pp. 1–22). New York: Springer.
Knapp, Mark L., & Hall, Judith A. (2010). Nonverbal Communication in Human Interaction (7th edn). Andover, UK: Cengage Learning.
Lausberg, Hedda & Sloetjes, Han (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841–849.
Marcos-Ramiro, Alvaro, Pizarro-Perez, Daniel, Romera, Marta Marrón, Nguyen, Laurent, & Gatica-Perez, Daniel (2013). Body communicative cue extraction for conversational analysis. In Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
McNeill, David (1985). So you think gestures are nonverbal? Psychological Review, 92(3), 350– 371.
McNeill, David (1992). Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press.
Mead, Ross, Atrash, Amin, & Matarić, Maja J. (2013). Automated proxemic feature extraction and behavior recognition: Applications in human-robot interaction. International Journal of Social Robotics, 5(3), 367–378.
Mehrabian, Albert (1968). Some referents and measures of nonverbal behavior. Behavior Research Methods, 1(6), 203–207.
Okwechime, Dumebi, Ong, Eng-Jon, Gilbert, Andrew, & Bowden, Richard (2011). Visualisation and prediction of conversation interest through mined social signals. Pages 951–956 of: Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG).
Park, Sunghyun, Scherer, Stefan, Gratch, Jonathan, Carnevale, Peter, & Morency, Louis-Philippe (2013). Mutual behaviors during dyadic negotiation: Automatic prediction of respondent reactions. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 423–428).
Paxton, Alexandra, & Dale, Rick (2013). Frame-differencing methods for measuring bodily synchrony in conversation. Behavior Research Methods, 45(2), 329–343.
Poppe, Ronald (2007). Vision-based human motion analysis: An overview. Computer Vision and Image Understanding, 108(1–2), 4–18.
Poppe, Ronald (2010). A survey on vision-based human action recognition. Image and Vision Computing, 28(6), 976–990.
Poppe, Ronald, Van Der Zee, Sophie, Heylen, Dirk K. J., & Taylor, Paul J. (2014). AMAB: Automated measurement and analysis of body motion. Behavior ResearchMethods, 46(3), 625–633.
Romera-Paredes, Bernardino, Aung, Hane, Pontil, Massimiliano, et al. (2013). Transfer learning to account for idiosyncrasy in face and body expressions. In Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
Rozensky, Ronald H., & Honor, Laurie Feldman (1982). Notation systems for coding nonverbal behavior: A review. Journal of Behavioral Assessment, 4(2), 119–132.
Sacks, Harvey, Schegloff, Emanuel A., & Jefferson, Gail (1974). A simplest systematics for the organisation of turn-taking for conversation. Language, 50(4), 696–735.
Scheflen, Albert E. (1964). The significance of posture in communicational systems. Psychiatry, 27(4), 316–331.
Scherer, Klaus R., & Ekman, Paul (2008). Methodological issues in studying nonverbal behavior. In J. A., Harrigan & R., Rosenthal (Eds), New Handbook of Methods in Nonverbal Behavior Research (pp. 471–504). Oxford: Oxford University Press.
Scherer, Stefan, Stratou, Giota, Mahmoud, Marwa, et al. (2013). Automatic behavior descriptors for psychological disorder analysis. Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
Shotton, Jamie, Fitzgibbon, Andrew, Cook, Mat, et al. (2011). Real-time human pose recognition in parts from single depth images. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1297–1304).
Veenstra, Arno, & Hung, Hayley (2011). Do they like me? Using video cues to predict desires during speed-dates. In Proceedings of the International Conference on Computer Vision (ICCV) Workshops (pp. 838–845).
Velloso, Eduardo, Bulling, Andreas, & Gellersen, Hans (2013). AutoBAP: Automatic coding of body action and posture units from wearable sensors. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 135–140).
Vinciarelli, Alessandro, Pantic, Maja, & Bourlard, Hervé (2009). Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 1743–1759.
Vondrak, Marek, Sigal, Leonid, & Jenkins, Odest Chadwicke (2013). Dynamical simulation priors for human motion tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 52–65.
von Laban, Rudolf (1975). Laban's Principles of Dance and Movement Notation (2nd edn). London: MacDonald and Evans.
Wallbott, Harald G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(6), 879–896.