Journal of Nonverbal Behavior

, Volume 38, Issue 2, pp 231–245 | Cite as

Nonverbal Social Sensing in Action: Unobtrusive Recording and Extracting of Nonverbal Behavior in Social Interactions Illustrated with a Research Example

  • Denise Frauendorfer
  • Marianne Schmid Mast
  • Laurent Nguyen
  • Daniel Gatica-Perez
Original Paper


Nonverbal behavior coding is typically conducted by “hand”. To remedy this time and resource intensive undertaking, we illustrate how nonverbal social sensing, defined as the automated recording and extracting of nonverbal behavior via ubiquitous social sensing platforms, can be achieved. More precisely, we show how and what kind of nonverbal cues can be extracted and to what extent automated extracted nonverbal cues can be validly obtained with an illustrative research example. In a job interview, the applicant’s vocal and visual nonverbal immediacy behavior was automatically sensed and extracted. Results show that the applicant’s nonverbal behavior can be validly extracted. Moreover, both visual and vocal applicant nonverbal behavior predict recruiter hiring decision, which is in line with previous findings on manually coded applicant nonverbal behavior. Finally, applicant average turn duration, tempo variation, and gazing best predict recruiter hiring decision. Results and implications of such a nonverbal social sensing for future research are discussed.


Ubiquitous social sensing platform Automated extraction Applicant nonverbal behavior Hiring decision Job interview 



We thank Dr. Florent Monay (Idiap) for the design and implementation of the sensing platform; Dr. Jean-Marc Odobez (Idiap) for his contribution to the sensing platform and the nodding recognition method; and Prof. Tanzeem Choudhury (Cornell University) for her contribution to the design of the job performance part of the study. This research was funded by the Swiss National Science Foundation through the Sinergia SONVB (Sensing and Analyzing Nonverbal Organizational Behavior) project.

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Anderson, N. R. (1992). Eight decades of employment interview research: A retrospective meta-review and prospective commentary. European Work and Organizational Psychologist, 2, 1–32. doi: 10.1080/09602009208408532.CrossRefGoogle Scholar
  2. Anderson, N. R., & Shackleton, V. J. (1990). Decision making in the graduate selection interview: A field study. Journal of Occupational Psychology, 63, 63–76. doi: 10.1111/j.2044-8325.1990.tb00510.x.CrossRefGoogle Scholar
  3. Ba, S., & Odobez, J.-M. (2011). Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Transaction on Pattern Analysis and Machine Intelligence, 3, 101–116.CrossRefGoogle Scholar
  4. Barrick, M. R., Shaffer, J. A., & DeGrassi, S. W. (2009). What you see may not be what you get: Relationships among self-presentations tactics and ratings of interview and job performance. Journal of Applied Psychology, 94, 1394–1411.PubMedCrossRefGoogle Scholar
  5. Basu, S. (2002). Conversational scene analysis. MIT Department of EECS, Cambridge, MA.
  6. Biel, J.-I., Aran, O., & Gatica-Perez, D. (2011). You are known by how you vlog: Personality impressions and nonverbal behavior in YouTube. Paper presented at the proceedings of international AAAI conference on weblogs and social media, Barcelona, Spain.Google Scholar
  7. Burgoon, J. K. (1994). Nonverbal signals. In M. L. Knapp & G. R. Miller (Eds.), Handbook of interpersonal communication (pp. 229–285). Thousand Oaks, CA: Sage.Google Scholar
  8. Burgoon, J. K., Birk, T., & Pfau, M. (1990). Nonverbal behaviors, persuasion, and credibility. Human Communication Research, 17, 140–169.CrossRefGoogle Scholar
  9. Campion, M. A., Cheraskin, L., & Stevens, M. J. (1994). Career-related antecedents and outcomes of job rotation. Academy of Management Journal, 37, 1518–1542.CrossRefGoogle Scholar
  10. Dovidio, J. F., Brown, C. E., Heltman, K., Ellyson, S. L., & Keating, C. F. (1988). Power displays between women and men in discussions of gender-linked tasks: A multichannel study. Journal of Personality and Social Psychology, 55(4), 580–587. doi: 10.1037/0022-3514.55.4.580.CrossRefGoogle Scholar
  11. Fillmore, C. J. (1979). On fluency. In D. Kempler & W. S. Y. Wang (Eds.), Individual differences in language ability and language behavior (pp. 85–102). New York: Academic Press.Google Scholar
  12. Forbes, R. J., & Jackson, P. R. (1980). Non-verbal behaviour and the outcome of selection interviews. Journal of Occupational Psychology, 53, 65–72.CrossRefGoogle Scholar
  13. Funes, K., & Odobez, J.-M. (2012). Gaze estimation from multimodal kinect data. Paper presented at the IEEE conference in computer vision and pattern recognition, Providence, RI, USA.Google Scholar
  14. Gatica-Perez, D., Guillaume, L., Odobez, J.-M., & McCowan, I. (2007). Audio-visual tracking of multiple speakers in meetings. IEEE Transaction on Audio Speech, and Language Processing, 15, 601–616.CrossRefGoogle Scholar
  15. Guerrero, L. K. (2005). Observer ratings of nonverbal involvement and immediacy. In V. Manusov (Ed.), The sourcebook of nonverbal measures. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  16. Hollandsworth, J. G., Kazelskis, R., Stevens, J., & Dressel, M. E. (1979). Relative contributions of verbal, articulative, and nonverbal communication to employment decisions in the job interview setting. Personnel Psychology, 32, 359–367.CrossRefGoogle Scholar
  17. Imada, A. S., & Hakel, M. D. (1977). Influence of nonverbal communication and rater proximity on impressions and decisions in simulated employment interviews. Journal of Applied Psychology, 62, 295–300.CrossRefGoogle Scholar
  18. Janz, T. (1982). Initial comparisons of patterned behavior description interviews versus unstructured interviews. Journal of Applied Psychology, 67, 577–580. doi: 10.1037/0021-9010.67.5.577.CrossRefGoogle Scholar
  19. Junker, H., Amft, O., Lukowicz, P., & Tröster, G. (2008). Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition, 41(6), 2010–2024. doi: 10.1016/j.patcog.2007.11.016.CrossRefGoogle Scholar
  20. Kleinke, C. L. (1986). Gaze and eye contact: A research review. Psychological Bulletin, 100, 78–100.PubMedCrossRefGoogle Scholar
  21. Knapp, M. L., & Hall, J. A. (2010). Nonverbal communication in human interaction (7th ed.). Wadsworth: Cengage Learning.Google Scholar
  22. Lathoud, G., & McCowan, I. A. (2003). Location based speaker segmentation. Paper presented at the meeting of acoustics, speech, and signal processing.Google Scholar
  23. Leathers, D. G. (1992). Successful nonvebal communication. London: Collier Macmillan.Google Scholar
  24. Leigh, T. W., & Summers, J. O. (2002). An initial evaluation of industrial buyers’ impressions of salespersons’ nonverbal cues. Journal of Personal Selling & Sales Management, 22, 41–53.Google Scholar
  25. Lu, H., Rabbi, M., Chittaranjan, G. T., Frauendorfer, D., Schmid Mast, M., Campbell, A. T., & Choudhury, T. (2012). Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. Paper presented at the UbiComp, Pittsburgh, USA.Google Scholar
  26. Marcos-Ramiro, A., Pizarro-Perez, D., Marron-Romera, M., Nguyen, L. S., & Gatica-Perez, D. (2013). Body communication cue extraction for conversational analysis. Paper presented at the IEEE international conference on automatic face and gesture recognition, Shanghai, China.Google Scholar
  27. McGovern, T. V., & Tinsley, H. E. A. (1978). Interviewer evaluations of interviewee nonverbal behavior. Journal of Vocational Behavior, 13, 163–171. doi: 10.1016/0001-8791(78)90041-6.CrossRefGoogle Scholar
  28. Mehrabian, A. (1972). Nonverbal communication. New Brunswick: AldineTransaction.Google Scholar
  29. Motowidlo, S. J., Carter, G. W., Dunnette, M. D., Tippins, N., Werner, S., Burnett, J. R., et al. (1992). Studies of the structured behavioral interview. Journal of Applied Psychology, 77, 571–587. doi: 10.1037/0021-9010.77.5.571.CrossRefGoogle Scholar
  30. Nguyen, L. S., Odobez, J.-M., & Gatica-Perez, D. (2012). Using self-context for multimodal detection of head nods in face-to-face interactions. Paper presented at the international conference on multimodal interactions, Santa Monica, CA, USA.Google Scholar
  31. Peterson, R. T. (2005). An examination of the relative effectiveness of training in nonverbal communication: Personal selling implications. Journal of Marketing Education, 27, 143–150. doi: 10.1177/0273475305276627.CrossRefGoogle Scholar
  32. Pudil, P., Novovicova, J., & Kittler, J. (1994). Floating search methods in feature selection. Pattern Recognition Letters, 15, 1119–1125.CrossRefGoogle Scholar
  33. Quené, H. (2007). On the just noticeable difference for tempo in speech. Journal of Phonetics, 35, 353–362.CrossRefGoogle Scholar
  34. Reis, H. T., & Charles, M. J. (2000). Handbook of research methods in social and personality psychology. Cambridge: Cambridge University Press.Google Scholar
  35. Taute, H. A., Heiser, R. S., & McArthur, D. N. (2011). The effect of nonverbal signals on student role-play evaluations. Journal of Marketing Education, 33, 28–40. doi: 10.1177/0273475310389153.CrossRefGoogle Scholar
  36. Tiedens, L. Z., & Fragale, A. R. (2003). Power moves: Complementarity in dominant and submissive nonverbal behavior. Journal of Personality and Social Psychology, 84(3), 558–568.PubMedCrossRefGoogle Scholar
  37. Vinciarelli, A., Pantic, M., Bourlard, H., & Pentland, A. (2008). Social signals, their function, and automatic analysis: A survey. Paper presented at the ICMI 2008, Chania, Crete, Greece.Google Scholar
  38. Wardhaugh, R. (1985). How conversation works. New York: Basil Blackwell.Google Scholar
  39. Wood, J. A. (2006). NLP revisited: Nonverbal communications and signals of trustworthiness. Journal of Personal Selling & Sales Management, 26, 197–204. doi: 10.2753/pss0885-3134260206.CrossRefGoogle Scholar
  40. Woolbert, C. (1920). The effects of various modes of public reading. Journal of Applied Psychology, 16, 162–185.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Denise Frauendorfer
    • 1
  • Marianne Schmid Mast
    • 1
  • Laurent Nguyen
    • 2
  • Daniel Gatica-Perez
    • 2
  1. 1.Department of Work and Organizational PsychologyUniversity of NeuchatelNeuchatelSwitzerland
  2. 2.Idiap Research InstituteEcole Polytechnique Federal de Lausanne (EPFL)LausanneSwitzerland

Personalised recommendations