Nonverbal behavior coding is typically conducted by “hand”. To remedy this time and resource intensive undertaking, we illustrate how nonverbal social sensing, defined as the automated recording and extracting of nonverbal behavior via ubiquitous social sensing platforms, can be achieved. More precisely, we show how and what kind of nonverbal cues can be extracted and to what extent automated extracted nonverbal cues can be validly obtained with an illustrative research example. In a job interview, the applicant’s vocal and visual nonverbal immediacy behavior was automatically sensed and extracted. Results show that the applicant’s nonverbal behavior can be validly extracted. Moreover, both visual and vocal applicant nonverbal behavior predict recruiter hiring decision, which is in line with previous findings on manually coded applicant nonverbal behavior. Finally, applicant average turn duration, tempo variation, and gazing best predict recruiter hiring decision. Results and implications of such a nonverbal social sensing for future research are discussed.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Anderson, N. R. (1992). Eight decades of employment interview research: A retrospective meta-review and prospective commentary. European Work and Organizational Psychologist, 2, 1–32. doi:10.1080/09602009208408532.
Anderson, N. R., & Shackleton, V. J. (1990). Decision making in the graduate selection interview: A field study. Journal of Occupational Psychology, 63, 63–76. doi:10.1111/j.2044-8325.1990.tb00510.x.
Ba, S., & Odobez, J.-M. (2011). Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Transaction on Pattern Analysis and Machine Intelligence, 3, 101–116.
Barrick, M. R., Shaffer, J. A., & DeGrassi, S. W. (2009). What you see may not be what you get: Relationships among self-presentations tactics and ratings of interview and job performance. Journal of Applied Psychology, 94, 1394–1411.
Basu, S. (2002). Conversational scene analysis. MIT Department of EECS, Cambridge, MA. http://alumni.media.mit.edu/~sbasu/papers.html.
Biel, J.-I., Aran, O., & Gatica-Perez, D. (2011). You are known by how you vlog: Personality impressions and nonverbal behavior in YouTube. Paper presented at the proceedings of international AAAI conference on weblogs and social media, Barcelona, Spain.
Burgoon, J. K. (1994). Nonverbal signals. In M. L. Knapp & G. R. Miller (Eds.), Handbook of interpersonal communication (pp. 229–285). Thousand Oaks, CA: Sage.
Burgoon, J. K., Birk, T., & Pfau, M. (1990). Nonverbal behaviors, persuasion, and credibility. Human Communication Research, 17, 140–169.
Campion, M. A., Cheraskin, L., & Stevens, M. J. (1994). Career-related antecedents and outcomes of job rotation. Academy of Management Journal, 37, 1518–1542.
Dovidio, J. F., Brown, C. E., Heltman, K., Ellyson, S. L., & Keating, C. F. (1988). Power displays between women and men in discussions of gender-linked tasks: A multichannel study. Journal of Personality and Social Psychology, 55(4), 580–587. doi:10.1037/0022-3522.214.171.1240.
Fillmore, C. J. (1979). On fluency. In D. Kempler & W. S. Y. Wang (Eds.), Individual differences in language ability and language behavior (pp. 85–102). New York: Academic Press.
Forbes, R. J., & Jackson, P. R. (1980). Non-verbal behaviour and the outcome of selection interviews. Journal of Occupational Psychology, 53, 65–72.
Funes, K., & Odobez, J.-M. (2012). Gaze estimation from multimodal kinect data. Paper presented at the IEEE conference in computer vision and pattern recognition, Providence, RI, USA.
Gatica-Perez, D., Guillaume, L., Odobez, J.-M., & McCowan, I. (2007). Audio-visual tracking of multiple speakers in meetings. IEEE Transaction on Audio Speech, and Language Processing, 15, 601–616.
Guerrero, L. K. (2005). Observer ratings of nonverbal involvement and immediacy. In V. Manusov (Ed.), The sourcebook of nonverbal measures. Mahwah, NJ: Lawrence Erlbaum Associates.
Hollandsworth, J. G., Kazelskis, R., Stevens, J., & Dressel, M. E. (1979). Relative contributions of verbal, articulative, and nonverbal communication to employment decisions in the job interview setting. Personnel Psychology, 32, 359–367.
Imada, A. S., & Hakel, M. D. (1977). Influence of nonverbal communication and rater proximity on impressions and decisions in simulated employment interviews. Journal of Applied Psychology, 62, 295–300.
Janz, T. (1982). Initial comparisons of patterned behavior description interviews versus unstructured interviews. Journal of Applied Psychology, 67, 577–580. doi:10.1037/0021-9010.67.5.577.
Junker, H., Amft, O., Lukowicz, P., & Tröster, G. (2008). Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition, 41(6), 2010–2024. doi:10.1016/j.patcog.2007.11.016.
Kleinke, C. L. (1986). Gaze and eye contact: A research review. Psychological Bulletin, 100, 78–100.
Knapp, M. L., & Hall, J. A. (2010). Nonverbal communication in human interaction (7th ed.). Wadsworth: Cengage Learning.
Lathoud, G., & McCowan, I. A. (2003). Location based speaker segmentation. Paper presented at the meeting of acoustics, speech, and signal processing.
Leathers, D. G. (1992). Successful nonvebal communication. London: Collier Macmillan.
Leigh, T. W., & Summers, J. O. (2002). An initial evaluation of industrial buyers’ impressions of salespersons’ nonverbal cues. Journal of Personal Selling & Sales Management, 22, 41–53.
Lu, H., Rabbi, M., Chittaranjan, G. T., Frauendorfer, D., Schmid Mast, M., Campbell, A. T., & Choudhury, T. (2012). Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. Paper presented at the UbiComp, Pittsburgh, USA.
Marcos-Ramiro, A., Pizarro-Perez, D., Marron-Romera, M., Nguyen, L. S., & Gatica-Perez, D. (2013). Body communication cue extraction for conversational analysis. Paper presented at the IEEE international conference on automatic face and gesture recognition, Shanghai, China.
McGovern, T. V., & Tinsley, H. E. A. (1978). Interviewer evaluations of interviewee nonverbal behavior. Journal of Vocational Behavior, 13, 163–171. doi:10.1016/0001-8791(78)90041-6.
Mehrabian, A. (1972). Nonverbal communication. New Brunswick: AldineTransaction.
Motowidlo, S. J., Carter, G. W., Dunnette, M. D., Tippins, N., Werner, S., Burnett, J. R., et al. (1992). Studies of the structured behavioral interview. Journal of Applied Psychology, 77, 571–587. doi:10.1037/0021-9010.77.5.571.
Nguyen, L. S., Odobez, J.-M., & Gatica-Perez, D. (2012). Using self-context for multimodal detection of head nods in face-to-face interactions. Paper presented at the international conference on multimodal interactions, Santa Monica, CA, USA.
Peterson, R. T. (2005). An examination of the relative effectiveness of training in nonverbal communication: Personal selling implications. Journal of Marketing Education, 27, 143–150. doi:10.1177/0273475305276627.
Pudil, P., Novovicova, J., & Kittler, J. (1994). Floating search methods in feature selection. Pattern Recognition Letters, 15, 1119–1125.
Quené, H. (2007). On the just noticeable difference for tempo in speech. Journal of Phonetics, 35, 353–362.
Reis, H. T., & Charles, M. J. (2000). Handbook of research methods in social and personality psychology. Cambridge: Cambridge University Press.
Taute, H. A., Heiser, R. S., & McArthur, D. N. (2011). The effect of nonverbal signals on student role-play evaluations. Journal of Marketing Education, 33, 28–40. doi:10.1177/0273475310389153.
Tiedens, L. Z., & Fragale, A. R. (2003). Power moves: Complementarity in dominant and submissive nonverbal behavior. Journal of Personality and Social Psychology, 84(3), 558–568.
Vinciarelli, A., Pantic, M., Bourlard, H., & Pentland, A. (2008). Social signals, their function, and automatic analysis: A survey. Paper presented at the ICMI 2008, Chania, Crete, Greece.
Wardhaugh, R. (1985). How conversation works. New York: Basil Blackwell.
Wood, J. A. (2006). NLP revisited: Nonverbal communications and signals of trustworthiness. Journal of Personal Selling & Sales Management, 26, 197–204. doi:10.2753/pss0885-3134260206.
Woolbert, C. (1920). The effects of various modes of public reading. Journal of Applied Psychology, 16, 162–185.
We thank Dr. Florent Monay (Idiap) for the design and implementation of the sensing platform; Dr. Jean-Marc Odobez (Idiap) for his contribution to the sensing platform and the nodding recognition method; and Prof. Tanzeem Choudhury (Cornell University) for her contribution to the design of the job performance part of the study. This research was funded by the Swiss National Science Foundation through the Sinergia SONVB (Sensing and Analyzing Nonverbal Organizational Behavior) project.
Conflict of interest
The authors declare that they have no conflict of interest.
About this article
Cite this article
Frauendorfer, D., Schmid Mast, M., Nguyen, L. et al. Nonverbal Social Sensing in Action: Unobtrusive Recording and Extracting of Nonverbal Behavior in Social Interactions Illustrated with a Research Example. J Nonverbal Behav 38, 231–245 (2014). https://doi.org/10.1007/s10919-014-0173-5
- Ubiquitous social sensing platform
- Automated extraction
- Applicant nonverbal behavior
- Hiring decision
- Job interview