Skip to main content
SpringerLink
Log in

Conceptual analysis of social signals: the importance of clarifying terminology

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

As a burgeoning field, Social Signal Processing (SSP) needs a solid grounding in the disciplines that have developed important concepts in the study of communication. However, the number and diversity of terms developed in linguistics, psychology, and the behavioural sciences may seem confusing for scholars who are not versed in the subtleties of conceptual analysis and theoretical developments. Indeed, different disciplines sometimes use the same term to mean different things or, conversely, use different terms to mean the same thing. The goals of this article are to present an overview of the different concepts developed in the various disciplines that studied animal and human communication, and to understand the differences and commonalities between concepts emerging from these disciplines. We conclude that such an understanding will greatly improve the efficiency of pluridisciplinary research projects, for the advancement of SSP requires that we look at the complexity of communication from different angles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Note that within the field of ethology, some authors consider signals as information carrier (e.g. [14, 103]); while other authors see signals as mostly influential [51, 65, 79].

References

  1. Allwood J (1976) Linguistic communication as action and cooperation. Gothenburg monographs in linguistics, vol 2. University of Göteborg, Dept of Linguistics

  2. Allwood J (1977) A critical look at speech act theory. In: Dahl O (ed) Logic, pragmatics, and grammar. Studentlitteatur, Lund, pp 53–99

    Google Scholar 

  3. Allwood J (2001) Cooperative multimodal communication, vol 2155. Springer, Berlin, Heidelberg

    Google Scholar 

  4. Ambady N, Rosenthal R (1992) Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychol Bull 111(2):256–274

    Google Scholar 

  5. Argyle M (1988) Bodily communication. Routledge, London

    Google Scholar 

  6. Ay N, Flack JC, Krakauer DC (2007) Robustness and complexity co-constructed in multimodal signalling networks. Philos Trans R Soc B 362:441–447

    Google Scholar 

  7. Austin JL (1962) How to do things with words. Oxford University Press, Oxford

    Google Scholar 

  8. Bachorowski J-A, Owren MJ (2001) Not all laughs are alike: voiced but not unvoiced laughter readily elicits positive affect. Psychol Sci 12(3):252–257

    Google Scholar 

  9. Bänziger T, Scherer KR (2007) Using actor portrayals to systematically study multimodal emotion expression: the GEMEP corpus. In: ACII 2007. Lecture notes in computer science, vol 4738. Springer, Berlin, Heidelberg, pp 476–487

    Google Scholar 

  10. Barrett K (1996) Infants’ use of conflicting emotion signals. Cogn Emot 10(2):113–136

    Google Scholar 

  11. Beer CG (1977) What is a display? Am Zool 17(1):155–165

    MathSciNet  Google Scholar 

  12. Berlo DK (1960) The process of communication: an introduction to theory and practice. Holt, Rinehart and Winston, New York

    Google Scholar 

  13. Bliege-Bird RL, Smith EA (2005) Signaling theory, strategic interaction, and symbolic capital. Curr Anthropol 46(2):221–248

    Google Scholar 

  14. Bradbury JW, Vehrencamp SL (1998) Principles of animal communication. Sinauer Associates, Sunderland

    Google Scholar 

  15. Brunswik E (1956) Perception and the representative design of psychological experiments. University of California Press, Berkeley

    Google Scholar 

  16. Bühler K (1934) Sprachtheorie: Die Darstellungsfunktion der Sprache. Fischer, Jena

    Google Scholar 

  17. Burgoon JK, Birk T, Pfau M (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17(1):140–169

    Google Scholar 

  18. Burgoon JK, Le Poire BA (1999) Nonverbal cues and interpersonal judgments: participant and observer perceptions of intimacy, dominance, composure, and formality. Commun Monogr 66(2):105–124

    Google Scholar 

  19. Caryl PG (1979) Communication by agonistic displays: what can games theory contribute to ethology? Behaviour 68(1–2):136–169

    Google Scholar 

  20. Castelfranchi C, Poggi I (1987) Communication beyond the cognitive approach and speech act theory. In: Verschueren J, Bertuccelli P (eds) The pragmatic perspective. John Benjamins, Amsterdam, pp 239–254

    Google Scholar 

  21. Chandler D (2007) Semiotics: the basics, 2nd edn. Routledge, Abingdon

    Google Scholar 

  22. Chartrand TL, Bargh JA (1999) The chameleon effect: the perception-behavior link and social interaction. J Pers Soc Psychol 76(6):893–910

    Google Scholar 

  23. Clark HH (1996) Using language. Cambridge University Press, Cambridge

    Google Scholar 

  24. Cole M (1995) Socio-cultural-historical psychology: some general remarks and a proposal for a new kind of culture-genetic methodology. In: Wertsch JV, Del Rio P, Alvarez A (eds) Sociocultural studies of mind. Harvard University Press, Cambridge, pp 187–214

    Google Scholar 

  25. Conte R, Castelfranchi C (1995) Cognitive and social action. University College Press, London

    Google Scholar 

  26. Dijksterhuis A, Bargh JA (2001) The perception-behavior expressway: automatic effects of social perception on social behavior. Adv Exp Soc Psychol 33:1–40

    Google Scholar 

  27. Dimberg U, Thunberg M, Elmehed K (2000) Unconscious facial reactions to emotional facial expressions. Psychol Sci 11(1):86–89

    Google Scholar 

  28. Duncan S (1972) Some signals and rules for taking speaking turns in conversations. J Pers Soc Psychol 23(2):283–292

    Google Scholar 

  29. Edinger JA, Patterson ML (1983) Nonverbal involvement and social control. Psychol Bull 93(1):30–56

    Google Scholar 

  30. Efron D (1941) Gesture and environment. King’s Crown, New York

    Google Scholar 

  31. Ekman P (1979) About brows: emotional and conversational signals. In: von Cranach M, Foppa K, Lepenies W, Ploog D (eds) Human ethology: claims and limits of a new discipline. Maison des Sciences de l’Homme and Cambridge University Press, Paris and Cambridge, pp 169–202

    Google Scholar 

  32. Ekman P (1992) Are there basic emotions? Psychol Rev 99(3):550–553

    Google Scholar 

  33. Ekman P (1997) Should we call it expression or communication? Innov Soc Sci Res 10(4):333–344

    MathSciNet  Google Scholar 

  34. Ekman P, Friesen WV (1969) The repertoire of nonverbal behavior: categories, origins, usage, and coding. Semiotica 1(1):49–98

    Google Scholar 

  35. Enquist M (1985) Communication during aggressive interactions with particular reference to variation in choice of behaviour. Anim Behav 33(4):1152–1161

    Google Scholar 

  36. Frege FLG (1892) Über sinn und bedeutung. Z Philos Philos Kritik 100:25–50

    Google Scholar 

  37. Ghiglione R, Bonnet C, Richard J-F (1990) Traité de psychologie cognitive. Dunod, Paris

    Google Scholar 

  38. Grice HP (1957) Meaning. Philos Rev 66:377–388

    Google Scholar 

  39. Grice HP (1969) Utterer’s meaning and intention. Philos Rev 78(2):147–177

    Google Scholar 

  40. Grice HP (1989) Studies in the way of words. Harvard University Press, Cambridge

    Google Scholar 

  41. Guilford T, Dawkins MS (1995) What are conventional signals? Anim Behav 49:1689–1695

    Google Scholar 

  42. Hall ET (1959) The silent language. Fawcett, New York

    Google Scholar 

  43. Harré R, Gillett G (1994) The discursive mind. Sage, London

    Google Scholar 

  44. Hasson O (1994) Cheating signals. J Theor Biol 167:223–238

    Google Scholar 

  45. Hauser MD (1996) The evolution of communication. MIT Press, Cambridge

    Google Scholar 

  46. Hausser RR (2001) Foundations of computational linguistics: human-computer communication in natural language. Springer, Berlin

    MATH  Google Scholar 

  47. Hinde R (1975) The concept of function. In: Baerends G, Manning A (eds) Function and evolution in behaviour. Clarendon Press, Oxford, pp 3–15

    Google Scholar 

  48. Hinde RA (1981) Animal signals: ethological and games-theory approaches are not incompatible. Anim Behav 29:535–542

    Google Scholar 

  49. Johnstone RA (1996) Multiple displays in animal communication: ‘Backup signals’ and ‘Multiple messages’. Philos Trans R Soc Lond B, Biol Sci 351(1337):329–338

    Google Scholar 

  50. Keltner D (1995) Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame. J Pers Soc Psychol 68(3):441–454

    Google Scholar 

  51. Krebs JR, Dawkins R (1984) Animal signals: mind-reading and manipulation. In: Krebs JR, Davies NB (eds) Behavioural ecology: an evolutionary approach, vol 2. Blackwell Scientific Publications, Oxford, pp 380–402

    Google Scholar 

  52. Kropp J, Haynes O (1987) Abusive and nonabusive mothers’ ability to identify general and specific emotion signals of infants. Child Dev 58(1):187–190

    Google Scholar 

  53. Maynard Smith J (1974) The theory of games and the evolution of animal conflicts. J Theor Biol 47(1):209–221

    MathSciNet  Google Scholar 

  54. Maynard Smith J, Harper DG (1988) The evolution of aggression: can selection generate variability? Philos Trans R Soc Lond B, Biol Sci 319(1196):557–570

    Google Scholar 

  55. Maynard Smith J, Harper DG (1995) Animal signals: models and terminology. J Theor Biol 177:305–311

    Google Scholar 

  56. Maynard Smith J, Harper DG (2003) Animal signals. Oxford University Press, Oxford

    Google Scholar 

  57. McNeill D (2005) Gesture and thought. University of Chicago Press, Chicago

    Google Scholar 

  58. Mehu M, Scherer KR (2012) A psycho-ethological approach to Social Signal Processing. Cogn Process. doi:10.1007/s10339-012-0435-2. Advance online publication

    Google Scholar 

  59. Mehrabian A (1971) Silent messages. Wadsworth, Belmont

    Google Scholar 

  60. Mininni G (2003) Il discorso come forma di vita. Guida, Napoli

    Google Scholar 

  61. Morris JS, Öhman A, Dolan RJ (1998) Conscious and unconscious emotional learning in the human amygdala. Nature 393(6684):467–470

    Google Scholar 

  62. Naccache L, Dehaene S (2001) Unconscious semantic priming extends to novel unseen stimuli. Cognition 80(3):215–229

    Google Scholar 

  63. Niedenthal PM, Barsalou LW, Winkielman P, Krauth-Gruber S, Ric F (2005) Embodiment in attitudes, social perception, and emotion. Personal Soc Psychol Rev 9(3):184–211

    Google Scholar 

  64. Ogden CK, Richards IA (1923) The meaning of meaning. A study of the influence of language upon thought and of the science of symbolism. Routledge & Kegan, London

    Google Scholar 

  65. Owings DH, Morton ES (1997) The role of information in communication: an assessment/management approach. In: Owings DH, Beecher MD, Thompson NS (eds) Perspectives in ethology: communication, vol 12. Plenum Press, New York, pp 359–390

    Google Scholar 

  66. Owren MJ, Bachorowski J-A (2003) Reconsidering the evolution of nonlinguistic communication: the case of laughter. J Nonverbal Behav 27(3):183–200

    Google Scholar 

  67. Owren MJ, Rendall D, Ryan MJ (2010) Redefining animal signaling: influence versus information in communication. Biol Philos 25(5):755–780

    Google Scholar 

  68. Patterson ML (1982) A sequential functional model of nonverbal exchange. Psychol Rev 89(3):231–249

    Google Scholar 

  69. Parker G (1974) Assessment strategy and the evolution of fighting behaviour. J Theor Biol 47:223–243

    Google Scholar 

  70. Partan SR, Marler P (1999) Communication goes multimodal. Science 283(5406):1272–1273

    Google Scholar 

  71. Peirce CS (1931–1935) Collected chapters. Cambridge University Press, Cambridge

    Google Scholar 

  72. Peirce CS (1909) A sketch of logical critics. In: Peirce Edition Project (ed) The essential Peirce: selected philosophical writings, vol 2. Indiana University Press, Bloomington & Indianapolis

    Google Scholar 

  73. Peirce Edition Project (1998) The essential Peirce: selected philosophical writings (1893–1913), vol 2. Indiana University Press, Bloomington & Indianapolis

    Google Scholar 

  74. Pentland A (2007) Social signal processing. IEEE Signal Process Mag 24(4):108–111

    Google Scholar 

  75. Pentland A (2008) Honest signals: how they shape our world. MIT Press, Cambridge

    Google Scholar 

  76. Poggi I (2007) Mind, hands, face and body. A goal and belief view of multimodal communication. Buchverlag, Weidler

    Google Scholar 

  77. Poggi I, D’Errico F (2010) Cognitive modelling of human social signals. In: Proceedings of SSPW 2010—social signal processing workshop, Firenze, Italy, 29 October 2010. ACM, Sheridan Press, New Jersey

    Google Scholar 

  78. Poggi I, D’Errico F (2012) Social signals. Definition and cognitive processes. Cogn Process (in press)

  79. Rendall D, Owren MJ, Ryan MJ (2009) What do animal signals mean? Anim Behav 78(2):233–240

    Google Scholar 

  80. Rich BA, Fromm SJ, Berghorst LH, Dickstein DP, Brotman MA, Pine DS, Leibenluft E (2008) Neural connectivity in children with bipolar disorder: impairment in the face emotion processing circuit. J Child Psychol Psychiatry 49(1):88–96

    Google Scholar 

  81. Russell JA, Fernandez-Dols J-M (1997) What does a facial expression mean? In: Russell JA, Fernandez-Dols J-M (eds) The psychology of facial expression. Cambridge University Press, Cambridge, pp 3–30

    Google Scholar 

  82. Sacks H, Schegloff EA, Jefferson G (1974) A simplest systematics for the organization of turn taking for conversation. Language 50:696–735

    Google Scholar 

  83. de Saussure F (1916) Cours de Linguistique générale. Payot, Paris

    Google Scholar 

  84. Scharlemann JP, Eckel CC, Kacelnik A, Wilson RK (2001) The value of a smile: game theory with a human face. J Econ Psychol 22:617–640

    Google Scholar 

  85. Scherer KR (1978) Personality inference from voice quality: the loud voice of extroversion. Eur J Soc Psychol 8(4):467–487

    Google Scholar 

  86. Scherer KR (1992) What does facial expression express? In: Strongman KT (ed) International review of studies of emotion, vol 2. Wiley, Chichester, pp 139–165

    Google Scholar 

  87. Scherer KR, Grandjean D (2008) Facial expressions allow inference of both emotions and their components. Cogn Emot 22(5):789–801

    Google Scholar 

  88. Searle JR (1969) Speech acts: an essay in the philosophy of language. Cambridge University Press, Cambridge

    Google Scholar 

  89. Seyfarth RM, Cheney DL (2003) Signalers and receivers in animal communication. Annu Rev Psychol 54:145–173

    Google Scholar 

  90. Shannon CE, Weaver W (1949) The mathematical theory of information. University of Illinois Press, Urbana

    Google Scholar 

  91. Shariff AF, Tracy JL (2011) What are emotion expressions for? Curr Dir Psychol Sci 20:395–399

    Google Scholar 

  92. Silvert L, Delplanque S, Bouwalerh H, Verpoort C, Sequeira H (2004) Autonomic responding to aversive words without conscious valence discrimination. Int J Psychol 53:135–145

    Google Scholar 

  93. Smith WJ (1997) The behavior of communicating, after twenty years. In: Owings DH, Beecher MD, Thompson NS (eds) Perspectives in ethology: communication, vol 12. Plenum Press, New York, pp 7–54

    Google Scholar 

  94. Tomasello M (2008) The origins of human communication. MIT Press, Cambridge

    Google Scholar 

  95. Tomkins SS (1962) Consciousness, imagery and affect, vol 1. Springer, New York

    Google Scholar 

  96. Vehrencamp SL (2000) Handicap, index, and conventional signal elements of bird song. In: Edpmark Y, Amundsen T, Rosenqvist GG (eds) Animal signals: signalling and signal design in animal communication. Tapir Academic Press, Trondheim, pp 277–300

    Google Scholar 

  97. Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comp J 27(12):1743–1759

    Google Scholar 

  98. Vygotsky LS (1986) Thought and language. MIT Press, Cambridge (newly revised and edited by A Kozulin)

    Google Scholar 

  99. White S (1989) Backchannels across cultures: A study of Americans and Japanese. Lang Soc 18(1):59–76

    Google Scholar 

  100. Wilczynski W, Ryan MJ (2010) The behavioral neuroscience of anuran social signal processing. Curr Opin Neurobiol 20(6):754–763

    Google Scholar 

  101. Woodworth RS (1961) Dynamics of behavior. Holt, New York

    Google Scholar 

  102. Yik MSM, Russell JA (1999) Interpretation of faces: a cross-cultural study of a prediction from Fridlund’s theory. Cogn Emot 13(1):93–104

    Google Scholar 

  103. Zahavi A (1975) Mate selection: selection for a handicap. J Theor Biol 53:205–214

    Google Scholar 

Download references

Acknowledgements

Work on this article was supported by the European Network of Excellence SSPNet (grant agreement No. 231287). The authors would like to thank Paul Brunet, Roderick Cowie, Hastings Donnan, Ellen Douglas-Cowie, Gary McKeown, Isabella Poggi, Marc Schröder, and Alessandro Vinciarelli for fruitful discussions during the meeting on conceptual issues in SSP held at Queen’s University Belfast, July 2010.

Author information

Authors and Affiliations

  1. Swiss Center for Affective Sciences, University of Geneva, Rue des Battoirs, 7, 1205, Geneva, Switzerland

    Marc Mehu

  2. Francesca D’Errico, Department of Education, University Roma Tre, Via Ostiense, 161, 00154, Rome, Italy

    Francesca D’Errico

  3. Department of Computing Science, Human Media Interaction, University of Twente, P.O. Box 217, 7500, AE Enschede, The Netherlands

    Dirk Heylen

Authors
  1. Marc Mehu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francesca D’Errico
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dirk Heylen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marc Mehu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mehu, M., D’Errico, F. & Heylen, D. Conceptual analysis of social signals: the importance of clarifying terminology. J Multimodal User Interfaces 6, 179–189 (2012). https://doi.org/10.1007/s12193-012-0091-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-012-0091-y

Keywords

Search

Navigation