Skip to main content
SpringerLink
Log in

Issues in Data Labelling

  • Chapter
  • First Online:
Emotion-Oriented Systems

Part of the book series: Cognitive Technologies ((COGTECH))

Abstract

Labelling emotion databases is not a purely technical matter. It is bound up with theoretical issues. Different issues affect labelling of emotional content, labelling of the signs that convey emotion, and labelling of the relevant context. Linked to these are representational issues, involving time course, consensus and divergence, and connections between states and events. From that background comes a wealth of resources for labelling emotion, involving not only everyday emotion words but also affect dimensions, and labels for combination types, appraisal categories, and authenticity. Resources for labelling signs of emotion cover linguistic, vocal, face descriptors, plus descriptors for gesture, and relevant physiological variables. Resources for labelling context are developing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Auberge V, Audibert N, Rillard A (2006) Auto-annotation: an alternative method to label expressive corpora. In: Proceedings LREC 2006, Genoa, pp 45–46

    Google Scholar 

  • Bachorowski JA (1999) Vocal expression and perception of emotion. Curr Dir Psychol Sci 8(2):53–57

    Article  Google Scholar 

  • Bänziger T, Tran V, Scherer KR (2005) The Geneva Emotion Wheel: a tool for the verbal report of emotional reactions. In: Poster presented at ISRE 2005, Bari

    Google Scholar 

  • Baron-Cohen S (2007) Mind reading: the interactive guide to emotions – version 1.3. Jessica Kingsley, London

    Google Scholar 

  • Batliner A, Steidl S, Schuller B, Seppi D, Laskowski K, Vogt T, Devillers L, Vidrascu L, Amir N, Kessous L, Aharonson V (2006) Combining efforts for improving automatic classification of emotional user states. In: Erjavec T, Gros J (eds) Language technologies, IS-LTC 2006. Infornacijska Druzba (Information Society), Ljubljana, Slovenia, pp 240–245

    Google Scholar 

  • Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59

    Article  Google Scholar 

  • Campbell N (2004) Extra-semantic protocols: input requirements for the synthesis of dialogue speech. In: Andre E, Dybkjaer L, Minker W, Heisterkamp P (eds) Affective dialogue systems. Lecture notes in artificial intelligence. Springer, Berlin, pp 221–228

    Chapter  Google Scholar 

  • Cohen JA (1960). A coefficient of agreement for nominal scales. Educ Psychol Meas, 20(1):37–46

    Article  Google Scholar 

  • Cowie R, Cornelius R (2003) Describing the emotional states that are expressed in speech. Speech Commun 40:5–32

    Article  MATH  Google Scholar 

  • Cowie R, Douglas-Cowie E, Apolloni B, Taylor J, Romano A, Fellenz W (1999) What a neural net needs to know about emotion words. In: Mastorakis N (ed) Computational intelligence and applications. World Scientific Engineering Society, Dallas, TX, pp 109–114

    Google Scholar 

  • Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schroeder M (2000) ‘FEELTRACE’: an instrument for recording perceived emotion in real time. In: Proceedings of the ISCA ITRW on speech and emotion: developing a conceptual framework, Newcastle, 5–7 Sept 2000, Textflow, Belfast, pp 19–24.

    Google Scholar 

  • Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor J (2001) Emotion recognition in human–computer interaction. IEEE Signal Process Mag 18(1):32–80

    Article  Google Scholar 

  • Craggs R, Wood M (2003) Annotating emotion in dialogue. In: Proceedings of the 4th SIGdial workshop on discourse and dialogue, Sapporo

    Google Scholar 

  • Craggs R, Wood M (2004) A 2 dimensional annotation scheme for emotion in dialogue. In: AAAI spring symposium on exploring attitude and affect in text: theories and applications. Stanford University, AAAI Press, pp 44–49

    Google Scholar 

  • Devillers L, Abrilian S, Martin J-C (2005) Representing real life emotions in audiovisual data with non basic emotional patterns and context features. In: Proceedings of the 1st international conference on affective computing and intelligent interaction (ACII’2005), Beijing, 22–24 Oct. Spinger, Berlin, pp 519–526

    Google Scholar 

  • Devillers L, Cowie R, Martin J-C, Douglas-Cowie E, Abrilian S, McRorie M (2006) Real life emotions in French and English TV video clips: an integrated annotation protocol combining continuous and discrete approaches. In: Proceedings LREC 2006, Genoa

    Google Scholar 

  • Devillers L, Martin J-C (2008) Coding emotional events in audiovisual corpora. In 6th international conference on language resources and evaluation (LREC 2008). Marrakech (Morocco)

    Google Scholar 

  • Douglas-Cowie E, Campbell N, Cowie R, Roach P (2003) Emotional speech: towards a new generation of databases. Speech Commun 40(1–2):33–60

    Article  MATH  Google Scholar 

  • Douglas-Cowie E, Devillers L, Martin J-C, Cowie R, Savvidou R, Abrilian S, Cox C (2005) Multimodal databases of everyday emotion: facing up to complexity. In: Proceedings of the Interspeech 2005, Lisbon, pp 813–816

    Google Scholar 

  • Ekman P (1992) An argument for basic emotions. Cogn Emot 6:169–200

    Article  Google Scholar 

  • Ekman P (1999) Basic emotions. In: Dalgleish T, Power MJ (eds) Handbook of cognition and emotion. Wiley, New York, pp 301–320

    Google Scholar 

  • Ekman P, Friesen W (1975) Unmasking the face: a guide to recognizing emotions from facial clues. Prentice-Hall, Englewood Cliffs, NJ

    Google Scholar 

  • Ekman P, Friesen WC, Hager JC (2002). Facial action coding system. The manual on CD ROM. Research Nexus Division of Network Information Research Corporation, Salt Lake City, UT

    Google Scholar 

  • El Kaliouby R, Robinson P (2004) Mind reading machines: automated inference of cognitive mental states from video. In: IEEE international conference on systems, man and cybernetics, vol 1, The Hague, pp 682–688

    Google Scholar 

  • Fragopanagos N, Taylor J (2005) Emotion recognition in human–computer interaction. Neural Netw 18:389–405

    Article  Google Scholar 

  • Hall JA, Matsumoto D (2004) Gender differences in judgments of multiple emotions from facial expressions. Emotion 4(2):201–206

    Article  Google Scholar 

  • Hanson NR (1958) Patterns of discovery: an inquiry into the conceptual foundations of science. Cambridge University Press, Cambridge, MA

    Google Scholar 

  • Hartmann B, Mancini M, Pelachaud C (2005) Implementing expressive gesture synthesis for embodied conversational agents. In: Gesture Workshop (GW’2005), Vannes, France, pp 1095–1096

    Google Scholar 

  • Ioannou S, Raouzaiou A, Tzouvaras V, Mailis T, Karpouzis K, Kollias S (2005) Emotion recognition through facial expression analysis based on a neurofuzzy network Neural Netw, 18, 423–435

    Google Scholar 

  • James W (1884) What is emotion? Mind 9:188–205

    Article  Google Scholar 

  • Kipp M (2001) Anvil – a generic annotation tool for multimodal dialogue. In: 7th European conference on speech communication and technology (Eurospeech’2001), Aalborg, Denmark, 3–7 Sept, pp 1367–1370

    Google Scholar 

  • Kipp M (2004). Gesture generation by imitation. From human behavior to computer character animation, Florida, Boca Raton, Dissertation.com. 1581122551

    Google Scholar 

  • Kita S, van Gijn I, van der Hulst H (1998). Movement phases in signs and co-speech gestures, and their transcription by human coders. Gesture and sign language in human computer interaction: Proceedings/international gesture workshop, Bielefeld, 17–19 Sept. Springer, Berlin, Heidelberg

    Google Scholar 

  • Ladd S, Silverman K, Bergmann G, Scherer K (1985) Evidence for independent function of intonation contour type, voice quality, and F0 in signalling speaker affect. J Acoust Soc Am 78(2):435–444

    Article  Google Scholar 

  • Martin J-C, Abrilian S, Devillers L (2005). Annotating multimodal behaviors occurring during non basic emotions. In: 1st international conference on affective computing and intelligent interaction (ACII’2005), Beijing, 22–24 Oct. Spinger, Berlin, pp 550–557

    Google Scholar 

  • Martin JC, Devillers L, Zara A, Maffiolo V, LeChenadec G (2006) The EmoTABOU corpus. Humaine Summer School, Genova, pp 22–28

    Google Scholar 

  • Matsumoto D (2001) Culture and emotion. In: Matsumoto D (ed) Handbook of culture and psychology. Oxford University Press, New York, NY, pp 171–194

    Google Scholar 

  • Mayer JD, Salovey P, Caruso D (2000) Models of emotional intelligence. In: Sternberg R (ed) Handbook of intelligence. Cambridge University Press, Cambridge, MA, pp 396–421

    Google Scholar 

  • McNeill D (1992) Hand and mind – what gestures reveal about thoughts. University of Chicago Press, Chicago, IL

    Google Scholar 

  • McNeill D (2005) Gesture and thought. University of Chicago Press, Chicago, IL

    Google Scholar 

  • Mozziconacci SJL (1998). Speech variability and emotion: production and perception. Ph.D. thesis, Eindhoven

    Google Scholar 

  • Ortony A, Clore G, Collins A (1988) The cognitive structure of emotions. Cambridge University Press, Cambridge, MA

    Google Scholar 

  • Ortony A, Turner TJ (1990) What’s basic about basic emotions? Psychol Rev 97:315–331

    Article  Google Scholar 

  • Rosenberg A, Binkowski E (2005). Augmenting the kappa statistic to determine interannotator reliability for multiple labelled data points. In: Proceeding of the HLT-NAACL, Boston

    Google Scholar 

  • Russell J, Barrett-Feldman L (1999) Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J Pers Soc Psychol 5:37–63

    Google Scholar 

  • Sander D, Grandjean D, Scherer KR (2005) A systems approach to appraisal mechanisms in emotion. Neural Netw 18:317–352

    Article  Google Scholar 

  • Savvidou S (submitted) Validation of the Feeltrace tool for recording impressions of expressed emotion. Thesis submitted for Ph.D., Queen’s University, Belfast.

    Google Scholar 

  • Scherer KR (ed) (1988a) Facets of emotion: recent research. Erlbaum, Hillsdale, NJ. Appendix F. Labels describing affective states in five major languages, revised by the members of the Geneva Emotion Research Group retrieved Dec 2007 from http://www.unige.ch/cisa/gerg/research.html

  • Scherer KR (ed) (1988b) Appendix B. Antecedent and reaction codes used in the “Emotion in Social Interaction.” In: Scherer KR (ed) Facets of emotion: recent research. Erlbaum, Hillsdale, NJ, pp 241–243. http://www.unige.ch/cisa/gerg/research.html

  • Schreuder M, van Eerten L, Gilbers D (2006) Music as a method of identifying emotional speech. In: LREC Research Workshop on Corpora on Emotion and Affect, Genoa

    Google Scholar 

  • Steidl S, Levit M, Batliner A, Nöth E, Niemann H (2005) “Of All Things the Measure is Man” – automatic classification of emotions and inter-labeller consistency. In: Proceedings of the ICASSP 2005, Philadelphia, pp 317–320

    Google Scholar 

  • Tabachnik BG, Fidell LS (2001) Using multivariate statistics, 4th edn. Allyn & Bacon, Boston, MA

    Google Scholar 

  • Wallbott HG (1998) Bodily expression of emotion. Eur J Soc Psychol 28:879–896

    Article  Google Scholar 

  • Whissell C (1989) The dictionary of affect in language. In: Plutchik R, Kellerman H (eds) Emotion: theory, research and experience: vol 4, The measurement of emotions. Academic, New York, NY, pp 113–131

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Queen’s University Belfast, Belfast, Northern Ireland, UK

    Roddy Cowie & Cate Cox

  2. Computer Sciences Laboratory for Mechanics and Engineering Sciences (LIMSI), Paris, France

    Jean-Claude Martin

  3. Lehrstuhl für Mustererkennung, Friedrich-Alexander-Universität Erlangen, Erlangen, Germany

    Anton Batliner

  4. Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands

    Dirk Heylen

  5. Image, Video and Multimedia Systems Lab, Institute of Communications and Computer Systems, National Technical University of Athens, Athens, Greece

    Kostas Karpouzis

Authors
  1. Roddy Cowie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cate Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Claude Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anton Batliner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dirk Heylen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kostas Karpouzis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roddy Cowie .

Editor information

Editors and Affiliations

  1. , Department of Psychology, Queen's University Belfast, Belfast, BT7 1NN, United Kingdom

    Roddy Cowie

  2. TELECOM ParisTech, CNRS - LTCI, rue Dareau 37, Paris, 75014, France

    Catherine Pelachaud

  3. for Artificial Intelligence (OFAI), Austrian Research Institute, Freyung 6/6, Vienna, 1010, Austria

    Paolo Petta

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Cowie, R., Cox, C., Martin, JC., Batliner, A., Heylen, D., Karpouzis, K. (2011). Issues in Data Labelling. In: Cowie, R., Pelachaud, C., Petta, P. (eds) Emotion-Oriented Systems. Cognitive Technologies. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15184-2_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15184-2_13

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15183-5

  • Online ISBN: 978-3-642-15184-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation