The LAST MINUTE Corpus as a Research Resource: From Signal Processing to Behavioral Analyses in User-Companion Interactions

  • Dietmar Rösner
  • Jörg Frommer
  • Andreas Wendemuth
  • Thomas Bauer
  • Stephan Günther
  • Matthias Haase
  • Ingo Siegert
Chapter
Part of the Cognitive Technologies book series (COGTECH)

Abstract

The LAST MINUTE Corpus (LMC) is one of the rare examples of a corpus with naturalistic human-computer interactions. It offers richly annotated data from N total = 130 experiments in a number of modalities. In this paper we present results from various investigations with data from the LMC using several primary modalities, e.g. transcripts, audio, questionnaire data.

We showed that sociodemographics (age, gender) have an influence on the global dialog success. Furthermore, distinct behavior during the initial phase of the experiment can be used to predict global dialog success during problem solving. Also, the influence of interventions on the dialog course was evaluated.

Additionally, the importance of discourse particles as prosodic markers could be shown. Especially during critical dialog situations, the use of these markers is increasing. These markers are furthermore influenced by user characteristics.

Thus, to enable future Companion-Systems to react appropriately to the user, these systems have to observe and monitor acoustic and dialogic markers and have to take into account the user’s characteristics, such as age, gender and personality traits.

Notes

Acknowledgements

This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).

References

  1. 1.
    Allwood, J., Nivre, J., Ahlsén, E.: On the semantics and pragmatics of linguistic feedback. J. Semant. 9, 1–26 (1992)CrossRefGoogle Scholar
  2. 2.
    Baayen, R.: Analyzing Linguistic Data – A Practical Introduction to Statistics Using R. Cambridge University Press, Cambridge (2008)CrossRefGoogle Scholar
  3. 3.
    Bakhtin, M.M., Holquist, M., McGee, V., Emerson, C.: Speech Genres and other Late Essays, vol. 8. University of Texas Press, Austin (1986)Google Scholar
  4. 4.
    Bickmore, T., Gruber, A., Picard, R.: Establishing the computer–patient working alliance in automated health behavior change interventions. Patient Educ. Couns. 59(1), 21–30 (2005)CrossRefGoogle Scholar
  5. 5.
    Corley, M., Stewart, O.W.: Hesitation Disfluencies in Spontaneous Speech: the Meaning of um. Lang. Ling. Compass. 2, 589–602 (2008)Google Scholar
  6. 6.
    Costa, P.T., McCrae, R.R.: Domains and facets: hierarchical personality assessment using the revised NEO personality inventory. J. Pers. Assess. 64, 21–50 (1995)CrossRefGoogle Scholar
  7. 7.
    Dalton, P., Fraenkel, N.: Gorillas we have missed: sustained inattentional deafness for dynamic events. Cognition 124(3), 367–372 (2012)CrossRefGoogle Scholar
  8. 8.
    Frommer, J., Rösner, D., Haase, M., Lange, J., Friesen, R., Otto, M.: Project A3 - Detection and Avoidance of Failures in Dialogs. Pabst Science Publisher, Lengerich (2012)Google Scholar
  9. 9.
    Funke, J.: Complex problem solving: a case for complex cognition? Cogn. Process. 11(2), 133–142 (2010)CrossRefGoogle Scholar
  10. 10.
    Georgila, K., Wolters, M., Moore, J., Logie, R.: The MATCH corpus: a corpus of older and younger users’ interactions with spoken dialogue systems. Lang. Resour. Eval. 44(3), 221–261 (2010)CrossRefGoogle Scholar
  11. 11.
    Grawe, K.: Grundriß einer Allgemeinen Psychotherapie. Psychotherapeut 40, 130–145 (1995)Google Scholar
  12. 12.
    Gries, S.T.: Quantitative Corpus Linguistics with R: A Practical Introduction. Routledge, Abingdon (2009)CrossRefGoogle Scholar
  13. 13.
    Grimm, M., Kroschel, K., Narayanan, S.: The Vera am Mittag German audio-visual emotional speech database. In: Proceedings of the 2008 IEEE ICME, pp. 865–868 (2008)Google Scholar
  14. 14.
    Halliday, M.A.K: Language as a Social Semiotic: The Social Interpretation of Language and Meaning. Hodder & Stoughton Educational, London (1976)Google Scholar
  15. 15.
    Hattie, J.: Visible Learning. A Bradford Book. Routledge, London (2009)Google Scholar
  16. 16.
    Hone, K.: Empathic agents to reduce user frustration: the effects of varying agent characteristics. Interacting Comput. 18(2), 227–245 (2006)CrossRefGoogle Scholar
  17. 17.
    Horowitz, L.M., Alden, L.E., Wiggins, J.S., Pincus, A.L.: Inventory of interpersonal problems manual. Psychological Cooperation, San Antonio (2000)Google Scholar
  18. 18.
    Horowitz, L.M., Strauß, B., Kordy, H.: Inventar zur Erfassung interpersonaler Probleme (IIPD), 2nd edn. Beltz, Weinheim (2000)Google Scholar
  19. 19.
    Jahnke, W., Erdmann, G., Kallus, K.: Stressverarbeitungsfragebogen mit SVF 120 und SVF 78, 3rd edn. Hogrefe, Göttingen (2002)Google Scholar
  20. 20.
    Kehrein, R., Rabanus, S.: Ein Modell zur funktionalen Beschreibung von Diskurspartikeln. In: Neue Wege der Intonationsforschung, Germanistische Linguistik, vol. 157–158, pp. 33–50. Georg Olms, Hildesheim (2001)Google Scholar
  21. 21.
    Klein, J., Moon, Y., Picard, R.W.: This computer responds to user frustration: theory, design and results. Interacting Comput. 14, 119–140 (2002)CrossRefGoogle Scholar
  22. 22.
    Kruskal, W., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47, 583–621 (1952)CrossRefMATHGoogle Scholar
  23. 23.
    Ladd, R.D.: Intonational Phonology. In: Studies in Linguistics, vol. 79. Cambridge University Press, Cambridge (1996)Google Scholar
  24. 24.
    Legát, M., Grůber, M., Ircing, P.: Wizard of oz data collection for the czech senior companion dialogue system. In: Fourth International Workshop on Human-Computer Conversation, pp. 1–4. University of Sheffield (2008)Google Scholar
  25. 25.
    Lotz, A.F., Siegert, I., Wendemuth, A.: Automatic differentiation of form-function-relations of the discourse particle “hm” in a naturalistic human-computer interaction. In: Proceedings of the 26th ESSV, Eichstätt (2015)Google Scholar
  26. 26.
    Nass, C., Moon, Y.: Machines and Mindlessness: Social Responses to Computers. J. Soc. Issues 56(1), 81–103 (2000)CrossRefGoogle Scholar
  27. 27.
    Paschen, H.: Die Funktion der Diskurspartikel HM. Master’s thesis, University Mainz (1995)Google Scholar
  28. 28.
    Prylipko, D., Egorow, O., Siegert, I., Wendemuth, A.: Application of image processing methods to filled pauses detection. In: Proceedings of INTERSPEECH’14, Singapore (2014)Google Scholar
  29. 29.
    Prylipko, D., Rösner, D., Siegert, I., Günther, S., Friesen, R., Haase, M., Vlasenko, B., Wendemuth, A.: Analysis of significant dialog events in realistic human-computer interaction. J. Multimodal User Interfaces 8(1), 75–86 (2014)CrossRefGoogle Scholar
  30. 30.
    R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2010). http://www.R-project.org
  31. 31.
    Rösner, D., Frommer, J., Friesen, R., Haase, M., Lange, J., Otto, M.: LAST MINUTE: a multimodal corpus of speech-based user-companion interactions. In: Proceedings of the 8th LREC, Istanbul, pp. 96–103 (2012)Google Scholar
  32. 32.
    Rösner, D., Friesen, R., Günther, S., Andrich, R.: Modeling and evaluating dialog success in the LAST MINUTE corpus. In: Proceedings of the 9th LREC, Reykjavik (2014)Google Scholar
  33. 33.
    Rösner, D., Andrich, R., Bauer, T., Friesen, R., Günther, S.: Annotation and Analysis of the LAST MINUTE corpus. In: Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology, pp. 112–121 (2015)Google Scholar
  34. 34.
    Rösner, D., Haase, M., Bauer, T., Günther, S., Krüger, J., Frommer, J.: Desiderata for the Design of Companion Systems – Insights from a Large Scale Wizard of Oz Experiment. Künstliche Intelligenz 30(1), 53–61 (2016). Online first: Oct 28 (2015). doi:10.1007/s13218-015-0410-zGoogle Scholar
  35. 35.
    Schmidt, J.E.: Bausteine der Intonation. In: Neue Wege der Intonationsforschung, Germanistische Linguistik, vol. 157–158, pp. 9–32. Georg Olms, Hildesheim, Germany (2001)Google Scholar
  36. 36.
    Schmidt, T., Schütte, W.: Folker: An annotation tool for efficient transcription of natural, multi-party interaction. In: Proceedings of the 7th LREC, Valletta (2010)Google Scholar
  37. 37.
    Siegert, I.: Emotional and user-specific cues for improved analysis of naturalistic interactions. Ph.D. thesis, Otto von Guericke University Magdeburg (2015)Google Scholar
  38. 38.
    Siegert, I., Böck, R., Philippou-Hübner, D., Vlasenko, B., Wendemuth, A.: Appropriate Emotional Labeling of Non-acted Speech Using Basic Emotions, Geneva Emotion Wheel and Self Assessment Manikins. In: Proceedings of the 2011 IEEE ICME, Barcelona (2011)Google Scholar
  39. 39.
    Siegert, I., Böck, R., Wendemuth, A.: The influence of context knowledge for multimodal annotation on natural material. In: Joint Proceedings of the IVA 2012 Workshops, pp. 25–32. Santa Cruz (2012)Google Scholar
  40. 40.
    Siegert, I., Hartmann, K., Philippou-Hübner, D., Wendemuth, A.: Human behaviour in HCI: complex emotion detection through sparse speech features. In: Salah, A., Hung, H., Aran, O., Gunes, H. (eds.) Human Behavior Understanding. Lecture Notes on Computer Science, vol. 8212, pp. 246–257. Springer, Berlin (2013)CrossRefGoogle Scholar
  41. 41.
    Siegert, I., Böck, R., Wendemuth, A.: Inter-Rater Reliability for Emotion Annotation in Human-Computer Interaction – Comparison and Methodological Improvements. J. Multimodal User Interfaces 8, 17–28 (2014)CrossRefGoogle Scholar
  42. 42.
    Siegert, I., Haase, M., Prylipko, D., Wendemuth, A.: Discourse particles and user characteristics in naturalistic human-computer interaction. In: Kurosu, M. (ed.) Human-Computer Interaction. Advanced Interaction Modalities and Techniques. Lecture Notes on Computer Science, vol. 8511, pp. 492–501. Springer, Berlin (2014)Google Scholar
  43. 43.
    Webb, N., Benyon, D., Bradley, J., Hansen, P., Mival, O.: Wizard of Oz experiments for a companion dialogue system: eliciting companionable conversation. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Paris (2010)Google Scholar
  44. 44.
    Wolters, M., Georgila, K., Moore, J., MacPherson, S.: Being old doesn’t mean acting old: how older users interact with spoken dialog systems. ACM Trans. Access. Comput. 2(1), 2:1–2:39 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Dietmar Rösner
    • 1
    • 2
  • Jörg Frommer
    • 3
  • Andreas Wendemuth
    • 4
    • 2
  • Thomas Bauer
    • 1
  • Stephan Günther
    • 1
  • Matthias Haase
    • 3
  • Ingo Siegert
    • 4
  1. 1.Institut für Wissens- und Sprachverarbeitung (IWS)Otto-von-Guericke UniversitätMagdeburgGermany
  2. 2.Center for Behavioral Brain SciencesMagdeburgGermany
  3. 3.Universitätsklinik für Psychosomatische Medizin und PsychotherapieOtto-von-Guericke Universität39120 MagdeburgGermany
  4. 4.Institut für Informations- und Kommunikationstechnik (IIKT)Otto-von-Guericke UniversitätMagdeburgGermany

Personalised recommendations