Advertisement

Language Resources and Evaluation

, Volume 40, Issue 1, pp 5–23 | Cite as

A corpus for studying addressing behaviour in multi-party dialogues

  • Natasa Jovanovic
  • Rieks op den Akker
  • Anton Nijholt
Original paper

Abstract

This paper describes a multi-modal corpus of hand-annotated meeting dialogues that was designed for studying addressing behaviour in face-to-face conversations. The corpus contains annotated dialogue acts, addressees, adjacency pairs and gaze direction. First, we describe the corpus design where we present the meetings collection, annotation scheme and annotation tools. Then, we present the analysis of the reproducibility and stability of the annotation scheme.

Keywords

Addressing Multi-party dialogues Multimodal corpora Annotation schemas Reliability analysis 

Notes

Acknowledgements

This work was partly supported by the European Union 6th FWP IST Integrated Project AMI (Augmented Multi-party Interaction, FP6-506811, publication AMI-160). We would like to thank Dennis Reidsma, Dennis Hofs, Lynn Packwood and the annotators who were involved in the corpus development. We are grateful to Klaus Krippendorff for useful discussions about reliability metrics.

References

  1. Bakx, I., van Turnhout, K., & Terken, J. (2003). Facial orientation during multi-party interaction with information Kiosks. In Proceedings of the 9th IFIP TC13 International Conference on Human–Computer Interaction (INTERACT), (pp. 701–704). Zurich, SwitzerlandGoogle Scholar
  2. Burger, S., & Sloane, Z. (2004). The ISL meeting corpus: Categorical features of communicative group interactions. In Proceedings of the NIST ICASSP Meeting Recognition Workshop. Montreal, CanadaGoogle Scholar
  3. Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249–254Google Scholar
  4. Carletta, J., Anderson, A. H., & Garrod, S. (2002). Seeing eye to eye: An account of grounding and understanding in work groups. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 9(1), 1–20Google Scholar
  5. Carletta, J., Ashby, S., Bourban, S. M., Flynn, M. G., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., & Wellner, P. (2006). The AMI meeting corpus: A pre-announcement. In S. Renals & S. Bengio (Eds.), Machine learning for multimodal interaction, Second International Workshop, MLMI 2005, Edinburgh, UK, July, 2005, revised selected papers, vol 3869 of Lecture Notes in Computer Science (pp. 28–39). Springer-Verlag. ISBN 3-540-32549-2Google Scholar
  6. Carletta, J., Evert, S., Heid, U., Kilgour, J., Robertson, J., & Voormann, H. (2003). The NITE XML toolkit: Flexible annotation for multi-modal language data. Behavior Research Methods, Instruments, and Computers, 35(3), 353–363Google Scholar
  7. Carletta, J., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G., & Anderson, A. (1997). The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1), 13–31Google Scholar
  8. Chen, L., Rose, R. T., Parrill, F., Han, X., Tu, J., Huang, Z., Harper, M., Quek, F., McNeill, D., Tuttle, R., & Huang, T. (2006). VACE multimodal meeting corpus. In S. Renals & S. Bengio (Eds.), Machine learning for multimodal interaction, Second International Workshop, MLMI 2005, Edinburgh, UK, July, 2005, revised selected papers, vol 3869 of Lecture Notes in Computer Science (pp. 40–51). Springer-Verlag. ISBN 3-540-32549-2Google Scholar
  9. Clark, H. H. & Carlson, T. B. (1992). Hearers and speech acts. In H. H. Clark (Ed.), Arenas of Language Use (pp. 205–247). University of Chicago Press and CSLIGoogle Scholar
  10. Clark, A., & Popescu-Belis, A. (2004). Multi-level dialogue act tags. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 163–170). Cambridge, MAGoogle Scholar
  11. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46Google Scholar
  12. Dhillon, R., Bhagat, S., Carvey, H., & Shriberg, E. (2004). Meeting recorder project: Dialogue act labeling guide. Technical report TR-04-002, International Computer Science Institute (ICSI), Berkeley, CA, USAGoogle Scholar
  13. Garofolo, J. S., Laprun, C. D., Michel, M., Stanford, V. M., & Tabassi, E. (2004) The NIST meeting room pilot corpus. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC) (pp. 1411–1414). Lisbon, PortugalGoogle Scholar
  14. Goffman, E. (1981). Footing. In Goffman, E., (Ed.), Forms of talk (pp. 124–159). Philadelphia, PA: University of Pennsylvania PressGoogle Scholar
  15. Goodwin, C. (1981). Conversational organization: Interaction between speakers and hearers. New York: Academic PressGoogle Scholar
  16. Hirschberg, J., & Nakatani, C. (1996). A prosodic analysis of discourse segments in direction-giving monologues. In In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics (ACL) (pp 286–293). Santa Cruz, CaliforniaGoogle Scholar
  17. Janin, A., Ang, J., Bhagat, S., Dhillon, R., Edwards, J., Macias-Guarasa, J., Morgan, N., Peskin, B., Shriberg, E., Stolcke, A., Wooters, C., & Wrede, B. (2004). The ICSI meeting project: Resources and research. In Proceedings of the NIST ICASSP Meeting Recognition Workshop. Montreal, CanadaGoogle Scholar
  18. Jovanovic, N., & op den Akker, R. (2004). Towards automatic addressee identification in multi-party dialogues. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 89–92). Cambridge, MAGoogle Scholar
  19. Jovanovic, N., op den Akker, R., & Nijholt, A. (2006). Addressee identification in face-to-face meetings. In Proceeding of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 169–176). Trento, ItalyGoogle Scholar
  20. Jurafsky, D., Shriberg, L., & Biasca, D. (1997). Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual, Draft 13. Technical report 97–02, University of Colorado, The Institute of Cognitive Science, Boulder, COGoogle Scholar
  21. Katzenmaier, M., Stiefelhagen, R., & Schultz, T. (2004). Identifying the addressee in human–human–robot interactions based on head pose and speech. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 144–151). State College, PAGoogle Scholar
  22. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Sage Publications, Beverly Hills, CAGoogle Scholar
  23. Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433Google Scholar
  24. Manning, C., & Schutze, H. (1999). Foundations of statistical natural language processing. The MIT Press, Cambridge, MAGoogle Scholar
  25. Moore, D. (2002). The IDIAP smart meeting room. Technical report IDIAP-COM-07, IDIAP, Martigny, SwitzerlandGoogle Scholar
  26. Otsuka, K., Takemae, Y., Yamato, J., & Murase, H. (2005). A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 191–198). Trento, ItalyGoogle Scholar
  27. Passonneau, R. (2004). Computing reliability for coreference annotation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC) (pp. 1503–1506). Lisbon, PortugalGoogle Scholar
  28. Passonneau, R. & Litman, D. (1997). Discourse segmentation by human and automated means. Computational Linguistics, 23(1), 103–139Google Scholar
  29. Reidsma, D., Hofs, D., & Jovanovic, N. (2005). A presentation of a set of new annotation tools based on the NXT API. In Proceedings of Measuring Behavior (pp. 512–513). Wageningen, The NetherlandsGoogle Scholar
  30. Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., & Carvey, H. (2004). The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 97–100). Boston, USAGoogle Scholar
  31. Traum, D. (2004). Issues in multi-party dialogues. In F. Dignum (Ed.), Advances in agent communication (pp. 201–211). Springer-Verlag LNCSGoogle Scholar
  32. van Turnhout, K., Terken, J., Bakx, I., & Eggen, B. (2005). Identifying the intended addressee in mixed human–human and human–computer interaction from non-verbal features. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 175–182). Trento, ItalyGoogle Scholar

Copyright information

© Springer Science+Business Media 2006

Authors and Affiliations

  • Natasa Jovanovic
    • 1
  • Rieks op den Akker
    • 1
  • Anton Nijholt
    • 1
  1. 1.Human Media Interaction GroupUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations