Skip to main content
SpringerLink
Log in

A corpus for studying addressing behaviour in multi-party dialogues

  • Original paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

This paper describes a multi-modal corpus of hand-annotated meeting dialogues that was designed for studying addressing behaviour in face-to-face conversations. The corpus contains annotated dialogue acts, addressees, adjacency pairs and gaze direction. First, we describe the corpus design where we present the meetings collection, annotation scheme and annotation tools. Then, we present the analysis of the reproducibility and stability of the annotation scheme.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. The M4 (MultiModal Meeting Manager) project: http://www.m4project.org

  2. The AMI (Augmented Multi-party Interaction) project: http://www.amiproject.org

  3. MMM File Server http: //www.mmm.idiap.ch

  4. http://www.mmm.idiap.ch/M4-Corpus/annotations/NXTbasedAnnotation/

  5. NXT Query Language: http://www.ims.uni-stuttgart.de/projekte/nite/

  6. The defined similarity measure is known as Dice coefficient (Manning & Schutze, 1999).

References

  • Bakx, I., van Turnhout, K., & Terken, J. (2003). Facial orientation during multi-party interaction with information Kiosks. In Proceedings of the 9th IFIP TC13 International Conference on Human–Computer Interaction (INTERACT), (pp. 701–704). Zurich, Switzerland

  • Burger, S., & Sloane, Z. (2004). The ISL meeting corpus: Categorical features of communicative group interactions. In Proceedings of the NIST ICASSP Meeting Recognition Workshop. Montreal, Canada

  • Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249–254

    Google Scholar 

  • Carletta, J., Anderson, A. H., & Garrod, S. (2002). Seeing eye to eye: An account of grounding and understanding in work groups. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 9(1), 1–20

    Google Scholar 

  • Carletta, J., Ashby, S., Bourban, S. M., Flynn, M. G., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., & Wellner, P. (2006). The AMI meeting corpus: A pre-announcement. In S. Renals & S. Bengio (Eds.), Machine learning for multimodal interaction, Second International Workshop, MLMI 2005, Edinburgh, UK, July, 2005, revised selected papers, vol 3869 of Lecture Notes in Computer Science (pp. 28–39). Springer-Verlag. ISBN 3-540-32549-2

  • Carletta, J., Evert, S., Heid, U., Kilgour, J., Robertson, J., & Voormann, H. (2003). The NITE XML toolkit: Flexible annotation for multi-modal language data. Behavior Research Methods, Instruments, and Computers, 35(3), 353–363

    Google Scholar 

  • Carletta, J., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G., & Anderson, A. (1997). The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1), 13–31

    Google Scholar 

  • Chen, L., Rose, R. T., Parrill, F., Han, X., Tu, J., Huang, Z., Harper, M., Quek, F., McNeill, D., Tuttle, R., & Huang, T. (2006). VACE multimodal meeting corpus. In S. Renals & S. Bengio (Eds.), Machine learning for multimodal interaction, Second International Workshop, MLMI 2005, Edinburgh, UK, July, 2005, revised selected papers, vol 3869 of Lecture Notes in Computer Science (pp. 40–51). Springer-Verlag. ISBN 3-540-32549-2

  • Clark, H. H. & Carlson, T. B. (1992). Hearers and speech acts. In H. H. Clark (Ed.), Arenas of Language Use (pp. 205–247). University of Chicago Press and CSLI

  • Clark, A., & Popescu-Belis, A. (2004). Multi-level dialogue act tags. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 163–170). Cambridge, MA

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46

    Google Scholar 

  • Dhillon, R., Bhagat, S., Carvey, H., & Shriberg, E. (2004). Meeting recorder project: Dialogue act labeling guide. Technical report TR-04-002, International Computer Science Institute (ICSI), Berkeley, CA, USA

  • Garofolo, J. S., Laprun, C. D., Michel, M., Stanford, V. M., & Tabassi, E. (2004) The NIST meeting room pilot corpus. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC) (pp. 1411–1414). Lisbon, Portugal

  • Goffman, E. (1981). Footing. In Goffman, E., (Ed.), Forms of talk (pp. 124–159). Philadelphia, PA: University of Pennsylvania Press

    Google Scholar 

  • Goodwin, C. (1981). Conversational organization: Interaction between speakers and hearers. New York: Academic Press

    Google Scholar 

  • Hirschberg, J., & Nakatani, C. (1996). A prosodic analysis of discourse segments in direction-giving monologues. In In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics (ACL) (pp 286–293). Santa Cruz, California

  • Janin, A., Ang, J., Bhagat, S., Dhillon, R., Edwards, J., Macias-Guarasa, J., Morgan, N., Peskin, B., Shriberg, E., Stolcke, A., Wooters, C., & Wrede, B. (2004). The ICSI meeting project: Resources and research. In Proceedings of the NIST ICASSP Meeting Recognition Workshop. Montreal, Canada

  • Jovanovic, N., & op den Akker, R. (2004). Towards automatic addressee identification in multi-party dialogues. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 89–92). Cambridge, MA

  • Jovanovic, N., op den Akker, R., & Nijholt, A. (2006). Addressee identification in face-to-face meetings. In Proceeding of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 169–176). Trento, Italy

  • Jurafsky, D., Shriberg, L., & Biasca, D. (1997). Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual, Draft 13. Technical report 97–02, University of Colorado, The Institute of Cognitive Science, Boulder, CO

  • Katzenmaier, M., Stiefelhagen, R., & Schultz, T. (2004). Identifying the addressee in human–human–robot interactions based on head pose and speech. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 144–151). State College, PA

  • Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Sage Publications, Beverly Hills, CA

    Google Scholar 

  • Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433

    Google Scholar 

  • Manning, C., & Schutze, H. (1999). Foundations of statistical natural language processing. The MIT Press, Cambridge, MA

    Google Scholar 

  • Moore, D. (2002). The IDIAP smart meeting room. Technical report IDIAP-COM-07, IDIAP, Martigny, Switzerland

  • Otsuka, K., Takemae, Y., Yamato, J., & Murase, H. (2005). A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 191–198). Trento, Italy

  • Passonneau, R. (2004). Computing reliability for coreference annotation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC) (pp. 1503–1506). Lisbon, Portugal

  • Passonneau, R. & Litman, D. (1997). Discourse segmentation by human and automated means. Computational Linguistics, 23(1), 103–139

    Google Scholar 

  • Reidsma, D., Hofs, D., & Jovanovic, N. (2005). A presentation of a set of new annotation tools based on the NXT API. In Proceedings of Measuring Behavior (pp. 512–513). Wageningen, The Netherlands

  • Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., & Carvey, H. (2004). The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (pp. 97–100). Boston, USA

  • Traum, D. (2004). Issues in multi-party dialogues. In F. Dignum (Ed.), Advances in agent communication (pp. 201–211). Springer-Verlag LNCS

  • van Turnhout, K., Terken, J., Bakx, I., & Eggen, B. (2005). Identifying the intended addressee in mixed human–human and human–computer interaction from non-verbal features. In Proceedings of the International Conference on Multimodal Interfaces (ICMI) (pp. 175–182). Trento, Italy

Download references

Acknowledgements

This work was partly supported by the European Union 6th FWP IST Integrated Project AMI (Augmented Multi-party Interaction, FP6-506811, publication AMI-160). We would like to thank Dennis Reidsma, Dennis Hofs, Lynn Packwood and the annotators who were involved in the corpus development. We are grateful to Klaus Krippendorff for useful discussions about reliability metrics.

Author information

Authors and Affiliations

  1. Human Media Interaction Group, University of Twente, PO Box 217, Enschede, 7500 AE, The Netherlands

    Natasa Jovanovic, Rieks op den Akker & Anton Nijholt

Authors
  1. Natasa Jovanovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rieks op den Akker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anton Nijholt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rieks op den Akker.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jovanovic, N., op den Akker, R. & Nijholt, A. A corpus for studying addressing behaviour in multi-party dialogues. Lang Resources & Evaluation 40, 5–23 (2006). https://doi.org/10.1007/s10579-006-9006-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-006-9006-4

Keywords

Search

Navigation