Analysing Multimodal Communication

Repair-Based Measures of Human Communicative Coordination
  • Patrick G. T. Healey
  • Marcus Colman
  • Mike Thirlwell
Part of the Text, Speech and Language Technology book series (TLTB, volume 30)


There are few techniques available to inform the design of systems to support human-human interaction. Psycholinguistic models have the potential to fill this gap however existing approaches have some conceptual and practical limitations. This chapter presents a technique, based on the conversation analytic model of breakdown and repair, for modality and task independent analysis of communicative exchanges. The rationale for the approach is presented and a protocol for coding repair is described. The potential of this approach for analysing multimodal interactions is discussed.


Communication conversation analysis repair evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson, A. H., O’Malley, C., Doherty-Sneddon, G., Langton, S., Newlands, A., Mullin, J., Fleming, A. M., and van der Velden, J. (1997). The impact of VMC on Collaborative Problem Solving: An Analysis of Task Performance, Communicative Process and User Satisfaction. In Finn, K. E., Sellen, A., and Wilbur, S. B., editors, Video-Mediated Communication, pages 133–156. Mahwah, New Jersey: Lawrence Erlbaum Associates.Google Scholar
  2. Bowers, J., Button, G., and Sharrock, W. (1995). Workflow from Within and Without: Technology and Co-operative Work on the Print Industry Shop Floor. In Proceedings of the fourth European conference on Computer-Supported Co-operative Work, pages 51–66. New York: ACM Press.Google Scholar
  3. Brennan, S. (1988). The Grounding Problem in Conversations with and through Computers. In Fussell, S. R. and Kreuz, R. J., editors, Social and Cognitive Approaches to Interpersonal Communication, pages 201–225. Mahwah, New Jersey: Lawrence Erlbaum Associates.Google Scholar
  4. Button, D. and Dourish, P. (1996). Technomethodology: Paradoxes and Possibilities. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 19–26. New York: ACM Press.Google Scholar
  5. Carletta, J., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G., and Anderson, A. (1996). The Reliability of a Dialogue Structure Coding Scheme. Computational Linguistics, 23(1):13–31.Google Scholar
  6. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press.Google Scholar
  7. Clark, H. H. and Brennan, S. (1991). Grounding in Communication. In Resnick, L., Levine, J., and Teasley, S., editors, Perspectives on Socially Shared Cognition, pages 127–150. American Psychological Association.Google Scholar
  8. Clark, H. H. and Schaefer, E. F. (1989). Contributing to Discourse. Cognitive Science, 13:259–294.CrossRefGoogle Scholar
  9. Clark, H. H. and Wilkes-Gibbs, D. (1986). Referring as a Collaborative Process. Cognition, 22:1–39.CrossRefGoogle Scholar
  10. Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliff: Prentice Hall.Google Scholar
  11. Heath, C. and Luff, P. (2000). Technology in Action. Cambridge University Press, Cambridge.Google Scholar
  12. Hughes, J., Randall, D., and Shapiro, D. (1992). Faltering from Ethnography to Design. In Proceedings of the Conference on Computer Supported Cooperative Work (CSCW), pages 115–122.Google Scholar
  13. Hutchins, E. (1995). How a Cockpit Remembers its Speeds. Cognitive Science, 19:265–288.CrossRefGoogle Scholar
  14. Jefferson, G. (1982). On Exposed and Embedded Correction in Conversation. In Button, G. and Lee, J. R., editors, Talk and Social Organisation, pages 86–100. Clevedon: Multilingual Matters.Google Scholar
  15. Kowtko, J. C., Isard, S. D., and Doherty, G. M (1991). Conversational Games within Dialogue. In Caenepeel, M., Delin, J. L., Oversteegen, L., Redeker, G., and Sanders, J., editors, Proceedings of the DANDI Workshop on Discourse, HCRC, Edinburgh, UK.Google Scholar
  16. Levinson, S. C. (1983). Pragmatics. Cambridge: Cambridge University Press.Google Scholar
  17. Nardi, B. and Miller, J. (1991). Twinkling Lights and Nested Loops: Distributed Problem Solving and Spreadsheet Development. International Journal of Man-Machine Studies, 34:161–184.CrossRefGoogle Scholar
  18. O’Conaill, B., Whittaker, S., and Wilbur, S. (1993). Conversations over Video Conferences: An Evaluation of the Spoken Aspects of Video-Mediated Communication. Human-Computer Interaction, 8:389–428.CrossRefGoogle Scholar
  19. O’Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G., and Bruce, V. (1996). Comparison of Face-to-Face and Video-Mediated Interaction. Interacting with Computers, 8(2):177–192.CrossRefGoogle Scholar
  20. Schegloff, E. A. (1982). Recycled Turn Beginnings: A Precise Repair Mechanism in Conversation’s Turn-Taking Organisation. In Button, G. and Lee, J. R., editors, Talk and Social Organisation, pages 70–85. Clevedon: Multilingual Matters.Google Scholar
  21. Schegloff, E. A. (1987). Some Sources of Misunderstanding in Talk-in-Interaction. Linguistics, 25:201–218.CrossRefGoogle Scholar
  22. Schegloff, E. A. (1992). Repair after Next Turn: The Last Structurally Provided Defense of Intersubjectivity in Conversation. American Journal of Sociology, 97(5):1295–1345.CrossRefGoogle Scholar
  23. Schegloff, E. A. (1993). Reflections on Quantification in the Study of Conversation. Research on Language and Social Interaction, 26(1):99–128.CrossRefGoogle Scholar
  24. Schegloff, E. A. (2000). When ‘Others’ Initiate Repair. Applied Linguistics, 21(2):205–243.CrossRefGoogle Scholar
  25. Schegloff, E. A., Jefferson, G., and Sacks, H. (1977). The Preference for Self-Correction in the Organization of Repair in Conversation. Language, 53(2):361–382.CrossRefGoogle Scholar
  26. Suchman, L. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press.Google Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  • Patrick G. T. Healey
    • 1
  • Marcus Colman
    • 1
  • Mike Thirlwell
    • 1
  1. 1.Interaction, Media, and Communication Research Group, Department of Computer Science Queen MaryUniversity of LondonUK

Personalised recommendations