Personal and Ubiquitous Computing

, Volume 14, Issue 8, pp 767–778 | Cite as

A multi-modal dialogue analysis method for medical interviews based on design of interaction corpus

  • Yuichi Koyama
  • Yuichi Sawamoto
  • Yasushi Hirano
  • Shoji Kajita
  • Kenji Mase
  • Tomio Suzuki
  • Kimiko Katsuyama
  • Kazunobu Yamauchi
Original Article


We propose a multi-modal dialogue analysis method for medical interviews that hierarchically interprets nonverbal interaction patterns in a bottom-up manner and simultaneously visualizes the topic structure. Our method aims to provide physicians with the clues generally overlooked by conventional dialogue analysis to form a cycle of dialogue practice and analysis. We introduce a motif and a pattern cluster in the designs of the hierarchical indices of interaction and exploit the Jensen–Shannon divergence (JSD) metric to reduce the number of usable indices. We applied the proposed interpretation method of interaction patterns to develop a corpus of interviews. The results of a summary reading experiment confirmed the validity of the developed indices. Finally, we discussed the integrated analysis of the topic structure and a nonverbal summary.


Multi-modal interaction Dialogue analysis Interaction corpus Topic structure 


  1. 1.
    Greenhalgh T et al (1998) Narrative based medicine: dialogue and discourse in clinical practice. BMJ Books, LondonGoogle Scholar
  2. 2.
    Roter D et al (2002) The Roter interaction analysis system (RIAS). Patient Educ Couns 46:243–251CrossRefGoogle Scholar
  3. 3.
    Mase K, Sumi Y, Toriyama T, Tsuchikawa M, Ito S, Iwasawa S, Koguure K, Hagita N (2006) Ubiquitous experience media. IEEE Multimedia, Oct–Dec, pp 20–29Google Scholar
  4. 4.
    Koyama Y, Hirano Y, Kajita S, Mase K, Katsuyama K, Yamauchi K (2006) Doctor-patient communication supporting method by visualizing topic structure, CSCW2006 Conference Supplement, pp 195–196Google Scholar
  5. 5.
    Lin J, Keogh E, Lonardi S, Patel P (2002) Finding motifs in time series. In: The 2nd workshop on Temporal data mining, the 8th ACM international conference on knowledge discovery and data mining, pp 23–26Google Scholar
  6. 6.
    Fuglede B, Topsoe F (2004) Jensen-Shannon divergence and Hilbert space embedding. In: Proceedings the international symposium on information theory, pp 31–36Google Scholar
  7. 7.
    Wren CR, Ivanov YA, Kaur I, Leigh D, Westhues J (2007) Social motion: measuring the hidden social life of a building, location- and context-awareness 2007. Lect Notes Comput Sci 4718:85–102CrossRefGoogle Scholar
  8. 8.
    Shiomi M, Kanda T, Ishiguro H, Hagita N (2007) Interactive humanoid robots for a science museum. IEEE Intell Syst 22(2):25–32CrossRefGoogle Scholar
  9. 9.
    Otsuka K, Takemae Y, Yamato J, Murase H (2005) A probabilistic inference of multiparty-conversation structure based on Markov-Switching models of gaze patterns, head directions, and utterances. In: Proceedings of the ACM international conference on multimodal interfaces (ICMI)’ 05, pp 191–198Google Scholar
  10. 10.
    Otsuka K, Sawada H, Yamato J (2007) Automatic inference of cross-modal nonverbal interactions in multiparty conversations. Proc ICMI 2007:255–262CrossRefGoogle Scholar
  11. 11.
    Zhiwen Yu, Zhiyong Yu, Yusa Ko, Xingshe Zhou, Yuichi Nakamura (2009) Inferring human interactions in meetings: a multimodal approach. In: The 6th international conference on ubiquitous intelligence and computing (UIC 2009), pp 14–24, July 7–9, Brisbane, AustraliaGoogle Scholar
  12. 12.
    Hillard D, Ostendorf M, Shriberg E (2003) Detection of Agreement vs. Disagreement in Meetings: Training with Unlabeled Data. Proc HLT-NAACL 2003:34–36Google Scholar
  13. 13.
    Tomobe H, Nagao K (2006) Discussion ontology: knowledge discovery from human activities in meetings. Proc Jpn Soc Artif Intell (JSAI) 2006:33–41Google Scholar
  14. 14.
    Ong LML, de Haes JCJM, Hoos AM, Lammes FB (1995) Doctor-patient communication: a review of the literature. Soc Sci Med 40(7):903–918CrossRefGoogle Scholar
  15. 15.
    Gorawara-Bhat R et al (2007) Nonverbal communication in doctor elderly patient transactions (NDEPT): development of a tool. Patient Educ Couns 66:223–234CrossRefGoogle Scholar
  16. 16.
    Dent E et al (2005) The Cancode interaction analysis system in the oncological setting: reliability and validity of video and audio tape coding. Patient Educ Couns 56:35–44CrossRefGoogle Scholar
  17. 17.
    Schmid Mast M (2007) On the importance of nonverbal communication in the physician-patient interaction. Patient Educ Couns 67:315–318CrossRefGoogle Scholar
  18. 18.
    Morita T, Hirano Y, Sumi Y, Kajita S, Mase K (2005) A pattern mining method for interpretation of interaction. In: Proceedings on international conference on multimodal interface (ICMI’05), pp 267–273Google Scholar
  19. 19.
    Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Saito S et al. (2003) Practice of narrative based medicine. Kongo Shuppan (in Japanese)Google Scholar
  21. 21.
    Katsuyama K, Koyama Y, Hirano Y, Mase K, Kato K, Mizuno S, Yamauchi K (2009) Computer analysis system of the physician-patient consultation process. Int J Health Care Quality Assurance 23(4)Google Scholar
  22. 22.
    Connor M et al (2009) The analysis of verbal interaction sequences in dyadic clinical communication: a review of methods. Patient Educ Couns 75(2):169–177CrossRefGoogle Scholar
  23. 23.
    Pinhanez C, Mase K, Bobick A (1997) Interval scripts: a design paradigm for story-based interactive systems. In: CHI97 Conference Proceedings, pp 287–294Google Scholar

Copyright information

© Springer-Verlag London Limited 2010

Authors and Affiliations

  • Yuichi Koyama
    • 1
  • Yuichi Sawamoto
    • 1
    • 2
  • Yasushi Hirano
    • 3
  • Shoji Kajita
    • 3
  • Kenji Mase
    • 1
  • Tomio Suzuki
    • 4
  • Kimiko Katsuyama
    • 5
  • Kazunobu Yamauchi
    • 6
  1. 1.Graduate School of Information ScienceNagoya UniversityNagoyaJapan
  2. 2.KDDI CorporationTokyoJapan
  3. 3.Information Technology CenterNagoya UniversityNagoyaJapan
  4. 4.Nagoya University HospitalNagoyaJapan
  5. 5.School of NursingOsaka Prefecture UniversityOsakaJapan
  6. 6.Fujita Health University CollegeAichiJapan

Personalised recommendations