A Multimodal Approach to Communicative Interactivity Classification

  • Tomasz M. Rutkowski
  • Danilo Mandic
  • Allan Kardec Barros


The problem of modality detection in so called communicative interactivity is addressed. Multiple audio and video recordings of human communication are analyzed within this framework, based on fusion of the extracted features. At the decision level, support vector machines (SVMs) are utilized to segregate between the communication modalities. The proposed approach is verified through simulations on real world recordings.


human communication analysis data fusion multimedia information processing audiovisual data fusion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    M. Chen, “Visualizing the Pulse of a Classroom,” in Proceedings of the Eleventh ACM International Conference on Multimedia, ACM Press, 2003, pp. 555–561.Google Scholar
  2. 2.
    J.J. Gibson, “The Theory of Affordances,” in Perceiving, Acting and Knowing, R. Shaw and J. Bransford (Eds.), Erlbaum, Hillsdale, NJ, 1977.Google Scholar
  3. 3.
    T.M. Rutkowski, M. Yokoo, D. Mandic, K. Yagi, Y. Kameda, K. Kakusho and M. Minoh, “Identification and Tracking of Active Speaker’s Position in Noisy Environments,” in Proceedings of International Workshop on Acoustic Echo and Noise Control (IWAENC2003), Kyoto, Japan, 2003, pp. 283–286.Google Scholar
  4. 4.
    T.M. Rutkowski, K. Kakusho, V.V. Kryssanov and M. Minoh, “Evaluation of the communication atmosphere,” Lect. Notes Comput. Sci., vol. 3213, 2004, pp. 364–370.CrossRefGoogle Scholar
  5. 5.
    T.M. Rutkowski, Y. Yamakata, K. Kakusho and M. Minoh, “Smart sensor mesh—intelligent sensor clusters configuration based on communicative affordances principle,” Lecture Notes in Artificial Intelligence, vol. 3490, 2005, pp. 147–157.Google Scholar
  6. 6.
    T.M. Rutkowski and D. Mandic, “Communicative interactivity—a multimodal communicative situation classification approach,” Lect. Notes Comput. Sci., vol. 3697, 2005, pp. 741–746.Google Scholar
  7. 7.
    S. Furui, “Digital Speech Processing, Synthesis, and Recognition—Second Edition, Revised and Expanded. 2nd edn. Signal Processing and Communications Series,” Marcell Dekker, Inc., New York, Basel, 2001.Google Scholar
  8. 8.
    D.D. Lee and H.S. Seung, “Learning the Parts of Objects by Non-Negative Matrix Factorization,” Nature, vol. 401, 1999, pp. 788–791.CrossRefGoogle Scholar
  9. 9.
    A. Hyvarinen, J. Karhunen, E. Oja, “Independent Component Analysis,” Wiley, 2001.Google Scholar
  10. 10.
    T.M., Rutkowski, S. Seki, Y. Yamakata, K. Kakusho and M. Minoh, “Toward the Human Communication Efficiency Monitoring from Captured Audio and Video Media in Real Environments,” Lect. Notes Comput. Sci., vol. 2774, 2003, pp. 1093–1100.CrossRefGoogle Scholar
  11. 11.
    C. Shannon and W. Weaver, “The Mathematical Theory of Communication,” University of Illinois Press, Urbana, 1949.MATHGoogle Scholar
  12. 12.
    V. Kryssanov and K. Kakusho, “From Semiotics of Hypermedia to Physics of Semiosis: A view from System Theory,” Semiotica, vol. 154, no. 1/4, 2005, pp. 11–38.Google Scholar
  13. 13.
    C.W. Hsu and C.J. Lin, “A Comparison of Methods for Multi-Class Support Vector Machines,” IEEE Trans. Neural Netw., vol. 13, 2002, pp. 415–425.CrossRefGoogle Scholar
  14. 14.
    V. Cherkassky and F. Mulier, “Learning from Data. Adaptive and Learning Systems for Signal Processing, Communication, and Control,” Wiley, USA (1998).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • Tomasz M. Rutkowski
    • 1
  • Danilo Mandic
    • 2
  • Allan Kardec Barros
    • 3
  1. 1.Brain Science InstituteRIKENSaitamaJapan
  2. 2.Department of Electrical and Electronic EngineeringImperial College of Science, Technology and MedicineLondonUK
  3. 3.Laboratory for Biological Information ProcessingUniversidade Federal do MaranhāoMaranhãoBrazil

Personalised recommendations