Multimodal Integration of Sensor Network

  • Joachim Neumann
  • Josep R. Casas
  • Dušan Macho
  • Javier Ruiz Hidalgo
Part of the IFIP International Federation for Information Processing book series (IFIPAICT, volume 204)


At the Universitat Politècnica de Catalunya (UPC), a Smart Room has been equipped with 85 microphones and 8 cameras. This paper describes the setup of the sensors, gives an overview of the underlying hardware and software infrastructure and indicates possibilities for high- and low-level multi-modal interaction. An example of usage of the information collected from the distributed sensor network is explained in detail: the system supports a group of students that have to solve a lab assignment related problem.


Linear Discriminant Analysis Gaussian Mixture Model Gesture Recognition Automatic Speech Recognition Acoustic Event 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Josep R. Casas, R. Stiefelhagen, et al, “Multi-camera/multi-microphone system design for continuous room monitoring,” CHIL-WP4-D4.1-V2.1-2004-07-08-CO, CHIL Consortium Deliverable D4.1, July 2004.Google Scholar
  2. 2.
    J-L. Landabaso, L-O. Xu, M. Pardas, Robust Tracking and Object Classification Towards Automated Video Surveillance, Proc. of International Conference on Image Analysis and Recognition ICIAR 2004, Porto, Portugal, September 29 — October 1, 2004, Proceedings, Part II, p. 463–470Google Scholar
  3. 3.
    J. L. Landabaso, M. Pardàs, L.-Q. Xu, Hierarchical Representation of Scenes using Activity Information, Proc of ICASSP 2005, March 18–23, Philadelphia, USA.Google Scholar
  4. 4.
    Josep R. Casas, O. Garcia, et al, “Initial multi-sensor selection strategy to get the best camera/microphone at any time,” CHIL-WP4-D4.2-V2.0-2004-10-18-CO, CHIL Deliverable D4.2, October 2004.Google Scholar
  5. 5.
    O. Garcia, J.R. Casas, “Functionalities for mapping 2D images and 3D world objects in a Multicamera Environment,” 6th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Montreux, Switzerland, April, 2005.Google Scholar
  6. 6.
    A. Laurentini, “The visual hull concept for silhouette-based image understanding,” IEEE Trans, Pattern Anal. Mach. Intell, 16(2): 150–162,1994.CrossRefGoogle Scholar
  7. 7.
    J.L. Landabaso, M. Pardas, “Foreground regions extraction and characterization towards real-time object tracking,” In Proceedings of Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI’ 05), 2005. 3Google Scholar
  8. 8.
    NIST smart space system, Scholar
  9. 9.
    Padrell J., Macho D., Nadeu C., “Robust Speech Activity Detection Using LDA Applied to FF Parameters”, Proc. ICASSP’05, Philadelphia, PA, USA, March 2005.Google Scholar
  10. 10.
    Macho D., Padrell J., Abad A., Nadeu C., Hernando J., McDonough J., Wölfel M., Klee U., Omologo M., Brutti A., Svaizer P., Potamianos G., Chu S.M., “Automatic Speech Activity Detection, Source Localization, and Speech Recognition on the CHIL Seminar Corpus”, Proc. ICME 2005, Amsterdam, The Netherlands, July 2005.Google Scholar
  11. 11.
    Omologo M., Svaizer P., “Acoustic event localization using a crosspower-spectrum phase based technique,” in Proc. ICASSP’94, Adelaide, 1994.Google Scholar
  12. 12.
    Abad A., Macho D., Segura C., Hernando J., Nadeu C., “Effect of Head Orientation on the Speaker Localization Performance in Smart-room Environment”, Proc. INTERSPEECH — EUROSPEECH 2005, Lisbon, Portugal, September 2005.Google Scholar
  13. 13.
    Temko A., Macho D., Nadeu C., “Selection of features and combination of classifiers using a fuzzy approach for acoustic event classification”, Proc. of 9th European Conference on Speech Communication and Technology, Interspeech 2005, Lisbon, Portugal, September 2005.Google Scholar
  14. 14.
    Temko A., Macho D., Nadeu C., “Improving the performance of acoustic event classification by selecting and combining information sources using the fuzzy integral”, Lecture Notes in Computer Science (LNCS), vol. 3869, February 2006Google Scholar
  15. 15.
    Temko A., Nadeu C., “Classification of Acoustic Events using SVM-based Clustering Schemes”, Pattern Recognition, in press, Elsevier, 2006Google Scholar

Copyright information

© International Federation for Information Processing 2006

Authors and Affiliations

  • Joachim Neumann
    • 1
  • Josep R. Casas
    • 1
  • Dušan Macho
    • 1
  • Javier Ruiz Hidalgo
    • 1
  1. 1.Signal Theory and Communications DepartmentUPC — Technical University of CataloniaBarcelonaSpain

Personalised recommendations