Skip to main content
SpringerLink
Log in

Interactive Multimedia for Technology-Enhanced Learning with Multimodal Feedback

  • Chapter
Musical Robots and Interactive Multimodal Systems

Part of the book series: Springer Tracts in Advanced Robotics ((STAR,volume 74))

Abstract

Musical performances are generally physically demanding with high degree of control (mental and motor) and accuracy. This chapter presents the i-Maestro (www.i-maestro.org) project which explored interactive multimedia environments for technology-enhanced music education. It discusses one of the key requirements for an interactive musical robot which is to analyze and provide feedback/interaction to a “performance”. This Chapter also serves as an example application of a musical robot in educational contexts. Music is not simply playing the right note at the right time. The multitude of interconnecting factors that influence and contribute to the nature of the playing is not easy to monitor nor analyze. Musical instrumentalists often use mirrors to observe themselves practicing. This Chapter briefly introduces the i-Maestro project and focuses on a gesture interface developed under the i-Maestro framework called the 3D Augmented Mirror (AMIR). AMIR captures, analyze and visualizes the performance in 3D. It offers a number of different analyses and feedback to support the learning and teaching of bowing technique and gesture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Baillie, S., Brewster, S., Hall, C., O’Donnell, J.: Motion space reduction in a haptic model of violin and viola bowing. In: 1st Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 525–526. IEEE CS (2005)

    Google Scholar 

  2. Bellini, P., Nesi, P., Zoia, G.: MPEG symbolic music representation: a solution for multimedia music applications. In: Ng, Nesi (eds.) Interactive Multimedia Music Technologies. IGI Global (2008), doi:10.4018/978-1-59904-150-6.

    Google Scholar 

  3. Bellini, P., Frosini, F., Mitolo, N., Nesi, P., Paolucci, M.: Collaborative working for music education. In: Ng (ed.) Proceedings of the 4th i-Maestro Workshop on Technology-Enhanced Music Education, Co-Located with the 8th International Conference on New Interfaces for Musical Expression (NIME 2008), Genova, Italy, pp. 5–10 (2008) ISBN: 9780853162698

    Google Scholar 

  4. Bevilacqua, F., Rasamimanana, N., Fléty, E., Lemouton, S., Baschet, F.: The augmented violin project: research, composition and performance report. In: Proceedings of the Int. Conference on New Interfaces for Musical Expression (NIME), Paris, France (2006)

    Google Scholar 

  5. Brewster, S.A., Wright, P.C., Edwards, A.D.N.: Parallel Earcons: reducing the length of audio messages. International Journal of Human-Computer Studies 43(22), 153–175 (1995)

    Article  Google Scholar 

  6. Brown, L.M., Brewster, S.A.: Drawing by ear: interpreting sonified line graphs. In: Proceedings of Int. Conference on Auditory Display (ICAD), Boston, Massachusetts (2003)

    Google Scholar 

  7. Cadoz, C., Wanderley, M.M.: Gesture-Music. In: Trends in Gestural Control of Music. IRCAM-Centre Pompidou (2000)

    Google Scholar 

  8. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 20–39. Springer, Heidelberg (2004a)

    Chapter  Google Scholar 

  9. Camurri, A., Mazzarino, B., Volpe, G.: Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 460–467. Springer, Heidelberg (2004b)

    Chapter  Google Scholar 

  10. Cont, A., Schwarz, D.: Score following at IRCAM. In: Cont, A., Schwarz, D. (eds.) MIREX 2006, Second Annual Music Information Retrieval Evaluation eXchange, Abstract Collection, The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL), p. 94. University of Illinois at Urbana-Champaign, Victoria, Canada (2006), http://www.music-r.org/evaluation/MIREX/2006abstracts/MIREX2006Abstracts.pdf (last accessed December 8, 2010)

    Google Scholar 

  11. Dahl, S., Friberg, A.: Visual perception of expressiveness in musicians’ body movements. Music Perception 24 (2004)

    Google Scholar 

  12. Delalande, F.: Le Geste, outil d’analyse: quelques enseignements d’une recherche sur la gestique de Glenn Gould. Analyse Musicale, 1er trimestre (1988)

    Google Scholar 

  13. Effenberg, A., Melzer, J., Weber, A., Zinke, A.: MotionLab Sonify: a framework for the sonification of human motion data. In: Proceedings of Ninth International Conference on Information Visualisation, IV 2005 (2005)

    Google Scholar 

  14. Ferguson, S.: Learning musical instrument skills through interactive sonification. In: Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2006), Paris, France (2006)

    Google Scholar 

  15. Gerle, R.: The art of bowing practice: the expressive bow technique. Stainer & Bell (1991)

    Google Scholar 

  16. Ghez, C., Rikakis, T., DuBois, L., Cook, P.: An auditory display system for aiding interjoint coordination. In: Proceedings of the International Conference on Auditory Display (2000)

    Google Scholar 

  17. Ho, T.K.-L.: A computer-assisted approach to the teaching of violin tone production. ACM SIGCUE Outlook 21(2), 73–83 (1991), doi:10.1145/122463.122472

    Article  Google Scholar 

  18. Hodgson, P.: Motion study and violin bowing. The Strad, Lonon (1934)

    Google Scholar 

  19. i-Maestro project website: http://www.i-maestro.org (last accessed December 8, 2010)

  20. Kapur, A., Tzanetakis, G., Virji-Babul, N., Wang, G., Cook, P.R.: A framework for sonification of vicon motion capture data. In: Proceedings of the International Conference on Digital Audio Effects DAFX 2005, Madrid, Spain (2005)

    Google Scholar 

  21. Larkin, O., Koerselman, T., Ong, B., Ng, K.: Sonification of bowing features for string instrument training. In: Proceedings of the 14th International Conference on Auditory Display, Paris, France (2008)

    Google Scholar 

  22. Manson, P.: Digital preservation research: An evolving landscape. In: The European Research Consortium for Informatics and Mathematics (ERCIM) News 80, Special Theme: Digital Preservation (2010), http://ercim-news.ercim.eu/en80

  23. Mansur, D.L.: Graphs in sound: a numerical data analysis method for the blind, M.Sc. Thesis, Department of Computing Science, University of California (1975)

    Google Scholar 

  24. Mora, J., Won-Sook, L., Comeau, G., Shirmohammadi, S., El Saddik, A.: Assisted piano pedagogy through 3D visualization of piano playing. In: Proceedings of HAVE 2006, Ottawa, Canada (2006)

    Google Scholar 

  25. Neuwirth, E.: Designing a pleasing sound mathematically. MATHMAG: Mathematics Magazine 74 (2001)

    Google Scholar 

  26. Ng, K.: Sensing and mapping for interactive performance. Organised Sound 7(2), 191–200 (2002)

    Article  Google Scholar 

  27. Ng, K.: Music via Motion (MvM): trans-domain mapping of motion and sound for interactive performances. Proceedings of the IEEE, Special Issue on: Engineering and Music – Supervisory Control & Auditory Communication 92(4), 645–655 (2004)

    Google Scholar 

  28. Ng, K., Larkin, O., Koerselman, T., Ong, B.: i-Maestro gesture and posture support: 3d motion data visualization for music learning and playing. In: Bowen, J.P., Keene, S., MacDonald, L. (eds.) Proceedings of EVA 2007 London International Conference, July 11-13, vol. 8, pp. pp. 20.1-20.8. London College of Communication, University of the Arts London, UK (2007a)

    Google Scholar 

  29. Ng, K., Larkin, O., Koerselman, T., Ong, B., Schwarz, D., Bevilacqua, F.: The 3D augmented mirror: motion analysis for string practice training. In: Proceedings of the International Computer Music Conference, ICMC 2007 – Immersed Music, Copenhagen, Denmark, Vol.II, pp. 53–56 (August 2007b), ISBN: 0-9713192-5-1

    Google Scholar 

  30. Ng, K., Weyde, T., Larkin, O., Neubarth, K., Koerselman, T., Ong, B.: 3D augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support. In: Proceedings of the 9th International Conference on Multimodal Interfaces, Nagoya, Japan, pp. 339–345. ACM SIGCHI (2007), http://doi.acm.org/10.1145/1322192.1322252 , ISBN: 978-1-59593-817-6

  31. Ng, K., Nesi, P. (eds.): Interactive multimedia music technologies, p. 394 pages. IGI Global, Information Science Reference, Library of Congress 2007023452 (2008) ISBN: 978-1-59904-150-6 (hardcover) 978-1-59904-152-0 (ebook)

    Google Scholar 

  32. Ng, K., Pham, T.V., Ong, B., Mikroyannidis, A., Giaretta, D.: Preservation of Interactive Multimedia Performances. International Journal of Metadata, Semantics and Ontologies (IJMSO) 2008 3(3), 183–196 (2009), doi:10.1504/IJMSO.2008.023567

    Article  Google Scholar 

  33. Ng, K.: Digital preservation of interactive multimedia performances. In: The European Research Consortium for Informatics and Mathematics (ERCIM) News 80, Special Theme: Digital Preservation (2010), http://ercim-news.ercim.eu/en80

  34. Ong, B., Ng, K., Mitolo, N., Nesi, P.: i-Maestro: interactive multimedia environments for music education. In: Proceedings of the 2nd Int. Conference on Automated Production of Cross Media Content for Multi-channel Distribution (AXMEDIS 2006), Workshops, Tutorials, Applications and Industrial, December 13-15, pp. 87–91. Firenze University Press (FUP), Leeds (2006)

    Google Scholar 

  35. Rasamimanana, N.: Gesture analysis of bow strokes using an augmented violin - mémoire de stage de DEA ATIAM année 2003-2004. IRCAM-Centre Pompidou (2004)

    Google Scholar 

  36. Rolland, P.: The teaching of action in string playing. Illinois String Research Associates (1974)

    Google Scholar 

  37. Sturm, H., Rabbath, F.: The art of the bow DVD (2006), http://www.artofthebow.com (last accessed December 18, 2009)

  38. Wanderley, M.M.: Non-obvious Performer Gestures in Instrumental Music. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, p. 37. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  39. Wanderley, M.M.: Quantitative analysis of non-obvious performer gestures. In: Wachsmuth, I., Sowa, T. (eds.) Gesture and Sign Language in Human-Computer Interaction, pp. 241–253. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  40. Weyde, T., Ng, K., Neubarth, K., Larkin, O., Koerselman, T., Ong, B.: A systemic approach to music performance learning with multimodal technology support. In: Bastiaens, T., Carliner, S. (eds.) Proceedings of E-Learn 2007, World Conference on E-Learning in Corporate, Government, Healthcare, & Higher Education (AACE), October 15-19, Québec City (2007)

    Google Scholar 

  41. Young, D.: Wireless sensor system for measurement of violin bowing parameters. In: Proceedings of the Stockholm Music Acoustics Conference, SMAC 2003, Stockholm (2003)

    Google Scholar 

  42. Young, D.: A methodology for investigation of bowed string performance through measurement of violin bowing technique, PhD Thesis, M.I.T (2007)

    Google Scholar 

  43. Young, D.: Capturing Bowing Gesture: Interpreting Individual Technique. In: Solis, J., Ng, K. (eds.) Musical robots and Interactive Multimodal Systems. Tracts in Advanced Robotics, vol. 74, Springer, Heidelberg (2011) (hard cover), ISBN: 978-3-642-22290-0

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing & School of Music, ICSRiM – University of Leeds, Leeds, LS2 9JT, UK

    Kia Ng

Authors
  1. Kia Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Humanoid Robotics Institute Faculty of Science and Engineering, Waseda University, 3-4-1 Ookubo, 168-8555, Shinjuku-ku, Tokyo, Japan

    Jorge Solis

  2. Interdisciplinary Centre for Scientific Research in Music (ICSRiM), University of Leeds School of Computing & School of Music, LS2 9JT, Leeds, UK

    Kia Ng

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Ng, K. (2011). Interactive Multimedia for Technology-Enhanced Learning with Multimodal Feedback. In: Solis, J., Ng, K. (eds) Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics, vol 74. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22291-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-22291-7_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-22290-0

  • Online ISBN: 978-3-642-22291-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation