Advertisement

Vibrotactile Feedback for an Open Air Music Controller

  • Håkon Knutzen
  • Tellef Kvifte
  • Marcelo M. Wanderley
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8905)

Abstract

In this paper we describe an approach for providing vibrotactile feedback for digital musical instruments (DMIs) that are controlled with open air hand motion. The hand motion was captured with infrared marker based motion capture technology. The marker position data was mapped to the control parameters of both sound and vibrotactile signal synthesis. Vibrotactile feedback was provided to the fingertips of the performer by sending the synthesized signals to voice coils actuators that were embedded in a glove. Vibrotactile strategies were developed for two DMI prototypes that focus on different ways of controlling musical sound. Results of an informal evaluation indicate that the synthesized vibrotactile stimuli can provide useful feedback on how the performer is playing the instrument, as well as enhancing the experience of playing the given DMIs.

Keywords

Haptic feedback Vibrotactile feedback Open air motion Open air music controller Motion capture Digital musical instrument DMI Mapping 

Notes

Acknowledgments

Marcello Giordano, Clayton Mamedes, Mark Zadel, Joseph Malloch, Stephen Sinclair, Aaron Krajeski, Darryl Cameron and Avrum Hollinger at McGill University for helping out with various technical issues. Also, thanks to the IDMIL students that participated in the evaluation.

References

  1. 1.
    Askenfelt, A., Jansson, E.V.: On vibration sensation and finger touch in stringed instrument playing. Music Percept. Interdisc. J. 9(3), 311–349 (1992)CrossRefGoogle Scholar
  2. 2.
    Berdahl, E., Steiner, H.C., Oldham, C.: Practical hardware and algorithms for creating haptic musical instruments. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, Genova, pp. 61–66 (2008)Google Scholar
  3. 3.
    Birnbaum, D.M.: Musical vibrotactile feedback. Master’s thesis, McGill University, Montréal (2007)Google Scholar
  4. 4.
    Birnbaum, D.M., Wanderley, M.M.: A systematic approach to musical vibrotactile feedback. In: Proceedings of the International Computer Music Conference, Copenhagen, vol. 2, pp. 397–404 (2007)Google Scholar
  5. 5.
    Brinkmann, P.: Making Musical Apps: Real-time audio synthesis on Android and iOS. O’Reilly Media Inc, Sebastopol (2012)Google Scholar
  6. 6.
    Brown, L., Brewster, S., Purchase, H.: A first investigation into the effectiveness of tactons. In: Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 167–176 (2005)Google Scholar
  7. 7.
    Bryan, N.J., Herrera, J., Oh, J., Wang, G.: MoMu: a mobile music toolkit. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, Sydney (2010)Google Scholar
  8. 8.
    Chafe, C.: Tactile audio feedback. In: Proceedings of the International Computer Music Conference, Japan, pp. 76–79 (1993)Google Scholar
  9. 9.
    Chafe, C., OModhrain, S.: Musical muscle memory and the haptic display of performance nuance. In: Proceedings of the International Computer Music Conference, Hong Kong, pp. 429–431 (1996)Google Scholar
  10. 10.
    Choi, S., Kuchenbecker, K.: Vibrotactile display: Perception, technology, and applications. In: Proceedings of the IEEE, pp. 1–12. IEEE (2012)Google Scholar
  11. 11.
    Cottle, D.M.: Beginner’s tutorial. In: Wilson, S., Cottle, D., Collins, N. (eds.) The SuperCollider Book, pp. 3–54. MIT Press, London (2011)Google Scholar
  12. 12.
    Egloff, D.C.: A vibrotactile music system based on sensory substitution. Master’s thesis, Rensselaer Polytechnic Institute, Troy (2011)Google Scholar
  13. 13.
    Geldard, F.A., Sherrick, C.E.: The cutaneous “rabbit”: a perceptual illusion. Science 178(4057), 178–179 (1972)CrossRefGoogle Scholar
  14. 14.
    Ghamsari, M., Pras, A., Wanderley, M.M.: Combining musical tasks and improvisation in evaluating novel digital musical instruments. In: Proceedings of the 10th International Symposium on Computer Music Multidisciplinary Research, Marseille, pp. 506–515 (2013)Google Scholar
  15. 15.
    Giordano, M., Wanderley, M.M.: Perceptual and technological issues in the design of vibrotactile-augmented interfaces for music technology and media. In: Oakley, I., Brewster, S. (eds.) HAID 2013. LNCS, vol. 7989, pp. 89–98. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Halata, Z., Baumann, K.I.: Anatomy of receptors. In: Grunwald, M. (ed.) Human Haptic Perception: Basics and Applications, pp. 85–92. Birkhuser, Basel (2008)CrossRefGoogle Scholar
  17. 17.
    Hunt, A., Kirk, R.: Mapping strategies for musical performance (reprint). In: Wanderley, M.M., Battier, M. (eds.) Trends in Gestural Control of Music, pp. 231–258. IRCAM, Centre Pompidou, Paris (2000)Google Scholar
  18. 18.
    Hunt, A., Wanderley, M.M., Kirk, R.: Towards a model for instrumental mapping in expert musical interaction. In: Proceedings of the International Computer Music Conference, Berlin, pp. 209–212 (2000)Google Scholar
  19. 19.
    Hunt, A., Wanderley, M.M., Paradis, M.: The importance of parameter mapping in electronic instrument design. J. New Music Res. 32, 429–440 (2003)CrossRefGoogle Scholar
  20. 20.
    Jensenius, A.R., Wanderley, M.M., Godøy, R.I., Leman, M.: Musical gestures: concepts and methods in research. In: Godøy, R.I., Leman, M. (eds.) Musical Gestures: Sound, Movement, and Meaning, pp. 12–35. Routledge, New York (2010)Google Scholar
  21. 21.
    Jordà, S.: Instruments and players: some thoughts on digital lutherie. J. New Music Res. 3, 321–341 (2004)CrossRefGoogle Scholar
  22. 22.
    Kim, Y., Cha, J., Ryu, J., Oakley, I.: A tactile glove design and authoring system for immersive multimedia. IEEE MultiMed. 17(3), 34–45 (2010)CrossRefGoogle Scholar
  23. 23.
    Knutzen, H.: Haptics in the Air - Exploring vibrotactile feedback for digital musical instruments with open air controllers. Master’s thesis, University of Oslo (2013)Google Scholar
  24. 24.
    Kvifte, T.: On images and representations. In: Instruments and the Electronic Age. Taragot Sounds, Oslo (2007)Google Scholar
  25. 25.
    Kvifte, T.: On the description of mapping structures. J. New Music Res. 37(4), 353–362 (2008)CrossRefGoogle Scholar
  26. 26.
    Kvifte, T., Jensenius, A.R.: Towards a coherent terminology and model of instrument description and design. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, Paris, pp. 220–225 (2006)Google Scholar
  27. 27.
    Libmapper. http://libmapper.github.io/. Accessed 4 July 2014
  28. 28.
    Magee, W.L., Burland, K.: An exploratory study of the use of electronic music technologies in clinical music therapy. Nord. J. Music Ther. 17(2), 124–141 (2008)CrossRefGoogle Scholar
  29. 29.
    Malloch, J., Sinclair, S., Wanderley, M.M.: Libmapper: (a library for connecting things). In: Extended Abstracts on Human Factors in Computing Systems, Paris, pp. 3087–3090 (2013)Google Scholar
  30. 30.
    Mamedes, C.R., Wanderley, M.M., Manzolli, J., Garcia, D.H.L.: Strategies for mapping control in interactive audiovisual installations. In: 10th International Symposium on Computer Music Multidisciplinary Research, Marseille, pp. 766–778 (2013)Google Scholar
  31. 31.
    Marshall, M.T., Wanderley, M.M.: Vibrotactile feedback in digital musical instruments. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, Paris, pp. 226–229 (2006)Google Scholar
  32. 32.
    Mathews, M.V.: The radio baton and conductor program, or: pitch, the most important and least expressive part of music. Comput. Music J. 15(4), 37–46 (1991)CrossRefGoogle Scholar
  33. 33.
    Moss, W., Cunitz, B.: Haptic theremin: developing a haptic musical controller using the sensable phantom omni. In: Proceedings of the International Computer Music Conference, Barcelona, pp. 275–277 (2005)Google Scholar
  34. 34.
    Nymoen, K., Skogstad, S.A., Jensenius, A.R.: SoundSaber - a motion capture instrument. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, Oslo, pp. 312–315 (2011)Google Scholar
  35. 35.
    Oakley, I., McGee, M.R., Brewster, S., Gray, P.: Putting the feel in “look and feel”. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 415–422, New York (2000)Google Scholar
  36. 36.
    Okazaki, R., Hachisu, T., Sato, M., Fukushima, S., Hayward, V., Kajimoto, H.: Judged consonance of tactile and auditory frequencies. In: Proceedings of the IEEE World Haptics Conference, pp. 663–666, Daejeon (2013)Google Scholar
  37. 37.
    O’Modhrain, M.S.: Playing by feel: incorporating haptic feedback into computer-based musical instruments. Ph.D. thesis, Stanford University, Stanford (2001)Google Scholar
  38. 38.
    Paradiso, J.A., Gershenfeld, N.: Musical applications of electric field sensing. Comput. Music J. 21(2), 69–89 (1997)CrossRefGoogle Scholar
  39. 39.
    Park, G., Choi, S.: Perceptual space of amplitude-modulated vibrotactile stimuli. In: IEEE World Haptics Conference, pp. 59–64 (2011)Google Scholar
  40. 40.
    Partan, S., Marler, P.: Communication goes multimodal. Science 283(5406), 1272–1273 (1999)CrossRefGoogle Scholar
  41. 41.
    Parts Express: Hiwave tactile actuator. http://www.parts-express.com/pe/showdetl.cfm?partnumber=297-228. Accessed 4 July 2014
  42. 42.
    Picinali, L., Feakes, C., Mauro, D.A., Katz, B.F.G.: Spectral discrimination thresholds comparing audio and haptics for complex stimuli. In: Magnusson, C., Szymczak, D., Brewster, S. (eds.) HAID 2012. LNCS, vol. 7468, pp. 131–140. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  43. 43.
    de Quay, Y., Skogstad, S., Jensenius, A.: Dance jockey: performing electronic music by dancing. Leonardo Music J. 21, 11–12 (2011)CrossRefGoogle Scholar
  44. 44.
    Rovan, J., Hayward, V.: Typology of tactile sounds and their synthesis in gesture-driven computer music performance. In: Wanderley, M.M., Battier, M. (eds.) Trends in Gestural Control of Music, pp. 355–368. IRCAM, Centre Pompidou, Paris (2000)Google Scholar
  45. 45.
    Russo, F.A., Ammirante, P., Fels, D.I.: Vibrotactile discrimination of musical timbre. J. Exp. Psychol. Hum. Percept. Perform. 38(4), 822–826 (2012)CrossRefGoogle Scholar
  46. 46.
    Sachs, D.M.: A forearm controller and tactile display. Master’s thesis, Massachusetts Institute of Technology, Cambridge (2005)Google Scholar
  47. 47.
    Schacher, J.C.: Gesture control of sounds in 3D space. In: Proceedings of the International Conference on New Inferfaces for Musical Expression, New York, pp. 358–362 (2007)Google Scholar
  48. 48.
    Sparkfun: Class D mono audio amplifier. https://www.sparkfun.com/products/11044. Accessed 4 July 2014
  49. 49.
    Sziebig, G., Solvang, B., Kiss, C., Korondi, P.: Vibro-tactile feedback for VR systems. In: 2nd Conference on Human System Interactions, pp. 406–410. IEEE, Catania (2009)Google Scholar
  50. 50.
    Vallbo, Å.B., Johansson, R.S.: Properties of cutaneous mechanoreceptors in the human hand related to touch sensation. Hum. Neurobiol. 3(1), 3–14 (1984)Google Scholar
  51. 51.
    Verrillo, R.T.: Vibration sensation in humans. Music Percept. 2(3), 281–302 (1992)CrossRefGoogle Scholar
  52. 52.
    Wanderley, M.M., Orio, N.: Evaluation of input devices for musical expression: borrowing tools from HCI. Comput. Music J. 26(3), 62–76 (2002)CrossRefGoogle Scholar
  53. 53.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Håkon Knutzen
    • 1
    • 2
  • Tellef Kvifte
    • 3
  • Marcelo M. Wanderley
    • 2
  1. 1.FourMs Lab, Department of MusicologyUniversity of OsloOsloNorway
  2. 2.IDMIL, CIRMMTMcGill UniversityMontrealCanada
  3. 3.Telemark University CollegeRaulandNorway

Personalised recommendations