Advertisement

Mobile Devices as Musical Instruments - State of the Art and Future Prospects

  • Georg EsslEmail author
  • Sang Won Lee
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11265)

Abstract

Mobile music making has established itself as a maturing field of inquiry, scholarship, and artistic practice. While mobile devices were used before the advent of the iPhone, its introduction no doubt drastically accelerated the field. We take a look at the current state of the art of mobile devices as musical instruments, and explore future prospects for research in the field.

Keywords

Mobile music technology State-of-the-art Survey Prospects 

Notes

Acknowledgments

We are grateful for helpful comments by the reviewers, which helped improve the manuscript.

References

  1. 1.
    Agiwal, M., Roy, A., Saxena, N.: Next generation 5g wireless networks: a comprehensive survey. IEEE Commun. Surveys Tuts. 18(3), 1617–1655 (2016)CrossRefGoogle Scholar
  2. 2.
    Allison, J., Dell, C.: AuRal: a mobile interactive system for geo-locative audio synthesis. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  3. 3.
    Axford, E.C.: Music Apps for Musicians and Music Teachers. Rowman & Littlefield (2015)Google Scholar
  4. 4.
    Baldan, S., Götzen, A.D., Serafin, S.: Sonic tennis: a rhythmic interaction game for mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  5. 5.
    Barraclough, T., Carnegie, D., Kapur, A.: Musical instrument design process for mobile technology. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)Google Scholar
  6. 6.
    Brinkmann, P., et al.: Embedding pure data with libpd. In: Proceedings of the Pure Data Convention (2011)Google Scholar
  7. 7.
    Bryan, N.J., Herrera, J., Oh, J., Wang, G.: MoMu: a mobile music toolkit. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  8. 8.
    Bryan, N.J., Wang, G.: Two turntables and a mobile phone. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  9. 9.
    de Carvalho Junior, A.D., Lee, S.W., Essl, G.: Understanding cloud support for the audience participation concert performance of crowd in c[loud]. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2016)Google Scholar
  10. 10.
    Clément, A.R., Ribeiro, F., Penha, R.: Bridging the gap between performers and the audience using networked smartphones: the a.bel system. In: Proceedings of the International Conference on Live Interfaces (2016)Google Scholar
  11. 11.
    d’Alessandro, N., et al.: A digital mobile choir: joining two interfaces towards composing and performing collaborative mobile music. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  12. 12.
    Derbinsky, N., Essl, G.: Cognitive architecture in mobile music interactions. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  13. 13.
    Derbinsky, N., Essl, G.: Exploring reinforcement learning for mobile percussive collaboration. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  14. 14.
    Dubus, G., Hansen, K.F., Bresin, R.: An overview of sound and music applications for android available on the market. In: Proceedings of the 9th Sound and Music Computing Conference (2012)Google Scholar
  15. 15.
    Erkut, C., Jylhä, A., Serafin, S.: (and sound) of simpe: Showcasing outcomes of a mobile audio programming seminar. In: Workshop on Speech and Sound in Mobile and Pervasive Environments (2013)Google Scholar
  16. 16.
    Essl, G.: UrMus - an environment for mobile instrument design and performance. In: Proceedings of the International Computer Music Conference (2010)Google Scholar
  17. 17.
    Essl, G.: UrSound - live patching of audio and multimedia using a multi-rate normed single-stream data-flow engine. In: Proceedings of the International Computer Music Conference (2010)Google Scholar
  18. 18.
    Essl, G.: Automated ad hoc networking for mobile and hybrid music performance. In: Proceedings of the International Computer Music Conference (2011)Google Scholar
  19. 19.
    Essl, G., Rohs, M., Kratz, S.: Use the force (or something) - pressure and pressure-like input for mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  20. 20.
    Essl, G., Rohs, M.: Interactivity for Mobile Music Making. Organ. Sound 14(2) (2009)CrossRefGoogle Scholar
  21. 21.
    Essl, G., Müller, A.: Designing mobile musical instruments and environments with urMus. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  22. 22.
    Essl, G., Rohs, M., Kratz, S.: Use the force (or something) - pressure and pressure - like input for mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  23. 23.
    Fabiani, M., Dubus, G., Bresin, R.: MoodifierLive: interactive and collaborative expressive music performance on mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  24. 24.
    Gaye, L., Holmquist, L.E., Behrendt, F., Tanaka, A.: Mobile music technology: report on an emerging community. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2006)Google Scholar
  25. 25.
    Gonzalez-Inostroza, M., Sylleros, A., Cádiz, R.F.: Understanding interaction in musical multi-touch surfaces. In: Proceedings of the International Computer Music Conference (2017)Google Scholar
  26. 26.
    Halpern, M., Zhu, Y., Reddi, V.J.: Mobile CPU’s rise to power: quantifying the impact of generational mobile CPU design trends on performance, energy, and user satisfaction. In: IEEE Symposium on High Performance Computer Architecture (2016)Google Scholar
  27. 27.
    Hamilton, R., Smith, J., Wang, G.: Social composition: musical data systems for expressive mobile music. Leonardo Music J. 21, 57–64 (2011)CrossRefGoogle Scholar
  28. 28.
    Hindle, A.: SWARMED: captive portals, mobile devices, and audience participation in multi-user music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  29. 29.
    Hindle, A.: CloudOrch: a portable soundcard in the cloud. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  30. 30.
    Hödl, O., Kayali, F., Fitzpatrick, G.: Designing interactive audience participation using smart phones in a musical performance. In: Proceedings of the International Computer Music Conference (2012)Google Scholar
  31. 31.
    John, D.: Updating the classifications of mobile music projects. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  32. 32.
    Keefe, P.O., Essl, G.: The visual in mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  33. 33.
    Kim, B., Yeo, W.S.: Interactive mobile music performance with digital compass. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  34. 34.
    Kim, H.S., Herrera, J., Wang, G.: Ping-pong: Musically discovering locations. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  35. 35.
    Kimmel, J.: Introduction to mobile and wearable displays. In: Chen, J., Cranton, W., Fihn, M. (eds.) Handbook of Visual Display Technology, pp. 1–12. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-642-35947-7_120-2CrossRefGoogle Scholar
  36. 36.
    Knapp, B., Bortz, B.: MobileMuse: integral music control goes mobile. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  37. 37.
    Kruge, N., Wang, G.: MadPad: a crowdsourcing system for audiovisual sampling. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  38. 38.
    Lazzarini, V., Yi, S., Timoney, J., Keller, D., Pimenta, M.S.: The mobile Csound platform. In: Proceedings of the International Computer Music Conference (2012)Google Scholar
  39. 39.
    Lee, S.W., de Carvalho Jr, A.D., Essl, G.: Crowd in c[loud]: audience participation music with online dating metaphor using cloud service. In: Proceedings of the Web Audio Conference (2016)Google Scholar
  40. 40.
    Lee, S.W., Essl, G.: Live coding the mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  41. 41.
    Lee, S.W., Essl, G.: Communication, control, and state sharing in collaborative live coding. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  42. 42.
    Lee, S.W., Essl, G., Mao, Z.M.: Distributing mobile music applications for audience participation using mobile ad-hoc network (MANET). In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  43. 43.
    Lee, S.W., Freeman, J.: echobo: audience participation using the mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  44. 44.
    Lee, S.W., Srinivasamurthy, A., Tronel, G., Shen, W., Freeman, J.: TOK!: a collaborative acoustic instrument using mobile phones. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  45. 45.
    Liu, D., van der Heide, E.: Interaction models for real-time participatory musical performance using mobile devices. In: Proceedings of the International Computer Music Conference (2017)Google Scholar
  46. 46.
    Martin, C.: Performing with a mobile computer system for vibraphone. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  47. 47.
    Michon, R., Smith, J.O., Wright, M., Chafe, C., Granzow, J., Wang, G.: Passively augmenting mobile devices towards hybrid musical instrument design. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 19–24 (2017)Google Scholar
  48. 48.
    Michon, R., Smith, J.O.I., Wright, M., Chafe, C.: Augmenting the iPad: the BladeAxe. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2016)Google Scholar
  49. 49.
    Michon, R., Smith III, J., Orlarey, Y.: MobileFaust: a set of tools to make musical mobile applications with the faust programming language. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)Google Scholar
  50. 50.
    Miranda, E.R., Wanderley, M.M.: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. A-R Editions (2006)Google Scholar
  51. 51.
    Oh, J., Wang, G.: Audience-participation techniques based on social mobile computing. In: Proceedings of the International Computer Music Conference (2011)Google Scholar
  52. 52.
    Ouzounian, G., Haworth, C., Bennett, P.: Speculative designs: towards a social music. In: Proceedings of the International Computer Music Conference (2017)Google Scholar
  53. 53.
    Park, S., Ban, S., Hong, D.R., Yeo, W.S.: Sound surfing network (SSN): mobile phone-based sound spatialization with audience collaboration. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  54. 54.
    Park, T.H., Nieto, O.: Fortissimo: force-feedback for mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  55. 55.
    Pugliese, R., Tahiroglu, K., Goddard, C., Nesfield, J.: Augmenting human-human interaction in mobile group improvisation. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  56. 56.
    Reus, J.: Crackle: a dynamic mobile multitouch topology for exploratory sound interaction. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)Google Scholar
  57. 57.
    Robaszkiewicz, S., Schnell, N.: Soundworks a playground for artists and developers to create collaborative mobile web performances (2015)Google Scholar
  58. 58.
    Roberts, C.: Control: aoftware for end-user interface programming and interactive performance. In: Proceedings of the International Computer Music Conference (2011)Google Scholar
  59. 59.
    Roberts, C., Wakefield, G., Wright, M.: Mobile controls On-The-Fly: an abstraction for distributed nimes. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  60. 60.
    Salazar, S., Wang, G.: Auraglyph: Handwritten computer music composition and design. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  61. 61.
    Salazar, S., Wang, G.: miniAudicle for iPad: touchscreen-based music software programming. In: Proceedings of the International Computer Music Conference (2014)Google Scholar
  62. 62.
    Savage, N.S., Ali, S.R., Chavez, N.E.: Mmmmm: a multi-modal mobile music mixer. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  63. 63.
    Shaw, T., Piquemal, S., Bowers, J.: Fields: an exploration into the use of mobile devices as a medium for sound diffusion. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)Google Scholar
  64. 64.
    Strylowski, B., Allison, J., Guessford, J.: Pitch Canvas: Touchscreen based mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  65. 65.
    Tanaka, A.: Mapping out instruments, affordances, and mobiles. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)Google Scholar
  66. 66.
    Tanaka, A., Parkinson, A., Settel, Z., Tahiroglu, K.: A survey and thematic analysis approach as input to the design of mobile music GUIs. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)Google Scholar
  67. 67.
    Tarakajian, S., Zicarelli, D., Clayton, J.: Mira: Liveness in iPad controllers for Max/Msp. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)Google Scholar
  68. 68.
    Taylor, B.: A history of the audience as a speaker array. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 481–486 (2017)Google Scholar
  69. 69.
    Taylor, B., et al.: Simplified expressive mobile development with nexusui, nexusup, and nexusdrop. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)Google Scholar
  70. 70.
    Vaillant, G.L., Villée, G., Dutoit, T.: Portable C++ framework for low-latency musical touch interaction with geometrical shapes. In: Proceedings of the International Computer Music Conference (2017)Google Scholar
  71. 71.
    Wang, G., et al.: Smule = sonic media: an intersection of the mobile, musical, and social. In: Proceedings of the International Computer Music Conference (2009)Google Scholar
  72. 72.
    Wang, G.: Ocarina: designing the iphone’s magic flute. Comput. Music. J. 38(2), 8–21 (2014)CrossRefGoogle Scholar
  73. 73.
    Wang, G.: Game design for expressive mobile music. In: Proceedings of the International Conference on New Interfaces for Musical Expression, vol. 16 (2016)Google Scholar
  74. 74.
    Wang, G., Salazar, S., Oh, J., Hamilton, R.: World stage: crowdsourcing paradigm for expressive social mobile music. J. New Music. Res. 44(2) (2015)CrossRefGoogle Scholar
  75. 75.
    Weinberg, G., Beck, A., Godfrey, M.: ZooZBeat: a gesture-based mobile music studio. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2009)Google Scholar
  76. 76.
    Weitzner, N., Freeman, J., Chen, Y.L., Garrett, S.: massMobile: towards a flexible framework for large-scale participatory collaborations in live performances. Organ. Sound 18(1), 30–42 (2013)CrossRefGoogle Scholar
  77. 77.
    Yang, Q.: Not all gestures are created equal: gesture and visual feedback in interaction spaces. Ph.D. thesis, University of Michigan (2015)Google Scholar
  78. 78.
    Yang, Q., Essl, G.: Representation-plurality in multi-touch mobile visual programming for music. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of Wisconsin – MilwaukeeMilwaukeeUSA
  2. 2.University of MichiganAnn ArborUSA

Personalised recommendations