RoboJam: A Musical Mixture Density Network for Collaborative Touchscreen Interaction

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10783)


RoboJam is a machine-learning system for generating music that assists users of a touchscreen music app by performing responses to their short improvisations. This system uses a recurrent artificial neural network to generate sequences of touchscreen interactions and absolute timings, rather than high-level musical notes. To accomplish this, RoboJam’s network uses a mixture density layer to predict appropriate touch interaction locations in space and time. In this paper, we describe the design and implementation of RoboJam’s network and how it has been integrated into a touchscreen music app. A preliminary evaluation analyses the system in terms of training, musical generation and user interaction.


Artificial neural networks Musical artificial intelligence Mobile music Collaboration Intelligent agents 



Supported by The Research Council of Norway as a part of the Engineering Predictability with Embodied Cognition (EPEC) project, under grant agreement 240862.


  1. 1.
    Mozer, M.C.: Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing. Connection Science 6(2–3), 247–280 (1994)CrossRefGoogle Scholar
  2. 2.
    Eck, D., Schmidhuber, J.: A first look at music composition using LSTM recurrent neural networks. Tech. Rep. IDSIA-07-02, Instituto Dalle Molle di studi sull’ intelligenza artificiale, Manno, Switzerland (2007)Google Scholar
  3. 3.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  4. 4.
    Karpathy, A.: The unreasonable effectiveness of recurrent neural networks. Published on Andrej Karpathy’s blog (May 2015),
  5. 5.
    Sturm, B.L., Santos, J.F., Ben-Tal, O., Korshunova, I.: Music transcription modelling and composition using deep learning. In: Proceedings of the 1st Conference on Computer Simulation of Musical Creativity (2016)Google Scholar
  6. 6.
    Hadjeres, G., Pachet, F., Nielsen, F.: DeepBach: a steerable model for Bach chorales generation. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 1362–1371. PMLR, International Convention Centre, Sydney, Australia (06–11 Aug 2017),
  7. 7.
    Colombo, F., Seeholzer, A., Gerstner, W.: Deep Artificial Composer: A Creative Neural Network Model for Automated Melody Generation. In: Correia, J., Ciesielski, V., Liapis, A. (eds.) EvoMUSART 2017. LNCS, vol. 10198, pp. 81–96. Springer, Cham (2017). CrossRefGoogle Scholar
  8. 8.
    Malik, I., Ek, C.H.: Neural translation of musical style. ArXiv e-prints (Aug 2017),
  9. 9.
    Hutchings, P., McCormack, J.: Using Autonomous Agents to Improvise Music Compositions in Real-Time. In: Correia, J., Ciesielski, V., Liapis, A. (eds.) EvoMUSART 2017. LNCS, vol. 10198, pp. 114–127. Springer, Cham (2017). CrossRefGoogle Scholar
  10. 10.
    Ha, D., Eck, D.: A neural representation of sketch drawings. ArXiv e-prints (Apr 2017),
  11. 11.
    Graves, A.: Generating sequences with recurrent neural networks. ArXiv e-prints (Aug 2013),
  12. 12.
    Bishop, C.M.: Mixture density networks. Tech. Rep. NCRG/97/004, Neural Computing Research Group, Aston University (1994)Google Scholar
  13. 13.
    Brando, A.: Mixture Density Networks (MDN) for distribution and uncertainty estimation. Master’s thesis, Universitat Politècnica de Catalunya (2017),
  14. 14.
    Biles, J.A.: Improvizing with genetic algorithms: GenJam. In: Miranda, E.R., Biles, J.A. (eds.) Evolutionary Computer Music, pp. 137–169. Springer, London (2007). CrossRefGoogle Scholar
  15. 15.
    Marchini, M., Pachet, F., Carré, B.: Rethinking reflexive looper for structured pop music. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 139–144. Aalborg University Copenhagen, Copenhagen (2017),
  16. 16.
    Pachet, F.: The continuator: Musical interaction with style. Journal of New Music Research 32(3), 333–341 (2003)CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Jensenius, A.R., Wanderley, M.M., Godøy, R.I., Leman, M.: Musical gestures: Concepts and methods in research. In: Musical Gestures: Sound, Movement, and Meaning. Routledge (2010)Google Scholar
  19. 19.
    Martin, C., Swift, B., Gardner, H.: Metatone-analysis: Touchscreen data corpus. Git Repository (2017),

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Robotics and Intelligent Systems (ROBIN) Group, Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations