Using Autonomous Agents to Improvise Music Compositions in Real-Time

  • Patrick HutchingsEmail author
  • Jon McCormack
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10198)


This paper outlines an approach to real-time music generation using melody and harmony focused agents in a process inspired by jazz improvisation. A harmony agent employs a Long Short-Term Memory (LSTM) artificial neural network trained on the chord progressions of 2986 jazz ‘standard’ compositions using a network structure novel to chord sequence analysis. The melody agent uses a rule-based system of manipulating provided, pre-composed melodies to improvise new themes and variations. The agents take turns in leading the direction of the composition based on a rating system that rewards harmonic consistency and melodic flow. In developing the multi-agent system it was found that implementing embedded spaces in the LSTM encoding process resulted in significant improvements to chord sequence learning.


Multi-agent systems Music composition Artificial neural networks 


  1. 1.
    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems (2015)., software available from
  2. 2.
    Barrett, F.J.: Coda–creativity and improvisation in jazz and organizations: implications for organizational learning. Organ. Sci. 9(5), 605–622 (1998)CrossRefGoogle Scholar
  3. 3.
    Bastien, D.T., Hostager, T.J.: Jazz as a process of organizational innovation. Commun. Res. 15(5), 582–602 (1988)CrossRefGoogle Scholar
  4. 4.
    Berliner, P.: Thinking in jazz: composing in the moment. Jazz Educ. J. 26, 241 (1994)Google Scholar
  5. 5.
    Biles, J.A.: Genjam in transition: from genetic jammer to generative jammer. In: Generative Art, vol. 2002 (2002)Google Scholar
  6. 6.
    Choi, K., Fazekas, G., Sandler, M.: Text-based LSTM networks for automatic music composition. arXiv preprint arXiv:1604.05358 (2016)
  7. 7.
    Eck, D., Schmidhuber, J.: A first look at music composition using LSTM recurrent neural networks. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale 103 (2002)Google Scholar
  8. 8.
    Eigenfeldt, A., Pasquier, P.: A realtime generative music system using autonomous melody, harmony, and rhythm agents. In: XIII Internationale Conference on Generative Arts, Milan, Italy (2009)Google Scholar
  9. 9.
    Eigenfeldt, A., Pasquier, P.: Realtime generation of harmonic progressions using controlled Markov selection. In: Proceedings of ICCC-X-Computational Creativity Conference, pp. 16–25 (2010)Google Scholar
  10. 10.
    Folkestad, G., Hargreaves, D.J., Lindström, B.: Compositional strategies in computer-based music-making. Br. J. Music Educ. 15(01), 83–97 (1998)CrossRefGoogle Scholar
  11. 11.
    Gers, F.A., Schraudolph, N.N., Schmidhuber, J.: Learning precise timing with LSTM recurrent networks. J. Mach. Learn. Res. 3, 115–143 (2002)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Johnson-Laird, P.N.: How jazz musicians improvise. Music Percept. Interdisc. J. 19(3), 415–442 (2002)CrossRefGoogle Scholar
  13. 13.
    Keller, R.M., Morrison, D.R.: A grammatical approach to automatic improvisation. In: Proceedings, Fourth Sound and Music Conference, Lefkada, Greece, July. Most of the soloists at Birdland had to wait for Parker’s next record in order to find out what to play next. What will they do now (2007)Google Scholar
  14. 14.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  15. 15.
    Levine, M.: The Jazz Theory Book. O’Reilly Media Inc., Sebastopol (2011)Google Scholar
  16. 16.
    Monson, I.: Jazz as political and musical practice. In: Musical Improvisation: Art, Education, and Society, pp. 21–37 (2009)Google Scholar
  17. 17.
    Pachet, F.: Enhancing individual creativity with interactive musical reflexive systems. In: Musical Creativity, pp. 359–375 (2006)Google Scholar
  18. 18.
    Pachet, F., Roy, P.: Imitative leadsheet generation with user constraints. In: ECAI, pp. 1077–1078 (2014)Google Scholar
  19. 19.
    Papadopoulos, A., Roy, P., Pachet, F.: Assisted lead sheet composition using FlowComposer. In: Rueher, M. (ed.) CP 2016. LNCS, vol. 9892, pp. 769–785. Springer, Cham (2016). doi: 10.1007/978-3-319-44953-1_48 CrossRefGoogle Scholar
  20. 20.
    Plans, D., Morelli, D.: Experience-driven procedural music generation for games. IEEE Trans. Comput. Intell. AI Games 4(3), 192–198 (2012)CrossRefGoogle Scholar
  21. 21.
    Rendel, A., Fernandez, R., Hoory, R., Ramabhadran, B.: Using continuous lexical embeddings to improve symbolic-prosody prediction in a text-to-speech front-end. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5655–5659. IEEE (2016)Google Scholar
  22. 22.
    Sturm, B.L., Santos, J.F., Ben-Tal, O., Korshunova, I.: Music transcription modelling and composition using deep learning. arXiv preprint arXiv:1604.08723 (2016)

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.sensiLab, Faculty of Information TechnologyMonash UniversityCaulfield EastAustralia

Personalised recommendations