Genetic Programming and Evolvable Machines

, Volume 18, Issue 4, pp 433–465 | Cite as

Affective evolutionary music composition with MetaCompose

  • Marco ScireaEmail author
  • Julian Togelius
  • Peter Eklund
  • Sebastian Risi


This paper describes the MetaCompose music generator, a compositional, extensible framework for affective music composition. In this context ‘affective’ refers to the music generator’s ability to express emotional information. The main purpose of MetaCompose is to create music in real-time that can express different mood-states, which we achieve through a unique combination of a graph traversal-based chord sequence generator, a search-based melody generator, a pattern-based accompaniment generator, and a theory for mood expression. Melody generation uses a novel evolutionary technique combining FI-2POP with multi-objective optimization. This allows us to explore a Pareto front of diverse solutions that are creatively equivalent under the terms of a multi-criteria objective function. Two quantitative user studies were performed to evaluate the system: one focusing on the music generation technique, and the other that explores valence expression, via the introduction of dissonances. The results of these studies demonstrate (i) that each part of the generation system improves the perceived quality of the music produced, and (ii) how valence expression via dissonance produces the perceived affective state. This system, which can reliably generate affect-expressive music, can subsequently be integrated in any kind of interactive application (e.g., games) to create an adaptive and dynamic soundtrack.


Evolutionary computing Genetic algorithm Music generation Affective music Creative computing 


  1. 1.
    S. Abrams, D.V. Oppenheim, D. Pazel, J. Wright, et al. Higher-level composition control in music sketcher: modifiers and smart harmony, in Proceedings of the ICMC. Citeseer (1999)Google Scholar
  2. 2.
    A. Alpern, Techniques for algorithmic composition of music (1995),
  3. 3.
    D. Arsenault, Guitar hero:” not like playing guitar at all. J. Can. Game Stud. Assoc. 1(2), 1–7 (2008)Google Scholar
  4. 4.
    J.J. Aucouturier, F. Pachet, M. Sandler, the way it sounds: timbre models for analysis and retrieval of music signals. IEEE Trans. Multimed. 7(6), 1028–1035 (2005)CrossRefGoogle Scholar
  5. 5.
    C.P.E. Bach, W.J. Mitchell, W. John, Essay on the True Art of Playing Keyboard Instruments (WW Norton, New York, 1949)Google Scholar
  6. 6.
    C.D. Batson, L.L. Shaw, K.C. Oleson, Differentiating affect, mood, and emotion: toward functionally based conceptual distinctions, in Emotion (Sage Publications, Inc., Thousand Oaks, CA, 1992), pp. 294–326Google Scholar
  7. 7.
    C. Beedie, P. Terry, A. Lane, Distinctions between emotion and mood. Cognit. Emot. 19(6), 847–878 (2005)CrossRefGoogle Scholar
  8. 8.
    J. Biles, Genjam: a genetic algorithm for generating jazz solos, in Proceedings of the International Computer Music Conference (International Computer Music Association, 1994), pp. 131–131Google Scholar
  9. 9.
    D. Birchfield, Generative model for the creation of musical emotion, meaning, and form, in Proceedings of the 2003 ACM SIGMM Workshop on Experiential Telepresence (2003), pp. 99–104Google Scholar
  10. 10.
    O. Bown, Experiments in modular design for the creative composition of live algorithms. Comput. Music J. 35(3), 73–85 (2011)CrossRefGoogle Scholar
  11. 11.
    C.R. Brewin, Cognitive change processes in psychotherapy. Psychol. Rev. 96(3), 379 (1989)CrossRefGoogle Scholar
  12. 12.
    D. Brown, Mezzo: an adaptive, real-time composition program for game soundtracks, in Proceedings of the AIIDE 2012 Workshop on Musical Metacreation (2012), pp. 68–72Google Scholar
  13. 13.
    G.C. Bruner, Music, mood, and marketing. J. Mark. 1, 94–104 (1990)Google Scholar
  14. 14.
    D. Butler, An historical investigation and bibliography of nineteenth century music psychology literature. Ph.D. thesis, Ohio State University (1973)Google Scholar
  15. 15.
    T. Byron, C. Stevens, Steps and leaps in human memory for melodies: the effect of pitch interval magnitude in a melodic contour discrimination task, in 9th International Conference on Music Perception and Cognition (ICMPC9), Bologna, Italy (Citeseer, 2006)Google Scholar
  16. 16.
    D. Chafekar, J. Xuan, K. Rasheed, Constrained multi-objective optimization using steady state genetic algorithms, in Genetic and Evolutionary Computation GECCO (Springer, 2003), pp. 813–824Google Scholar
  17. 17.
    H. Chan, D.A. Ventura, Automatic composition of themed mood pieces, in Proceedings of the International Joint Workshop on Computational Creativity (2008), pp. 19–115Google Scholar
  18. 18.
    K. Collins, An introduction to procedural music in video games. Contemp. Music Rev. 28(1), 5–15 (2009). doi: 10.1080/07494460802663983 MathSciNetCrossRefGoogle Scholar
  19. 19.
    D. Cope, Algorithmic music composition, in Patterns of Intuition, ed. by G. Nierhaus (Springer Netherlands, 2015), pp. 405–416. doi: 10.1007/978-94-017-9561-6_19
  20. 20.
    P. Dahlstedt, Autonomous evolution of complete piano pieces and performances, in Proceedings of Music AL Workshop (Citeseer, 2007)Google Scholar
  21. 21.
    B. De Haas, R.C. Veltkamp, F. Wiering, Tonal pitch step distance: a similarity measure for chord progressions, in ISMIR (2008), pp. 51–56Google Scholar
  22. 22.
    K. Deb, Multi-objective Optimization Using Evolutionary Algorithms, vol. 16 (Wiley, Hoboken, 2001)zbMATHGoogle Scholar
  23. 23.
    K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: Nsga-II. IEEE Trans. Evolut. Comput. 6(2), 182–197 (2002)CrossRefGoogle Scholar
  24. 24.
    K. Deb, A. Pratap, T. Meyarivan, Constrained test problems for multi-objective evolutionary optimization, in Evolutionary Multi-Criterion Optimization (Springer, 2001), pp. 284–298Google Scholar
  25. 25.
    P. Doornbusch, A brief survey of mapping in algorithmic composition, in Proceedings of the International Computer Music Conference (2002),
  26. 26.
    M. Edwards, Algorithmic composition: computational thinking in music. Commun. ACM 54(7), 58–67 (2011). doi: 10.1145/1965724.1965742 CrossRefGoogle Scholar
  27. 27.
    A.E. Eiben, J. Smith, From evolutionary computation to the evolution of things. Nature 521(7553), 476–482 (2015)CrossRefGoogle Scholar
  28. 28.
    P. Ekman, Are there basic emotions? Psychol. Rev. 99(3), 550–553 (1992). doi: 10.1037/0033-295X.99.3.550 CrossRefGoogle Scholar
  29. 29.
    P. Ekman, An argument for basic emotions. Cognit. Emot. 6(3–4), 169–200 (1992)CrossRefGoogle Scholar
  30. 30.
    M. Eladhari, R. Nieuwdorp, M. Fridenfalk, The soundtrack of your mind: mind music-adaptive audio for game characters, in Proceedings of Advances in Computer Entertainment Technology (2006)Google Scholar
  31. 31.
    B. Eno, The ship (2016),
  32. 32.
    P.R. Farnsworth, The Social Psychology of Music (Dryden, Oxford, 2003), p. 304Google Scholar
  33. 33.
    A. Gabrielsson, P.N. Juslin, Emotional Expression in Music (Oxford University Press, Oxford, 2003)Google Scholar
  34. 34.
    J.M. Grey, J.W. Gordon, Perceptual effects of spectral modifications on musical timbres. J. Acoust. Soc. Am. 63(5), 1493–1500 (1978)CrossRefGoogle Scholar
  35. 35.
    R.H. Gundlach, Factors determining the characterization of musical phrases. Am. J. Psychol. 47(4), 624–643 (1935)CrossRefGoogle Scholar
  36. 36.
    K. Hevner, The affective character of the major and minor modes in music. Am. J. Psychol. 47(1), 103–118 (1935)CrossRefGoogle Scholar
  37. 37.
    K. Hevner, Experimental studies of the elements of expression in music. Am. J. Psychol. 48(2), 246–268 (1936)CrossRefGoogle Scholar
  38. 38.
    K. Hevner, The affective value of pitch and tempo in music. Am. J. Psychol. 49(4), 621–630 (1937)CrossRefGoogle Scholar
  39. 39.
    G. Husain, W.F. Thompson, E.G. Schellenberg, Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. Interdiscip. J. 20(2), 151–171 (2002)CrossRefGoogle Scholar
  40. 40.
    G. Ilie, W.F. Thompson, A comparison of acoustic cues in music and speech for three dimensions of affect. Music Percept. Interdiscip. J. 23(4), 319–330 (2006)CrossRefGoogle Scholar
  41. 41.
    A. Isaacs, T. Ray, W. Smith, Blessings of maintaining infeasible solutions for constrained multi-objective optimization problems, in IEEE Congress on Evolutionary Computation (IEEE, 2008), pp. 2780–2787Google Scholar
  42. 42.
    F. Jimenez, A.F. Gómez-Skarmeta, G. Sánchez, K. Deb, An evolutionary algorithm for constrained multi-objective optimization. In: Proceedings of the Congress on Evolutionary Computation, pp. 1133–1138. IEEE (2002)Google Scholar
  43. 43.
    P.N. Juslin, S. Liljeström, D. Västfjäll, G. Barradas, A. Silva, An experience sampling study of emotional reactions to music: listener, music, and situation. Emotion 8(5), 668 (2008)CrossRefGoogle Scholar
  44. 44.
    H. Katayose, M. Imai, S. Inokuchi, Sentiment extraction in music, in Proceedings of the 9th International Conference on Pattern Recognition (1988), pp. 1083–1087Google Scholar
  45. 45.
    S.O. Kimbrough, G.J. Koehler, M. Lu, D.H. Wood, On a feasible-infeasible two-population (fi-2pop) genetic algorithm for constrained optimization: distance tracing and no free lunch. Eur. J. Oper. Res. 190(2), 310–327 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    A. Kirke, E.R. Miranda, A survey of computer systems for expressive music performance. ACM Comput. Surv. 42(1), 3:1–3:41 (2009). doi: 10.1145/1592451.1592454 CrossRefGoogle Scholar
  47. 47.
    V.J. Konečni, Does music induce emotion? A theoretical and methodological analysis. Psychol. Aesthet. Creat. Arts 2(2), 115 (2008)CrossRefGoogle Scholar
  48. 48.
    A.E. Krause, A.C. North, L.Y. Hewitt, Music-listening in everyday life: devices and choice. Psychol. Music 43(2), 155–170 (2015)CrossRefGoogle Scholar
  49. 49.
    G. Kreutz, U. Ott, D. Teichmann, P. Osawa, D. Vaitl, Using music to induce emotions: influences of musical preference and absorption. Psychol. Music 36(1), 101–126 (2008)CrossRefGoogle Scholar
  50. 50.
    C.L. Krumhansl, An exploratory study of musical emotions and psychophysiology. Can. J. Exp. Psychol. Revue canadienne de psychologie expérimentale 51(4), 336 (1997)CrossRefGoogle Scholar
  51. 51.
    C.G. Lange, W. James, The Emotions (Williams & Wilkins, Baltimore, 1922)CrossRefGoogle Scholar
  52. 52.
    T. Langlois, G. Marques, A music classification method based on timbral features, in ISMIR (2009), pp. 81–86Google Scholar
  53. 53.
    R.S. Lazarus, Emotion and Adaptation (Oxford University Press, Oxford, 1991)Google Scholar
  54. 54.
    F. Lerdahl, Tonal pitch space. Music Percept. 5, 315–349 (1988)CrossRefGoogle Scholar
  55. 55.
    J.S. Lerner, D. Keltner, Beyond valence: toward a model of emotion-specific influences on judgement and choice. Cognit. Emot. 14(4), 473–493 (2000)CrossRefGoogle Scholar
  56. 56.
    E. Lindström, P.N. Juslin, R. Bresin, A. Williamon, Expressivity comes from within your soul: a questionnaire study of music students’ perspectives on expressivity. Res. Stud. Music Educ. 20(1), 23–47 (2003)CrossRefGoogle Scholar
  57. 57.
    D. Liu, L. Lu, H.J. Zhang, Automatic mood detection from acoustic music data, in Proceedings of the International Symposium on Music Information Retrieval (2003), pp. 81–87Google Scholar
  58. 58.
    S.R. Livingstone, A.R. Brown, Dynamic response: real-time adaptation for music emotion, in Proceedings of the 2nd Australasian Conference on Interactive Entertainment (2005), pp. 105–111Google Scholar
  59. 59.
    R. Loughran, J. McDermott, M. O’Neill, Tonality driven piano compositions with grammatical evolution, in IEEE Congress on Evolutionary Computation (CEC) (IEEE, 2015), pp. 2168–2175Google Scholar
  60. 60.
    B.A. Martin, The influence of gender on mood effects in advertising. Psychol. Mark. 20(3), 249–273 (2003)CrossRefGoogle Scholar
  61. 61.
    H.P. Martinez, G.N. Yannakakis, J. Hallam, Don’t classify ratings of affect; rank them!. IEEE Trans. Affect. Comput. 5(3), 314–326 (2014)CrossRefGoogle Scholar
  62. 62.
    S.K. Meier, J.L. Briggs, System for real-time music composition and synthesis. US Patent 5,496,962 (1996)Google Scholar
  63. 63.
    L.B. Meyer, Emotion and Meaning in Music (University of Chicago Press, Chicago, 2008)Google Scholar
  64. 64.
    K. Miller, Schizophonic performance: guitar hero, rock band, and virtual virtuosity. J. Soc. Am. Music 3(04), 395–429 (2009)CrossRefGoogle Scholar
  65. 65.
    E.R. Miranda, Readings in Music and Artificial Intelligence, vol. 20 (Routledge, London, 2013)Google Scholar
  66. 66.
    E.R. Miranda, A. Biles, Evolutionary Computer Music (Springer, Berlin, 2007)CrossRefGoogle Scholar
  67. 67.
    K. Monteith, T. Martinez, D. Ventura, Automatic generation of music for inducing emotive response, in Proceedings of the International Conference on Computational Creativity (Citeseer, 2010), pp. 140–149Google Scholar
  68. 68.
    S. Mugglin, Chord charts and maps, Accessed 14 Sept 2015
  69. 69.
    A.C. North, D.J. Hargreaves, J.J. Hargreaves, Uses of music in everyday life. Music Percept. Interdiscip. J. 22(1), 41–77 (2004)CrossRefGoogle Scholar
  70. 70.
    G. Papadopoulos, G. Wiggins, AI methods for algorithmic composition: a survey, a critical view and future prospects, in AISB Symposium on Musical Creativity, Edinburgh, UK (1999), pp. 110–117Google Scholar
  71. 71.
    G. Perle, Serial Composition and Atonality: An Introduction to the Music of Schoenberg, Berg, and Webern (Univ of California Press, Berkeley, 1972)Google Scholar
  72. 72.
    J. Posner, J.A. Russell, B.S. Peterson, The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(03), 715–734 (2005)CrossRefGoogle Scholar
  73. 73.
    M. Puckette, et al. Pure data: another integrated computer music environment, in Proceedings of the Second Intercollege Computer Music Concerts (1996), pp. 37–41Google Scholar
  74. 74.
    A.P. Rigopulos, E.B. Egozy, Real-time music creation system. US Patent 5,627,335 (1997)Google Scholar
  75. 75.
    J. Robertson, A. de Quincey, T. Stapleford, G. Wiggins, Real-time music generation for a virtual environment, in Proceedings of ECAI-98 Workshop on AI/Alife and Entertainment (Citeseer, 1998)Google Scholar
  76. 76.
    R. Rosenthal, D.B. Rubin, A simple, general purpose display of magnitude of experimental effect. J. Educ. Psychol. 74(2), 166 (1982)CrossRefGoogle Scholar
  77. 77.
    J.A. Russell, A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  78. 78.
    E.G. Schellenberg, A.M. Krysciak, R.J. Campbell, Perceiving emotion in melody: interactive effects of pitch and rhythm. Music Percept. Interdiscip. J. 18(2), 155–171 (2000)CrossRefGoogle Scholar
  79. 79.
    K.R. Scherer, A. Schorr, T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research (Oxford University Press, Oxford, 2001)Google Scholar
  80. 80.
    H. Schlosberg, Three dimensions of emotion. Psychol. Rev. 61(2), 81 (1954)CrossRefGoogle Scholar
  81. 81.
    M. Scirea, Mood dependent music generator, in Proceedings of Advances in Computer Entertainment (2013), pp. 626–629Google Scholar
  82. 82.
    M. Scirea, G.A. Barros, N. Shaker, J. Togelius, Smug: scientific music generator, in Proceedings of the Sixth International Conference on Computational Creativity (2015), p. 204Google Scholar
  83. 83.
    M. Scirea, M.J. Nelson, J. Togelius, Moody music generator: characterising control parameters using crowdsourcing, in Evolutionary and Biologically Inspired Music, Sound, Art and Design (Springer, 2015), pp. 200–211Google Scholar
  84. 84.
    M. Scirea, J. Togelius, P. Eklund, S. Risi, Metacompose: a compositional evolutionary music composer, in International Conference on Evolutionary and Biologically Inspired Music and Art (Springer, 2016), pp. 202–217Google Scholar
  85. 85.
    J.A. Sloboda, S.A. O’Neill, Emotions in everyday listening to music, in Music and Emotion: Theory and Research (Oxford University Press, New York, NY, 2001), pp. 415–429Google Scholar
  86. 86.
    A. Smaill, G. Wiggins, M. Harris, Hierarchical music representation for composition and analysis. Comput. Humanit. 27(1), 7–17 (1993)CrossRefGoogle Scholar
  87. 87.
    K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
  88. 88.
    R.E. Thayer, The Biopsychology of Mood and Arousal (Oxford University Press, Oxford, 1989)Google Scholar
  89. 89.
    S.S. Tomkins, Affect Imagery Consciousness: Volume I: The Positive Affects, vol. 1 (Springer, Berlin, 1962)Google Scholar
  90. 90.
    G.T. Toussaint, et al. The Euclidean algorithm generates traditional musical rhythms, in Proceedings of BRIDGES: Mathematical Connections in Art, Music and Science (2005), pp. 47–56Google Scholar
  91. 91.
    L.J. Trainor, B.M. Heinmiller, The development of evaluative responses to music: infants prefer to listen to consonance over dissonance. Infant Behav. Dev. 21(1), 77–88 (1998)CrossRefGoogle Scholar
  92. 92.
    D. Watson, A. Tellegen, Toward a consensual structure of mood. Psychol. Bull. 98(2), 219 (1985)CrossRefGoogle Scholar
  93. 93.
    G. Wiggins, M. Harris, A. Smaill, Representing music for analysis and composition. University of Edinburgh, Department of Artificial Intelligence (1990)Google Scholar
  94. 94.
    R. Wooller, A.R. Brown, E. Miranda, J. Diederich, R. Berry, A framework for comparison of process in algorithmic music systems, in Generative Arts Practice 2005—A Creativity & Cognition Symposium (2005)Google Scholar
  95. 95.
    W. Wundt, Outlines of Psychology (Springer, Berlin, 1980)CrossRefGoogle Scholar
  96. 96.
    G.N. Yannakakis, J. Togelius, Experience-driven procedural content generation. IEEE Trans. Affect. Comput. 2(3), 147–161 (2011)CrossRefGoogle Scholar
  97. 97.
    E. Zitzler, K. Deb, L. Thiele, Comparison of multiobjective evolutionary algorithms: empirical results. Evol. Comput. 8(2), 173–195 (2000)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.IT University of CopenhagenCopenhagenDenmark
  2. 2.New York UniversityNew YorkUSA

Personalised recommendations