Integration of nonparametric fuzzy classification with an evolutionary-developmental framework to perform music sentiment-based analysis and composition

  • Ralph Abboud
  • Joe TekliEmail author
Methodologies and Application


Over the past years, several approaches have been developed to create algorithmic music composers. Most existing solutions focus on composing music that appears theoretically correct or interesting to the listener. However, few methods have targeted sentiment-based music composition: generating music that expresses human emotions. The few existing methods are restricted in the spectrum of emotions they can express (usually to two dimensions: valence and arousal) as well as the level of sophistication of the music they compose (usually monophonic, following translation-based, predefined templates or heuristic textures). In this paper, we introduce a new algorithmic framework for autonomous music sentiment-based expression and composition, titled MUSEC, that perceives an extensible set of six primary human emotions (e.g., anger, fear, joy, love, sadness, and surprise) expressed by a MIDI musical file and then composes (creates) new polyphonic (pseudo) thematic, and diversified musical pieces that express these emotions. Unlike existing solutions, MUSEC is: (i) a hybrid crossover between supervised learning (SL, to learn sentiments from music) and evolutionary computation (for music composition, MC), where SL serves at the fitness function of MC to compose music that expresses target sentiments, (ii) extensible in the panel of emotions it can convey, producing pieces that reflect a target crisp sentiment (e.g., love) or a collection of fuzzy sentiments (e.g., 65% happy, 20% sad, and 15% angry), compared with crisp-only or two-dimensional (valence/arousal) sentiment models used in existing solutions, (iii) adopts the evolutionary-developmental model, using an extensive set of specially designed music-theoretic mutation operators (trille, staccato, repeat, compress, etc.), stochastically orchestrated to add atomic (individual chord-level) and thematic (chord pattern-level) variability to the composed polyphonic pieces, compared with traditional evolutionary solutions producing monophonic and non-thematic music. We conducted a large battery of tests to evaluate MUSEC’s effectiveness and efficiency in both sentiment analysis and composition. It was trained on a specially constructed set of 120 MIDI pieces, including 70 sentiment-annotated pieces: the first significant dataset of sentiment-labeled MIDI music made available online as a benchmark for future research in this area. Results are encouraging and highlight the potential of our approach in different application domains, ranging over music information retrieval, music composition, assistive music therapy, and emotional intelligence.


Music sentiment analysis MIDI Evolutionary algorithms Algorithmic composition Supervised learning Fuzzy classification 



This study is partly funded by the National Council for Scientific Research - Lebanon (CNRS-L, grant number: NCSRLAU#887), by LAU (Grant Number: SOERC1516R003), as well as the Fulbright Visiting Scholar program (sponsored by the US Department of State, grant number: PS00232737). Special thanks go to music experts: Anthony Bou Fayad, Robert Lamah, and Joseph Khalifé, who helped evaluate the synthetic compositions, as well as Jean Marie Riachi for his participation in a live demonstration of the system. We would also like to thank the non-expert testers (including LAU students, faculty, staff, and friends) who volunteered to participate in the experimental evaluation.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with animals performed by any of the authors.

Informed consent

Additional informed consent was obtained from all individual participants for whom identifying information is included in this article.


  1. Abbasi A, Chen H, Thoms S, Fu T (2008) Affect analysis of web forums and blogs using correlation ensembles. IEEE Trans Knowl Data Eng 20(9):1168–1180CrossRefGoogle Scholar
  2. Abboud R, Tekli J (2018) MUSE prototype for music sentiment expression. In: IEEE international conference on cognitive computing (ICCC'18). San Francisco, pp 106–109Google Scholar
  3. Abu AO (2017) Adaptation of reproducing kernel algorithm for solving fuzzy Fredholm–Volterra integrodifferential equations. Neural Comput Appl 28(7):1591–1610CrossRefGoogle Scholar
  4. Abu AO, Abo-Hammour ZS (2014) Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm. Inf Sci 279:396–415MathSciNetzbMATHCrossRefGoogle Scholar
  5. Abu AO, Al-Smadi M, Momani S, Hayat T (2016) Numerical solutions of fuzzy differential equations using reproducing kernel Hilbert space method. Soft Comput 20(8):3283–3302zbMATHCrossRefGoogle Scholar
  6. Adiloglu K, Alpaslan FN (2007) A Machine Learning Approach to Two-Voice Counterpoint Composition. Knowl-Based Syst 20(3):300–309CrossRefGoogle Scholar
  7. Amin F, Fahmi A, Abdullah S, Ali A, Ahmed R, Ghani F (2018) Triangular cubic linguistic hesitant fuzzy aggregation operators and their application in group decision making. J Intell Fuzzy Syst 34(4):2401–2416Google Scholar
  8. Ayadi MG, Bouslimi R, Akaichi J (2016) A medical image retrieval scheme with relevance feedback through a medical social network. Soc Netw Anal Min 6(1):53:1–53:23CrossRefGoogle Scholar
  9. Baeza-Yates R, Ribeiro-Neto B (2011) Modern information retrieval: the concepts and technology behind search, 2nd edn. ACM Press Books, New YorkGoogle Scholar
  10. Barrett FS, Grimm KJ, Robins RW, Wildschut T, Sedikides C (2010) Music-evoked nostalgia: affect, memory, and personality. Emotion 10(3):390–403CrossRefGoogle Scholar
  11. Bas De Haas W, Veltkamp RC, Wiering F (2008) Tonal pitch step distance: a similarity measure for chord progressions. In: International society of music information retrieval (ISMIR), pp 51–56Google Scholar
  12. Bas De Haas W, Wiering F, Veltkamp RC (2013) A geometrical distance measure for determining the similarity of musical harmony. Int J Multimed Inf Retr 2(3):189–202CrossRefGoogle Scholar
  13. Berrett LF (2017) How emotions are made: the secret life of the brain. Macmillan, LondonGoogle Scholar
  14. Boden MA (1994) Precis of the creative mind: myths and mechanisms. Behav Brain Sci 17(3):519–570MathSciNetCrossRefGoogle Scholar
  15. Bradley M, Lang P (1999) Affective norms for English Words (ANEW): instruction manual and affective ratings. Technical report C-1, Center for Research in Psychophysiology, University of FloridaGoogle Scholar
  16. Burton AR (1998) A hybrid neuro-genetic pattern evolution system applied to musical composition. Ph.D. thesis, University of Surrey, UKGoogle Scholar
  17. Cai Z, Hu H (2018) Session-aware music recommendation via a generative model approach. Soft Comput 22(3):1023–1031zbMATHCrossRefGoogle Scholar
  18. Cao Y, Jia L, Chen Y, Lin N, Yang C, Zhang B, Liu Z, Li X, Dai H (2019) Recent advances of generative adversarial networks in computer vision. IEEE Access 7:14985–15006CrossRefGoogle Scholar
  19. Carnie A (2013) Syntax: a generative introduction, 3rd edn. Wiley, MaldenGoogle Scholar
  20. Chen Y, Garcia E, Gupta M, Rahimi A, Cazzanti L (2009) Similarity-based classification: concepts and algorithms. J Mach Learn Res 10:747–776MathSciNetzbMATHGoogle Scholar
  21. Chivadshetti P, Sadafale K, Thakare K (2015) Content based video retrieval using integrated feature extraction and personalization of results. In: International conference on information processing (ICIP’15).
  22. Cormen TH, Leiserson CE, Rivest RL, Stein C (2009) Introduction to algorithms, 3rd edn. MIT Press, CambridgezbMATHGoogle Scholar
  23. Costa Y, Oliveira L, Silla C Jr (2017) An evaluation of convolutional neural networks for music classification using spectrograms. Appl Soft Comput 52:28–38CrossRefGoogle Scholar
  24. Danhauser A (1994) Theory of music (French). Henri Lemoine, Paris (original edition published in 1950) Google Scholar
  25. Dell AG, Newton DA, Petroff JG (2011) Assistive technology in the classroom: enhancing the school experiences of students with disabilities, 2nd edn. Pearson, New DelhiGoogle Scholar
  26. Demopoulos RJ, Katchabaw MJ (2007) Music information retrieval: a survey of issues and approaches. Technical report #677, Department of Computer Science, University of Western OntarioGoogle Scholar
  27. Di Nunzio A (2014) Illiac suite for string quartet. Accessed July 2017
  28. Diaz-Jerez G (2011) Composing with melomics: delving into the computational world for musical inspiration. MIT Press J 21:3–14Google Scholar
  29. Dubois RL (2003) Applications of generative string-substitution systems in computer music. Ph.D. dissertation, Columbia UniversityGoogle Scholar
  30. Ekman P (1993) Facial expression of emotion. Am Psychol 48:384–392CrossRefGoogle Scholar
  31. Epstein R, Roberts G, Beber G (2009) Parsing the Turing test: philosophical and methodological issues in the quest for the thinking computer, 2009th edn. Springer, BerlinzbMATHCrossRefGoogle Scholar
  32. Fahmi A, Abdullah S, Amin F, Siddiqui N, Ali A (2017) Aggregation operators on triangular cubic fuzzy numbers and its application to multi-criteria decision making problems. J Intell Fuzzy Syst 33(6):3323–3337CrossRefGoogle Scholar
  33. Fahmi A, Abdullah S, Amin F, Ali A, Khan WA (2018) Some geometric operators with triangular cubic linguistic hesitant fuzzy number and their application in group decision-making. J Intell Fuzzy Syst 35(2):2485–2499CrossRefGoogle Scholar
  34. Fahmi A, Abdullah S, Amin F, Sajjad Ali Khan M (2019) Trapezoidal cubic fuzzy number Einstein hybrid weighted averaging operators and its application to decision making. Soft Comput 24(14):5753–5783zbMATHCrossRefGoogle Scholar
  35. Fernandez J, Vico F (2013) AI methods in algorithmic composition: a comprehensive survey. J Artif Intell Res 48:513–582MathSciNetCrossRefGoogle Scholar
  36. Fleischman MB, Deb KR (2013) Displaying estimated social interest in time-based media. U.S. patent no. 8,516,374Google Scholar
  37. Freeman J (2015) Survey of music technology. Coursera. Accessed Jul 2017
  38. Ghosh A, Strehl J (2003) Cluster ensembles—a knowledge reuse framework for combining multiple partitions. J Mach Learn Res 3:583–617MathSciNetzbMATHGoogle Scholar
  39. Gkonou C, Mercer S (2017) Understanding emotional and social intelligence among English language teachers. ELT research papers 17.03, British Council, ISBN 978-0-86355-842-9Google Scholar
  40. Goldberg D (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, ReadingzbMATHGoogle Scholar
  41. Goleman D (2005) Emotional intelligence: why it can matter more than IQ, Bantam Books 10th Anniversary edn. Bloomsbury Publishing, LondonGoogle Scholar
  42. Hauger D, Schedl M, Kosir A, Tkalcic M (2013) The million musical tweet dataset: what we can learn from microblogs. In: Proceedings of the 14th international society for music information retrieval conference (ISMIR’13)Google Scholar
  43. Hevner K (1935) The affective character of the major and minor modes in music. Am J Psychol 47(1):03–118CrossRefGoogle Scholar
  44. Hiller L (1970) Music composed with computers: a historical survey. In: Lincoln HB (ed) The computer and music. Cornell University Press, Ithaca, pp 42–97Google Scholar
  45. Hiller L, Isaaccson L (1959) Experimental music: composition with an electronic computer. McGraw-Hill, New YorkGoogle Scholar
  46. Hoeberechts M, Shantz J (2009) Realtime emotional adaptation in automated composition. In: Proceedings of audio mostly, pp 1–8Google Scholar
  47. Holland S, Wilkie K, Mulholland P, Seago A (2013) Music and human–computer interaction. Springer series on cultural computing. Springer, Berlin. CrossRefGoogle Scholar
  48. Hopfield J, Tank D (1985) Neural computation of decisions in optimization problems. Biol Cybern 52(3):52–141zbMATHGoogle Scholar
  49. Hovy E (2015) What are sentiment, affect, and emotion? Applying the methodology of Michael Zock to sentiment analysis. In: Gala N et al (eds) Language production, cognition, and the lexicon, text, speech and language technology, vol 48. Springer, Berlin, pp 13–24CrossRefGoogle Scholar
  50. Huang C, Lin E (2013) An emotion-based method to perform algorithmic composition. In: The 3rd international conference on music & emotion, pp. 244–247Google Scholar
  51. Husarik S (1983) John Cage and LeJaren Hiller: HPSCHD, 1969. Am Music 1(2):1–21CrossRefGoogle Scholar
  52. Iakovidou C, Anagnostopoulos N, Kapoutsis A, Chatzichristofis Y, Boutalis Y (2014) Searching images with MPEG-7 (& MPEG-7-like) powered localized descriptors: the SIMPLE answer to effective content based image retrieval. In: 12th international workshop on content-based multimedia indexing (CBMI). pp 18–20Google Scholar
  53. Iren D, Liem C, Yang J, Bozzon A (2016) Using social media to reveal social and collective perspectives on music. In: International ACM conference on web sciencs (WebSci’16), Hannover, Germany, pp 296–300Google Scholar
  54. Katayose H, Kato H, Imai M, Inokuchi S (1989) An approach to an artificial music expert. In: International computer music conference, pp 138–146Google Scholar
  55. Keller JM, Gray MR, Givens JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE Trans Syst Man Cybern 4:580–585CrossRefGoogle Scholar
  56. Kim J, Wigram T, Gold C (2009) Emotional, motivational and interpersonal responsiveness of children with autism in improvisational music therapy. Autism 13(4):389–409CrossRefGoogle Scholar
  57. Kirke A, Miranda ER (2009) A survey of computer systems for expressive music performance. ACM Comput Surv (CSUR) 42(1):3CrossRefGoogle Scholar
  58. Kirke A, Miranda E (2011) Combining EEG frontal asymmetry studies with affective algortihmic composition and expressive performance model. In: International computer music conference, HuddersfieldGoogle Scholar
  59. Kirke A, Miranda E (2017) Aiding soundtrack composer creativity through automated film script-profiled algorithmic composition. J Creat Music Syst 1(2)Google Scholar
  60. Kotsiantis SB (2007) Supervised machine learning: a review of classification techniques. Informatica 31:249–268MathSciNetzbMATHGoogle Scholar
  61. Kyogu L (2008) A system for acoustic chord transcription and key extraction from audio using hidden Markov models trained on synthesized audio. Dissertation, Department of Music, Stanford UniversityGoogle Scholar
  62. L’Hadj LS, Boughanem M, Amrouche K (2016) Enhancing information retrieval through concept-based language modeling and semantic smoothing. J Assoc Inf Sci Technol (JASIST) 67(12):2909–2927CrossRefGoogle Scholar
  63. Lin CL, Shih YH, Tzeng GH, Yu HC (2016) A service selection model for digital music service platforms using a hybrid MCDM approach. Appl Soft Comput 48:385–430CrossRefGoogle Scholar
  64. Liu J, Zhong W, Jiao L (2010) A multiagent evolutionary algorithm for combinatorial optimization problems. IEEE Trans Syst Man Cybern 40(1):229–240CrossRefGoogle Scholar
  65. Livingstone SR et al (2010) Changing musical emotion: a computational rule system for modifying score and performance. Comput Music J 34(1):41–64CrossRefGoogle Scholar
  66. Manousakis S (2006) Musical L-systems. Koninklijk Conservatorium, The Hague Master ThesisGoogle Scholar
  67. Marques M, Oliveira V, Vieira S, Rosa AC (2000) Music composition using genetic evolutionary algorithms. In: Proceedings of the IEEE conference on evolutionary computation. IEEE Press, New York, NYGoogle Scholar
  68. Matic D (2010) A genetic algorithm for composing music. Yugoslav J Oper Res 20(1):157–177MathSciNetzbMATHCrossRefGoogle Scholar
  69. McAndrew S, Everett M (2015) Music as collective invention: a social network analysis of composers. Cult Sociol J 9(1):56–80. CrossRefGoogle Scholar
  70. McChord KA (2004) Moving beyond “that’s all i can do”: encouraging musical creativity in children with learning disabilities. Bull Counsil Res Music Educ 159:23–32Google Scholar
  71. McCormack J (1996) Grammar-based music composition. In: Stocker S et al (eds) Complex systems. IOS Press, Amsterdam, pp 321–336Google Scholar
  72. Molina A, Daniel D, Moya JC, Vico FJ (2016) An Evo-Devo system for algorithmic composition that actually works. In: Proceedings of the 2016 on genetic and evolutionary computation conference companion. ACM, pp 37–38Google Scholar
  73. Morreale F, de Angeli A (2016) Collaborating with an autonomous agent to generate affective music. ACM Trans Comput Entertain (ACM CIE) 14(3):1–21Google Scholar
  74. Mühling M et al (2016) Content-based video retrieval in historical collections of the German broadcasting archive. In: International conference on theory and practice of digital libraries (TPLD’16), pp 67–78CrossRefGoogle Scholar
  75. O’Connor B, Balasubramanyan R, Routledge BR, Smith NA (2010) From tweets to polls: linking text sentiment to public opinion time series. In: Proceedings of the fourth international AAAI conference on weblogs and social media, pp 122–129Google Scholar
  76. Orio N (2006) Music retrieval: a tutorial and review. Found Trends Inf Retr 1(11):90zbMATHGoogle Scholar
  77. Ozcan E, Erçal T (2008) A genetic algorithm for generating improvized music. Lecture notes in computer science. Springer, Heidelberg, p 4926Google Scholar
  78. Panda R, Malheiro R, Rocha B, Oliveira A, Paiva RP (2013) Multi-modal music emotion recognition: a new dataset, methodology and comparative analysis. In: 10th international symposium on computer music multidisciplinary research (CMMR), pp 1–13Google Scholar
  79. Papadopoulos G, Wiggins G (1999) AI methods for algorithmic composition: a survey, a critical view and future prospects. In: AISB symposium on musical creativity, pp 110–117Google Scholar
  80. Pavlov S, Olsson C, Svensson C, Anderling V, Wikner J, Andreasson O (2014) Generation of music through genetic algorithms. Bachelor’s Thesis, University of Gothenburg, SwedenGoogle Scholar
  81. Prusinkiewicz P, Lindenmayer A (1990) The algorithmic beauty of plants. Springer, New YorkzbMATHCrossRefGoogle Scholar
  82. Rahim A, Civelek I, Liang FH (2015) A model of department chairs’ social intelligence & faculty members’ turnover intention. Intelligence 53:65–71CrossRefGoogle Scholar
  83. Ravi K, Ravi V (2015) A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowl Based Syst 89:14–46CrossRefGoogle Scholar
  84. Reimer MA, Garnett GE (2014) A hierarchical system for autonomous musical creation. In: Tenth artificial intelligence and interactive digital entertainment conference, pp 45–49Google Scholar
  85. Russell J (1980) A circumplex model of affect. J Pers Soc Psychol 6(39):1161–1980CrossRefGoogle Scholar
  86. SACEM (Society of Auhors, C., and Editors of Music) (2016) AIVA: artificial intelligence virtual artist. Accessed May 2018
  87. Sandred O, Laurson M, Kuuskankare M (2009) Revisiting the Illiac suite—a rule-based approach to stochastic processes. Sonic Ideas/Ideas Sonicas 2:42–46Google Scholar
  88. Schank RC, Cleary C (1995) Making machines creative. In: Smith S, Ward TB, Finke RA (eds) The creative cognition approach. MIT Press, Cambridge, pp 229–247Google Scholar
  89. Schedl M, Flexer A, Urbano J (2013) The neglected user in music information retrieval research. J Intell Inf Syst (JIIS) 41(3):523–539CrossRefGoogle Scholar
  90. Schedl M, Gómez E, Urbano J (2014) Music information retrieval: recent developments and applications. Found Trends Inf Retr 8(2–3):127–161CrossRefGoogle Scholar
  91. See CM (2012) The use of music and movement therapy to modify behaviour of children with autism. Pertanika J Soc Sci Hum 20(4):1103–1116Google Scholar
  92. Serra MH (1993) Stochastic composition and stochastic timbre: Gendy3 by Iannis Xenakis. Perspect New Music 237–257Google Scholar
  93. Shang W et al (2005) An improved kNN algorithm—fuzzy kNN. In: Computational intelligence and security, pp 741–746CrossRefGoogle Scholar
  94. Song Y, Dixon S, Pearce M (2012) A survey of music recommendation systems and future perspectives. In: 9th international symposium on computer music modeling and retrieval, pp 395–410Google Scholar
  95. Subasic P, Huettner A (2001) Affect analysis of text using fuzzy semantic typing. IEEE Trans Fuzzy Syst 9(4):483–496CrossRefGoogle Scholar
  96. Temperley D (2002) A Bayesian approach to key-finding. In: International conference on music and artificial intelligence, LNAI 2445, pp 195–206CrossRefGoogle Scholar
  97. Troiano L, Birtolo C, Armenise R (2017) Modeling and predicting the user next input by Bayesian reasoning. Soft Comput 21(6):1583–1600CrossRefGoogle Scholar
  98. Verbeurgt K, Fayer M, Dinolfo M (2004) A hybrid neural-markov approach for learning to compose music by example. In: Conference of the Canadian society for computational studies of intelligence, pp 480–484CrossRefGoogle Scholar
  99. Wan CY et al (2011) Auditory-motor mapping training as an intervention to facilitate speech output in non-verbal children with autism: a proof of concept study. PLoS ONE 6(9):e25505. CrossRefGoogle Scholar
  100. Whipple J (2004) Music in intervention for children and adolescents with autism: a meta-analysis. J Music Ther 41(2):90–106CrossRefGoogle Scholar
  101. Whitley D, Sutton AM (2012) Genetic algorithms—a survey of models and methods. In: Rozenberg G, Bäck T, Kok JN (eds) Handbook of natural computing. Springer, Berlin, pp 637–671CrossRefGoogle Scholar
  102. Wohlfahrt-Laymanna J, Heimbürgerb A (2017) Content aware music analysis with multi-dimensional similarity measure. Inf Model Knowl Bases XXVIII:292–303Google Scholar
  103. Wolfram Tones Inc (2005) WolframTones: how it works—scientific foundations. Accessed June 2019
  104. Worth P, Stepney S (2005) Growing music: musical interpretations of L-systems. In: Workshop on applications of evolutionary computing, pp 545–550Google Scholar
  105. Xiao H, Downie SJ (2010) Improving mood classification in music digital libraries by combining lyrics and audio. In: Proceedings of the 10th annual joint conference on digital libraries. ACM, pp 159–168Google Scholar
  106. Yiu R (2013) A composer’s imagining of musical tradition and the reinvention of heritage. Doctoral thesis, City, University of London Institutional RepositoryGoogle Scholar
  107. Yuanyuan W (2014) Music emotion cognition model and interactive technology. In: IEEE workshop on electronics, computer and applications, Ottawa, Canada
  108. Zangerle E, Pichl M, Gassler W, Specht G (2014) #nowplaying music dataset: extracting listening behavior from Twitter. In: Proceedings of the first international workshop on internet-scale multimedia management (WISMM), ACM Multimedia, Orlando, Florida, USAGoogle Scholar
  109. Zentner M, Eerola T (2010) Self-report measures and models. In: Juslin PN, Sloboda JA (eds) Handbook of music and emotion: theory, research, applications. Oxford University Press, New York, pp 187–221Google Scholar
  110. Zenz V (2007) Automatic chord detection in polyphonic audio data. Master’s thesis University of Wien, AustriaGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.E.C.E. Department, School of EngineeringLebanese American University (LAU), Byblos CampusByblosLebanon

Personalised recommendations