Advertisement

Multimodal mood classification of Hindi and Western songs

  • Braja Gopal Patra
  • Dipankar Das
  • Sivaji Bandyopadhyay
Article

Abstract

Music mood classification is one of the most interesting research areas in music information retrieval, and it has many real-world applications. Many experiments have been performed in mood classification or emotion recognition of Western music; however, research on mood classification of Indian music is still at initial stage due to scarcity of digitalized resources. In the present work, a mood taxonomy is proposed for Hindi and Western songs; both audio and lyrics were annotated using the proposed mood taxonomy. Differences in mood were observed during the annotation of the audio and lyrics for Hindi songs only. The detailed studies on mood classification of Hindi and Western music are presented for the requirement of the recommendation system. LibSVM and Feed-forward neural networks have been used to develop mood classification systems based on audio, lyrics, and a combination of them. The multimodal mood classification systems using Feed-forward neural networks for Hindi and Western songs obtained the maximum F-measures of 0.751 and 0.835, respectively.

Keywords

Music information retrieval Multimodal mood classification Mood taxonomy Hindi and Western songs Feed-forward neural networks 

Notes

Acknowledgements

The work reported in this paper is supported by a grant from the “Visvesvaraya Ph.D. Scheme for Electronics and IT” funded by Media Lab Asia of Ministry of Electronics and Information Technology (MeitY), Government of India. The authors are thankful to Afif Ahmed, Anit, Arijit Das, and Niloy Mukherjee, for helping in data collection. The authors are also thankful to the anonymous reviewers for their helpful comments.

References

  1. Abburi, H., Akkireddy, E.S.A., Gangashetty, S.V., & Mamidi, R. (2016). Multimodal sentiment analysis of telugu songs. In Proceedings of the 4th workshop on sentiment analysis where AI meets psychology (SAAIP 2016) (pp. 48–52).Google Scholar
  2. Bertin-Mahieux, T., Ellis, D.P.W., Whitman, B., & Lamere, P. (2011). The million song dataset. In Proceedings of the 12th international society for music information retrieval conference (ISMIR 2011) (pp. 591–596).Google Scholar
  3. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.MathSciNetCrossRefGoogle Scholar
  4. Duncan, N., & Fox, M. (2005). Computer–aided music distribution: the future of selection, retrieval and transmission. First Monday, 10(4).  https://doi.org/10.5210/fm.v10i4.1220.
  5. Eyben, F., Wöllmer, M., & Schuller, B. (2010). Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia (pp. 1459–1462).Google Scholar
  6. Hall, M.A. (1999). Correlation-based feature selection for machine learning. PhD dissertation: The University of Waikato.Google Scholar
  7. Hampiholi, V. (2012). A method for music classification based on perceived mood detection for Indian bollywood music. International Journal of Computer, Electrical, Automation, Control and Information Engineering, 6(12), 1636–1643.Google Scholar
  8. Hevner, K. (1936). Experimental studies of the elements of expression in music. The American Journal of Psychology, 48(2), 246–268.CrossRefGoogle Scholar
  9. Hu, X. (2010). Music and mood: where theory and reality meet. In Proceedings of the iConference 2010.Google Scholar
  10. Hu, X., & Downie, J.S. (2010a). Improving mood classification in music digital libraries by combining lyrics and audio. In Proceedings of the 10th annual joint conference on digital libraries (pp. 159–168).Google Scholar
  11. Hu, X., & Downie, J.S. (2010b). When lyrics outperform audio for music mood classification: a feature analysis. In Proceedings of the 11th international society for music information retrieval conference (ISMIR 2010) (pp. 619–624).Google Scholar
  12. Hu, X., Downie, J.S., Laurier, C., Bay, M., & Ehmann, A.F. (2008). The 2007 MIREX audio mood classification task: lessons learned. In Proceedings of the 9th international society for music information retrieval conference (ISMIR 2008) (pp. 462–467).Google Scholar
  13. Hu, X., Choi, K., & Downie, J.S. (2017). A framework for evaluating multimodal music mood classification. Journal of the Association for Information Science and Technology, 68(2), 273–285.  https://doi.org/10.1002/asi.23649.CrossRefGoogle Scholar
  14. Katayose, H., Imai, M., & Inokuchi, S. (1988). Sentiment extraction in music. In Proceedings of the 9th international conference on pattern recognition (pp. 1083–1087).Google Scholar
  15. Kim, Y.E., Schmidt, E.M., & Emelle, L. (2008). MoodSwings: a collaborative game for music mood label collection. In Proceedings of the 9th international society for music information retrieval conference (ISMIR 2008) (pp. 231–236).Google Scholar
  16. Kim, Y.E., Schmidt, E.M., Migneco, R., Morton, B.G., Richardson, P., Scott, J., Speck, J.A., & Turnbull, D. (2010). Music emotion recognition: a state of the art review. In Proceedings of the 11th international society for music information retrieval conference (ISMIR 2010) (pp. 255–266).Google Scholar
  17. Koduri, G.K., & Indurkhya, B. (2010). A behavioral study of emotions in south Indian classical music and its implications in music recommendation systems. In Proceedings of the 2010 ACM workshop on social, adaptive and personalized multimedia interaction and access (pp. 55–60).Google Scholar
  18. Lamere, P. (2008). Social tagging and music information retrieval. Journal of New Music Research, 37(2), 101–114.CrossRefGoogle Scholar
  19. Lang, P.J., Bradley, M.M., & Cuthbert, B.N. (1998). Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology. Biological Psychiatry, 44(12), 1248–1263.CrossRefGoogle Scholar
  20. Laurier, C., & Herrera, P. (2007). Audio music mood classification using support vector machine. MIREX task on Audio Mood Classification, 2–4.Google Scholar
  21. Laurier, C., Grivolla, J., & Herrera, P. (2008). Multimodal music mood classification using audio and lyrics. In Proceedings of the 7th international conference on machine learning and applications (ICMLA’08) (pp. 688–693).Google Scholar
  22. Laurier, C., Sordo, M., Serra, J., & Herrera, P. (2009). Music mood representations from social tags. In Proceedings of the 10th international society for music information retrieval conference (ISMIR 2009) (pp. 381–386).Google Scholar
  23. Liu, D., Lu, L., & Zhang, H. (2003). Automatic mood detection from acoustic music data. In Proceedings of the 6th international conference on music information retrieval (ISMIR-2003) (pp. 81–87).Google Scholar
  24. Lu, L., Liu, D., & Zhang, H.J. (2006). Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing, 14(1), 5–18.CrossRefGoogle Scholar
  25. Mathematica Neural NetworksTrain and Analyze Neural Networks to Fit Your Data. 2005. Wolfram Research Inc., First Edition, Champaign, Illinois, USA.Google Scholar
  26. Mayer, R., Neumayer, R., & Rauber, A. (2008). Combination of audio and lyrics features for genre classification in digital audio collections. In Proceedings of the 16th ACM international conference on multimedia (pp. 159–168).Google Scholar
  27. McKay, C., Fujinaga, I., & Depalle, P. (2005). jAudio: a feature extraction library. In Proceedings of the 6th international conference on music information retrieval (pp. 600–603).Google Scholar
  28. Patra, B.G., Das, D., & Bandyopadhyay, S. (2013a). Automatic music mood classification of Hindi songs. In Proceedings of the 3rd workshop on sentiment analysis where AI meets psychology (SAAIP 2013) (pp. 24–28).Google Scholar
  29. Patra, B.G., Das, D., & Bandyopadhyay, S. (2013b). Unsupervised approach to Hindi music mood classification. In Proceedings of the mining intelligence and knowledge exploration (pp. 62–69).Google Scholar
  30. Patra, B.G., Maitra, P., Das, D., & Bandyopadhyay, S. (2015a). MediaEval 2015: music emotion recognition based on feed-forward neural network. In Proceedings of MediaEval 2015 Workshop.Google Scholar
  31. Patra, B.G., Das, D., & Bandyopadhyay, S. (2015b). Music emotion recognition system. In Proceedings of the international symposium frontiers of research speech and music (FRSM-2015) (pp. 114–119).Google Scholar
  32. Patra, B.G., Das, D., & Bandyopadhyay, S. (2015c). Mood classification of Hindi songs based on lyrics. In Proceedings of the 12th international conference on natural language processing (ICON-2015) (pp. 261–267).Google Scholar
  33. Patra, B.G., Das, D., & Bandyopadhyay, S. (2016a). Multimodal mood classification framework for Hindi songs. Computación y Sistemas, 20(3), 515–526.CrossRefGoogle Scholar
  34. Patra, B.G., Das, D., & Bandyopadhyay, S. (2016b). Labeling data and developing supervised framework for Hindi music mood analysis. Journal of Intelligent Information Systems, 48(3), 633–651.  https://doi.org/10.1007/s10844-016-0436-1.CrossRefGoogle Scholar
  35. Patra, B.G., Das, D., & Bandyopadhyay, S. (2016c). Multimodal mood classification - a case study of differences in Hindi and Western songs. In Proceedings of the 26th international conference on computational linguists (COLING-2016) (pp. 1980–1989).Google Scholar
  36. Posner, J., Russell, J.A., & Peterson, B.S. (2005). The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and Psychopathology, 17(03), 715–734.CrossRefGoogle Scholar
  37. Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.CrossRefzbMATHGoogle Scholar
  38. Russell, J.A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.CrossRefGoogle Scholar
  39. Soleymani, M., Caro, M.N., Schmidt, E.M., Sha, C.Y., & Yang, Y.H. (2013). 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM international workshop on crowdsourcing for multimedia (pp. 1–6).Google Scholar
  40. Thayer, R.E. (1990). The biopsychology of mood and arousal. Oxford University Press.Google Scholar
  41. Ujlambkar, A.M. (2012). Automatic mood classification of Indian popular music Master’s Thesis. College of Engineering, Pune.Google Scholar
  42. Velankar, M.R., & Sahasrabuddhe, H.V. (2012). A pilot study of Hindustani music sentiments. In Proceedings of the 2nd workshop on sentiment analysis where AI meets psychology (SAAIP-2012) (pp. 91–98).Google Scholar
  43. Watson, D., Wiese, D., Vaidya, J., & Tellegen, A. (1999). The two general activation systems of affect: structural findings, evolutionary considerations, and psychobiological evidence. Journal of Personality and Social Psychology, 76(5), 820–838.CrossRefGoogle Scholar
  44. Yang, Y.H., Lin, Y.C., Cheng, H.T., Liao, I-Bin, & Ho, Y.C. (2008). Toward multi-modal music emotion classification. In Proceedings of the pacific-rim conference on multimedia (pp. 70–79).Google Scholar
  45. Zaanen, M.V., & Kanters, P. (2010). Automatic mood classification using TF*IDF based on lyrics. In Proceedings of the 11th international society for music information retrieval conference (ISMIR 2010) (pp. 75–80).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Biomedical InformaticsUTHealthHoustonUSA
  2. 2.Department of Computer Science and EngineeringJadavpur UniversityKolkataIndia

Personalised recommendations