Journal of Intelligent Information Systems

, Volume 41, Issue 3, pp 499–521 | Cite as

Toward an understanding of the history and impact of user studies in music information retrieval

  • Jin Ha LeeEmail author
  • Sally Jo Cunningham


Most Music Information Retrieval (MIR) researchers will agree that understanding users’ needs and behaviors is critical for developing a good MIR system. The number of user studies in the MIR domain has been gradually increasing since the early 2000s, reflecting this growing appreciation of the need for empirical studies of users. However, despite the growing number of user studies and the wide recognition of their importance, it is unclear how great their impact has been in the field: on how systems are developed, how evaluation tasks are created, and how MIR system developers in particular understand critical concepts such as music similarity or music mood. In this paper, we present our analysis on the growth, publication and citation patterns, topics, and design of 198 user studies. This is followed by a discussion of a number of issues/challenges in conducting MIR user studies and distributing the research results. We conclude by making recommendations to increase the visibility and impact of user studies in the field.


Music MIR User study Citation analysis Co-authorship analysis 



We thank Gary Gao, Tiffany Huang, and Ben Farabelli at University of Washington for their valuable contributions to this project, participants of the ISMIR 2012 Conference for their helpful feedback, David Bainbridge at the University of Waikato for creating the initial Greenstone digital library of MIR user studies, and the anonymous reviewers for their comments and suggestions.


  1. Agresti, A., & Finlay, B. (2009). Statistical methods for the social sciences. Upper Saddle River: Pearson Prentice Hall.Google Scholar
  2. Berenzweig, A., Logan, B., Ellis, D.P.W., Whitman, B. (2004). A large-scale evaluation of acoustic and subjective music-similarity measures. Computer Music Journal, 28(2), 63–76.CrossRefGoogle Scholar
  3. Boltz, M., Schulkind, M., Kantra, S. (1991). Effects of background music on the remembering of filmed events. Memory & Cognition, 19(6), 593–606.CrossRefGoogle Scholar
  4. Bradford, S. (1985). Sources of information on specific subjects. Journal of Information Science, 10(4), 173–180.CrossRefGoogle Scholar
  5. Bull, M. (2005). No dead air! The iPod and the culture of mobile listening. Leisure Studies, 24(4), 343–355.CrossRefGoogle Scholar
  6. Cunningham, S.J. (2003). User studies: A first step in designing an MIR testbed. In The MIR/MDL evaluation project white paper collection, GSLIS (pp. 19–21). Champaign, IL.Google Scholar
  7. Cunningham, S.J., Reeves, N., Britland, M. (2003). An ethnographic study of music information seeking: Implications for the design of a music digital library. In Proceedings of the 3rd ACM/IEEE-CS joint conference on digital libraries (pp. 5–16).Google Scholar
  8. Dervin, B. (2003). Human studies and user studies: A call for methodological inter-disciplinarity. Information Research, 9(1).
  9. Dickson, W.J., & Roethlisberger, F.J. (1966). Counseling in an organization: A sequel to the Hawthorn Researches. Boston: Harvard University.Google Scholar
  10. Downie, J.S. (2003). Music information retrieval. In: B. Cronin (Ed.), Annual review of information science and technology 37 (pp. 295–340). Medford: Information Today.Google Scholar
  11. Drott, M.C. (1981). Bradford’s law: Theory, empiricism and the gaps between. In Library trends summer (Special issue on bibliometrics) (pp. 41–52).Google Scholar
  12. Ellis, D.P.W., Whitman, B., Berenzweig, A., Lawrence, S. (2002). The quest for ground truth in musical artist similarity. In Proceedings of the 3rd international society for music information retrieval (ISMIR) conference (pp. 170–177).Google Scholar
  13. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.CrossRefGoogle Scholar
  14. Futrelle, F., & Downie, J.S. (2002). Interdisciplinary communities and research issues in music information retrieval. In Proceedings of the 3rd ISMIR conference (pp. 215–221).Google Scholar
  15. Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (some of the time). In Proceedings of the CHI’08 (pp. 111–120).Google Scholar
  16. Gomes, C., Schneider, D., Moraes, K., De Souza, J. (2012). Crowdsourcing for music: Survey and taxonomy. In Proceedings of the IEEE SMCS conference.Google Scholar
  17. Hansen, D.L., Schneiderman, B., Smith, M.A. (2011). Analyzing social media networks with NodeXL: Insights from a connected world. Burlington: Morgan Kaufmann.Google Scholar
  18. Hiemstra, D., Hauff, C., De Jong, F., Kraaij, W. (2007). SIGIR’s 30th anniversary: An analysis of trends in IR research and the topology of its community. ACM SIGIR Forum, 41(2), 18–24.CrossRefGoogle Scholar
  19. Lee, J.H. (2011). How similar is too similar?: Exploring users’ perceptions of similarity in playlist evaluation. In Proceedings of the 12th ISMIR conference (pp. 109–114).Google Scholar
  20. Lee, J.H., & Downie, J.S. (2004). Survey of music information needs, uses, and seeking behaviours: Preliminary findings. In Proceedings of the 5th ISMIR conference (pp. 441–446).Google Scholar
  21. Lee, J.H., Hill, T., Work, L. (2012). What does music mood mean for real users? In Proceedings of the 2012 iConference (pp. 112–119).Google Scholar
  22. Lee, J.H., Jones, M.C., Downie, J.S. (2009). An analysis of ISMIR proceedings: Patterns of authorship, topic, and citation. In Proceedings of the 10th ISMIR conference (pp. 57–62).Google Scholar
  23. Lee, J.H., & Waterman, N.M. (2012). Understanding user requirements for music information services. In Proceedings of the 13th ISMIR conference (pp. 253–258).Google Scholar
  24. Levitin, D.J. (1994). Absolute memory for musical pitch: evidence from the production of learned melodies. Perception & Psychophysics, 54(4), 414–423.CrossRefGoogle Scholar
  25. Levitt, J.M., & Thelwall, M. (2009). The most highly cited Library and Information Science articles: interdisciplinary, first authors and citation patterns. Scientometrics, 78(1), 45–67.CrossRefGoogle Scholar
  26. Liu, X., Bollen, J., Nelson, M.L., Van de Sompel, H. (2005). Co-authorship networks in the digital library research community. Information Processing & Management, 41(6), 1462–1480.CrossRefGoogle Scholar
  27. McNab, R.J., Smith, L.A., Witten, I.H., Henderson, C.L., Cunningham, S.J. (1996). Towards the digital music library: Tune retrieval from acoustic input. In Proceedings of the 1st ACM international conference on digital libraries (pp. 11–18).Google Scholar
  28. North, A.C., Hargreaves, D.J., Hargreaves, J.J. (2004). Uses of music in everyday life. Music Perception: An Interdisciplinary Journal, 22(1), 41–77.CrossRefGoogle Scholar
  29. North, A.C., Hargreaves, D.J., O’Neill, S.A. (2000). The importance of music to adolescents. British Journal of Educational Psychology, 70, 255–272.CrossRefGoogle Scholar
  30. Pauws, S., & Eggen, B. (2002). PATS: Realization and user evaluation of an automatic playlist generator. In Proceedings of the 3rd ISMIR conference (pp. 13–17).Google Scholar
  31. Porter, M. (1997). An algorithm for suffix stripping. In: K. Sparck-Jones, & P. Willet (Eds.), Readings in information retrieval (pp. 313–316). San Francisco: Morgan Kaufmann.Google Scholar
  32. Rentfrow, P.J., & Gosling, S.D. (2003). The Do Re Mi’s of everyday life: the structure and personality correlates of music preference. Journal of Psychology and Social Psychology, 84(6), 1236–1256.Google Scholar
  33. Schedl, M., & Flexer, A. (2012). Putting the user in the center of music information retrieval. In Proceedings of the 13th ISMIR conference (pp. 385–390).Google Scholar
  34. Shardanand, U., & Maes, P. (1995). Social information filtering: Algorithms for automating “word of mouth”. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 210–217).Google Scholar
  35. Sloboda, J.A., O’Neill, S.A., Ivaldi, A. (2001). Functions of music in everyday life: an exploratory study using the Experience Sampling Method. Musicae Scientiae, 5(1), 9–32.Google Scholar
  36. Smeaton, A.F., Keogh, G., Gurrin, C., McDonald, K., Sødring, T. (2002). Analysis of papers from twenty-five years of SIGIR conferences: what have we been doing for the last quarter of a century? ACM SIGIR Forum, 37(1), 49–53.CrossRefGoogle Scholar
  37. Thorne, F.C. (1977). The citation index: another case of spurious validity. Journal of Clinical Psychology, 33(4), 1157–1161.MathSciNetCrossRefGoogle Scholar
  38. Voida, A., Grinter, R.E., Ducheneaut, N., Edwards, W.K., Newman, M.W. (2005). Listening in: Practices surrounding iTunes music sharing. In CHI’ 05 Proceedings of the SIGCHI conference on human factors in computing systems (pp. 191–200).Google Scholar
  39. Weigl, D.M., & Guastavino, C. (2011). User studies in the music information retrieval literature. In Proceedings of the 12th ISMIR conference (pp. 335–340).Google Scholar
  40. Weinstock, M. (1971). Citation indexes. In Encyclopedia of library and information science 5 (pp. 16–41). New York: Marcel Dekker.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Information SchoolUniversity of WashingtonSeattleUSA
  2. 2.Department of Computer ScienceUniversity of WaikatoHamiltonNew Zealand

Personalised recommendations