Advertisement

The Music Information Retrieval Evaluation eXchange: Some Observations and Insights

  • J. Stephen Downie
  • Andreas F. Ehmann
  • Mert Bay
  • M. Cameron Jones
Part of the Studies in Computational Intelligence book series (SCI, volume 274)

Abstract

Advances in the science and technology of Music Information Retrieval (MIR) systems and algorithms are dependent on the development of rigorous measures of accuracy and performance such that meaningful comparisons among current and novel approaches can be made. This is the motivating principle driving the efforts of the International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) and the annual Music Information Retrieval Evaluation eXchange (MIREX). Since it started in 2005, MIREX has fostered great advancements not only in many specific areas of MIR, but also in our general understanding of how MIR systems and algorithms are to be evaluated. This chapter outlines some of the major highlights of the past four years of MIREX evaluations, including its organizing principles, the selection of evaluation metrics, and the evolution of evaluation tasks. The chapter concludes with a brief introduction of how MIREX plans to expand into the future using a suite of Web 2.0 technologies to automated MIREX evaluations.

Keywords

MIREX Music Information Retrieval Evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cano, P., Gomez, E., Gouyon, F., Herrera, P., Koppenberger, M., Ong, B., Serra, X., Streich, S., Wack, N.: ISMIR 2004 audio description contest. MTG Technical Report, MTG-TR-2006-02 (Music Technology Group, Barcelona, Spain) (2004)Google Scholar
  2. 2.
    Downie, J.S. (ed.): The exploratory workshop on music information retrieval. Graduate School of Library and Information Science, Champaign (1999)Google Scholar
  3. 3.
    Downie, J.S. (ed.): The MIR/MDL evaluation project white paper collection, edition #3: Establishing music information retrieval (MIR) and music digital library (MDL) evaluation frameworks: Preliminary foundations and infrastructures. Graduate School of Library and Information Science, Champaign (2003)Google Scholar
  4. 4.
    Downie, J.S.: The scientific evaluation of music information retrieval systems: Foundations and future. Computer Music Journal 28(3), 12–23 (2004)CrossRefGoogle Scholar
  5. 5.
    Downie, J.S.: The Music Information Retrieval Evaluation Exchange (2005-2007): A window into music information retrieval research. Acoustical Science and Technology 29(4), 247–255 (2008)CrossRefGoogle Scholar
  6. 6.
    Downie, J.S., Bay, M., Ehmann, A.F., Jones, M.C.: Audio cover song identification: MIREX 2006-2007 results and analyses. In: Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2008), pp. 468–473 (2008)Google Scholar
  7. 7.
    Downie, J.S., Lee, J.H., Gruzd, A.A., Jones, M.C.: Toward an understanding of similarity judgments for music digital library evaluation. In: Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2007), pp. 307–308 (2007)Google Scholar
  8. 8.
    Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378–382 (1971)CrossRefGoogle Scholar
  9. 9.
    Gruzd, A.A., Downie, J.S., Jones, M.C., Lee, J.H.: Evalutron 6000: Collecting music relevance judgments. In: Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2007), p. 507 (2007)Google Scholar
  10. 10.
    Jones, M.C., Downie, J.S., Ehmann, A.F.: Understanding human judgments of similarity in music information retrieval. In: Proceedings of the Eighth International Conference on Music Information Retrieval (ISMIR 2007), pp. 539–542 (2007)Google Scholar
  11. 11.
    Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    McEnnis, D., McKay, C., Fujinaga, I.: Overview of OMEN. In: Proceedings of the Seventh International Conference on Music Information Retrieval (ISMIR 2006), pp. 7–12 (2006)Google Scholar
  13. 13.
    McKinney, M.F., Moelants, D.: Tempo perception and musical content: What makes a piece fast, slow or temporally ambiguous? In: Proceedings of the International Conference on Music Perception and Cognition, Evanston, IL, USA (2004)Google Scholar
  14. 14.
    Pampalk, E.: Audio-based music similarity and retrieval: Combining a spectral similarity model with information extracted from fluctuation patters. In: MIREX 2006, Submission Abstracts (2006), http://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pampalk.pdf
  15. 15.
    Répertoire International des Sources Musicales (RISM). Serie A/II, manuscrits musicaux après 1600. München. K. G. Saur Verlag, Germany (2002)Google Scholar
  16. 16.
    Typke, R., den Hoed, M., de Nooijer, J., Wiering, F., Veltkamp, R.C.: A ground truth for half a million musical incipits. Journal of Digital Information Management 3(1), 34–39 (2005)Google Scholar
  17. 17.
    Typke, R., Veltkamp, R., Wiering, F.: A measure for evaluating retrieval techniques based on partially ordered ground truth lists. In: IEEE International Conference on Multimedia and Expo (ICME 2006), pp. 1793–1796 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • J. Stephen Downie
    • 1
  • Andreas F. Ehmann
    • 1
  • Mert Bay
    • 1
  • M. Cameron Jones
    • 1
  1. 1.International Music Information Retrieval Systems Evaluation Laboratory, Graduate School of Library and Information ScienceUniversity of Illinois at Urbana-ChampaignChampaignUSA

Personalised recommendations