Abstract
Advances in the science and technology of Music Information Retrieval (MIR) systems and algorithms are dependent on the development of rigorous measures of accuracy and performance such that meaningful comparisons among current and novel approaches can be made. This is the motivating principle driving the efforts of the International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) and the annual Music Information Retrieval Evaluation eXchange (MIREX). Since it started in 2005, MIREX has fostered great advancements not only in many specific areas of MIR, but also in our general understanding of how MIR systems and algorithms are to be evaluated. This chapter outlines some of the major highlights of the past four years of MIREX evaluations, including its organizing principles, the selection of evaluation metrics, and the evolution of evaluation tasks. The chapter concludes with a brief introduction of how MIREX plans to expand into the future using a suite of Web 2.0 technologies to automated MIREX evaluations.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Cano, P., Gomez, E., Gouyon, F., Herrera, P., Koppenberger, M., Ong, B., Serra, X., Streich, S., Wack, N.: ISMIR 2004 audio description contest. MTG Technical Report, MTG-TR-2006-02 (Music Technology Group, Barcelona, Spain) (2004)
Downie, J.S. (ed.): The exploratory workshop on music information retrieval. Graduate School of Library and Information Science, Champaign (1999)
Downie, J.S. (ed.): The MIR/MDL evaluation project white paper collection, edition #3: Establishing music information retrieval (MIR) and music digital library (MDL) evaluation frameworks: Preliminary foundations and infrastructures. Graduate School of Library and Information Science, Champaign (2003)
Downie, J.S.: The scientific evaluation of music information retrieval systems: Foundations and future. Computer Music Journal 28(3), 12–23 (2004)
Downie, J.S.: The Music Information Retrieval Evaluation Exchange (2005-2007): A window into music information retrieval research. Acoustical Science and Technology 29(4), 247–255 (2008)
Downie, J.S., Bay, M., Ehmann, A.F., Jones, M.C.: Audio cover song identification: MIREX 2006-2007 results and analyses. In: Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2008), pp. 468–473 (2008)
Downie, J.S., Lee, J.H., Gruzd, A.A., Jones, M.C.: Toward an understanding of similarity judgments for music digital library evaluation. In: Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2007), pp. 307–308 (2007)
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378–382 (1971)
Gruzd, A.A., Downie, J.S., Jones, M.C., Lee, J.H.: Evalutron 6000: Collecting music relevance judgments. In: Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2007), p. 507 (2007)
Jones, M.C., Downie, J.S., Ehmann, A.F.: Understanding human judgments of similarity in music information retrieval. In: Proceedings of the Eighth International Conference on Music Information Retrieval (ISMIR 2007), pp. 539–542 (2007)
Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)
McEnnis, D., McKay, C., Fujinaga, I.: Overview of OMEN. In: Proceedings of the Seventh International Conference on Music Information Retrieval (ISMIR 2006), pp. 7–12 (2006)
McKinney, M.F., Moelants, D.: Tempo perception and musical content: What makes a piece fast, slow or temporally ambiguous? In: Proceedings of the International Conference on Music Perception and Cognition, Evanston, IL, USA (2004)
Pampalk, E.: Audio-based music similarity and retrieval: Combining a spectral similarity model with information extracted from fluctuation patters. In: MIREX 2006, Submission Abstracts (2006), http://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pampalk.pdf
Répertoire International des Sources Musicales (RISM). Serie A/II, manuscrits musicaux après 1600. München. K. G. Saur Verlag, Germany (2002)
Typke, R., den Hoed, M., de Nooijer, J., Wiering, F., Veltkamp, R.C.: A ground truth for half a million musical incipits. Journal of Digital Information Management 3(1), 34–39 (2005)
Typke, R., Veltkamp, R., Wiering, F.: A measure for evaluating retrieval techniques based on partially ordered ground truth lists. In: IEEE International Conference on Multimedia and Expo (ICME 2006), pp. 1793–1796 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Downie, J.S., Ehmann, A.F., Bay, M., Jones, M.C. (2010). The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. In: Raś, Z.W., Wieczorkowska, A.A. (eds) Advances in Music Information Retrieval. Studies in Computational Intelligence, vol 274. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11674-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-11674-2_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11673-5
Online ISBN: 978-3-642-11674-2
eBook Packages: EngineeringEngineering (R0)