Learning to Tag from Open Vocabulary Labels

  • Edith Law
  • Burr Settles
  • Tom Mitchell
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6322)


Most approaches to classifying media content assume a fixed, closed vocabulary of labels. In contrast, we advocate machine learning approaches which take advantage of the millions of free-form tags obtainable via online crowd-sourcing platforms and social tagging websites. The use of such open vocabularies presents learning challenges due to typographical errors, synonymy, and a potentially unbounded set of tag labels. In this work, we present a new approach that organizes these noisy tags into well-behaved semantic classes using topic modeling, and learn to predict tags accurately using a mixture of topic classes. This method can utilize an arbitrary open vocabulary of tags, reduces training time by 94% compared to learning from these tags directly, and achieves comparable performance for classification and superior performance for retrieval. We also demonstrate that on open vocabulary tasks, human evaluations are essential for measuring the true performance of tag classifiers, which traditional evaluation methods will consistently underestimate. We focus on the domain of tagging music clips, and demonstrate our results using data collected with a human computation game called TagATune.


Human Computation Music Information Retrieval Tagging Algorithms Topic Modeling 


  1. 1.
    Bergstra, J., Lacoste, A., Eck, D.: Predicting genre labels for artists using freedb. In: ISMIR, pp. 85–88 (2006)Google Scholar
  2. 2.
    Bertin-Mahieux, T., Eck, D., Maillet, F., Lamere, P.: Autotagger: a model for predicting social tags from acoustic features on large music databases. TASLP 37(2), 115–135 (2008)Google Scholar
  3. 3.
    Blei, D., McAuliffe, J.D.: Supervised topic models. In: NIPS (2007)Google Scholar
  4. 4.
    Blei, D., Ng, A., Jordan, M.: Latent dirichlet allocation. Journal of Machine Learning Research 3, 993–1022 (2003)zbMATHCrossRefGoogle Scholar
  5. 5.
    Csiszar, I.: Maxent, mathematics, and information theory. In: Hanson, K., Silver, R. (eds.) Maximum Entropy and Bayesian Methods. Kluwer Academic Publishers, Dordrecht (1996)Google Scholar
  6. 6.
    Dannenberg, R.B., Hu, N.: Understanding search performance in query-by-humming systems. In: ISMIR, pp. 41–50 (2004)Google Scholar
  7. 7.
    Eisenberg, G., Batke, J.M., Sikora, T.: Beatbank – an mpeg-7 compliant query by tapping system. Audio Engineering Society Convention, 6136 (2004)Google Scholar
  8. 8.
    Goto, M., Hirata, K.: Recent studies on music information processing. Acoustic Science and Technology, 419–425 (2004)Google Scholar
  9. 9.
    Herrera, P., Peeters, G., Dubnov, S.: Automatic classification of music instrument sounds. Journal of New Music Research, 3–21 (2003)Google Scholar
  10. 10.
    Hoffman, M., Blei, D., Cook, P.: Easy as CBA: A simple probabilistic model for tagging music. In: ISMIR, pp. 369–374 (2009)Google Scholar
  11. 11.
    Iwata, T., Yamada, T., Ueda, N.: Modeling social annotation data with content relevance using a topic model. In: NIPS (2009)Google Scholar
  12. 12.
    Lamere, P.: Social tagging and music information retrieval. Journal of New Music Research 37(2), 101–114 (2008)CrossRefGoogle Scholar
  13. 13.
    Laurier, C., Sordo, M., Serra, J., Herrera, P.: Music mood representations from social tags. In: ISMIR, pp. 381–386 (2009)Google Scholar
  14. 14.
    Law, E., von Ahn, L.: Input-agreement: A new mechanism for collecting data using human computation games. In: CHI, pp. 1197–1206 (2009)Google Scholar
  15. 15.
    Law, E., West, K., Mandel, M., Bay, M., Downie, S.: Evaluation of algorithms using games: The case of music tagging. In: ISMIR, pp. 387–392 (2009)Google Scholar
  16. 16.
    Levy, M., Sandler, M.: A semantic space for music derived from social tags. In: ISMIR (2007)Google Scholar
  17. 17.
    Li, T., Ogihara, M., Li, Q.: A comparative study on content-based music genre classification. In: SIGIR, pp. 282–289 (2003)Google Scholar
  18. 18.
    Mandel, M., Ellis, D.: Song-level features and support vector machines for music classification. In: ISMIR (2005)Google Scholar
  19. 19.
    Mandel, M., Ellis, D.: Labrosa’s audio classification submissions (2009)Google Scholar
  20. 20.
    Mandel, M., Ellis, D.: A web-based game for collecting music metadata. Journal of New Music Research 37(2), 151–165 (2009)CrossRefGoogle Scholar
  21. 21.
    Mimno, D., McCallum, A.: Topic models conditioned on arbitrary features with dirichlet-multinomial regression. In: UAI (2008)Google Scholar
  22. 22.
    Steyvers, M., Griffiths, T.: Probabilistic topic models. In: Landauer, T., McNamara, D.S., Dennis, S., Kintsch, W. (eds.) Handbook of Latent Semantic Analysis. Erlbaum, Hillsdale (2007)Google Scholar
  23. 23.
    Trohidis, K., Tsoumakas, G., Kalliris, G., Vlahavas, I.: Multi-label classification of music emotions. In: ISMIR, pp. 325–330 (2008)Google Scholar
  24. 24.
    Turnbull, D., Barrington, L., Torres, D., Lanckriet, G.: Semantic annotation and retrieval of music and sound effects. TASLP 16(2), 467–476 (2008)Google Scholar
  25. 25.
    Turnbull, D., Liu, R., Barrington, L., Lanckriet, G.: A game-based approach for collecting semantic annotations of music. In: ISMIR, pp. 535–538 (2007)Google Scholar
  26. 26.
    Tzanetakis, G., Cook, P.: Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing 10(5), 293–302 (2002)CrossRefGoogle Scholar
  27. 27.
    von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: CHI, pp. 319–326 (2004)Google Scholar
  28. 28.
    Whitman, B., Smaragdis, P.: Combining musical and cultural features for intelligent style detection. In: ISMIR (2002)Google Scholar
  29. 29.
    Yao, L., Mimno, D., McCallum, A.: Efficient methods for topic model inference on streaming document collections. In: KDD, pp. 937–946 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Edith Law
    • 1
  • Burr Settles
    • 1
  • Tom Mitchell
    • 1
  1. 1.Machine Learning DepartmentCarnegie Mellon University 

Personalised recommendations