Advertisement

A digital library framework for heterogeneous music collections: from document acquisition to cross-modal interaction

  • David Damm
  • Christian Fremerey
  • Verena Thomas
  • Michael Clausen
  • Frank Kurth
  • Meinard Müller
Article

Abstract

In this paper, we present a digital library system for managing heterogeneous music collections. The heterogeneity refers to various document types and formats as well as to different modalities, e. g., CD-audio recordings, scanned sheet music, and lyrics. The system offers a full-fledged, widely automated document processing chain: digitization, indexing, annotation, access, and presentation. Our system is implemented as a generic and modular music repository based on a service-oriented software architecture. As a particular strength of our approach, the various documents representing aspects of a piece of music are jointly considered in all stages of the document processing chain. Our user interfaces allow for a multimodal and synchronized presentation of documents (WYSIWYH: what you see is what you hear), a score- or lyrics-based navigation in audio, as well as a cross- and multimodal retrieval. Hence, our music repository may be called a truly cross-modal library system. In our paper, we describe the system components, outline the techniques of the document processing chain, and illustrate the implemented functionalities for user interaction. We describe how the system is put into practice at the Bavarian State Library (BSB) Munich as a part of the German PROBADO Digital Library Initiative (PDLI).

Keywords

Music digital library system Multimodality Cross-modal navigation Content-based retrieval Music synchronization Music information retrieval 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Allamanche, E., Herre, J., Fröba, B., Cremer, M.: AudioID: Towards content-based identification of audio material. In: Proceedings of the 110th Audio Engineering Society (AES) Convention (2001)Google Scholar
  2. 2.
    Arifi V., Clausen M., Kurth F., Müller M.: Synchronization of music data in score-, MIDI- and PCM-format. Comput. Musicol. 13, 9–33 (2004)Google Scholar
  3. 3.
    Baggi, D., Barate, A., Haus, G., Ludovico, L.A.: NINA—navigating and interacting with notation and audio. In: Proceedings of the 2nd International Workshop on Semantic Media Adaptation and Personalization (SMAP), pp. 134–139. IEEE Computer Society, Washington, DC, USA (2007). doi: 10.1109/SMAP.2007.28
  4. 4.
    Bainbridge, D., Thompson, J., Witten, I.H.: Assembling and enriching digital library collections. In: Proceedings of the 3rd ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), pp. 323–334. IEEE Computer Society, Washington, DC, USA (2003)Google Scholar
  5. 5.
    Bartsch M.A., Wakefield G.H.: Audio thumbnailing of popular music using chroma-based representations. IEEE Trans. Multimed. 7(1), 96–104 (2005)CrossRefGoogle Scholar
  6. 6.
    Birmingham, W.P., Pardo, B., Meek, C., Shifrin, J.: The MusArt music-retrieval system: an overview. D-Lib Magazine 8(2) (2002). doi: 10.1045/february2002birmingham. URL http://www.dlib.org/dlib/february02/birmingham/02birmingham.html
  7. 7.
    Birmingham, W.P., O’Malley, K., Dunn, J.W., Scherle, R.: V2V: a second variation on query-by-humming. In: Proceedings of the 3rd ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), pp. 380–380. IEEE Computer Society, Washington, DC, USA (2003)Google Scholar
  8. 8.
    Blümel, I., Krottmaier, H., Wessel, R.: The PROBADO framework: a repository for architectural 3D-models. In: International Conference on Online Repositories in Architecture. Fraunhofer irb Verlag (2008)Google Scholar
  9. 9.
    Byrd, D., Schindele, M.: Prospects for improving OMR with multiple recognizers. In: Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), pp. 41–46 (2006)Google Scholar
  10. 10.
    Cano, P., Battle, E., Kalker, T., Haitsma, J.: A review of audio fingerprinting. In: Proceedings of the 5th IEEE Workshop on Multimedia Signal Processing (MMSP) (2002)Google Scholar
  11. 11.
    Choudhury, G., DiLauro, T., Droettboom, M., Fujinaga, I., Harrington, B., MacMillan, K.: Optical music recognition system within a large-scale digitization project. In: Proceedings of the 1st International Symposium on Music Information Retrieval (ISMIR) (2000)Google Scholar
  12. 12.
    Clausen M., Kurth F.: A unified approach to content-based and fault-tolerant music recognition. IEEE Trans. Multimed. 6(5), 717–731 (2004)CrossRefGoogle Scholar
  13. 13.
    D’Aguanno A., Vercellesi G.: Automatic music synchronization using partial score representation based on IEEE 1599. J. Multimed. 4(1), 19–24 (2009)Google Scholar
  14. 14.
    Damm, D., Kurth, F., Fremerey, C., Clausen, M.: A concept for using combined multimodal queries in digital music libraries. In: Proceedings of the 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL) (2009)Google Scholar
  15. 15.
    Damnjanovic, I., Reiss, J., Barry, D.: Enabling access to sound archives through integration, enrichment , and retrieval. In: Proceedings of the 2008 IEEE International Conference on Multimedia and Expo (ICME), pp. 1597–1598 (2008). doi: 10.1109/ICME.2008.4607756
  16. 16.
    Dannenberg, R.B., Raphael, C.: Music score alignment and computer accompaniment. In: Pardo, B. (ed.): Special Issue: Music Information Retrieval, vol. 49, pp. 38–43. ACM, New York, NY, USA (2006). doi: 10.1145/1145287.1145311
  17. 17.
    Diet, J., Kurth, F.: The PROBADO music repository at the Bavarian State Library. In: Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR), pp. 501–504 (2007)Google Scholar
  18. 18.
    Dixon, S., Widmer, G.: MATCH: A music alignment tool chest. In: Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR) (2005)Google Scholar
  19. 19.
    Dunn, J.W., Byrd, D., Notess, M., Riley, J., Scherle, R.: Variations2: Retrieving and using music in an academic setting. In: Pardo, B. (ed.): Special Issue: Music Information Retrieval, vol. 49, pp. 53–58. ACM, New York, NY, USA (2006). doi: 10.1145/1145287.1145314
  20. 20.
    European Union: EUROPEANA (2007). http://www.europeana.eu/portal/index.html
  21. 21.
    Fremerey, C., Müller, M., Kurth, F., Clausen, M.: Automatic mapping of scanned sheet music to audio recordings. In: Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR), pp. 413–418. Philadelphia, USA (2008)Google Scholar
  22. 22.
    Fremerey, C., Clausen, M., Ewert, S., Müller, M.: Sheet music-audio identification. In: Proceedings of the 10th International Conference on Music Information Retrieval (ISMIR), pp. 645–650. Kobe, Japan (2009a)Google Scholar
  23. 23.
    Fremerey, C., Müller, M., Clausen, M.: Towards bridging the gap between sheet music and audio. In: Selfridge-Field, E., Wiering, F., Wiggins, G.A. (eds.) Knowledge Representation for Intelligent Music Processing, no. 09051 in Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany, Dagstuhl, Germany (2009b). http://drops.dagstuhl.de/opus/volltexte/2009/1965
  24. 24.
    Fremerey, C., Müller, M., Clausen, M.: Handling repeats and jumps in score-performance synchronization. In: Proceedings of the 11th International Conference on Music Information Retrieval (ISMIR). Utrecht, the Netherlands (2010)Google Scholar
  25. 25.
    Good, M.: MusicXML: An internet-friendly format for sheet music. In: Proceedings XML Conference and Exposition (2001). http://www.idealliance.org/papers/xml2001/papers/html/03-04-05.html
  26. 26.
    Google Inc.: Google Book Search (2007). http://books.google.com
  27. 27.
    Goto, M.: A chorus-section detecting method for musical audio signals. In: Proceedings of the IEEE Internatinal Conference on Acoustics, Speech, and Signal Processing ICASSP, pp. 437–440 (2003)Google Scholar
  28. 28.
    Gracenote: Music Search (2008). http://www.gracenote.com/
  29. 29.
    Hankinson, A., Pugin, L., Fujinaga, I.: Interfaces for document representation in digital music libraries. In: Proceedings of the 10th International Conference on Music Information Retrieval (ISMIR), pp. 39–44 (2009)Google Scholar
  30. 30.
    Hu, N., Dannenberg, R., Tzanetakis, G.: Polyphonic audio matching and alignment for music retrieval. In: Proceedings of the 4th IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) (2003)Google Scholar
  31. 31.
    Huber D.M.: The MIDI Manual. Focal Press, Boston (1999)Google Scholar
  32. 32.
    IFLA Study Group: Functional requirements for bibliographic records: Final report. UBCIM Publications-New Series 19 (1998). http://www.ifla.org/VII/s13/frbr/frbr.htm
  33. 33.
    Kahle, B.: Internet Archive (1996). http://www.archive.org/index.php
  34. 34.
    Klapuri, A., Davy, M. (eds): Signal Processing Methods for Music Transcription. Springer, New York (2006)Google Scholar
  35. 35.
    Krajewski, E.: DE-PARCON softwaretechnologie (2008). http://www.de-parcon.de/
  36. 36.
    Krottmaier, H., Kurth, F., Steenweg, T., Appelrath, H.J., Fellner, D.: PROBADO—a generic repository integration framework. In: Proceedings of the 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL) (2007)Google Scholar
  37. 37.
    Kurth, F., Müller, M., Fremerey, C.: Audio Matching für symbolische Musikdaten. In: Fortschritte der Akustik, Tagungsband der DAGA (2007a).http://www.cs.uni-bonn.de/~meinard/publications/07_KuMuFr_DAGA_SymbAudioMatch.pdf
  38. 38.
    Kurth, F., Müller, M., Fremerey, C., Chang, Y., Clausen, M.: Automated synchronization of scanned sheet music with audio recordings. In: Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR), pp. 261–266 (2007b)Google Scholar
  39. 39.
    Kurth F., Müller M.: Efficient index-based audio matching. IEEE Trans. Audio Speech Lang. Process. 16(2), 382–395 (2008)CrossRefGoogle Scholar
  40. 40.
    Landone, C., J., H., Reiss, J.: Enabling access to sound archives through integration, enrichment and retrieval: the EASAIER project. In: Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR), pp. 159–160 (2007)Google Scholar
  41. 41.
    Ludovico L.A.: IEEE 1599: a multi-layer approach to music description. J. Multimed. 4(1), 9–14 (2009)Google Scholar
  42. 42.
    Maddage, N.C., Xu, C., Kankanhalli, M.S., Shao, X.: Content-based music structure analysis with applications to music semantics understanding. In: Proceedings of the ACM Multimedia, pp. 112–119. New York, NY, USA (2004). doi: 10.1145/1027527.1027549
  43. 43.
    Müller M.: Information Retrieval for Music and Motion. Springer, New York (2007)CrossRefGoogle Scholar
  44. 44.
    Müller, M., Appelt, D.: Path-constrained partial music synchronization. In: Proceedings of the 34th International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 1, pp. 65–68. Las Vegas, Nevada, USA (2008)Google Scholar
  45. 45.
    Müller, M., Clausen, M.: Transposition-invariant self-similarity matrices. In: Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR 2007), pp. 47–50 (2007)Google Scholar
  46. 46.
    Müller M., Kurth F.: Towards structural analysis of audio recordings in the presence of musical variations. EURASIP J. Appl. Signal Process. 2007(89686), 18 (2007)Google Scholar
  47. 47.
    Müller, M., Kurth, F., Röder, T.: Towards an efficient algorithm for automatic score-to-audio synchronization. In: Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR), pp. 365–372. Barcelona, Spain (2004)Google Scholar
  48. 48.
    Müller, M., Kurth, F., Damm, D., Fremerey, C., Clausen, M.: Lyrics-based audio retrieval and multimodal navigation in music collections. In: Proceedings of the 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL) (2007)Google Scholar
  49. 49.
    Orio, N.: Alignment of performances with scores aimed at content-based music access and retrieval. In: Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), pp. 479–492. Rome, Italy (2002)Google Scholar
  50. 50.
    Orio, N., Lemouton, S., Schwarz, D.: Score following: State of the art and new developments. In: Proceedings of the Conference of New Interfaces for Musical Expression (NIME), pp. 36–41. Montreal, CA (2003)Google Scholar
  51. 51.
    Pardo, B.: Introduction. In: Pardo, B. (ed.): Special Issue: Music Information Retrieval, vol. 49, pp. 28–31. ACM, New York, NY, USA (2006). doi: 10.1145/1145287.1145309
  52. 52.
    Peeters, G., Burthe, A.L., Rodet, X.: Toward automatic music audio summary generation from signal analysis. In: Proceedings of the 3th International Conference on Music Information Retrieval (ISMIR) (2002)Google Scholar
  53. 53.
    Pickens, J., Bello, J.P., Monti, G., Crawford, T., Dovey, M., Sandler, M.: Polyphonic score retrieval using polyphonic audio queries: a harmonic modeling approach. In: Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR), pp. 140–149. Paris, France (2002)Google Scholar
  54. 54.
    Pinto A.: Multi-model music content description and retrieval using IEEE 1599 XML standard. J. Multimed. 4(1), 30–39 (2009)Google Scholar
  55. 55.
    Raphael, C.: A hybrid graphical model for aligning polyphonic audio with musical scores. In: Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR) (2004)Google Scholar
  56. 56.
    Rauber, A., Frühwirth, M.: Automatically analyzing and organizing music archives. In: Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), Springer Lecture Notes in Computer Science. Springer, Darmstadt, Germany (2001). http://www.ifs.tuwien.ac.at/ifs/research/publications.html
  57. 57.
    Selfridge-Field, E. (eds): Beyond MIDI: The Handbook of Musical Codes. MIT Press, Cambridge (1997)Google Scholar
  58. 58.
    Soulez, F., Rodet, X., Schwarz, D.: Improving polyphonic and poly-instrumental music to score alignment. In: Proceedings of the 4th International Conference on Music Information Retrieval (ISMIR) (2003)Google Scholar
  59. 59.
    Suyoto I.S.H., Uitdenbogerd A.L., Scholer F.: Searching musical audio using symbolic queries. IEEE Trans. Audio Speech Lang. Process. 16(2), 372–381 (2008). doi: 10.1109/TASL.2007.911644 CrossRefGoogle Scholar
  60. 60.
    Turetsky, R.J., Ellis, D.P.: Force-aligning MIDI syntheses for polyphonic music transcription generation. In: Proceedings of the 4th International Conference on Music Information Retrieval (ISMIR) (2003a)Google Scholar
  61. 61.
    Turetsky, R.J., Ellis, D.P.W.: Ground-truth transcriptions of real music from force-aligned MIDI syntheses. In: Proceedings of the 4th International Conference on Music Information Retrieval (ISMIR) (2003b)Google Scholar
  62. 62.
    Typke, R., Wiering, F., Veltkamp, R.C.: A survey of music information retrieval systems. In: Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR), pp. 153–160 (2005)Google Scholar
  63. 63.
    Union der deutschen Akademien der Wissenschaften: Neue Mozart Ausgabe (2007). http://www.nma.at/
  64. 64.
    United States: World Digital Library (2009). http://www.wdl.org/en/
  65. 65.
    University of Chicago Library: Chopin Early Edition (2004). http://chopin.lib.uchicago.edu/
  66. 66.
    University of Rochester Libraries: UR research—Sibley Music Library (2009). https://urresearch.rochester.edu/home.action
  67. 67.
    W3C: Web Services. http://www.w3.org/2002/ws/
  68. 68.
    Wang, A.L.C.: An industrial-strength audio search algorithm (2003). http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf
  69. 69.
    Wang, Y., Kan, M.Y., Nwe, T.L., Shenoy, A., Yin, J.: LyricAlly: automatic synchronization of acoustic musical signals and textual lyrics. In: Proceedings of the 12th annual ACM International Conference on Multimedia, pp. 212–219. ACM Press, New York, NY, USA (2004). http://doi.acm.org/10.1145/1027527.1027576
  70. 70.
    Wiener Wissenschafts-, Forschungs- und Technologiefonds: Schubert-Autographe. http://www.schubert-online.at/
  71. 71.
    Witten I.H., Moffat A., Bell T.C.: Managing Gigabytes. 2nd edn. Van Nostrand Reinhold, New York (1999)Google Scholar
  72. 72.
    Witten, I.H., Mcnab, R.J., Boddie, S.J., Bainbridge, D.: Greenstone: A comprehensive open-source digital library software system. In: Proceedings of the 5th ACM International Conference on Digital Libraries (2000). http://citeseer.ist.psu.edu/witten99greenstone.html

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • David Damm
    • 1
  • Christian Fremerey
    • 1
  • Verena Thomas
    • 1
  • Michael Clausen
    • 1
  • Frank Kurth
    • 2
  • Meinard Müller
    • 3
  1. 1.Department of Computer Science IIIUniversity of BonnBonnGermany
  2. 2.Fraunhofer Institute for CommunicationInformation Processing and Ergonomics (FKIE)WachtbergGermany
  3. 3.Saarland University and Max-Planck-Institut für InformatikSaarbrückenGermany

Personalised recommendations