The LEMO annotation framework: weaving multimedia annotations with the web

  • Bernhard Haslhofer
  • Wolfgang Jochum
  • Ross King
  • Christian Sadilek
  • Karin Schellner
Regular Paper


Cultural institutions and museums have realized that annotations contribute valuable metadata for search and retrieval, which in turn can increase the visibility of the digital items they expose via their digital library systems. By exploiting annotations created by others, visitors can discover content they would not have found otherwise, which implies that annotations must be accessible and processable for humans and machines. Currently, however, there exists no widely adopted annotation standard that goes beyond specific media types. Most institutions build their own in-house annotation solution and employ proprietary annotation models, which are not interoperable with those of other systems. As a result, annotation data are usually stored in closed data silos and visible and processable only within the scope of a certain annotation system. As the main contribution of this paper, we present the LEMO Annotation Framework. It (1) provides a uniform annotation model for multimedia contents and various types of annotations, (2) can address fragments of various content-types in a uniform, interoperable manner and (3) pulls annotations out of closed data silos and makes them available as interoperable, dereferencable Web resources. With the LEMO Annotation Framework annotations become part of the Web and can be processed, linked, and referenced by other services. This in turn leads to even higher visibility and increases the potential value of annotations.


Annotations Semantics Interoperability Fragment identification Multimedia Web 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adobe Inc.: Adobe acrobat professional.
  2. 2.
    Agosti, M., Benfante, L., Orio, N.: Ipsa: A digital archive of herbals to support scientific research. In: Proceedings of the International Conference on Asian Digital Libraries (ICADL), pp. 253–264 (2003)Google Scholar
  3. 3.
    Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Orio, N., Panizzi, E., Pejtersen, A.M., Thiel, U.: DiLAS: a digital library annotation service. In: Proceedings of Annotation for Collaboration—A Workshop on Annotation Models, Tools and Practices (2005)Google Scholar
  4. 4.
    Agosti M., Ferro N.: Annotations: Enriching a digital library. In: Koch, T., Sølvberg, I.(eds) ECDL, Lecture Notes in Computer Science, vol. 2769, pp. 88–100. Springer, Berlin (2003)Google Scholar
  5. 5.
    Agosti M., Ferro N.: Annotations as context for searching documents. In: Crestani, F., Ruthven, I.(eds) CoLIS, Lecture Notes in Computer Science, vol. 3507, pp. 155–170. Springer, Berlin (2005)Google Scholar
  6. 6.
    Agosti M., Ferro N.: Search strategies for finding annotations and annotated documents: The fast service. In: Larsen, H.L., Pasi, G., Arroyo, D.O., Andreasen, T., Christiansen, H.(eds) FQAS, Lecture Notes in Computer Science, vol. 4027, pp. 270–281. Springer, Berlin (2006)Google Scholar
  7. 7.
    Agosti, M., Ferro, N.: A formal model of annotations of digital content. ACM Trans. Inf. Syst. 26(1) (2007)Google Scholar
  8. 8.
    Agosti M., Ferro N., Frommholz I., Thiel U.: Annotations in digital libraries and collaboratories—facets, models and usage. In: Heery, R., Lyon, L.(eds) ECDL, Lecture Notes in Computer Science, vol. 3232, pp. 244–255. Springer, Berlin (2004)Google Scholar
  9. 9.
    Baker, T., Dekkers, M., Heery, R., Patel, M., Salokhe, G.: What terms does your metadata use? Application profiles as machine-understandable narratives. J. Digit. Inform. 2(2) (2001).
  10. 10.
    Berners-Lee, T.: Linked data (2006).
  11. 11.
    Bottoni, P., Levialdi, S., Labella, A., Panizzi, E., Trinchese, R., Gigli, L.: Madcow: a visual interface for annotating web pages. In: AVI ’06: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 314–317. ACM Press, New York (2006)Google Scholar
  12. 12.
    Brocks, H., Stein, A., Thiel, U., Frommholz, I., Dirsch- Weigand, A.: How to incorporate collaborative discourse in cultural digital libraries. In: Proceedings of the ECAI 2002 Workshop on Semantic Authoring, Annotation & Knowledge Markup (SAAKM02), Lyon, France (2002)Google Scholar
  13. 13.
    Brush, A.J.B., Bargeron, D., Gupta, A., Cadiz, J.J.: Robust annotation positioning in digital documents. In: CHI, pp. 285–292 (2001)Google Scholar
  14. 14.
    Bush V.: As we may think. Atl. Mon. 176(1), 101–108 (1945)Google Scholar
  15. 15.
    Candela, L., Castelli, D., Ferro, N., Ioannidis, Y., Koutrika, G., Meghini, C., Pagano, P., Ross, S., Soergel, D., Agosti, M., Dobreva, M., Katifori, V., Schuldt, H.: The DELOS digital library reference model—foundations for digital libraries, version 0.98. Tech. rep., DELOS—A Network of Excellence on Digital Libraries (2007).
  16. 16.
    DCMI: Dublin Core Metadata Element Set, Version 1.1. Dublin Core Metadata Initiative (2006).
  17. 17.
    Denoue, L., Vignollet, L.: An annotation tool for web browsers and its applications to information retrieval (2000)Google Scholar
  18. 18.
    Frisse, M.E.: Searching for information in a hypertext medical handbook. In: Hypertext’87 Proceedings, November 13–15, 1987, Chapel Hill, North Carolina, USA, pp. 57–66. ACM, New York (1987)Google Scholar
  19. 19.
    Frommholz, I., Fuhr, N.: Probabilistic, object-oriented logics for annotation-based retrieval in digital libraries. In: JCDL ’06: Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 55–64. ACM, New York, NY, USA (2006)Google Scholar
  20. 20.
    Frommholz, I., Fuhr, N.: Probabilistic, object-oriented logics for annotation-based retrieval in digital libraries. In: Marchionini, G., Nelson, M.L., Marshall, C.C. (eds.) JCDL, pp. 55–64. ACM, New York (2006)Google Scholar
  21. 21.
    Geurts, J., van Ossenbruggen, J., Hardman, L.: Requirements for practical multimedia annotation. In: Workshop on Multimedia and the Semantic Web, pp. 4–11. Heraklion, Crete (2005).
  22. 22.
    Gonçalves M.A., Fox E.A., Watson L.T., Kipp N.A.: Streams, structures, spaces, scenarios, societies (5s): a formal model for digital libraries. ACM Trans. Inf. Syst. 22(2), 270–312 (2004)CrossRefGoogle Scholar
  23. 23.
    Google Inc.: Google notebook.
  24. 24.
    Grahl Software Design: Pdf annotator.
  25. 25.
    Heery, R., Patel, M.: Application profiles: mixing and matching metadata schemas (2000).
  26. 26.
    Hunter, J., Schroeter, R., Guerin, J., Henderson, M.: Vannotea.
  27. 27.
    ISO/IEC: Introducing MPEG-21 Part 17—An Overview. International Organization for Standardization—MPEG Working Group (ISO/IEC JTC 1/SC 29/WG 11/N7221) (2005).
  28. 28.
    ISO/IEC: Multimedia framework (MPEG-21)—Part 17: Fragment Identification of MPEG Resources. (International Organization for Standardization, Geneva 2006)Google Scholar
  29. 29.
    Jacobs, I., Walsh, N.: Architecture of the World Wide Web, Volume One (2004).
  30. 30.
    Jochum, W.: Requirements of fragment identification. In: International Conferences on New Media Technology and Semantic Systems, pp. 172–179. Springer, Berlin (2007)Google Scholar
  31. 31.
    Jochum W., Kaiser M., Schellner K., Wirl F.: Living memory annotation tool—image annotations for digital libraries. In: Kovács, L., Fuhr, N., Meghini, C.(eds) ECDL, Lecture Notes in Computer Science, vol. 4675, pp. 549–550. Springer, Berlin (2007)Google Scholar
  32. 32.
    Kahan, J., Koivunen, M.R.: Annotea: an open RDF infrastructure for shared Web annotations. In: WWW ’01: Proceedings of the 10th International Conference on World Wide Web, pp. 623–632. ACM Press, New York (2001)Google Scholar
  33. 33.
    Kautz, H., Selman, B., Shah, M.: Referral web: combining social networks and collaborative filtering. Commun. ACM 40(3), 63–65 (1997).
  34. 34.
    Koivunen, M.R.: Semantic authoring by tagging with annotea social bookmarks and topics. In: The 5th International Semantic Web Conference (ISWC2006)—1st Semantic Authoring and Annotation Workshop (SAAW2006), Athens (2006)Google Scholar
  35. 35.
    Kruk S.R., Decker S., Decker L.Z.S., Zieborak L.: Jeromedl—adding semantic web technologies to digital libraries. In: Andersen, K.V., Debenham, J.K., Wagner, R.(eds) DEXA, Lecture Notes in Computer Science, vol. 3588, pp. 716–725. Springer, Berlin (2005)Google Scholar
  36. 36.
    Lagoze C., Payette S., Shin E., Payette C.W.S., Shin E., Wilper C.: Fedora: an architecture for complex objects and their relationships. Int. J. Digit. Libr. 6(2), 124–138 (2006)CrossRefGoogle Scholar
  37. 37.
    Library of Congress: Library of Congress Subject Headings (LCSH) (2007).
  38. 38.
    Marshall, C.C.: Annotation: from paper books to the digital library. In: DL ’97: Proceedings of the Second ACM International Conference on Digital Libraries, pp. 131–140. ACM Press, New York (1997)Google Scholar
  39. 39.
    Marshall, C.C.: Toward an ecology of hypertext annotation. In: UK Conference on Hypertext, pp. 40–49 (1998)Google Scholar
  40. 40.
    Marshall, C.C.: The future of annotation in a digital (paper) world. In: Twidale, H. (eds.) Successes and Failures of Digital Libraries, pp. 97–117. Urbana-Champaign, University of Illinois (2000).
  41. 41.
    Marshall, C.C., Brush, B.A.J.: Exploring the relationship between personal and public annotations. In: JCDL ’04: Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 349–357. ACM Press, New York (2004)Google Scholar
  42. 42.
    Maryland Information and Network Dynamics Lab Semantc Web Agents Project (MINDSWAP): Photostuff project homepage.
  43. 43.
    Mojiti Inc.: Mojiti.
  44. 44.
    Nichols D.M., Pemberton D., Dalhoumi S., Larouk O., Belisle C., Twidale M.: Debora: Developing an interface to support collaboration in a digital library. In: Borbinha, J.L., Baker, T.(eds) ECDL, Lecture Notes in Computer Science, vol. 1923, pp. 239–248. Springer, Berlin (2000)Google Scholar
  45. 45.
    OCLC Online Computer Library Center: Dewey Decimal Classification (DDC) (2007).
  46. 46.
    Ovsiannikov I.A., Arbib M.A., McNeill T.H.: Annotation technology. Int. J. Hum.-Comput. Stud. 50(4), 329–362 (1999)CrossRefGoogle Scholar
  47. 47.
    Payette, S., Lagoze, C.: Flexible and extensible digital object and repository architecture (fedora). In: ECDL ’98: Proceedings of the Second European Conference on Research and Advanced Technology for Digital Libraries, pp. 41–59. Springer, London (1998)Google Scholar
  48. 48.
    Petridis, K., Anastasopoulos, D., Saathoff, C., Timmermann, N., Kompatsiaris, Y., Staab, S.: M-ontomat-annotizer: Image annotation linking ontologies and multimedia low-level features. In: Gabrys, B., Howlett, R.J., Jain, L.C. (eds.) KES (3), Lecture Notes in Computer Science, vol. 4253, pp. 633–640. Springer, Berlin (2006)Google Scholar
  49. 49.
    Pfeiffer, S., Parker, C., Pang, A.: Specifying time intervals in uri queries and fragments of time-based web resources. Network Working Group, Internet-Draft (2005).
  50. 50.
    Phelps, T., Wilensky, R.: Robust intra-document locations (2000).
  51. 51.
    Phelps, T.A., Wilensky, R.: Multivalent annotations. In: European Conference on Digital Libraries, pp. 287–303 (1997)Google Scholar
  52. 52.
    Salembier, P., Smith, J.: Mpeg-7 multimedia description schemes. In: IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, pp. 748–759 (2001)Google Scholar
  53. 53.
    Schilit, B.N., Golovchinsky, G., Price, M.N.: Beyond paper: Supporting active reading with free form digital ink annotations. In: CHI, pp. 249–256 (1998)Google Scholar
  54. 54.
    Schroeter, R., Hunter, J., Newman, A.: Annotating relationships between multiple mixed-media digital objects by extending annotea. In: Franconi, E., Kifer, M., May, W. (eds.) ESWC, vol. 4519. Springer, Berlin (2007)Google Scholar
  55. 55.
    The Bricks Community: BRICKS—Building Resources for Integrated Cultural Knowledge Services (2007).
  56. 56.
    The FotoNotes Project: Fotonotes.
  57. 57.
    Troncy, R., Harman, L., van Ossenbruggen, J., Hausenblas, M.: Identifying spatial and temporal media fragments on the web. In: W3C Video on the Web Workshop (2007).
  58. 58.
    Viddler Inc.: Viddler.
  59. 59.
    Wilde, E., Baschnagel, M.: Fragment identifiers for plain text files. In: HYPERTEXT ’05: Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, pp. 211–213. ACM Press, New York (2005)Google Scholar
  60. 60.
    Witten, I.H., Boddie, S.J., Bainbridge, D., McNab, R.J.: Greenstone: a comprehensive open-source digital library software system. In: DL ’00: Proceedings of the fifth ACM conference on Digital libraries, pp. 113–121. ACM Press, New York (2000). doi:10.1145/336597.336650
  61. 61.
    Wu, X., Zhang, L., Yu, Y.: Exploring social annotations for the semantic web. In: WWW ’06: Proceedings of the 15th International Conference on World Wide Web, pp. 417–426. ACM Press, New York (2006)Google Scholar
  62. 62.
    Yahoo Inc.: Flickr.
  63. 63.
    Zoomify Inc.: Zoomify annotation system.

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Bernhard Haslhofer
    • 1
  • Wolfgang Jochum
    • 1
  • Ross King
    • 2
  • Christian Sadilek
    • 2
  • Karin Schellner
    • 3
  1. 1.Department of Distributed and Multimedia SystemsUniversity of ViennaViennaAustria
  2. 2.Digital Memory EngineeringAustrian Research Centers GmbH, ARCSeibersdorfAustria
  3. 3.ViennaAustria

Personalised recommendations