Advertisement

International Journal on Digital Libraries

, Volume 9, Issue 2, pp 83–100 | Cite as

The aDORe federation architecture: digital repositories at scale

  • Herbert Van de Sompel
  • Ryan Chute
  • Patrick Hochstenbach
Regular Paper
First Online:

Abstract

The need to federate repositories emerges in two distinctive scenarios. In one scenario, scalability-related problems in the operation of a repository reach a point beyond which continued service requires parallelization and hence federation of the repository infrastructure. In the other scenario, multiple distributed repositories manage collections of interest to certain communities or applications, and federation is an approach to present a unified perspective across these repositories. The high-level, 3-Tier aDORe federation architecture can be used as a guideline to federate repositories in both cases. This paper describes the architecture, consisting of core interfaces for federated repositories in Tier-1, two shared infrastructure components in Tier-2, and a single-point of access to the federation in Tier-3. The paper also illustrates two large-scale deployments of the aDORe federation architecture: the aDORe Archive repository (over 100,000,000 digital objects) at the Los Alamos National Laboratory and the Ghent University Image Repository federation (multiple terabytes of image files).

Keywords

Interoperability Repository federation OAI-PMH OpenURL 
Download to read the full article text

References

  1. 1.
    Apps A.: The JISC information environment service registry. ASSIGNation 22(3), 9–11 (2005)Google Scholar
  2. 2.
    Bekaert, J., Hochstenbach, P., Van de Sompel, H.: Using MPEG-21 DIDL to represent complex digital objects in the Los Alamos National Laboratory Digital Library. D-Lib Mag. 9(11), (2003). doi: 10.1045/november2003-bekaert
  3. 3.
    Bekaert, J., Balakireva, L., Hochstenbach, P., Van de Sompel, H.: Using MPEG-21 and NISO OpenURL for the dynamic dissemination of complex digital objects in the Los Alamos National Laboratory Digital Library. D-Lib Mag. 9(11), (2004). doi: 10.1045/february2004-bekaert
  4. 4.
    Bekaert, J., Van de Sompel, H.: A standards-based solution for the accurate transfer of digital assets. D-Lib Mag. 11(6), (2005). doi: 10.1045/june2005-bekaert
  5. 5.
    Bekaert, J.: Standards-based interfaces for harvesting and obtaining assets from digital repositories. Ph.D. thesis, Ghent University (2006). Retrieved from http://hdl.handle.net/1854/4833
  6. 6.
    Bekaert, J., De Kooning, E., Van de Sompel, H.: Representing digital assets using MPEG-21 digital item declaration. Int. J. Digit. Libr. 6(2), 159–173 (2006). doi: 10.1007/s00799-005-0133-0 Google Scholar
  7. 7.
    Caplan P., Guenther R.: Practical preservation: the PREMIS experience. Libr. Trends 54(1), 111–124 (2005)CrossRefGoogle Scholar
  8. 8.
    Davis, J.R., Lagoze, C.: NCSTRL: design and deployment of a globally distributed digital library. J. Am. Soc. Inf. Sci. 31(3), 273–280 (1999). doi:10.1002/(SICI)1097-4571(2000)51:3<273::AID-ASI6>3.0.CO;2-6Google Scholar
  9. 9.
    DRIVER: Digital Repository Infrastructure Vision for European Research (2006). Retrieved from http://www.driver-repository.eu/
  10. 10.
    International Organization for Standardization: ISO/IEC 21000-2:2003. Information Technology—Multimedia Framework (MPEG-21)—Part 2: Digital Item Declaration, 1st edn. Geneva, Switzerland (2003)Google Scholar
  11. 11.
    International Organization for Standardization: ISO/IEC 21000-3:2003: Information Technology—Multimedia Framework (MPEG-21)—Part 3: Digital Item Identification, 1st edn. Geneva, Switzerland (2003)Google Scholar
  12. 12.
    International Press Telecommunications Council: “IPTC Core” Schema for XMP (2005). Retrieved from http://www.iptc.org/IPTC4XMP/
  13. 13.
    Japan Electronic Industries Development Association: Exchangeable Image File Format v 2.1 (1998). Retrieved from http://www.exif.org
  14. 14.
    Jerez, H., Liu, X., Hochstenbach, P., Van de Sompel, H.: The multi-faceted use of the OAI-PMH in the LANL repository. In: Joint Conference on Digital Libraries Proceedings, pp. 11–20 (2004). doi: 10.1109/JCDL.2004.1336089
  15. 15.
    Jerez, H., Manepalli, G., Blanchi, C., Lannom, L.: ADL-R: the First CORDRA Registry. D-Lib Mag. 12(2), (2006). doi: 10.1045/February2006-jerez
  16. 16.
    Joint Information Systems Committee: Information Environment Service Registry Metadata (2006). Retrieved from http://iesr.ac.uk/metadata/
  17. 17.
    Kahn, R., Wilensky, R.: A framework for distributed digital object services (1995). Retrieved from http://hdl.handle.net/cnri.dlib/tn95-01
  18. 18.
    Kahn, R., Wilensky, R.: A framework for distributed digital object services. Int. J. Digit. Libr. 6(2), 115–123 (1995). doi: 10.1007/s00799-005-0128-x CrossRefGoogle Scholar
  19. 19.
    Kindberg, T., Hawke, S.: RFC 4151: the ‘tag’ URI scheme (2005). Retrieved from http://www.ietf.org/rfc/rfc4151.txt
  20. 20.
    Kunze, J., Arvidson, A., Mohr, G., Stack, M.: The WARC File Format Version 0.9 (2006). Retrieved from http://archive-access.sourceforge.net/warc/warc_file_format-0.9.html
  21. 21.
    Kunze, J., Rodgers, R.P.C.: Internet draft: ARK identifier scheme (2007). Retrieved from http://www.ietf.org/internet-drafts/draft-kunze-ark-14.txt
  22. 22.
    Lagoze C., Davis J.R.: Dienst: an architecture for distributed document libraries. Commun. ACM 38(4), 47 (1995)CrossRefGoogle Scholar
  23. 23.
    Lagoze, C., Van de Sompel, H.: The open archives initiative: building a low-barrier interoperability framework. In: Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 54–62, (2001). doi: 10.1145/379437.379449
  24. 24.
    Lagoze, C., Van de Sompel, H., Nelson, M.L., Warner, S. (eds.): The open archives initiative protocol for metadata harvesting, 2nd edn. (2003). Retrieved from http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm
  25. 25.
    Lagoze, C., Payette, S., Shin, E., Wilper C.: Fedora: an architecture for complex objects and their relationships. Int. J. Digit. Libr. 6(2), 124–138 (2006). doi: 10.1007/s00799-005-0130-3 CrossRefGoogle Scholar
  26. 26.
    Lagoze, C., Van de Sompel, H., Nelson, M.L., Sanderson, R., Warner, S. (eds.): ORE specification—resource map profile of atom (2007). Retrieved from http://www.openarchives.org/ore/0.1/atom
  27. 27.
    Library of Congress, Preservation Metadata Maintenance Activity: PREMIS (2007). Retrieved from http://www.loc.gov/standards/premis/
  28. 28.
    Leach, P., Mealling, M., Salz, R.: RFC 4122: a Universally Unique IDentifier (UUID) URN Namespace. Retrieved from http://www.ietf.org/rfc/rfc4122.txt (2005)
  29. 29.
    Liu, X., Balakireva, L., Van de Sompel, H.: File-based storage of digital objects and constituent datastreams: XMLtapes and internet archive ARC files. Lect. Notes Comput. Sci. 3652, 254–265 (2005). doi: 10.1007/11551362_23 CrossRefGoogle Scholar
  30. 30.
    Los Alamos National Laboratory Research Library: aDORe Archive (2006). Retrieved from http://african.lanl.gov/aDORe/projects/adoreArchive/
  31. 31.
    Los Alamos National Laboratory Research Library: DIDLTools (2006). Retrieved from http://african.lanl.gov/aDORe/projects/DIDLTools/
  32. 32.
    Manepalli, G., Jerez, H., Nelson, M.L.: FeDCOR: an institutional CORDRA registry. D-Lib Mag. 12(2), (2006). doi: 10.1045/february2006-manepalli
  33. 33.
    McDonough, J.P.: METS: standardized encoding for digital library objects. Int. J. Digit. Libr. 6(2), 148–158 (2006). doi: 10.1007/s00799-005-0132-1 CrossRefGoogle Scholar
  34. 34.
    Nelson, M.L., Van de Sompel, H.: IJDL special issue on complex digital objects: Guest editors’ introduction. Int. J. Digit. Libr. 6(2), 113–114 (2006). doi: 10.1007/s00799-005-0127-y CrossRefGoogle Scholar
  35. 35.
    National Information Standards Organization. ANSI/NISO Z39.88-2004: The OpenURL Framework for Context-Sensitive Services. NISO Press, BethesdaGoogle Scholar
  36. 36.
    Rehak, D., Daniel, R., Lannom, R.: A model and infrastructure for federated learning content repositories. Interoperability of web-based educational systems workshop. In: CEUR Workshop Proceedings, vol. 143 (2005). Retrieved from http://cordra.net/cordra/information/publications/2005/www2005/cordrawww2005.pdf
  37. 37.
    Tansley, R., Bass, M., Stuve, D., Branschofsky, M., Chudnov, D., McClellan, G., Smith, M.: The Dspace institutional digital repository system: current functionality. In: Joint Conference on Digital Libraries Proceedings, pp. 87–97 (2003)Google Scholar
  38. 38.
    Tansley, R.: Building a distributed, standards-based repository federation. D-Lib Mag. 12(7/8), (2006). doi: 10.1045/july2006-tansley
  39. 39.
    Universiteitsbiliotheek Gent.: Topografische Collectie. Retrieved from http://adore.ugent.be/topo/ (2006)
  40. 40.
    Universiteitsbibliotheek Gent, Google.: Google and Ghent University Library to make hundreds of thousands of Dutch and French books available online. Press Release. Retrieved from http://lib.ugent.be/info/en/project-google.shtml
  41. 41.
    Van de Sompel, H., Payette, S., Ericksson, J., Lagoze, C., Warner, S.: Rethinking scholarly communication: building the system that scholars deserve. D-Lib Mag. 10(9), (2004). doi: 10.1045/september2004-vandesompel
  42. 42.
    Van de Sompel, H., Nelson, M.L., Lagoze, C., Warner, S.: Resource harvesting within the OAI-PMH framework. D-Lib Mag. 10(12), (2004). doi: 10.1045/december2004-vandesompel
  43. 43.
    Van de Sompel, H., Bekaert, J., Liu, X., Balakireva, L., Schwander, T.: aDORe: a modular, standards-based digital object repository. Comput. J. 48(5), 514–535 (2005). doi: 10.1093/comjnl/bxh114 CrossRefGoogle Scholar
  44. 44.
    Van de Sompel, H., Hammond, T., Neylon, E., Weibel, S.: RFC 4452: The “info” URI scheme for information assets with identifiers in public namespaces (2006). Retrieved from http://www.ietf.org/rfc/rfc4452.txt
  45. 45.
    Van de Sompel, H., Lagoze, C., Bekaert, J., Liu, X., Payette, S., Warner, S.: An interoperable fabric for scholarly value chains. D-Lib Mag. 12(10), (2006). doi: 10.1045/October2006-vandesompel
  46. 46.
    Van de Sompel, H., Lagoze, C.: Interoperability for the discovery, use, and re-use of units of scholarly communication. CT Watch Q. 3(3), (2007). Retrieved from http://www.ctwatch.org/quarterly/articles/2007/08/interoperability-for-the-discovery-use-and-re-use-of-units-of-scholarly-communication/
  47. 47.
    Warner, S., Bekaert, J., Lagoze, C., Liu, X., Payette, S., Van de Sompel, H.: Pathways: augmenting interoperability across scholarly repositories. Int. J. Digit. Libr. 7(1–2), 35–52 (2007). doi: 10.1007/s00799-007-0025-6 CrossRefGoogle Scholar

Copyright information

© Los Alamos National Laboratory 2008

Authors and Affiliations

  • Herbert Van de Sompel
    • 1
  • Ryan Chute
    • 1
  • Patrick Hochstenbach
    • 2
  1. 1.Digital Library Research and Prototyping TeamLos Alamos National LaboratoryLos AlamosUSA
  2. 2.Universiteitsbibliotheek, Universiteit GentGhentBelgium

Personalised recommendations