Digital Preservation Metadata Practice for Web Archives

  • Clément OuryEmail author
  • Karl-Rainer Blumenthal
  • Sébastien Peyrard


Twenty years after the pioneering experiments performed by Internet Archive and few national libraries, web archiving has become a common activity of many scientific, cultural, and heritage institutions. They are using a set of tools, generally open source, to identify, harvest, store, index, make available to end users, and preserve internet content over the long term. Institutions seeking to preserve web archives are however facing major challenges: not only the huge amount of collected data, but also the lack of fully reliable metadata, which are crucial to understand the web archives and inform future preservation actions upon them. Web archives are generally stored in container formats, notably the ARC file format and its successor, the WARC format—an ISO standard. Context and Provenance information, generated prior to or as part of the harvesting process, is stored in these container formats, but other metadata—especially information on the formats of the collected files—may be generated afterwards. To store and archive these assets in digital repositories, it is necessary to record and manage their metadata. Therefore, institutions need to make data and metadata modeling choices, which should be consistent not only with the design of their own repository and the kind and amount of data they have to preserve, but also with their conceptual view of the nature of web archives. This paper presents the choices and achievements of the National Library of France, called “container modeling”. It then compares it to the approaches of other members of the International Internet Preservation Consortium and to the projects of the New York Art Resources Consortium. It underlines how the different solutions are implemented with PREMIS and concludes with the use of format identification tools and metadata vocabularies for emulation strategies.


National Library Semantic Unit Digital Preservation Digital Repository Provenance Information 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bonnel S, Oury C (2014) Selecting websites in an encyclopaedic national library: a shared collection policy for internet legal deposit at the BnF. Paper presented at the 80th IFLA WLIC conference, Lyon, France. Accessed 22 Mar 2016
  2. 2.
    Hockx-Yu, Helen (2014) Archiving social media in the context of non-print legal deposit. Paper presented at the 80th IFLA WLIC conference, Lyon, France. Accessed 22 Mar 2016
  3. 3.
    Brügger N (2005) Archiving websites: general considerations and strategies. Center for Internetforskning, Aarhus Universitet, Aarhus, 76 ppGoogle Scholar
  4. 4.
    Brügger N (2015) Humanities, digital humanities, media studies, internet studies: an inaugural lecture. Center for Internetforskning, Aarhus Universitet, Aarhus, 16 ppGoogle Scholar
  5. 5.
    Graham S, Milligan I, Weingart S (2015) Exploring big historical data: the historian’s macroscope. Imperial College Press, London, 400 ppGoogle Scholar
  6. 6.
    Illien G, Sanz P, Sepetjan S, Stirling P (2011) The state of e-legal deposit in France: looking back at five years of putting new legislation into practice and envisioning the future. IFLA J 38(1). Accessed 23 Oct 2016
  7. 7.
    IIPC: International Internet Preservation Consortium (2016) Official web site. Accessed 01 Feb 2016
  8. 8.
    Consultative Committee for Space Data Systems, International Organization for Standardization (2012) Reference model for an Open Archival Information System (OAIS): issue 2. Accessed 06 Jan 2016
  9. 9.
  10. 10.
    Preservica (2016) Accessed 20 Mar 2016
  11. 11.
    The Internet Archive (2016) Wayback machine. Accessed 01 Feb 2016
  12. 12.
    National Library of New Zealand, British Library (2015) Web curator tool. Accessed 01 Feb 2016
  13. 13.
    Royal Library of Denmark, State and University Library of Aarhus (2015) NetarchiveSuite. Accessed 01 Feb 2015
  14. 14.
    The Internet Archive (2016) ArchiveIt! Accessed 01 Feb 2016
  15. 15.
    Bermès E, Fauduet L, Peyrard S (2010) A data first approach to digital preservation: the SPAR project. Paper presented at the 76th IFLA general conference and assembly, Gothenburg. Accessed 01 Feb 2016
  16. 16.
    Le Follic A, Stirling P, Wendland B (2013) Putting it all together: creating a unified web harvesting workflow at the Bibliothèque nationale de France.
  17. 17.
    BnF (2009) The WARC file format (ISO 28500): information, maintenance, drafts. Accessed 01 Feb 2016
  18. 18.
    Burner M, Kahle B (1996) Arc file format. Accessed 01 Feb 2016
  19. 19.
    Oury C (2010) Large-scale collections under the magnifying glass: format identification for web archives. Paper presented at the 7th international conference on preservation of digital objects (iPRES), Vienna. Accessed 01 Feb 2016
  20. 20.
    Abrams S, Cramer T, Morrissey S (2009) “What? So What”: the next-generation JHOVE2 architecture for format-aware characterization. Int J Digit Curation 4(3):132–136. doi: 10.2218/ijdc.v4i3.122 CrossRefGoogle Scholar
  21. 21.
  22. 22.
    Open Planets Foundation (2014) JHOVE2: the next-generation architecture for format-aware characterization. Accessed 01 Feb 2016
  23. 23.
    Gones J, Oury C, Steinke T (2012) Ensuring long-term access to the memory of the web: preservation working group of the international internet preservation consortium. Int Preserv News 28:34–37. Accessed 02 Feb 2016
  24. 24.
    Oury C, Peyrard S (2011) From the World Wide Web to digital library stacks. Paper presented at the 8th international conference on preservation of digital objects (iPRES), Singapore. Accessed 02 Feb 2016
  25. 25.
    Wikimedia Foundation (2016) File (command). Accessed 13 Feb 2016
  26. 26.
    Hockx-Yu H (2015) The unknown aspects of web archives. Accessed 22 Mar 2016
  27. 27.
    PREMIS Editorial Committee (2015) PREMIS data dictionary for preservation metadata version 3.0. Accessed 05 Jan 2016
  28. 28.
    Bibliothèque nationale de France (2011) ContainerMD. Accessed 03 Jan 2016
  29. 29.
    Pearson D, Webb C (2008) Defining file format obsolescence: a risk journey. Int J Digit Curation 3(1):89–106. doi: 10.2218/ijdc.v3i1.44 CrossRefGoogle Scholar
  30. 30.
    Jackson A (2015) Ten years of the UK web archive: what have we saved? UK web archive blog. Accessed 02 Feb 2016
  31. 31.
    New York Art Resources Consortium (NYARC) (2015) Official website. Accessed 02 Feb 2015Google Scholar
  32. 32.
    Internet Archive Webteam (2015) Archive-it storage and preservation policy. Accessed 02 Feb 2016
  33. 33.
    Duncan S (2015) Preserving born-digital catalogues raisonnés: web archiving at the New York art resources consortium. Art Librar J 40(2):50–55. Accessed 02 Feb 2016
  34. 34.
    Persons S (2015) Turns 20: archiving two decades of exhibition sites. INSIDE/OUT, 25 May 2015. Accessed 02 Feb 2016
  35. 35.
    Kreymer I (nd) Accessed 10 Apr 2010
  36. 36.
    University of Illinois at Urbana-Champaign, Grainger Engineering Library Information Center (2006) ECHO Dep METS profile for web site captures. Accessed 02 Feb 2016
  37. 37.
    Guenther R, Myrick L (2008) Archiving web sites for preservation and access: MODS, METS and MINERVA. J Arch Organ 4(1–2):141–166. doi: 10.1300/J201v04n01_08 Google Scholar
  38. 38.
  39. 39.
    The National Archives (2016) The technical registry PRONOM. Accessed 02 Feb 2016
  40. 40.
    Lehane R (2016) Siegfried. Accessed 02 Feb 2016
  41. 41.
    Rhizome (2016) Welcome to webrecorder beta. Accessed 02 Feb 2016
  42. 42.
    McKeehan M (2015) Preserving variability. The National Digital Stewardship Residency, New York. Accessed 02 Feb 2016
  43. 43.
    Goethals A, Oury C, Pearson D, Sierman B, Steinke T (2015) Facing the challenge of web archives preservation collaboratively: the role and work of the IIPC Preservation Working Group. D-Lib Mag. doi: 10.1045/may2015-goethals Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Clément Oury
    • 1
    Email author
  • Karl-Rainer Blumenthal
    • 2
  • Sébastien Peyrard
    • 3
  1. 1.International ISSN CentreParisFrance
  2. 2.New York Art Resources ConsortiumNew YorkUSA
  3. 3.Bibliothèque nationale de FranceParis Cedex 13France

Personalised recommendations