Skip to main content
Log in

Abstract

The Internet Archive’s (IA) Wayback Machine is the largest and oldest public Web archive and has become a significant repository of our recent history and cultural heritage. Despite its importance, there has been little research about how it is discovered and used. Based on Web access logs, we analyze what users are looking for, why they come to IA, where they come from, and how pages link to IA. We find that users request English pages the most, followed by the European languages. Most human users come to Web archives because they do not find the requested pages on the live Web. About 65 % of the requested archived pages no longer exist on the live Web. We find that more than 82 % of human sessions connect to the Wayback Machine via referrals from other Web sites, while only 15 % of robots have referrers. Most of the links (86 %) from Websites are to individual archived pages at specific points in time, and of those 83 % no longer exist on the live Web. Finally, we find that users who come from search engines browse more pages than users who come from external Web sites.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. Apache “Combined Log File Format”.

  2. https://archive.org/details/911.

References

  1. Bounce rate. http://en.wikipedia.org/wiki/Bounce_rate

  2. Category: All articles with dead external links. http://en.wikipedia.org/w/index.php?title=Category:All_articles_with_dead_external_links

  3. Wikipedia: Link rot. http://en.wikipedia.org/wiki/Wikipedia:Link_rot

  4. Wikipedia: Using the Wayback Machine. http://en.wikipedia.org/wiki/Wikipedia:Using_the_Wayback_Machine

  5. Internet Archive appeals for donations after \(600,000\) in fire damage. http://www.theguardian.com/technology/2013/nov/08/internet-archive-appeals-donations-fire-damage (2013)

  6. Internet Archive building damaged by fire. http://www.bbc.co.uk/news/technology-24848907 (2013)

  7. Wikimedia Report Card. http://reportcard.wmflabs.org/ (2014)

  8. Ainsworth, S.G., Alsum, A., SalahEldeen, H., Weigle, M.C., Nelson, M.L.: How much of the web is archived? In: Proceedings of the 11th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’11, p. 133. ACM Press, USA (2011)

  9. AlNoamany, Y., Weigle, M.C., Nelson, M.L.: Access patterns for robots and humans in web archives. In: Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’13, pp. 339–348. ACM, USA. doi:10.1145/2467696.2467722. http://doi.acm.org/10.1145/2467696.2467722 (2013) ISBN 978-1-4503-2077-1

  10. Alsum, A., Weigle, M., Nelson, M., Sompel, H.: Profiling web archive coverage for top-level domain and content language. Research and advanced technology for digital libraries. Lecture notes in computer science, pp. 60–71. Springer, Berlin (2013)

  11. Anick, P.: Using terminological feedback for web search refinement—a log-based study. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, SIGIR ’03, pp. 88–95. ACM, USA (2003)

  12. Aye, T.T.: Web log cleaning for mining of web usage patterns. In: IEEE 3rd International Conference on Computer Research and Development, ICCRD, pp. 490–494 (2011)

  13. Banos, V., Kim, Y., Ross, S., Manolopoulos, Y.: CLEAR: a credible method to evaluate website archivability. In: Proceedings of the 9th International Conference on Preservation of Digital Objects, iPRES (2013)

  14. Bar-Yossef, Z., Broder, A.Z., Kumar, R., Tomkins, A.: Sic Transit Gloria Telae: towards an understanding of the web’s decay. In: Proceedings of the 13th International Conference on World Wide Web, WWW ’04, pp. 328–337. ACM, USA (2004)

  15. Broache, A.: FBI rescinds secret order for Internet Archive records. http://news.cnet.com/8301-10784_3-9938603-7.html (2008)

  16. Brown, R.: Selecting and weighting N-grams to identify 1100 languages. Text, speech, and dialogue. Lecture notes in computer science, pp. 475–483. Springer, Berlin (2013)

    Google Scholar 

  17. Carmel, D., Yom-Tov, E., Roitman, H.: Enhancing digital libraries using missing content analysis. In: Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’08, pp. 1–10. ACM, USA (2008)

  18. Castellano, G., Fanelli, A.M., Torsello, M.A.: LODAP: A LOg DAta Preprocessor for mining Web browsing patterns. In: Proceedings of the 6th WSEAS International Conference on Artificial Intelligence, Knowledge Engineering and Data Bases, AIKED ’07, vol. 6, pp. 12–17 (2007)

  19. Catledge, L.D., Pitkow, J.E.: Characterizing browsing strategies in the world-wide web. Comput. Netw. ISDN Syst. 27(6), 1065–1073 (1995)

    Google Scholar 

  20. Cooley, R., Mobasher, B.: Data preparation for mining World Wide Web browsing patterns. Knowl. Inf. Syst. 1, 5–32 (1999)

    Article  Google Scholar 

  21. Costa, M., J. Silva, M.: Characterizing search behavior in Web Archives. In: Proceedings of Temporal Web Analytics Workshop, TWAW (2011)

  22. Costa, M., Silva, M.J.: Understanding the information needs of web archive users. In: Proceedings of the 10th International Web Archiving, Workshop, pp. 9–16 (2010)

  23. Dikaiakos, M.D., Stassopoulou, A., Papageorgiou, L.: An investigation of web crawler behavior: characterization and metrics. Comput. Commun. 28(8), 880–897 (2005)

    Article  Google Scholar 

  24. Doran, D., Gokhale, S.S.: Web robot detection techniques: overview and limitations. Data Min. Knowl. Discov. 22(1–2), 183–210 (2010)

    Google Scholar 

  25. Fukuda, K., Cho, K., Esaki, H.: The impact of residential broadband traffic on Japanese ISP backbones. SIGCOMM Comput. Commun. Rev. 35(1), 15–22 (2005)

    Article  Google Scholar 

  26. Harrison, T.L., Nelson, M.L.: Just-in-time recovery of missing Web Pages. In: Proceedings of the 17th Conference on Hypertext and Hypermedia, HYPERTEXT ’06, pp. 145–156. ACM, USA (2006)

  27. Horrigan, J.: Broadband adoption and use in America. Federal Commun. Comm. (2010)

  28. Kahle, B.: Preserving the Internet. Sci. Am. 276(3), 82–83 (1997)

    Article  Google Scholar 

  29. Kahle, B.: Fire update: lost many cameras, 20 boxes. No one hurt. https://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ (2013)

  30. Kahle, B.: Reader privacy at the Internet Archive. http://blog.archive.org/2013/10/25/reader-privacy-at-the-internet-archive/ (2013)

  31. Kahle, B.: Wayback Machine: now with 240,000,000,000 URLs. http://blog.archive.org/2013/01/09/updated-wayback/ (2013)

  32. Kemper, E.A., Stringfield, S., Teddlie, C.: Mixed methods sampling strategies in social science research. Handbook of mixed methods in social and behavioral research, pp. 273–296 (2003)

  33. Koehler, W.: Web page change and persistence-a four-year longitudinal study. J. Am. Soc. Inf. Sci. Technol. 53(2), 162–171 (2002)

    Article  Google Scholar 

  34. Kramer-Smyth, J., Nishigaki, M., Anglade, T.: ArchivesZ: Visualizing archival collections. http://archivesz.com/ArchivesZ.pdf (2007)

  35. Krzywinski, M., Schein, J., Birol, İ., Connors, J., Gascoyne, R., Horsman, D., Jones, S.J., Marra, M.A.: Circos: an information aesthetic for comparative genomics. Genome Res. 19(9), 1639–1645 (2009)

    Article  Google Scholar 

  36. Kumar, R., Tomkins, A.: A characterization of online browsing behavior. In: Proceedings of the 19th International World Wide Web Conference, WWW ’10, pp. 561–570. ACM, USA (2010)

  37. Liu, H., Kešelj, V.: Combined mining of web server logs and web contents for classifying user navigation patterns and predicting users’ future requests. Data Knowl. Eng. 61(2), 304–330 (2007)

    Article  Google Scholar 

  38. Markov, Z., Larose, D.T.: Data mining the web: uncovering patterns in wWeb content, structure, and usage. Wiley, New York (2007)

    Book  Google Scholar 

  39. Negulescu, K.C.: Web archiving @ the Internet Archive. In: Presentation at the 2010 Digital Preservation Partners Meeting, http://www.digitalpreservation.gov/meetings/documents/ndiipp10/NDIIPP072110FinalIA.ppt (2010)

  40. Nelson, M.L., Allen, B.D.: Object persistence and availability in digital libraries. D-Lib Mag. 8(1) (2002).10.1045/january2002-nelson

  41. Nithya, P., Sumathi, P.: Novel pre-processing technique for web log mining by removing global noise, cookies and web robots. Int. J. Comput. Appl. 53(17), 1–6 (2012)

    Google Scholar 

  42. Omodei, M.: Trends in use of Pandora Archive. International Internet Preservation Consortium. http://netpreserve.org/sites/default/files/resources/IIPC-GA-NLA-presentation_m.pdf (2012)

  43. Padia, K., AlNoamany, Y., Weigle, M.C.: Visualizing digital collections at Archive-It. In: Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’12, pp. 15–18. ACM, USA (2012)

  44. Reddy, K.S., Varma, G.P.S., Babu, I.R.: Preprocessing the Web Server Logs: an illustrative approach for effective usage mining. ACM SIGSOFT Softw. Eng. Notes 37(3), 1–5 (2012)

    Google Scholar 

  45. Reisinger, D.: Netflix gobbles a third of peak Internet traffic in North America. CNET, http://news.cnet.com/8301-1023_3-57546405-93/netflix-gobbles-a-third-of-peak-internet-traffic-in-north-america/ (2012)

  46. Rossi, A.: Fixing broken links on the Internet | Internet Archive Blogs. https://blog.archive.org/2013/10/25/fixing-broken-links/ (2013)

  47. SalahEldeen, H.M., Nelson, M.L.: Carbon dating the web: estimating the age of web resources. In: Proceedings of 3rd Temporal Web Analytics Workshop, TempWeb ’13, pp. 1075–1082 (2013)

  48. Sanderson, R., Phillips, M., Van de Sompel, H.: Analyzing the Persistence of referenced web resources with Memento. Tech. Rep. arXiv:1105.3459 (2011)

  49. Shuyo, N.: Language Detection Library - 99% over precision for 49 languages. http://www.slideshare.net/shuyo/language-detection-library-for-java (2010)

  50. Shuyo, N.: Language Detection Library for Java. http://code.google.com/p/language-detection/ (2012)

  51. Silva, A.J.C., Gonçalves, M.A., Laender, A.H.F., Modesto, M.A.B., Cristo, M., Ziviani, N.: Finding what is missing from a digital library: a case study in the computer science field. Inf. Process. Manage. 45(3), 380–391 (2009)

    Article  Google Scholar 

  52. Smith, A.: Home broadband 2010. Technical report, Pew Internet & American Life Project, An initiative of the Pew Research Center (2010)

  53. Spiliopoulou, M., Mobasher, B., Berendt, B., Nakagawa, M.: A framework for the evaluation of session reconstruction heuristics in web-usage analysis. INFORMS J. Comput. 15(2), 171–190 (2003)

    Article  MATH  Google Scholar 

  54. Srivastava, J., Cooley, R., Deshpande, M., Tan, P.N.: Web usage mining: discovery and applications of usage patterns from Web data. ACM SIGKDD Explor. Newslett. 1(2), 12 (2000). doi:10.1145/846183.846188

  55. Stassopoulou, A., Dikaiakos, M.D.: Web robot detection: a probabilistic reasoning approach. Comput. Netw. 53(3), 265–278 (2009)

    Google Scholar 

  56. Streitfeld, A.: Internet Archive will shield visitors - NYTimes.com. http://bits.blogs.nytimes.com/2013/10/24/internet-archive-will-shield-visitors/ (2013)

  57. Tan, P.N., Kumar, V.: Discovery of web robot sessions based on their navigational patterns. Data Min. Knowl. Discov. 6(1), 9–35 (2002)

    Google Scholar 

  58. Tanasa, D., Trousse, B.: Advanced data preprocessing for intersites Web usage mining. IEEE Intell. Syst. 19(2), 59–65 (2004)

    Article  Google Scholar 

  59. Teddlie, C., Yu, F.: Mixed methods sampling: a typology with examples. J. Mixed Methods Res. 1(1), 77–100 (2007)

    Article  Google Scholar 

  60. Thelwall, M., Vaughan, L.: A fair history of the Web? Examining country balance in the Internet Archive. Libr. Inf. Sci. Res. 26(2), 162–176 (2004)

    Article  Google Scholar 

  61. Tofel, B.: Wayback for accessing Web Archives. In: Proceedings of International Web Archiving Workshop, IWAW (2007)

  62. Tongco, M., Dolores, C.: Purposive sampling as a tool for informant selection. Ethnobot. Res. Appl. 5, 147–158 (2008)

    Google Scholar 

  63. Van de Sompel, H., Nelson, M.L., Sanderson, R.: RFC 7089—HTTP framework for time-based access to resource states—Memento. http://tools.ietf.org/html/rfc7089 (2013)

  64. Van de Sompel, H., Nelson, M.L., Sanderson, R., Balakireva, L.L., Ainsworth, S., Shankar, H.: Memento: Time Travel for the Web. Tech. Rep. arXiv:0911.1112 (2009)

  65. Van Ryzin, G.G.: Cluster analysis as a basis for purposive sampling of projects in case study evaluations. Am. J. Eval. 16(2), 109–119 (1995)

    Article  Google Scholar 

  66. Wasserman, T.: Netflix takes up 32.7% of Internet bandwidth. Mashable, http://mashable.com/2011/10/27/netflix-takes-up-32-7-of-internet-bandwidth-study/ (2011)

  67. Whitelaw, M.: Exploring archival collections with interactive visualisation. In: Proceedings of E-Research Australasia Conference (2009)

  68. Zhuang, Z., Wagle, R., Giles, C.: What’s there and what’s not? Focused crawling for missing documents in digital libraries. In: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’05, pp. 301–310 (2005)

Download references

Acknowledgments

This work was supported in part by the NSF (IIS 1009392) and the Library of Congress. We thank Kris Carpenter Negulescu (Internet Archive) for access to the anonymized Wayback Machine logs.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yasmin AlNoamany.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

AlNoamany, Y., AlSum, A., Weigle, M.C. et al. Who and what links to the Internet Archive. Int J Digit Libr 14, 101–115 (2014). https://doi.org/10.1007/s00799-014-0111-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-014-0111-5

Keywords

Navigation