Abstract
The heart of any substantial search engine is a crawler. A crawler is a program that collects web pages by following links from one web page to the next. Due to our complete dependence on search engines for finding information and insights into every aspect of human endeavors, from finding cat videos to the deep mysteries of the universe, we tend to overlook the enormous complexities of today’s search engines powered by the web crawlers to index and aggregate everything found on the internet. The sheer scale and technological innovation that enabled the vast body of knowledge on the internet to be indexed and easily accessible upon queries is constantly evolving. In this paper, we look at the current state of the massive apparatus of crawling the internet, specifically focusing on deep web crawling, given the explosion of information behind an interface that cannot be extracted from raw text. We also explore distributed search engines and the way forward for finding information in the age of large language models like ChatGPT or Bard. Our primary goal is to explore the junction of large-scale web crawling and search engines in an integrative approach to identify the emerging challenges and scopes in massive data where recent advancements in AI upend traditional means of information retrieval. Finally, we present the design of a new asynchronous crawler that can extract information from any domain into a structured format.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30, 107–117 (1998). https://doi.org/10.1016/S0169-7552(98)00110-X
Armstrong, M.: Number of websites. https://www.statista.com/chart/19058/number-of-websites-online. Accessed 13 Aug 2023
Hernández, I., Rivero, C.R., Ruiz, D.: Deep Web crawling: a survey. World Wide Web 22, 1577–1610 (2019)
Rattan, D., Bhatia, R., Kumar, M.: A survey of web crawlers for information retrieval. WIREs Data Min. Knowl. Discov. 7, e1218 (2017). https://doi.org/10.1002/widm.1218
Sato, N., Uehara, M., Sakai, Y., Mori, H.: Distributed information retrieval by using cooperative meta search engines. In: Proceedings 21st International Conference on Distributed Computing Systems Workshops, pp. 345–350 (2001)
Guo, Y., Li, K., Zhang, G., Zhang, K.: Board forum crawling: a web crawling method for web forum. In: IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings) (WI’06), pp. 745–748 (2006)
Heydon, A., Najork, M.: High-Performance Web Crawling. Springer, Cham (2002)
How Google Search organizes information
Koloveas, P., Chantzios, T., Tryfonopoulos, C., Skiadopoulos, S.: A crawler architecture for harvesting the clear, social, and dark web for IoT-related cyber-threat intelligence. In: IEEE World Congress on Services (SERVICES), pp. 3–8 (2019)
Ren, X., Wang, H., Dai, D.: A summary of research on web data acquisition methods based on distributed crawler. In: IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1682–1688 (2020)
Yin, F., He, X., Zhixin, L.: Research on scrapy-based distributed crawler system for crawling semi-structure information at high speed. In: IEEE 4th International Conference on Computer and Communications (ICCC), pp. 1356–1359 (2018)
Apache Nutch Homepage. https://nutch.apache.org. Accessed 13 Aug 2023
Apache Spark Homepage. https://spark.apache.org. Accessed 13 Aug 2023
Liu, F., Xin, W.: Implementation of distributed crawler system based on spark for massive data mining. In: 5th International Conference on Computer and Communication Systems (ICCCS), pp. 482–485 (2020)
Common Crawl Homepage. https://commoncrawl.org. Accessed 13 Aug 2023
Common Crawl. https://commoncrawl.org/2023/04/mar-apr-2023-crawl-archive-now-available. Accessed 13 Aug 2023
Brown, T.B., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Gold, Z., Mark, L.: Robots welcome: ethical and legal considerations for web crawling and scraping. Wash. JL Tech. Arts 13, 275 (2017)
Knowledge Coin. https://rorur.com/kcoin2.pdf. Accessed 13 Aug 2023
Cambazoglu, B.B., Varol, E., Kayaaslan, E., Aykanat, C., Baeza-Yates, R.: Query forwarding in geographically distributed search engines. In: SIGIR 2010 Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 90–97 (2010).https://doi.org/10.1145/1835449.1835467
Li, J., Loo, B.T., Hellerstein, J.M., Kaashoek, M.F., Karger, D.R., Morris, R.: On the feasibility of peer-to-peer web indexing and search. In: International Workshop on Peer-to-Peer Systems, pp. 207–215 (2003)
Apache Lucene Homepage. https://lucene.apache.org. Accessed 13 Aug 2023
Elasticsearch Homepage. https://www.elastic.co/what-is/elasticsearch. Accessed 13 Aug 2023
Introducing ChatGPT. https://openai.com/blog/chatgpt. Accessed 13 Aug 2023
Language Model and Search Engine. https://www.technologyreview.com/2021/05/14/1024918/language-models-gpt3-search-engine-google. Accessed 13 Aug 2023
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Al Galib, A., Mehedi, M.H.K., Rhythm, E.R., Rasel, A.A. (2024). Large Scale Web Crawling and Distributed Search Engines: Techniques, Challenges, Current Trends, and Future Prospects. In: Zakaria, N.H., Mansor, N.S., Husni, H., Mohammed, F. (eds) Computing and Informatics. ICOCI 2023. Communications in Computer and Information Science, vol 2001. Springer, Singapore. https://doi.org/10.1007/978-981-99-9589-9_2
Download citation
DOI: https://doi.org/10.1007/978-981-99-9589-9_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9588-2
Online ISBN: 978-981-99-9589-9
eBook Packages: Computer ScienceComputer Science (R0)