Abstract
This paper discusses the feasibility of peer-to-peer full-text keyword search of the Web. Two classes of keyword search techniques are in use or have been proposed: flooding of queries over an overlay network (as in Gnutella), and intersection of index lists stored in a distributed hash table. We present a simple feasibility analysis based on the resource constraints and search workload. Our study suggests that the peer-to-peer network does not have enough capacity to make naive use of either of search techniques attractive for Web search. The paper presents a number of existing and novel optimizations for P2P search based on distributed hash tables, estimates their effects on performance, and concludes that in combination these optimizations would bring the problem to within an order of magnitude of feasibility. The paper suggests a number of compromises that might achieve the last order of magnitude.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Google Press Center: Technical Highlights, http://www.google.com/press/highlights.html
The Deep Web: Surfacing Hidden Value, http://www.press.umich.edu/jep/07-01/bergman.html
Gnutella, http://gnutella.wego.com
Kazaa, http://www.kazza.com
Ingram: Record Industry Plays Hardball with Kazaa, http://www.globeandmail.com/
Why Gnutella Can’t Scale. No, Really, http://www.darkridge.com/jpr5/doc/gnutella.html
Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: Scalable Peer-To-Peer Lookup Service for Internet Applications. In: Proceedings of the ACM SIGCOMM Conference, pp. 149–160 (2001)
Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A Scalablen Content Addressable Network. In: Proceedings of the ACM SIGCOM Conference (2001)
Rowstron, A., Druschel, P.: Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, p. 329. Springer, Heidelberg (2001)
Zhao, B.Y., Kubiatowicz, J.D., Joseph, A.D.: Tapestry: An Infrastructure for Fault-tolerant Wide-area Location and Routing. UC Berkeley Technical Report (2001) UCB/CSD-01-1141
Reynolds, P., Vahdat, A.: Efficient Peer-to-Peer Keyword Searching (2002) (unpublished manuscript)
Tang, C., Xu, Z., Mahalingam, M.: pSearch: Information Retrieval in Structured Overlays. In: The First Workshop on Hot Topics in Networks, HotNets-I (2002)
Harren, M., Hellerstein, J.M., Huebsch, R., Loo, B.T., Shenker, S., Stoica, I.: Complex Queries in DHT-based Peer-to-Peer Networks. In: 1st International Workshop on Peer-to-Peer Systems, IPTPS (2002)
Gnawali, O.D.: A Keyword Set Search System for Peer-to-Peer Networks. Master Thesis, Massachusetts Institute of Technology (2002)
Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Library Technologies Project (1998)
Thompson, K., Miller, G., Wilder, R.: Wide-area Traffic Patterns and Characteristics. IEEE Network 11(6), 10–23 (1997)
Boardwatch Magazine’s Directory of Internet Service Providers (1999)
Mitzenmacher, M.: Compressed Bloom Filters. In: Twentieth ACM Symposium on Principles of Distributed Computing (2001)
Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images (1999)
Demaine, E.D., López-Ortiz, A., Ian Munro, J.: Adaptive Set Intersections, Unions, and Differences. In: Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2000 (2000)
Hofmann, T.: Probabilistic Latent Semantic Analysis. In: Proceedings of Uncertainty in Artificial Intelligence (1999)
Fagin, R., Lotem, A., Naor, M.: Optimal Aggregation Algorithms for Middleware. In: Symposium on Principles of Database Systems (2001)
Hellerstein, J.M., Avnur, R., Chou, A., Olston, C., Raman, V., Roth, T., Hidber, C., Haas, P.: Interactive Data Analysis with CONTROL. IEEE Computer (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Li, J., Loo, B.T., Hellerstein, J.M., Kaashoek, M.F., Karger, D.R., Morris, R. (2003). On the Feasibility of Peer-to-Peer Web Indexing and Search. In: Kaashoek, M.F., Stoica, I. (eds) Peer-to-Peer Systems II. IPTPS 2003. Lecture Notes in Computer Science, vol 2735. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45172-3_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-45172-3_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40724-9
Online ISBN: 978-3-540-45172-3
eBook Packages: Springer Book Archive