On the Feasibility of Peer-to-Peer Web Indexing and Search

  • Jinyang Li
  • Boon Thau Loo
  • Joseph M. Hellerstein
  • M. Frans Kaashoek
  • David R. Karger
  • Robert Morris
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2735)


This paper discusses the feasibility of peer-to-peer full-text keyword search of the Web. Two classes of keyword search techniques are in use or have been proposed: flooding of queries over an overlay network (as in Gnutella), and intersection of index lists stored in a distributed hash table. We present a simple feasibility analysis based on the resource constraints and search workload. Our study suggests that the peer-to-peer network does not have enough capacity to make naive use of either of search techniques attractive for Web search. The paper presents a number of existing and novel optimizations for P2P search based on distributed hash tables, estimates their effects on performance, and concludes that in combination these optimizations would bring the problem to within an order of magnitude of feasibility. The paper suggests a number of compromises that might achieve the last order of magnitude.


Compression Ratio Communication Cost Ranking Function Distribute Hash Table Bloom Filter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Google Press Center: Technical Highlights,
  2. 2.
    The Deep Web: Surfacing Hidden Value,
  3. 3.
  4. 4.
  5. 5.
    Ingram: Record Industry Plays Hardball with Kazaa,
  6. 6.
    Why Gnutella Can’t Scale. No, Really,
  7. 7.
    Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: Scalable Peer-To-Peer Lookup Service for Internet Applications. In: Proceedings of the ACM SIGCOMM Conference, pp. 149–160 (2001)Google Scholar
  8. 8.
    Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A Scalablen Content Addressable Network. In: Proceedings of the ACM SIGCOM Conference (2001)Google Scholar
  9. 9.
    Rowstron, A., Druschel, P.: Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, p. 329. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  10. 10.
    Zhao, B.Y., Kubiatowicz, J.D., Joseph, A.D.: Tapestry: An Infrastructure for Fault-tolerant Wide-area Location and Routing. UC Berkeley Technical Report (2001) UCB/CSD-01-1141Google Scholar
  11. 11.
    Reynolds, P., Vahdat, A.: Efficient Peer-to-Peer Keyword Searching (2002) (unpublished manuscript)Google Scholar
  12. 12.
    Tang, C., Xu, Z., Mahalingam, M.: pSearch: Information Retrieval in Structured Overlays. In: The First Workshop on Hot Topics in Networks, HotNets-I (2002)Google Scholar
  13. 13.
    Harren, M., Hellerstein, J.M., Huebsch, R., Loo, B.T., Shenker, S., Stoica, I.: Complex Queries in DHT-based Peer-to-Peer Networks. In: 1st International Workshop on Peer-to-Peer Systems, IPTPS (2002)Google Scholar
  14. 14.
    Gnawali, O.D.: A Keyword Set Search System for Peer-to-Peer Networks. Master Thesis, Massachusetts Institute of Technology (2002)Google Scholar
  15. 15.
    Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Library Technologies Project (1998)Google Scholar
  16. 16.
    Thompson, K., Miller, G., Wilder, R.: Wide-area Traffic Patterns and Characteristics. IEEE Network 11(6), 10–23 (1997)CrossRefGoogle Scholar
  17. 17.
    Boardwatch Magazine’s Directory of Internet Service Providers (1999) Google Scholar
  18. 18.
    Mitzenmacher, M.: Compressed Bloom Filters. In: Twentieth ACM Symposium on Principles of Distributed Computing (2001)Google Scholar
  19. 19.
    Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images (1999)Google Scholar
  20. 20.
    Demaine, E.D., López-Ortiz, A., Ian Munro, J.: Adaptive Set Intersections, Unions, and Differences. In: Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2000 (2000)Google Scholar
  21. 21.
    Hofmann, T.: Probabilistic Latent Semantic Analysis. In: Proceedings of Uncertainty in Artificial Intelligence (1999)Google Scholar
  22. 22.
    Fagin, R., Lotem, A., Naor, M.: Optimal Aggregation Algorithms for Middleware. In: Symposium on Principles of Database Systems (2001)Google Scholar
  23. 23.
    Hellerstein, J.M., Avnur, R., Chou, A., Olston, C., Raman, V., Roth, T., Hidber, C., Haas, P.: Interactive Data Analysis with CONTROL. IEEE Computer (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Jinyang Li
    • 1
  • Boon Thau Loo
    • 2
  • Joseph M. Hellerstein
    • 2
  • M. Frans Kaashoek
    • 1
  • David R. Karger
    • 1
  • Robert Morris
    • 1
  1. 1.MIT Lab for Computer Science 
  2. 2.UCBerkeley

Personalised recommendations