Shepherdable Indexes and Persistent Search Services for Mobile Users

  • Michael Higgins
  • Dominic Widdows
  • Magesh Balasubramanya
  • Peter Lucas
  • David Holstius
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4276)


We describe a range of designs for supporting rich search queries in a peer-to-peer network. Our implementation is based upon uniquely identified data objects which are replicated upon request by agents called Shepherds. Several abstract data structures are built upon this framework, supporting dataset management, lexical search, and distributed GIS interfaces in an application called the Geobrowser. Our results demonstrate that it is possible to layer higher-level data structures upon a basic peer-to-peer transport and replication layer. When users perform a given query, parts of the index as well as the query results themselves are shepherded to the user’s local venue. A natural benefit of this approach is that mobile users can repeat previous searches if they become disconnected from the rest of the network. Some of the data structures that prove to be successful are peer-to-peer adaptations of traditional indexing structures. We review some of the properties that lead to successful designs in this domain, giving examples of deployed systems in the Geobrowser.


Mobile User Search Tree Index Structure Inverted Index Test Query 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ripeanu, M.: Peer-to-peer architecture case study: Gnutella network. Technical report, University of Chicago (2001)Google Scholar
  2. 2.
    Kubiatowicz, J., Bindel, D., Chen, Y., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., Wells, C., Zhao, B.: Oceanstore: An architecture for global-scale persistent storage. In: Proceedings of ACM ASPLOS. ACM, New York (2000)Google Scholar
  3. 3.
    de Kretser, O., Moffat, A., Shimmin, T., Zobel, J.: Methodologies for distributed information retrieval. In: International Conference on Distributed Computing Systems, pp. 66–73 (1998)Google Scholar
  4. 4.
    Melnik, S., Raghavan, S., Yang, B., Garcia-Molina, H.: Building a distributed full-text index for the web. In: World Wide Web, pp. 396–406 (2001)Google Scholar
  5. 5.
    Rowstron, A., Druschel, P.: Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, p. 329. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  6. 6.
    Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: A scalable Peer-To-Peer lookup service for internet applications. In: Proceedings of the 2001 ACM SIGCOMM Conference, pp. 149–160 (2001)Google Scholar
  7. 7.
    Harvey, N., Jones, M.B., Saroiu, S., Theimer, M., Wolman, A.: Skipnet: A scalable overlay network with practical locality properties. In: Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems (USITS 2003), Seattle, WA (2003)Google Scholar
  8. 8.
    Aberer, K., Cudré-Mauroux, P., Datta, A., Despotovic, Z., Hauswirth, M., Punceva, M., Schmidt, R., Wu, J.: Advanced peer-to-peer networking: The P-Grid System and its Applications. PIK Journal - Praxis der Informationsverarbeitung und Kommunikation, Special Issue on P2P Systems (2003)Google Scholar
  9. 9.
    Johnson, T., Colbrook, A.: A distributed data-balanced dictionary based on the b-link tree. In: Proceedings of the 6th International Parallel Processing Symposium, Washington, DC, USA, pp. 319–324. IEEE Computer Society, Los Alamitos (1992)CrossRefGoogle Scholar
  10. 10.
    Cosway, P.R.: Replication control in distributed B-trees. Technical Report MIT/LCS/TR-705 (1997)Google Scholar
  11. 11.
    Lindemann, C., Waldhorst, O.: A distributed search service for peer-to-peer file sharing in mobile applications. In: Proc. 2nd IEEE Conf. on Peer-to-Peer Computing (P2P 2002) (2002)Google Scholar
  12. 12.
    Crespo, A., Garcia-Molina, H.: Archival storage for digital libraries. In: Third ACM International Conference on Digital Libraries (1998)Google Scholar
  13. 13.
    Kroll, B., Widmayer, P.: Distributing a search tree among a growing number of processors. In: SIGMOD 1994: Proceedings of the 1994 ACM SIGMOD international conference on Management of data, pp. 265–276. ACM Press, New York (1994)CrossRefGoogle Scholar
  14. 14.
    Lindemann, C., Waldhorst, O.P.: Exploiting epidemic data dissemination for consistent lookup operations in mobile applications. SIGMOBILE Mob. Comput. Commun. Rev. 8, 44–56 (2004)CrossRefGoogle Scholar
  15. 15.
    Saito, Y., Shapiro, M.: Optimistic replication. ACM Computing Surveys 37 (2005)Google Scholar
  16. 16.
    Lucas, P., Senn, J., Widdows, D.: Distributed knowledge representation using universal identity and replication. Technical Report MAYA-05007, MAYA Design (2005)Google Scholar
  17. 17.
    Roth, S., Lucas, P., Senn, J., Gomberg, C., Burks, M., Stroffolino, P., Kolojejchick, J., Dunmire, C.: Visage: A user interface environment for exploring information. In: Proceedings of Information Visualization, San Francisco, pp. 3–12. IEEE, Los Alamitos (1996)Google Scholar
  18. 18.
    Viles, C.L., French, J.C.: Dissemination of collection wide information in a distributed information retrieval system. In: SIGIR 1995: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 12–20. ACM Press, New York (1995)CrossRefGoogle Scholar
  19. 19.
    Lucas, P., Widdows, D., Hughes, J., Lucas, W.: Roles in the universal database: Data and metadata in a distributed semantic network. Technical Report MAYA-05009, MAYA Design (2005)Google Scholar
  20. 20.
    Sedgewick, R.: Algorithms in C. Addison-Wesley, Reading (1990)MATHGoogle Scholar
  21. 21.
    Guttman, A.: R-trees: a dynamic index structure for spatial searching. In: Proceedings of SIGMOD, pp. 45–47 (1984)Google Scholar
  22. 22.
    Lomet, D.B., Salzberg, B.: Concurrency and recovery for index trees. VLDB Journal: Very Large Data Bases 6, 224–240 (1997)CrossRefGoogle Scholar
  23. 23.
    Higgins, M., Roth, S., Senn, J., Lucas, P., Widdows, D.: Managing distributed collaboration in a peer-to-peer network. In: 14th International Conference on Cooperative Information Systems (CoopIS 2006) (2006)Google Scholar
  24. 24.
    Manola, F., Miller, E.: RDF primer (2004)Google Scholar
  25. 25.
    Higgins, M., Lucas, P., Senn, J.: VisageWeb: Visualizing WWW Data in Visage. In: Symposium on Information Visualization (Infovis), pp. 100–107. IEEE, Los Alamitos (1999)Google Scholar
  26. 26.
    Project, D.: Command post of the future (CPOF) (2005),
  27. 27.
    Allegheny County Department of Human Services: (2006),
  28. 28.
    Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: A scalable Peer-To-Peer lookup service for internet applications. In: Proceedings of the 2001 ACM SIGCOMM Conference, pp. 149–160 (2001)Google Scholar
  29. 29.
    Li, J., Stribling, J., Morris, R., Kaashoek, M.F.: Bandwidth-efficient management of DHT routing tables. In: Proceedings of the 2nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 2005), Boston, Massachusetts (2005)Google Scholar
  30. 30.
    Pugh, W.: Skip lists: A probabilistic alternative to balanced trees. In: Workshop on Algorithms and Data Structures, pp. 437–449 (1989)Google Scholar
  31. 31.
    Balasubramanya, M., Higgins, M., Lucas, P., Senn, J., Widdows, D.: Collaborative annotation that lasts forever: Using peer-to-peer technology for disseminating corpora and language resources. In: Fifth International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy (2006)Google Scholar
  32. 32.
    Rigaux, P., Scholl, M., Voisard, A.: Spatial Databases. Morgan Kauffmann / Academic Press (2002)Google Scholar
  33. 33.
    Arge, L., de Berg, M., Haverkort, H., Yi, K.: The priority r-tree: A practically efficient and worst-case optimal r-tree. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data (SIGMOD 2004), Paris, France, pp. 347–358 (2004)Google Scholar
  34. 34.
    Lucas, P., Balasubramanya, M., Widdows, D., Higgins, M.: The Information Commons Gazetteer: A public resource of populated places and worldwide administrative divisions. In: Fifth International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy (2006)Google Scholar
  35. 35.
    Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the number of points required to represent a line or its caricature. The Canadian Cartographer 10, 112–122 (1973)Google Scholar
  36. 36.
    Foley, J.D., van Dam, A., Feiner, S.K., Hughes, J.F.: Computer Graphics. Addison-Wesley, Reading (1990)Google Scholar
  37. 37.
    Haines, E.: Point in polygon strategies. In: Heckbert, P. (ed.) Graphics Gems, vol. IV, pp. 24–46. Academic Press, London (1994)Google Scholar
  38. 38.
    Hormann, K., Agathos, A.: The point in polygon problem for arbitrary polygons. Computational Geometry 20, 131–144 (2001)MATHCrossRefMathSciNetGoogle Scholar
  39. 39.
    Gabbay, D.M., Woods, J.: The New Logic. Journal of the Interest Group in Pure and Applied Logics 9(2), 157–190 (2001)MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Michael Higgins
    • 1
  • Dominic Widdows
    • 1
  • Magesh Balasubramanya
    • 1
  • Peter Lucas
    • 1
  • David Holstius
    • 1
  1. 1.MAYA Design Inc. 

Personalised recommendations