A Novel Distributed Collaborative Filtering Algorithm and Its Implementation on P2P Overlay Network

  • Peng Han
  • Bo Xie
  • Fan Yang
  • Jiajun Wang
  • Ruimin Shen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3056)

Abstract

Collaborative filtering (CF) has proved to be one of the most effective information filtering techniques. However, as their calculation complexity increased quickly both in time and space when the record in user database increases, traditional centralized CF algorithms has suffered from their shortage in scalability. In this paper, we first propose a novel distributed CF algorithm called PipeCF through which we can do both the user database management and prediction task in a decentralized way. We then propose two novel approaches: significance refinement (SR) and unanimous amplification (UA), to further improve the scalability and prediction accuracy of PipeCF. Finally we give the algorithm framework and system architecture of the implementation of PipeCF on Peer-to-Peer (P2P) overlay network through distributed hash table (DHT) method, which is one of the most popular and effective routing algorithm in P2P. The experimental data show that our distributed CF algorithm has much better scalability than traditional centralized ones with comparable prediction efficiency and accuracy.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Goldberg, D., Nichols, D., Oki, B.M., Terry, D.: Using collaborative filtering to weave an information tapestry. Communications of the ACM 35(12), 61–70 (1992)CrossRefGoogle Scholar
  2. 2.
    Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 230–237 (1999)Google Scholar
  3. 3.
    Breese, J., Heckerman, D., Kadie, C.: Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In: Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pp. 43–52 (1998)Google Scholar
  4. 4.
    Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: GroupLens: an open architecture for collaborative filtering of netnews. In: Proceedings of the 1994 ACM conference on Computer supported cooperative work, October 22-26, pp. 175–186. Chapel Hill, North Carolina (1994)CrossRefGoogle Scholar
  5. 5.
    Shardanand, U., Maes, P.: Social information filtering: algorithms for automating “word of mouth”. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Denver, Colorado, United State, May 7-11, pp. 210–217 (1995)Google Scholar
  6. 6.
    Eachmovie collaborative filtering data set, http://research.compaq.com/SRC/eachmovie
  7. 7.
    Tveit, A.: Peer-to-peer based Recommendations for Mobile Commerce. In: Proceedings of the First International Mobile Commerce Workshop, July 2001, pp. 26–29. ACM Press, Rome (2001)CrossRefGoogle Scholar
  8. 8.
    Olsson, T.: Bootstrapping and Decentralizing Recommender Systems, Licentiate Thesis 2003-006, Department of Information Technology, Uppsala University and SICS (2003)Google Scholar
  9. 9.
    Canny, J.: Collaborative filtering with privacy. In: Proceedings of the IEEE Symposium on Research in Security and Privacy, Oakland, CA, May 2002, pp. 45–57 (2002); IEEE Computer Society, Technical Committee on Security and Privacy, IEEE Computer Society PressGoogle Scholar
  10. 10.
    Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A scalable contentaddressable network. In: SIGCOMM (August 2001)Google Scholar
  11. 11.
    Stocal, I., et al.: Chord: A scalable peer-to-peer lookup service for Internet applications. In: ACM SIGCOMM, San Diego, CA, USA, pp. 149–160 (2001)Google Scholar
  12. 12.
    Rowstron, A., Druschel, P.: Pastry: Scalable, distributed object location and routing for large scale peer-to-peer systems. In: IFIP/ACM Middleware, Hedelberg, Germany (2001)Google Scholar
  13. 13.
    Zhao, B.Y., et al.: Tapestry: An infrastructure for fault-tolerant wide-area location and routing. Tech. Rep. UCB/CSB-0-114, UC Berkeley, EECS 2001 (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Peng Han
    • 1
  • Bo Xie
    • 1
  • Fan Yang
    • 1
  • Jiajun Wang
    • 1
  • Ruimin Shen
    • 1
  1. 1.Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations