A Scalable P2P RIA Crawling System with Partial Knowledge

  • Khaled Ben Hafaiedh
  • Gregor von Bochmann
  • Guy-Vincent Jourdan
  • Iosif Viorel Onut
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8593)


Rich Internet Applications are widely used as they are interactive and user friendly. Automated tools for crawling Rich Internet Applications have become needed for many reasons such as content indexing or testing for correctness and security. Due to the large size of RIAs, distributed crawling has been introduced to reduce the amount of time required for crawling. However, having one controller may result in a performance bottleneck resulting from a single database simultaneously accessed by many crawlers. It may also be vulnerable to complete data loss if a node failure occurs at the storage unit. We present a distributed decentralized scheme for crawling large-scale RIAs capable of partitioning the search space among several controllers in which the information is partially stored, which allows for fault tolerance and for the scalability of the system. Our results are significantly better than for non-distributed crawling, and outperforms the distributed crawling using one coordinator.


Rich Internet Applications Web crawling Web application modeling Graph exploration Distributed crawling P2P networks 



This work is supported in part by IBM and the Natural Science and Engineering Research Council of Canada.

Disclaimer. The views expressed in this article are the sole responsibility of the authors and do not necessarily reflect those of IBM.

Trademarks. IBM, Rational and AppScan are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at


  1. 1.
    Mirtaheri, S.M., von Bochmann, G., Jourdan, G.V., Onut, I.V.: GDist-RIA Crawler: a greedy distributed crawler for Rich Internet Applications. In: Noubir, G., Raynal, M. (eds.) NETYS 2014, LNCS 8593, pp. 200–214. Springer, Heidelberg (2014)Google Scholar
  2. 2.
    Zhang, X., Wang, H.: AJAX crawling scheme based on Document Object Model. In: 4th International Conference on Computational and Information Sciences (ICCIS), China, Chongqing, pp. 1198–1201 (2012)Google Scholar
  3. 3.
    Choudhary, S., Dincturk, M.E., Mirtaheri, S.M., Moosavi, A., von Bochmann, G., Jourdan, G.V., Onut, I.V.: Crawling rich Internet applications: the state of the art. In: Conference of the Center for Advanced Studies on Collaborative Research, Toronto, Markham, pp. 146–160 (2012)Google Scholar
  4. 4.
    Benjamin, K., von Bochmann, G., Dincturk, M.E., Jourdan, G.V., Onut, V.: Some modeling challenges when testing rich internet applications for security. In: 1st International Workshop on Modeling and Detection of Vulnerabilities, France, Paris (2010)Google Scholar
  5. 5.
    Xiao, X., Zhang, W.Z., Zhang, H.L., Fang, B.X.: A forwarding-based task scheduling algorithm for distributed web crawling over DHTs. In: Proceedings of the 15th International Conference on Parallel and Distributed Systems (ICPADS), China, Shenzhen, pp. 854–859 (2009)Google Scholar
  6. 6.
    Paulson, L.D.: Building rich web applications with Ajax. Computer 38, 14–17 (2005). (IEEE Computer Society)CrossRefGoogle Scholar
  7. 7.
    Cho, J., Garcia-Molina, H.: Parallel crawlers. In: Proceedings of the 11th International Conference on World Wide Web, WWW, Hawaii, Honolulu, vol. 2 (2002)Google Scholar
  8. 8.
    Schollmeier, R.: A definition of peer-to-peer networking for the classification of peer-to-peer architectures and applications. In: Proceedings of the IEEE 2001 International Conference on Peer-to-Peer Computing (P2P2001), Sweden, Linkping (2001)Google Scholar
  9. 9.
    Stoica, I., Morris, R., Karger, D., Frans Kaashoek, M., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for Internet applications. In: Proceedings of ACM SIGCOMM 2001, California, San Deigo (2001)Google Scholar
  10. 10.
    Brin, S., Page, L.: The anatomy of a large-scale hypertextual Web search engine. In: Proceedings of the 7th International Conference on World Wide Web, WWW, Australia, Brisbane, vol. 7 (1998)Google Scholar
  11. 11.
    Misra, J.: Detecting termination of distributed computations using markers. In: PODC’83, Proceedings of the 2nd Annual ACM Symposium on Principles of Distributed Computing, vol. 22, pp. 290–294. ACM, New York (1983)Google Scholar
  12. 12.
    Marini, J.: Document Object Model: Processing Structured Documents. McGraw-Hill/Osborne, New York (2002)Google Scholar
  13. 13.
    Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1, 269–271 (1959)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Loo, B.T., Owen, L., Krishnamurthy, C.S.: Distributed web crawling over DHTs. Technical report (2004)Google Scholar
  15. 15.
    Carzanig, A., Rutheford, M.: SSim: a simple Discrete-event Simulation Library, University of Colorado (2003).

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Khaled Ben Hafaiedh
    • 1
    • 3
  • Gregor von Bochmann
    • 1
    • 3
  • Guy-Vincent Jourdan
    • 1
    • 3
  • Iosif Viorel Onut
    • 2
    • 3
  1. 1.School of Electrical Engineering and Computer ScienceUniversity of OttawaOttawaCanada
  2. 2.R&D IBM Security AppScan® EnterpriseOttawaCanada
  3. 3.Software Security Research GroupOttawaCanada

Personalised recommendations