A Scalable P2P RIA Crawling System with Partial Knowledge

  • Khaled Ben Hafaiedh
  • Gregor von Bochmann
  • Guy-Vincent Jourdan
  • Iosif Viorel Onut
Conference paper

DOI: 10.1007/978-3-319-09581-3_13

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8593)
Cite this paper as:
Ben Hafaiedh K., von Bochmann G., Jourdan GV., Onut I.V. (2014) A Scalable P2P RIA Crawling System with Partial Knowledge. In: Noubir G., Raynal M. (eds) Networked Systems. Lecture Notes in Computer Science, vol 8593. Springer, Cham

Abstract

Rich Internet Applications are widely used as they are interactive and user friendly. Automated tools for crawling Rich Internet Applications have become needed for many reasons such as content indexing or testing for correctness and security. Due to the large size of RIAs, distributed crawling has been introduced to reduce the amount of time required for crawling. However, having one controller may result in a performance bottleneck resulting from a single database simultaneously accessed by many crawlers. It may also be vulnerable to complete data loss if a node failure occurs at the storage unit. We present a distributed decentralized scheme for crawling large-scale RIAs capable of partitioning the search space among several controllers in which the information is partially stored, which allows for fault tolerance and for the scalability of the system. Our results are significantly better than for non-distributed crawling, and outperforms the distributed crawling using one coordinator.

Keywords

Rich Internet Applications Web crawling Web application modeling Graph exploration Distributed crawling P2P networks 

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Khaled Ben Hafaiedh
    • 1
    • 3
  • Gregor von Bochmann
    • 1
    • 3
  • Guy-Vincent Jourdan
    • 1
    • 3
  • Iosif Viorel Onut
    • 2
    • 3
  1. 1.School of Electrical Engineering and Computer ScienceUniversity of OttawaOttawaCanada
  2. 2.R&D IBM Security AppScan® EnterpriseOttawaCanada
  3. 3.Software Security Research GroupOttawaCanada

Personalised recommendations