A Scalable P2P RIA Crawling System with Partial Knowledge
Rich Internet Applications are widely used as they are interactive and user friendly. Automated tools for crawling Rich Internet Applications have become needed for many reasons such as content indexing or testing for correctness and security. Due to the large size of RIAs, distributed crawling has been introduced to reduce the amount of time required for crawling. However, having one controller may result in a performance bottleneck resulting from a single database simultaneously accessed by many crawlers. It may also be vulnerable to complete data loss if a node failure occurs at the storage unit. We present a distributed decentralized scheme for crawling large-scale RIAs capable of partitioning the search space among several controllers in which the information is partially stored, which allows for fault tolerance and for the scalability of the system. Our results are significantly better than for non-distributed crawling, and outperforms the distributed crawling using one coordinator.
KeywordsRich Internet Applications Web crawling Web application modeling Graph exploration Distributed crawling P2P networks
This work is supported in part by IBM and the Natural Science and Engineering Research Council of Canada.
Disclaimer. The views expressed in this article are the sole responsibility of the authors and do not necessarily reflect those of IBM.
Trademarks. IBM, Rational and AppScan are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml.
- 1.Mirtaheri, S.M., von Bochmann, G., Jourdan, G.V., Onut, I.V.: GDist-RIA Crawler: a greedy distributed crawler for Rich Internet Applications. In: Noubir, G., Raynal, M. (eds.) NETYS 2014, LNCS 8593, pp. 200–214. Springer, Heidelberg (2014)Google Scholar
- 2.Zhang, X., Wang, H.: AJAX crawling scheme based on Document Object Model. In: 4th International Conference on Computational and Information Sciences (ICCIS), China, Chongqing, pp. 1198–1201 (2012)Google Scholar
- 3.Choudhary, S., Dincturk, M.E., Mirtaheri, S.M., Moosavi, A., von Bochmann, G., Jourdan, G.V., Onut, I.V.: Crawling rich Internet applications: the state of the art. In: Conference of the Center for Advanced Studies on Collaborative Research, Toronto, Markham, pp. 146–160 (2012)Google Scholar
- 4.Benjamin, K., von Bochmann, G., Dincturk, M.E., Jourdan, G.V., Onut, V.: Some modeling challenges when testing rich internet applications for security. In: 1st International Workshop on Modeling and Detection of Vulnerabilities, France, Paris (2010)Google Scholar
- 5.Xiao, X., Zhang, W.Z., Zhang, H.L., Fang, B.X.: A forwarding-based task scheduling algorithm for distributed web crawling over DHTs. In: Proceedings of the 15th International Conference on Parallel and Distributed Systems (ICPADS), China, Shenzhen, pp. 854–859 (2009)Google Scholar
- 7.Cho, J., Garcia-Molina, H.: Parallel crawlers. In: Proceedings of the 11th International Conference on World Wide Web, WWW, Hawaii, Honolulu, vol. 2 (2002)Google Scholar
- 8.Schollmeier, R.: A definition of peer-to-peer networking for the classification of peer-to-peer architectures and applications. In: Proceedings of the IEEE 2001 International Conference on Peer-to-Peer Computing (P2P2001), Sweden, Linkping (2001)Google Scholar
- 9.Stoica, I., Morris, R., Karger, D., Frans Kaashoek, M., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for Internet applications. In: Proceedings of ACM SIGCOMM 2001, California, San Deigo (2001)Google Scholar
- 10.Brin, S., Page, L.: The anatomy of a large-scale hypertextual Web search engine. In: Proceedings of the 7th International Conference on World Wide Web, WWW, Australia, Brisbane, vol. 7 (1998)Google Scholar
- 11.Misra, J.: Detecting termination of distributed computations using markers. In: PODC’83, Proceedings of the 2nd Annual ACM Symposium on Principles of Distributed Computing, vol. 22, pp. 290–294. ACM, New York (1983)Google Scholar
- 12.Marini, J.: Document Object Model: Processing Structured Documents. McGraw-Hill/Osborne, New York (2002)Google Scholar
- 14.Loo, B.T., Owen, L., Krishnamurthy, C.S.: Distributed web crawling over DHTs. Technical report (2004)Google Scholar
- 15.Carzanig, A., Rutheford, M.: SSim: a simple Discrete-event Simulation Library, University of Colorado (2003). http://www.inf.usi.ch/carzaniga/ssim/index.html