DataRover: An Automated System for Extracting Product Information From Online Catalogs

  • Syed Toufeeq Ahmed
  • Srinivas Vadrevu
  • Hasan Davulcu
Part of the Studies in Computational Intelligence book series (SCI, volume 23)


The increasing number of e-commerce Web sites on the Web introduces numerous challenges in organizing and searching the product information across multiple Web sites. This problem is further exacerbated by various presentation templates that different Web sites use in presenting their product information, and different ways of product information they store in their catalogs. This paper describes the DataRover system, which can automatically crawl and extract all products from online catalogs. DataRover is based on pattern mining algorithms and domain specific heuristics which utilize the navigational and presentation regularities to identify taxonomy, list-of-product and single-product segments within an online catalog. Next, it uses the inferred patterns to extract data from all such data segments and to automatically transform an online catalog into a database of categorized products. We also provide experimental results to demonstrate the efficacy of the DataRover.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    J. Hammer, H. Garcia-Molina, S. Nestorov, R. Yerneni, M. M. Breunig, and V. Vas-salos. Template-based wrappers in the tsimmis system. In ACM SIGMOD, 1997.Google Scholar
  2. 2.
    Gustavo O. Arocena and Alberto O. Mendelzon. Weboql: Restructuring documents, databases, and webs. In ICDE, pages 24–33, 1998.Google Scholar
  3. 3.
    Nickolas Kushmerick, Daniel S. Weld, and Robert B. Doorenbos. Wrapper induction for information extraction. In Intl. Joint Conference on Artificial Intelligence (IJCAI), pages 729–737, 1997.Google Scholar
  4. 4.
    Robert B. Doorenbos, Oren Etzioni, and Daniel S. Weld. A scalable comparison-shopping agent for the world-wide web. In W. Lewis Johnson and Barbara Hayes-Roth, editors, Proceedings of the First International Conference on Autonomous Agents (Agents’97), pages 39–48, Marina del Rey, CA, USA, 1997. ACM Press.Google Scholar
  5. 5.
    Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. Roadrunner: Towards automatic data extraction from large web sites. In Intl. Conf. on Very Large Data Bases, 2001.Google Scholar
  6. 6.
    A. Arasu and H. Garcia-Molina. Extracting structured data from web pages. In ACM SIGMOD, 2003.Google Scholar
  7. 7.
    Hasan Davulcu, Srinivas Vadrevu, and Saravanakumar Nagarajan. Ontominer: Bootstrapping and populating ontologies from domain specific web sites. IEEE Intelligent Systems, 18(5), September 2003.Google Scholar
  8. 8.
    Hasan Davulcu, Sukumar Koduri, and Saravanakumar Nagarajan. Datarover: A taxonomy based crawler for automated data extraction from data-intensive web sites. In Proceedings of the ACM International Workshop on Web Information and Data Management, pages 9–14, 2003.Google Scholar
  9. 9.
    D. W. Embley, Y. Jiang, and Y.-K. Ng. Record-boundary discovery in Web documents. pages 467–478, 1999.Google Scholar
  10. 10.
    Christina Yip Chung, Michael Gertz, and Neel Sundaresan. Reverse engineering for web data: From visual to semantic structures. In Intl. Conf. on Data Engineering, 2002.Google Scholar
  11. 11.
    R. C. Berwick and S. Pilato. Learning syntax by automata induction. In Machine Learning 2, pages 9–38, 1987.Google Scholar
  12. 12.
    E. Mark Gold. Complexity of automaton identification from given sets. In Information and Control, pages 37:302–320, 1978.MATHMathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Syed Toufeeq Ahmed
    • 1
  • Srinivas Vadrevu
    • 1
  • Hasan Davulcu
    • 1
  1. 1.Department of Computer Science and EngineeringArizona State UniversityTempeUSA

Personalised recommendations