DataRover: An Automated System for Extracting Product Information From Online Catalogs
The increasing number of e-commerce Web sites on the Web introduces numerous challenges in organizing and searching the product information across multiple Web sites. This problem is further exacerbated by various presentation templates that different Web sites use in presenting their product information, and different ways of product information they store in their catalogs. This paper describes the DataRover system, which can automatically crawl and extract all products from online catalogs. DataRover is based on pattern mining algorithms and domain specific heuristics which utilize the navigational and presentation regularities to identify taxonomy, list-of-product and single-product segments within an online catalog. Next, it uses the inferred patterns to extract data from all such data segments and to automatically transform an online catalog into a database of categorized products. We also provide experimental results to demonstrate the efficacy of the DataRover.
Unable to display preview. Download preview PDF.
- 1.J. Hammer, H. Garcia-Molina, S. Nestorov, R. Yerneni, M. M. Breunig, and V. Vas-salos. Template-based wrappers in the tsimmis system. In ACM SIGMOD, 1997.Google Scholar
- 2.Gustavo O. Arocena and Alberto O. Mendelzon. Weboql: Restructuring documents, databases, and webs. In ICDE, pages 24–33, 1998.Google Scholar
- 3.Nickolas Kushmerick, Daniel S. Weld, and Robert B. Doorenbos. Wrapper induction for information extraction. In Intl. Joint Conference on Artificial Intelligence (IJCAI), pages 729–737, 1997.Google Scholar
- 4.Robert B. Doorenbos, Oren Etzioni, and Daniel S. Weld. A scalable comparison-shopping agent for the world-wide web. In W. Lewis Johnson and Barbara Hayes-Roth, editors, Proceedings of the First International Conference on Autonomous Agents (Agents’97), pages 39–48, Marina del Rey, CA, USA, 1997. ACM Press.Google Scholar
- 5.Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. Roadrunner: Towards automatic data extraction from large web sites. In Intl. Conf. on Very Large Data Bases, 2001.Google Scholar
- 6.A. Arasu and H. Garcia-Molina. Extracting structured data from web pages. In ACM SIGMOD, 2003.Google Scholar
- 7.Hasan Davulcu, Srinivas Vadrevu, and Saravanakumar Nagarajan. Ontominer: Bootstrapping and populating ontologies from domain specific web sites. IEEE Intelligent Systems, 18(5), September 2003.Google Scholar
- 8.Hasan Davulcu, Sukumar Koduri, and Saravanakumar Nagarajan. Datarover: A taxonomy based crawler for automated data extraction from data-intensive web sites. In Proceedings of the ACM International Workshop on Web Information and Data Management, pages 9–14, 2003.Google Scholar
- 9.D. W. Embley, Y. Jiang, and Y.-K. Ng. Record-boundary discovery in Web documents. pages 467–478, 1999.Google Scholar
- 10.Christina Yip Chung, Michael Gertz, and Neel Sundaresan. Reverse engineering for web data: From visual to semantic structures. In Intl. Conf. on Data Engineering, 2002.Google Scholar
- 11.R. C. Berwick and S. Pilato. Learning syntax by automata induction. In Machine Learning 2, pages 9–38, 1987.Google Scholar