Abstract
Wrapper induction is currently the main technology for data extraction from semi-structured web pages. However, wrapper induction has the limitation of requiring training Web pages, and the information extraction process is quite complex involving pattern induction, data extraction and data transformation. This paper introduces a new approach that achieves automatic data extraction by applying clustering to detecting similar text tokens, developing a new method to label text tokens to capture the hierarchical structure of HTML pages, and developing an algorithm for transforming labelled text tokens to XML. The approach is examined and compared with a number of existing wrapper induction systems on three different sets of web pages. The results suggest that the new approach is effective for data extraction and that it outperforms existing approaches on these web sites. This approach has the advantages of requiring no training and has no explicit processes for pattern induction or data extraction, therefore the whole process has been simplified.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kushmerick, N., Weld, D.S., Doorenbos, R.B.: Wrapper induction for information extraction. In: IJCAI 1997. Intl. Joint Conference on Artificial Intelligence, pp. 729–737 (1997)
Muslea, I., Minton, S., Knoblock, C.: A hierarchical approach to wrapper induction. In: Etzioni, O., Müller, J.P., Bradshaw, J.M. (eds.) Agents 1999. Proceedings of the Third International Conference on Autonomous Agents, pp. 190–197. ACM Press, Seattle, WA (1999)
Crescenzi, V., Mecca, G., Merialdo, P.: Roadrunner: Towards automatic data extraction from large web sites. In: Proceedings of 27th International Conference on Very Large Data Bases, pp. 109–118 (2001)
Gao, X., Andreae, P., Collins, R.: Approximately repetitive structure detection for wrapper induction. In: Zhang, C., W. Guesgen, H., Yeap, W.-K. (eds.) PRICAI 2004. LNCS (LNAI), vol. 3157, pp. 585–594. Springer, Heidelberg (2004)
Carme, J., Ceresna, M., Frlich, O., Gottlob, G., Hassan, T., Herzog, M., Holzinger, W., Krpl, B.: The lixto project: Exploring new frontiers of web data extraction. In: Bell, D., Hong, J. (eds.) Flexible and Efficient Information Handling. LNCS, vol. 4042, pp. 1–15. Springer, Heidelberg (2006)
Muslea, I.: Extraction patterns for information extraction tasks: A survey. In: Proceedings of AAAI Workshop on Machine Learning for Information Extraction, Orlando, Florida (July 1999)
Liu, B., Grossman, R., Zhai, Y.: Mining data records in web pages. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 601–606. ACM Press, New York (2003)
Lerman, K., Knoblock, C., Minton, S.: Automatic data extraction from lists and tables in web sources. In: IJCAI 2001. Proceedings of the workshop on Advances in Text Extraction and Mining (2001)
Cheeseman, P., Stutz, J.: Bayesian classification (autoclass): Theory and results. In: Advances in Knowledge Discovery and Data Mining. American Association for Artificial Intelligence USA, pp. 153–180 (1996)
Vuong, L.P.B., Gao, X., Zhang, M.: Data extraction from semi-structured web pages by clustering. In: proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 374–377. The IEEE Computer Society Press, Los Alamitos (2006)
Object Management Group: W3c document object model (2005), http://www.w3.org/dom/
Muslea, I.: Repository of online information sources used in information extraction tasks (2005), http://www.isi.edu/info-agents/rise/repository.html
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gao, X., Vuong, L.P.B., Zhang, M. (2007). Automatic Data Record Detection in Web Pages. In: Zhang, Z., Siekmann, J. (eds) Knowledge Science, Engineering and Management. KSEM 2007. Lecture Notes in Computer Science(), vol 4798. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-76719-0_35
Download citation
DOI: https://doi.org/10.1007/978-3-540-76719-0_35
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-76718-3
Online ISBN: 978-3-540-76719-0
eBook Packages: Computer ScienceComputer Science (R0)