Skip to main content
SpringerLink
Log in

Crawling Data-Intensive Web Sources Using Structure Information

  • Conference paper
Business Information Systems Workshops (BIS 2013)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 160))

Included in the following conference series:

  • 1658 Accesses

Abstract

As more and more information resources are being published on the Web, companies, in order to fulfil their information needs, run a monitoring process of data-intensive Web sources. Although current Web monitoring solutions (Web crawlers) address many challenges of the information resources retrieval, they seem to miss an important business-related factor – the utility of the resulting resources’ collection. This article presents a concept of utility-aware Web crawler that can use Web site structure information to build a collection of information resources of high utility for a company and maintain the utility level when the resources change.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Levene, M.: An Introduction to Search Engines and Web Navigation. John Wiley & Sons (2010)

    Google Scholar 

  2. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. In: Proceedings of the Seventh International Conference on World Wide Web 7, WWW7, pp. 107–117. Elsevier Science Publishers B. V., Amsterdam (1998)

    Google Scholar 

  3. Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: a new approach to topic-specific web resource discovery. Comput. Netw. 31(11-16), 1623–1640 (1999)

    Article  Google Scholar 

  4. Abramowicz, W.: Filtrowanie informacji. Wydawnictwo Uniwersytetu Ekonomicznego w Poznaniu (2008)

    Google Scholar 

  5. Rennie, J., McCallum, A.: Using reinforcement learning to spider the web efficiently. In: Proceedings of the Sixteenth International Conference on Machine Learning, ICML 1999, pp. 335–343. Morgan Kaufmann Publishers Inc., San Francisco (1999)

    Google Scholar 

  6. Diligenti, M., Coetzee, F., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: Focused crawling using context graphs. In: Proceedings of the 26th International Conference on Very Large Data Bases, VLDB 2000, pp. 527–534. Morgan Kaufmann Publishers Inc., San Francisco (2000)

    Google Scholar 

  7. Haveliwala, T.H.: Topic-sensitive pagerank. In: Proceedings of the 11th International Conference on World Wide Web, WWW 2002, pp. 517–526. ACM, New York (2002)

    Google Scholar 

  8. Feng, S., Zhang, L., Xiong, Y., Yao, C.: Focused crawling using navigational rank. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM 2010, pp. 1513–1516. ACM, New York (2010)

    Chapter  Google Scholar 

  9. Pirkola, A., Talvensaari, T.: Addressing the limited scope problem of focused crawling using a result merging approach. In: Proceedings of the 2010 ACM Symposium on Applied Computing, SAC 2010, pp. 1735–1740. ACM, New York (2010)

    Chapter  Google Scholar 

  10. Barbosa, L., Bangalore, S.: Focusing on novelty: a crawling strategy to build diverse language models. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 755–764. ACM, New York (2011)

    Google Scholar 

  11. Kolari, P., Finin, T., Joshi, A.: SVMs for the blogosphere: Blog identification and splog detection. In: AAAI Spring Symposium on Computational Approaches to Analysing Weblogs (2006)

    Google Scholar 

  12. Cai, R., Yang, J.M., Lai, W., Wang, Y., Zhang, L.: iRobot: an intelligent crawler for web forums. In: Proceedings of the 17th International Conference on World Wide Web, WWW 2008, pp. 447–456. ACM, New York (2008)

    Chapter  Google Scholar 

  13. Yang, J.M., Cai, R., Wang, C., Huang, H., Zhang, L., Ma, W.Y.: Incorporating site-level knowledge for incremental crawling of web forums: a list-wise strategy. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2009, pp. 1375–1384. ACM, New York (2009)

    Chapter  Google Scholar 

  14. Jiang, J., Yu, N., Lin, C.Y.: Focus: learning to crawl web forums. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012 Companion, pp. 33–42. ACM, New York (2012)

    Chapter  Google Scholar 

  15. Catanese, S.A., De Meo, P., Ferrara, E., Fiumara, G., Provetti, A.: Crawling facebook for social network analysis purposes. In: Proceedings of the International Conference on Web Intelligence, Mining and Semantics, WIMS 2011, pp. 52:1–52:8. ACM, New York (2011)

    Google Scholar 

  16. Boanjak, M., Oliveira, E., Martins, J., Mendes Rodrigues, E., Sarmento, L.: Twitterecho: a distributed focused crawler to support open research with twitter data. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012 Companion, pp. 1233–1240. ACM, New York (2012)

    Chapter  Google Scholar 

  17. Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: Proceedings of the 27th International Conference on Very Large Data Bases, VLDB 2001, pp. 129–138. Morgan Kaufmann Publishers Inc., San Francisco (2001)

    Google Scholar 

  18. Ntoulas, A., Zerfos, P., Cho, J.: Downloading textual hidden web content through keyword queries. In: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital libraries, JCDL 2005, pp. 100–109. ACM, New York (2005)

    Google Scholar 

  19. Wu, P., Wen, J.R., Liu, H., Ma, W.Y.: Query selection techniques for efficient crawling of structured web sources. In: Proceedings of the 22nd International Conference on Data Engineering, ICDE 2006, p. 47. IEEE Computer Society, Washington, DC (2006)

    Google Scholar 

  20. Barbosa, L., Freire, J.: Searching for hidden-web databases. In: Doan, A., Neven, F., McCann, R., Bex, G.J. (eds.) WebDB, pp. 1–6 (2005)

    Google Scholar 

  21. Barbosa, L., Freire, J.: An adaptive crawler for locating hidden-web entry points. In: Proceedings of the 16th International Conference on World Wide Web, WWW 2007, pp. 441–450. ACM, New York (2007)

    Chapter  Google Scholar 

  22. Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., Halevy, A.: Google’s deep web crawl. Proc. VLDB Endow. 1(2), 1241–1252 (2008)

    Google Scholar 

  23. Liu, W., Xiao, J.: Incremental structured web database crawling via history versions. In: Chen, L., Triantafillou, P., Suel, T. (eds.) WISE 2010. LNCS, vol. 6488, pp. 524–533. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  24. Flejter, D.: Semi-Automatic Web Information Extraction. PhD thesis, Poznań University of Economics (2011)

    Google Scholar 

  25. Cho, J., Garcia-Molina, H.: Estimating frequency of change. ACM Trans. Internet Technol. 3(3), 256–290 (2003)

    Article  Google Scholar 

  26. Boehm, B.: A spiral model of software development and enhancement. SIGSOFT Softw. Eng. Notes 11(4), 14–24 (1986)

    Article  Google Scholar 

  27. Kaczmarek, T., Węckowski, D.G.: Web forums change analysis. In: Proceedings of the 9th International Conference on Web Information Systems and Technologies, Aachen, Germany (2013)

    Google Scholar 

  28. Cho, J., Garcia-Molina, H.: The evolution of the web and implications for an incremental crawler. In: Proceedings of the 26th International Conference on Very Large Data Bases, VLDB 2000, pp. 200–209. Morgan Kaufmann Publishers Inc., San Francisco (2000)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Systems, Poznań University of Economics, al. Niepodległości 10, 61-875, Poznań, Poland

    Dawid Grzegorz Węckowski

Authors
  1. Dawid Grzegorz Węckowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Information Systems, Poznań University of Economics, Poznań, Poland

    Witold Abramowicz

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Węckowski, D.G. (2013). Crawling Data-Intensive Web Sources Using Structure Information. In: Abramowicz, W. (eds) Business Information Systems Workshops. BIS 2013. Lecture Notes in Business Information Processing, vol 160. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41687-3_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-41687-3_19

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-41686-6

  • Online ISBN: 978-3-642-41687-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation