Skip to main content
SpringerLink
Log in

Automated Organic Web Harvesting on Web Data for Analytics

  • Conference paper
  • First Online:
Data Science and Security

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 290))

  • 514 Accesses

Abstract

Automated Web search and web data extraction has become an inevitable part of research in the area of web mining. The web scraping has immense influence on ecommerce, market research, web indexing and much more. Most of the web information is presented in an unstructured or free format. Web scraping helps every user to retrieve, analyze and use the data suitably according to their requirement. There exist different methodologies for web scraping. Major web scraping tools are rule based systems. In the proposed work, an automated method for web information extraction using Computer Vision is proposed and developed. The proposed automated web scraping method comprises of automated URL extraction virtual extraction of required data and storing the data in a structured format which is useful in market research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. https://www.webharvy.com/articles/what-is-web-scraping.html

  2. https://wscraper.com/what-is-data-harvesting-and-how-to-prevent-it/

  3. Ashiwal P, Tandan SR, Tripathi P, Miri R (2016) Web information retrieval using python and beautifulsoup. Int J Res Appl Sci Eng Technol 4(VI). ISSN: 2321–9653

    Google Scholar 

  4. Peterson A (2021) BeautifulSoup: Web Scraping with Python

    Google Scholar 

  5. https://www.shieldsquare.com/what-are-the-different-scraping-techniques/

  6. https://towardsdatascience.com/https-medium-com-hiren787-patel-web-scraping-applications-a6f370d316f4

  7. https://www.import.io/post/web-scraping-explained/

  8. Sirisurya S (2015) A comparative study on web scraping. In: Proceedings of 8th International Research Conference, KDU

    Google Scholar 

  9. https://www.analyticsvidhya.com/blog/2020/04/5-popular-python-libraries-web-scraping

  10. https://yoast.com/what-is-a-snippet/

  11. https://pypi.org/project/pytesseract/

  12. Liu W, Meng X, Meng W (2010) ViDE: a vision-based approach for deep web data extraction. IEEE Trans Knowl Data Eng 22:447–460. https://doi.org/10.1109/TKDE.2009.109

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Christ University, Bangalore, India

    Lija Jacob & K. T. Thomas

Authors
  1. Lija Jacob
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. T. Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lija Jacob .

Editor information

Editors and Affiliations

  1. Christ University, Bangalore, India

    Samiksha Shukla

  2. Digital Monozukuri, Stanford Alumni, Palo Alto, CA, USA

    Aynur Unal

  3. Christ University, Bangalore, India

    Joseph Varghese Kureethara

  4. Computer Science and Engineering, Sri Aurobindo Institute of Technology, Indore, Madhya Pradesh, India

    Durgesh Kumar Mishra

  5. School of Electronics Engineering, Kyungpook National University, Daegu, Korea (Republic of)

    Dong Seog Han

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jacob, L., Thomas, K.T. (2021). Automated Organic Web Harvesting on Web Data for Analytics. In: Shukla, S., Unal, A., Kureethara, J.V., Mishra, D.K., Han, D.S. (eds) Data Science and Security. Lecture Notes in Networks and Systems, vol 290. Springer, Singapore. https://doi.org/10.1007/978-981-16-4486-3_14

Download citation

Publish with us

Policies and ethics

Search

Navigation