Abstract
The purpose of this paper is to present the design and results of experiments that focus on universal, autonomous data extraction (web scraping) system fed by publicly available online job listings. In particular, methods of automated crawling, preprocessing and classifying data from job offers will be presented together with the aggregation of the acquired data stored in large-scale, structured databases. We tested two models to classify the content of job portals: fastText and XGBoost. We obtained promising results in the experimental phase, with 88% accuracy by both methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahmadi, A., Fotouhi, M., Khaleghi, M.: Intelligent classification of web pages using contextual and visual features. Appl. Soft Comput. 11(2), 1638–1647 (2011)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Chen, T., Guestrin, C.: XGboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
Drozda, P., Talun, A., Bukowski, L.: Emplobot - design of the system. In: Proceedings of the 28th International Workshop on Concurrency, Specification and Programming, Olsztyn, Poland, 24–26th September 2019 (2019)
Dumais, S., Chen, H.: Hierarchical classification of web content. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 256–263 (2000)
Dziwiński, P., Bartczuk, Ł., Paszkowski, J.: A new auto adaptive fuzzy hybrid particle swarm optimization and genetic algorithm. J. Artif. Intell. Soft Comput. Res. 10(2), 95–111 (2020)
Gabryel, M.: The bag-of-words method with different types of image features and dictionary analysis. J. UCS 24(4), 357–371 (2018)
Gabryel, M., Grzanek, K., Hayashi, Y.: Browser fingerprint coding methods increasing the effectiveness of user identification in the web traffic. J. Artif. Intell. Soft Comput. Res. 10(4), 243–253 (2020)
Gabryel, M., Przybyszewski, K.: The dynamically modified BoW algorithm used in assessing clicks in online ads. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11509, pp. 350–360. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20915-5_32
Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)
Koren, O., Hallin, C.A., Perel, N., Bendet, D.: Decision-making enhancement in a big data environment: application of the k-means algorithm to mixed data. J. Artif. Intell. Soft Comput. Res. 9(4), 293–302 (2019)
Korytkowski, M., Senkerik, R., Scherer, M.M., Angryk, R.A., Kordos, M., Siwocha, A.: Efficient image retrieval by fuzzy rules from boosting and metaheuristic. J. Artif. Intell. Soft Comput. Res. 10(1), 57–69 (2020)
Kumar, R., Jain, A., Agrawal, C.: Survey of web crawling algorithms. Adv. Vis. Comput.: Int. J. (AVC) 1(2/3) (2014)
Ludwig, S.A.: Applying a neural network ensemble to intrusion detection. J. Artif. Intell. Soft Comput. Res. 9(3), 177–188 (2019)
Mahdi, D.A.F., Ahmed, R.K.A.: A new technique for web crawling in multimedia web sites. Int. J. Comput. Eng. Res. 4(2) (2014)
Malhotra, R., Sharma, A.: Quantitative evaluation of web metrics for automatic genre classification of web pages. Int. J. Syst. Assur. Eng. Manag. 8(2), 1567–1579 (2017)
Tambouratzis, G., Vassiliou, M.: Swarm algorithms for NLP - the case of limited training data. J. Artif. Intell. Soft Comput. Res. 9(3), 219–234 (2019)
Vijayarani, S., Suganya, M.E.: Web crawling algorithms–a comparative study. Int. J. Sci. Adv. Res. Technol. 2(10) (2016)
Acknowledgements
This work is a part of the Emplobot project number POIR.01.01.01-00-1135/17 “Development of autonomous artificial intelligence using the learning of deep neural networks with strengthening, automating recruitment processes” funded by the National Centre for Research and Development.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Talun, A., Drozda, P., Bukowski, L., Scherer, R. (2020). FastText and XGBoost Content-Based Classification for Employment Web Scraping. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2020. Lecture Notes in Computer Science(), vol 12416. Springer, Cham. https://doi.org/10.1007/978-3-030-61534-5_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-61534-5_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61533-8
Online ISBN: 978-3-030-61534-5
eBook Packages: Computer ScienceComputer Science (R0)