Abstract
Geological knowledge can provide support for knowledge discovery, knowledge inference and mineralization predictions of geological big data. Entity identification and relationship extraction from geological data description text are the key links for constructing knowledge graphs. Given the lack of publicly annotated datasets in the geology domain, this paper illustrates the construction process of geological entity datasets, defines the types of entities and interconceptual relationships by using the geological entity concept system, and completes the construction of the geological corpus. To address the shortcomings of existing language models (such as Word2vec and Glove) that cannot solve polysemous words and have a poor ability to fuse contexts, we propose a geological named entity recognition and relationship extraction model jointly with Bidirectional Encoder Representation from Transformers (BERT) pretrained language model. To effectively represent the text features, we construct a BERT- bidirectional gated recurrent unit network (BiGRU)-conditional random field (CRF)-based architecture to extract the named entities and the BERT-BiGRU-Attention-based architecture to extract the entity relations. The results show that the F1-score of the BERT-BiGRU-CRF named entity recognition model is 0.91 and the F1-score of the BERT-BiGRU-Attention relationship extraction model is 0.84, which are significant performance improvements when compared to classic language models (e.g., word2vec and Embedding from Language Models (ELMo)).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References Cited
Bengio, Y., Ducharme, R., Vincent, P., 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3: 1137–1155
Bojanowski, P., Grave, E., Joulin, A., et al., 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5: 135–146. https://doi.org/10.1162/tacl_a_00051
Bouvrie, J., 2006. Notes on Convolutional Neural Networks, Neural Nets. http://web.mit.edu/jvb/www/papers/cnn_tutorial.pdf
Cao, P., Chen, Y., Liu, K., et al., 2018. Adversarial Transfer Learning for Chinese Named Entity Recognition with Self-Attention Mechanism. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, October 31–November 4, Brusssels
Chiticariu, L., Krishnamurthy, R., Li, Y. Y., et al., 2010. Domain Adaptation of Rule-Based Annotators for Named-Entity Recognition Tasks. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. October 9–11, 2010, Cambridge. 1002–1012. https://doi.org/10.5555/1870658.1870756
Chiu, J. P., Nichols, E., 2016. Named Entity Recognition with Bidirectional LSTM-CNNS. Transactions of the Association for Computational Linguistics, 4: 357–370
Deng, C., Jia, Y. T., Xu, H., et al., 2021. GAKG: A Multimodal Geoscience Academic Knowledge Graph. Proceedings of the 30th ACM International Conference on Information & Knowledge Management. November 1–5, 2021, Virtual Event, New York. 4445–4454. https://doi.org/10.1145/3459637.3482003
Devlin, J., Chang, M. W., Lee, K., et al., 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805. https://arxiv.org/abs/1810.04805
Enkhsaikhan, M., Liu, W., Holden, E. J., et al., 2021. Auto-Labelling Entities in Low-Resource Text: A Geological Case Study. Knowledge and Information Systems, 63(3): 695–715. https://doi.org/10.1007/s10115-020-01532-6
Fan, J., Shen, S., Erwin, D. H., et al., 2020. A High-Resolution Summary of Cambrian to Early Triassic Marine Invertebrate Biodiversity. Science, 367(6475): 272–277
Fang, W. L., Ma, L., Love, P. E. D., et al., 2020. Knowledge Graph for Identifying Hazards on Construction Sites: Integrating Computer Vision with Ontology. Automation in Construction, 119: 103310. https://doi.org/10.1016/j.autcon.2020.103310
Gayen, V., Sarkar, K., 2014. An HMM Based Named Entity Recognition System for Indian Languages: The JU System at ICON 2013. arXiv: 1405.7397. https://arxiv.org/abs/1405.7397
Gers, F. A., Schmidhuber, J., Cummins, F., 2000. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10): 2451–2471. https://doi.org/10.1162/089976600300015015
Ghahabi, O., Hernando, J., 2018. Restricted Boltzmann Machines for Vector Representation of Speech in Speaker Recognition. Computer Speech & Language, 47: 16–29. https://doi.org/10.1016/j.csl.2017.06.007
Hashimoto, K., Miwa, M., Tsuruoka, Y., et al., 2013. Simple Customization of Recursive Neural Networks for Semantic Relation Classification. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 18–21
Hochreiter, S., Schmidhuber, J., 1997. Long Short-Term Memory. Neural Computation, 9(8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Jauregi Unanue, I., Zare Borzeshi, E., Piccardi, M., 2017. Recurrent Neural Networks with Specialized Word Embeddings for Health-Domain Named-Entity Recognition. Journal of Biomedical Informatics, 76: 102–109. https://doi.org/10.1016/j.jbi.2017.11.007
Lai, T., Ji, H., Zhai, C. X., et al., 2021. Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference. arXiv: 2105.13456. https://arxiv.org/abs/2105.13456
Lawley, C. J. M., Raimondo, S., Chen, T. Y., et al., 2022. Geoscience Language Models and Their Intrinsic Evaluation. Applied Computing and Geosciences, 14: 100084. https://doi.org/10.1016/j.acags.2022.100084
Li, P. F., Mao, K. Z., 2019. Knowledge-Oriented Convolutional Neural Network for Causal Relation Extraction from Natural Language Texts. Expert Systems with Applications, 115: 512–523. https://doi.org/10.1016/j.eswa.2018.08.009
Lin, Y., Shen, S., Liu, Z., et al., 2016. Neural Relation Extraction with Selective Attention Over Instances. The 54th Annual Meeting of the Association for Computational Linguistics, August 7–12, Berlin
Liu, Z. J., Yang, M., Wang, X. L., et al., 2017. Entity Recognition from Clinical Texts via Recurrent Neural Network. BMC Medical Informatics and Decision Making, 17(Suppl 2): 67. https://doi.org/10.1186/s12911-017-0468-7
Lü, X., Xie, Z., Xu, D., et al., 2022. Chinese Named Entity Recognition in the Geoscience Domain Based on BERT. Earth and Space Science, 9(3): e2021EA002166
Ma, K., Tan, Y. J., Tian, M., et al., 2022a. Extraction of Temporal Information from Social Media Messages Using the BERT Model. Earth Science Informatics, 15(1): 573–584. https://doi.org/10.1007/s12145-021-00756-6
Ma, K., Tan, Y. J., Xie, Z., et al., 2022b. Chinese Toponym Recognition with Variant Neural Structures from Social Media Messages Based on BERT Methods. Journal of Geographical Systems, 24(2): 143–169. https://doi.org/10.1007/s10109-022-00375-9
Ma, K., Tian, M., Tan, Y. J., et al., 2022c. What is This Article About? Generative Summarization with the BERT Model in the Geosciences Domain. Earth Science Informatics, 15(1): 21–36. https://doi.org/10.1007/s12145-021-00695-2
Ma, X. G., 2019. Geo-Data Science: Leveraging Geoscience Research with Geoinformatics, Semantics and Open Data. Acta Geologica Sinica-English Edition, 93(S3): 44–47. https://doi.org/10.1111/1755-6724.14240
Ma, X. G., Ma, C., Wang, C. B., 2020. A New Structure for Representing and Tracking Version Information in a Deep Time Knowledge Graph. Computers & Geosciences, 145: 104620. https://doi.org/10.1016/j.cageo.2020.104620
Mikolov, T., Chen, K., Corrado, G., et al., 2013. Efficient Estimation of Word Representations in Vector Space. arXiv: 1301.3781. https://arxiv.org/abs/1301.3781
Miwa, M., Bansal, M., 2016. End-to-End Relation Extraction Using LSTMS on Sequences and Tree Structures. arXiv: 1601.00770. https://arxiv.org/abs/1601.00770
Nguyen, T. H., Grishman, R., 2015. Relation Extraction: Perspective from Convolutional Neural Networks. Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing. Association for Computational Linguistics, Denver
Nieh, E. H., Schottdorf, M., Freeman, N. W., et al., 2021. Geometry of Abstract Learned Knowledge in the Hippocampus. Nature, 595(7865): 80–84. https://doi.org/10.1038/s41586-021-03652-7
Oramas, S., Ostuni, V. C., Di Noia, T., et al., 2017. Sound and Music Recommendation with Knowledge Graphs. ACM Transactions on Intelligent Systems and Technology, 8(2): 1–21. https://doi.org/10.1145/2926718
Palumbo, E., Monti, D., Rizzo, G., et al., 2020. Entity2rec: Property-Specific Knowledge Graph Embeddings for Item Recommendation. Expert Systems with Applications, 151: 113235. https://doi.org/10.1016/j.eswa.2020.113235
Peng, N. Y., Dredze, M., 2016. Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning. arXiv: 1603.00786. https://arxiv.org/abs/1603.00786
Pennington, J., Socher, R., Manning, C., 2014. Glove: Global Vectors for Word Representation Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha
Peters, M. E., Neumann, M., Iyyer, M., et al., 2018. Deep Contextualized Word Representations. arXiv: 1802.05365. https://arxiv.org/abs/1802.05365
Qiu, Q. J., Xie, Z., Wu, L., et al., 2018. DGeoSegmenter: A Dictionary-Based Chinese Word Segmenter for the Geoscience Domain. Computers & Geosciences, 121: 1–11. https://doi.org/10.1016/j.cageo.2018.08.006
Qiu, Q. J., Xie, Z., Wu, L., et al., 2019a. BiLSTM-CRF for Geological Named Entity Recognition from the Geoscience Literature. Earth Science Informatics, 12(4): 565–579. https://doi.org/10.1007/s12145-019-00390-3
Qiu, Q. J., Xie, Z., Wu, L., et al., 2019b. Geoscience Keyphrase Extraction Algorithm Using Enhanced Word Embedding. Expert Systems With Applications, 125: 157–169. https://doi.org/10.1016/j.eswa.2019.02.001
Qiu, Q. J., Xie, Z., Wu, L. A., et al., 2019c. GNER: A Generative Model for Geological Named Entity Recognition without Labeled Data Using Deep Learning. Earth and Space Science, 6(6): 931–946. https://doi.org/10.1029/2019ea000610
Qiu, Q. J., Xie, Z., Wu, L., et al., 2020a. Automatic Spatiotemporal and Semantic Information Extraction from Unstructured Geoscience Reports Using Text Mining Techniques. Earth Science Informatics, 13(4): 1393–1410. https://doi.org/10.1007/s12145-020-00527-9
Qiu, Q. J., Xie, Z., Wu, L. A., et al., 2020b. Dictionary-Based Automated Information Extraction from Geological Documents Using a Deep Learning Algorithm. Earth and Space Science, 7(3): e2019EA000993. https://doi.org/10.1029/2019ea000993
Qu, J. F., Ouyang, D. T., Hua, W., et al., 2018. Distant Supervision for Neural Relation Extraction Integrated with Word Attention and Property Features. Neural Networks, 100: 59–69. https://doi.org/10.1016/j.neunet.2018.01.006
Radford, A., Narasimhan, K., 2018. Improving Language Understanding by Generative Pre-Training, preprint. https://gwern.net/doc/www/s3-us-west-2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf
Radford, A., Wu, J., Child, R., et al., 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8): 9
Santos, R., Murrieta-Flores, P., Calado, P., et al., 2018. Toponym Matching through Deep Neural Networks. International Journal of Geographical Information Science, 32(2): 324–348. https://doi.org/10.1080/13658816.2017.1390119
Singhal, A., 2012. Introducing the Knowledge Graph: Things, not Strings. Google Blog. https://www.blog.google/products/search/introducing-knowledge-graph-things-not/
Sun, C., Yang, Z. H., Wang, L., et al., 2021. Biomedical Named Entity Recognition Using BERT in the Machine Reading Comprehension Framework. Journal of Biomedical Informatics, 118: 103799. https://doi.org/10.1016/j.jbi.2021.103799
Surdeanu, M., Tibshirani, J., Nallapati, R., et al., 2012. Multi-Instance Multi-Label Learning for Relation Extraction. The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. July 12–14, 2012, Jeju Island. https://doi.org/10.5555/2390948.2391003
Vaswani, A., Shazeer, N., Parmar, N., et al., 2017. Attention is All You Need. Advances in Neural Information Processing Systems, 30: 5998–6008
Vincent, P., Larochelle, H., Bengio, Y., et al., 2008. Extracting and Composing Robust Features with Denoising Autoencoders. The 25th International conference on Machine Learning. July 5–9, 2008, Helsinki. https://doi.org/10.1145/1390156.1390294
Wang, C., Hazen, R. M., Cheng, Q., et al., 2021. The Deep-Time Digital Earth Program: Data-Driven Discovery in Geosciences. National Science Review, 8(9): nwab027. https://doi.org/10.1093/nsr/nwab027
Wu, S., Song, X. N., Feng, Z. H., 2021. MECT: Multi-Metadata Embedding Based Cross-Transformer for Chinese Named Entity Recognition. arXiv: 2107.05418. https://arxiv.org/abs/2107.05418
Xu, Y., Mou, L. L., Li, G., et al., 2015. Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path. arXiv: 1508.03720. https://arxiv.org/abs/1508.03720
Yao, L., Zhang, Y., Chen, Q. F., et al., 2017. Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data. Engineering Applications of Artificial Intelligence, 64: 432–439. https://doi.org/10.1016/j.engappai.2017.06.024
Zeng, D., Liu, K., Lai, S., et al., 2014. Relation Classification via Convolutional Deep Neural Network. The 25th International Conference on Computational Linguistics: Technical Papers, March 25–28, Tokyo
Zhang, W., Du, Y. H., Yoshida, T., et al., 2019. DeepRec: A Deep Neural Network Approach to Recommendation with Item Embedding and Weighted Loss Function. Information Sciences, 470: 121–140. https://doi.org/10.1016/j.ins.2018.08.039
Zhang, X. Y., Ye, P., Wang, S., et al., 2018. Geological Entity Recognition Method Based on Deep Belief Networks. Acta Petrologica Sinica, 34(2): 343–351
Zheng, S. C., Hao, Y. X., Lu, D. Y., et al., 2017a. Joint Entity and Relation Extraction Based on a Hybrid Neural Network. Neurocomputing, 257: 59–66. https://doi.org/10.1016/j.neucom.2016.12.075
Zheng, S. C., Wang, F., Bao, H. Y., et al., 2017b. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. arXiv: 1706.05075. https://arxiv.org/abs/1706.05075
Zhou, C. H., Wang, H., Wang, C. S., et al., 2021. Geoscience Knowledge Graph in the Big Data Era. Science China Earth Sciences, 64(7): 1105–1114. https://doi.org/10.1007/s11430-020-9750-4
Zhou, P., El-Gohary, N., 2017. Ontology-Based Automated Information Extraction from Building Energy Conservation Codes. Automation in Construction, 74: 103–117
Zhou, P., Xu, J., Qi, Z., et al., 2018. Distant Supervision for Relation Extraction with Hierarchical Selective Attention. Neural Networks, 108:240. https://doi.org/10.1016/j.neunet.2018.08.016
Acknowledgments
This study was financially supported by the National Key R & D Program of China (No. 2022YFF0711601), the Natural Science Foundation of Hubei Province of China (No. 2022CFB640), the Open Fund of Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources (No. KF-2022-07-014), the Opening Fund of Hubei Key Laboratory of Intelligent Vision-Based Monitoring for Hydroelectric Engineering (No. 2022SDSJ04), and the Beijing Key Laboratory of Urban Spatial Information Engineering (No. 20220108). The final publication is available at Springer via https://doi.org/10.1007/s12583-022-1724-z.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Ma, K., Tian, M., Tan, Y. et al. Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports. J. Earth Sci. 34, 1390–1405 (2023). https://doi.org/10.1007/s12583-022-1724-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12583-022-1724-z