Abstract
The record meterials used in some industries including medical service and insurance, whose information has high commercial and scientific research value, are still mainly paper-based. Recent progress in deep learning makes it easier to parse visually-rich document. Compared with traditional manual input, the application of this technology contributes to improvement of work efficiency and reduction of the training cost of business personnel. In previous work, the task of visual document parsing was divided into two stages which are composed of Optical Character Recognition (OCR) and Natural language understanding (NLU). In order to solve a series of problems in OCR, such as high computational costs, multi-language inflexibility and backward propagation of OCR errors, OCR-free multimodal visual document understanding method based on deep learning has been proposed recently. Through fine-tuning the pre-trained model, it can perform well in many downstream tasks. However, such approach is still limited in the specific context by 1) the language and context in which the encoder and decoder are pre-trained; 2) the image input size of the pre-trained encoder. In view of the above two problems, in this paper, we put forward the corresponding solutions, as a result, our proposed scheme won the second place in the “Identification of Electronic Medical Paper Documents (ePaper)” (IEMPD) task in the Eighth China Health Information Processing Conference (CHIP 2022) in an end-to-end way.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baek, J., et al.: What is wrong with scene text recognition model comparisons? dataset and model analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4715–4723 (2019)
Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9365–9374 (2019)
Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: Bros: a pre-trained language model focusing on text and layout for better key information extraction from documents. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 10767–10775 (2022)
Hwang, W., et al.: Post-ocr parsing: building simple and robust parser via bio tagging. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
Hwang, W., Lee, H., Yim, J., Kim, G., Seo, M.: Cost-effective end-to-end information extraction for semi-structured document images. arXiv preprint arXiv:2104.08041 (2021)
Hwang, W., Yim, J., Park, S., Yang, S., Seo, M.: Spatial dependency parsing for semi-structured document information extraction. arXiv preprint arXiv:2005.00642 (2020)
Kim, G., et al.: Ocr-free document understanding transformer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022, ECCV 2022. LNCS, vol. 13688, pp. 498–517. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_29
Lewis, M., et al.: Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
Li, J., Xu, Y., Lv, T., Cui, L., Zhang, C., Wei, F.: Dit: self-supervised pre-training for document image transformer. arXiv preprint arXiv:2203.02378 (2022)
Liu, Chang, Z., et al.: Information extraction of medical materials: an overview of the track of medical materials medocr. In: Health Information Processing (2022)
Liu, Chang, Z., et al.: Medocr: the dataset for extraction of optical character recognition elements for medical materials. J. Med. Inf. 43(12), 4 (2022)
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Majumder, B.P., Potti, N., Tata, S., Wendt, J.B., Zhao, Q., Najork, M.: Representation learning for information extraction from form-like documents. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6495–6504 (2020)
Xu, Y., et al.: Layoutlmv2: multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740 (2020)
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192–1200 (2020)
Yim, M., Kim, Y., Cho, H.-C., Park, S.: SynthTIGER: synthetic text image GEneratoR towards better text recognition models. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 109–124. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_8
Zhang, K., Shasha, D.: Simple fast algorithms for the editing distance between trees and related problems. SIAM J. Comput. 18(6), 1245–1262 (1989)
Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 564–580. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_34
Zong, L.L., et al.: Overview of technology evaluation dataset for medical multimodal information extraction. J. Med. Inf. 43(12), 6 (2022)
Acknowledgements
This work was supported by the Infinity Security, Hangzhou, China.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lu, Y., Qiu, W., Hong, Y., Wang, J. (2023). Multimodal End-to-End Visual Document Parsing. In: Tang, B., et al. Health Information Processing. Evaluation Track Papers. CHIP 2022. Communications in Computer and Information Science, vol 1773. Springer, Singapore. https://doi.org/10.1007/978-981-99-4826-0_15
Download citation
DOI: https://doi.org/10.1007/978-981-99-4826-0_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4825-3
Online ISBN: 978-981-99-4826-0
eBook Packages: Computer ScienceComputer Science (R0)