Abstract
Unsupervised learning algorithms (e.g., self-supervised learning, auto-encoder, contrastive learning) allow deep learning models to learn effective image representations from large-scale unlabeled data. In medical image analysis, even unannotated data can be difficult to obtain for individual labs. Fortunately, national-level efforts have been made to provide efficient access to obtain biomedical image data from previous scientific publications. For instance, NIH has launched the Open-i\(^\circledR \) search engine that provides a large-scale image database with free access. However, the images in scientific publications consist of a considerable amount of compound figures with subplots. To extract and curate individual subplots, many different compound figure separation approaches have been developed, especially with the recent advances in deep learning. However, previous approaches typically required resource extensive bounding box annotation to train detection models. In this paper, we propose a simple compound figure separation (SimCFS) framework that uses weak classification annotations from individual images. Our technical contribution is three-fold: (1) we introduce a new side loss that is designed for compound figure separation; (2) we introduce an intra-class image augmentation method to simulate hard cases; (3) the proposed framework enables an efficient deployment to new classes of images, without requiring resource extensive bounding box annotations. From the results, the SimCFS achieved a new state-of-the-art performance on the ImageCLEF 2016 Compound Figure Separation Database. The source code of SimCFS is made publicly available at https://github.com/hrlblab/ImageSeperation.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Apostolova, E., You, D., Xue, Z., Antani, S., Demner-Fushman, D., Thoma, G.R.: Image retrieval from scientific publications: text and image content processing to separate multipanel figures. J. Am. Soc. Inform. Sci. Technol. 64(5), 893–908 (2013)
Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: YOLOv4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)
Bueno, G., Fernandez-Carrobles, M.M., Gonzalez-Lopez, L., Deniz, O.: Glomerulosclerosis identification in whole slide images using semantic segmentation. Comput. Methods Programs Biomed. 184, 105273 (2020)
Celebi, M.E., Aydin, K.: Unsupervised Learning Algorithms. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-24211-8
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Davila, K., Setlur, S., Doermann, D., Bhargava, U.K., Govindaraju, V.: Chart mining: a survey of methods for automated chart analysis. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Demner-Fushman, D., Antani, S., Simpson, M., Thoma, G.R.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6(2), 168–177 (2012)
Gadermayr, M., Dombrowski, A.K., Klinkhammer, B.M., Boor, P., Merhof, D.: CNN cascades for segmenting whole slide images of the kidney. arXiv preprint arXiv:1708.00251 (2017)
Ginley, B., et al.: Computational segmentation and classification of diabetic glomerulosclerosis. J. Am. Soc. Nephrol. 30(10), 1953–1967 (2019)
Govind, D., Ginley, B., Lutnick, B., Tomaszewski, J.E., Sarder, P.: Glomerular detection and segmentation from multimodal microscopy images using a butterworth band-pass filter. In: Medical Imaging 2018: Digital Pathology, vol. 10581, p. 1058114. International Society for Optics and Photonics (2018)
GarcÃa Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: Working Notes of CLEF 2016 (Cross Language Evaluation Forum), September 2016
Huang, W., Tan, C.L., Leow, W.K.: Associating text and graphics for scientific chart understanding. In: Eighth International Conference on Document Analysis and Recognition (ICDAR 2005), pp. 580–584. IEEE (2005)
Huo, Y., Deng, R., Liu, Q., Fogo, A.B., Yang, H.: AI applications in renal pathology. Kidney Int. 99, 1309–1320 (2021)
Jiang, W., Schwenker, E., Spreadbury, T., Ferrier, N., Chan, M.K., Cossairt, O.: A two-stage framework for compound figure separation. arXiv preprint arXiv:2101.09903 (2021)
Kalpathy-Cramer, J., de Herrera, A.G.S., Demner-Fushman, D., Antani, S., Bedrick, S., Müller, H.: Evaluating performance of biomedical image retrieval systems–an overview of the medical image retrieval task at ImageCLEF 2004–2013. Comput. Med. Imaging Graph. 39, 55–61 (2015)
Kannan, S., et al.: Segmentation of glomeruli within trichrome images using deep learning. Kidney Int. Rep. 4(7), 955–962 (2019)
Koziell, A., et al.: Genotype/phenotype correlations of NPHS1 and NPHS2 mutations in nephrotic syndrome advocate a functional inter-relationship in glomerular filtration. Hum. Mol. Genet. 11(4), 379–388 (2002)
Lee, P.-S., Howe, B.: Detecting and dismantling composite visualizations in the scientific literature. In: Fred, A., De Marsico, M., Figueiredo, M. (eds.) ICPRAM 2015. LNCS, vol. 9493, pp. 247–266. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-27677-9_16
Lee, P.S., Howe, B.: Dismantling composite visualizations in the scientific literature. In: ICPRAM (2), pp. 79–91. Citeseer (2015)
Li, P., Jiang, X., Kambhamettu, C., Shatkay, H.: Compound image segmentation of published biomedical figures. Bioinformatics 34(7), 1192–1199 (2017). https://doi.org/10.1093/bioinformatics/btx611
Li, P., Jiang, X., Kambhamettu, C., Shatkay, H.: Segmenting compound biomedical figures into their constituent panels. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 199–210. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_20
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Sathya, R., Abraham, A.: Comparison of supervised and unsupervised learning algorithms for pattern classification. Int. J. Adv. Res. Artif. Intell. 2(2), 34–38 (2013)
Shi, X., Wu, Y., Cao, H., Burns, G., Natarajan, P.: Layout-aware subfigure decomposition for complex figures in the biomedical literature. In: 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2019, pp. 1343–1347. IEEE (2019)
Tsutsui, S., Crandall, D.J.: A data driven approach for compound figure separation using convolutional neural networks. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 533–540. IEEE (2017)
Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 408–416. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_47
Zou, J., Thoma, G., Antani, S.: Unified deep neural network for segmentation and labeling of multipanel biomedical figures. J. Am. Soc. Inf. Sci. 71(11), 1327–1340 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yao, T. et al. (2021). Compound Figure Separation of Biomedical Images with Side Loss. In: Engelhardt, S., et al. Deep Generative Models, and Data Augmentation, Labelling, and Imperfections. DGM4MICCAI DALI 2021 2021. Lecture Notes in Computer Science(), vol 13003. Springer, Cham. https://doi.org/10.1007/978-3-030-88210-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-88210-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88209-9
Online ISBN: 978-3-030-88210-5
eBook Packages: Computer ScienceComputer Science (R0)