Abstract
Deep learning based pipelines for semantic segmentation often ignore structural information available on annotated images used for training. We propose a novel post-processing module enforcing structural knowledge about the objects of interest to improve segmentation results provided by deep learning. This module corresponds to a “many-to-one-or-none” inexact graph matching approach, and is formulated as a quadratic assignment problem. Using two standard measures for evaluation, we show experimentally that our pipeline for segmentation of 3D MRI data of the brain outperforms the baseline CNN (U-Net) used alone. In addition, our approach is shown to be resilient to small training datasets that often limit the performance of deep learning.
This research was conducted in the framework of the regional program Atlanstic 2020, Research, Education and Innovation in Pays de la Loire, supported by the French Region Pays de la Loire and the European Regional Development Fund.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The open-source code and data are to be shared with the community https://github.com/Jeremy-Chopin/APACoSI/.
- 2.
The IBSR annotated public dataset can be downloaded at the following address: https://www.nitrc.org/projects/ibsr.
References
Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017). https://doi.org/10.1109/TPAMI.2016.2644615
Bloch, I.: Fuzzy sets for image processing and understanding. Fuzzy Sets Syst. 281, 280–291 (2015). https://doi.org/10.1016/j.fss.2015.06.017
Chopin, J., Fasquel, J.B., Mouchère, H., Dahyot, R., Bloch, I.: Semantic image segmentation based on spatial relationships and inexact graph matching. In: 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6 (2020). https://doi.org/10.1109/IPTA50016.2020.9286611
Colliot, O., Camara, O., Bloch, I.: Integration of fuzzy spatial relations in deformable models - application to brain MRI segmentation. Pattern Recogn. 39, 1401–1414 (2006). https://doi.org/10.1016/j.patcog.2006.02.022
Delaye, A., Anquetil, E.: Fuzzy relative positioning templates for symbol recognition. In: International Conference on Document Analysis and Recognition, Beijing, China, September 2011. https://doi.org/10.1109/ICDAR.2011.246
Deruyver, A., Hodé, Y.: Qualitative spatial relationships for image interpretation by using a conceptual graph. Image Vis. Comput. 27(7), 876–886 (2009). https://doi.org/10.1016/j.imavis.2008.10.002, 7th IAPR-TC15 Workshop on Graph-based Representations (GbR 2007)
Fasquel, J.B., Agnus, V., Moreau, J., Soler, L., Marescaux, J.: An interactive medical image segmentation system based on the optimal management of regions of interest using topological medical knowledge. Comput. Methods Programs Biomed. 82, 216–230 (2006). https://doi.org/10.1016/j.cmpb.2006.04.004
Fasquel, J.B., Delanoue, N.: An approach for sequential image interpretation using a priori binary perceptual topological and photometric knowledge and k-means based segmentation. J. Opt. Soc. Am. A 35(6), 936–945 (2018). https://doi.org/10.1364/JOSAA.35.000936
Fasquel, J.B., Delanoue, N.: A graph based image interpretation method using a priori qualitative inclusion and photometric relationships. IEEE Trans. Pattern Anal. Mach. Intell. 41(5), 1043–1055 (2019). https://doi.org/10.1109/TPAMI.2018.2827939
Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Appl. Soft Comput. 70, 41–65 (2018). https://doi.org/10.1016/j.asoc.2018.05.018
Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Julca-Aguilar, F., Mouchère, H., Viard-Gaudin, C., Hirata, N.S.T.: A general framework for the recognition of online handwritten graphics. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 143–160 (2020). https://doi.org/10.1007/s10032-019-00349-6
Kunze, L., et al.: Combining top-down spatial reasoning and bottom-up object class recognition for scene understanding. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2910–2915. IEEE (2014). https://doi.org/10.1109/IROS.2014.6942963
Kushibar, K., et al.: Automated sub-cortical brain structure segmentation combining spatial and deep convolutional features. Med. Image Anal. 48, 177–186 (2018). https://doi.org/10.1016/j.media.2018.06.006
Lee, B., Yamanakkanavar, N., Choi, J.Y.: Automatic segmentation of brain MRI using a novel patch-wise U-Net deep architecture. PLOS ONE 15(8), 1–20 (2020). https://doi.org/10.1371/journal.pone.0236493
Lezoray, O., Leo, L.: Image Processing and Analysis with Graphs: Theory and Practice. CRC Press, Cambridge (2012)
Maciel, J., Costeira, J.P.: A global solution to sparse correspondence problems. IEEE Trans. Pattern Anal. Mach. Intell. 25(2), 187–199 (2003). https://doi.org/10.1109/TPAMI.2003.1177151
Moreno, A., Takemura, C., Colliot, O., Camara, O., Bloch, I.: Using anatomical knowledge expressed as fuzzy constraints to segment the heart in CT images. Pattern Recogn. 41(8), 2525–2540 (2008). https://doi.org/10.1016/j.patcog.2008.01.020
Nempont, O., Atif, J., Bloch, I.: A constraint propagation approach to structural model based image segmentation and recognition. Inf. Sci. 246, 1–27 (2013). https://doi.org/10.1016/j.ins.2013.05.030
Noma, A., Graciano, A.B., Cesar, R.M., Jr., Consularo, L.A., Bloch, I.: Interactive image segmentation by matching attributed relational graphs. Pattern Recogn. 45(3), 1159–1179 (2012). https://doi.org/10.1016/j.patcog.2011.08.017
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Pérez-García, F., Sparks, R., Ourselin, S.: TorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. arXiv:2003.04696 [cs, eess, stat], March 2020
Weiss, K., Khoshgoftaar, T.M., Wang, D.D.: A survey of transfer learning. J. Big Data 3(1), 1–40 (2016). https://doi.org/10.1186/s40537-016-0043-6
Zanfir, A., Sminchisescu, C.: Deep learning of graph matching. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2684–2693 (2018). https://doi.org/10.1109/CVPR.2018.00284
Zhou, F., De la Torre, F.: Factorized graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 38(9), 1774–1789 (2016). https://doi.org/10.1109/TPAMI.2015.2501802
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Chopin, J., Fasquel, JB., Mouchère, H., Dahyot, R., Bloch, I. (2022). Improving Semantic Segmentation with Graph-Based Structural Knowledge. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)