Abstract
Modern medical image segmentation methods primarily use discrete representations in the form of rasterized masks to learn features and generate predictions. Although effective, this paradigm is spatially inflexible, scales poorly to higher-resolution images, and lacks direct understanding of object shapes. To address these limitations, some recent works utilized implicit neural representations (INRs) to learn continuous representations for segmentation. However, these methods often directly adopted components designed for 3D shape reconstruction. More importantly, these formulations were also constrained to either point-based or global contexts, lacking contextual understanding or local fine-grained details, respectively—both critical for accurate segmentation. To remedy this, we propose a novel approach, SwIPE (Segmentation with Implicit Patch Embeddings), that leverages the advantages of INRs and predicts shapes at the patch level—rather than at the point level or image level—to enable both accurate local boundary delineation and global shape coherence. Extensive evaluations on two tasks (2D polyp segmentation and 3D abdominal organ segmentation) show that SwIPE significantly improves over recent implicit approaches and outperforms state-of-the-art discrete methods with over 10x fewer parameters. Our method also demonstrates superior data efficiency and improved robustness to data shifts across image resolutions and datasets. Code is available on Github.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Multi-atlas labeling beyond the cranial vault (2015). https://www.synapse.org/#!Synapse:syn3193805/wiki/89480. Accessed Jan 2021
Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Med. Imaging Graph. 43, 99–111 (2015)
Chabra, R., et al.: Deep local shapes: learning local SDF priors for detailed 3D reconstruction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 608–625. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_36
Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3D shape reconstruction and completion. CVPR, pp. 6968–6979 (2020)
Dupont, E., Goliński, A., Alizadeh, M., Teh, Y.W., Doucet, A.: COIN: COmpression with Implicit Neural representations. arXiv preprint arXiv:2103.03123 (2021)
Fan, D.-P., et al.: PraNet: parallel reverse attention network for polyp segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12266, pp. 263–273. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59725-2_26
Gao, S.H., et al.: Res2Net: a new multi-scale backbone architecture. IEEE TPAMI 43(2), 652–662 (2019)
Gu, P., Zheng, H., Zhang, Y., Wang, C., Chen, D.Z.: kCBAC-Net: Deeply Supervised Complete Bipartite Networks with Asymmetric Convolutions for Medical Image Segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12901, pp. 337–347. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87193-2_32
Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., Shi, H.: Escaping the big data paradigm with compact Transformers. ArXiV:2104.05704 (2021)
Hatamizadeh, A., et al.: UNETR: transformers for 3D medical image segmentation. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 574–584 (2022)
Hu, H., et al.: Learning implicit feature alignment function for semantic segmentation. In: ECCV, pp. 487–505. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19818-2_28
Isensee, F., Jaeger, P., Kohl, S., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203–211 (2021)
Jha, D., et al.: A comprehensive study on colorectal polyp segmentation with ResUNet++, conditional random field and test-time augmentation. IEEE J. Biomed. Health Inform. 25(6), 2029–2040 (2021)
Ji, Y., et al.: AMOS: a large-scale abdominal multi-organ benchmark for versatile medical image segmentation. ArXiv:2206.08023 (2022)
Jiang, C., Sud, A., Makadia, A., Huang, J., Nießner, M., Funkhouser, T., et al.: Local implicit grid representations for 3D scenes. In: CVPR, pp. 6001–6010 (2020)
Khan, M., Fang, Y.: Implicit neural representations for medical imaging segmentation. In: MICCAI (2022)
Liu, S., Huang, D., et al.: Receptive field block net for accurate and fast object detection. In: ECCV, pp. 385–400 (2018)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2017)
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: CVPR, pp. 4455–4465 (2018)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24
Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., Geiger, A.: Texture fields: learning texture representations in function space. In: ICCV, pp. 4531–4540 (2019)
Park, J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: CVPR, pp. 165–174 (2019)
Pasupathy, A.: The neural basis of image segmentation in the primate brain. Neuroscience 296, 101–109 (2015)
Raju, A., Miao, S., Jin, D., Lu, L., Huang, J., Harrison, A.P.: Deep implicit statistical shape models for 3D medical image delineation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2135–2143 (2022)
Reich, C., Prangemeier, T., Cetin, O., Koeppl, H.: OSS-Net: memory efficient high resolution semantic segmentation of 3D medical data. In: British Machine Vision Conference (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shen, L., Pauly, J., Xing, L.: NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Trans. Neural Networks Learn. Syst. (2022)
Sørensen, K., Camara, O., Backer, O., Kofoed, K., Paulsen, R.: NUDF: neural unsigned distance fields for high resolution 3D medical image segmentation. ISBI, pp. 1–5 (2022)
Wiesner, D., Suk, J., Dummer, S., Svoboda, D., Wolterink, J.M.: Implicit neural representations for generative modeling of living cell shapes. In: MICCAI, pp. 58–67. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16440-8_6
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: ECCV, pp. 3–19 (2018)
Yang, J., Wickramasinghe, U., Ni, B., Fua, P.: ImplicitAtlas: learning deformable shape templates in medical imaging. In: CVPR, pp. 15861–15871 (2022)
Zhang, Y., Sapkota, N., Gu, P., Peng, Y., Zheng, H., Chen, D.Z.: Keep your friends close & enemies farther: Debiasing contrastive learning with spatial priors in 3D radiology images. In: 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1824–1829. IEEE (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Gu, P., Sapkota, N., Chen, D.Z. (2023). SwIPE: Efficient and Robust Medical Image Segmentation with Implicit Patch Embeddings. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14224. Springer, Cham. https://doi.org/10.1007/978-3-031-43904-9_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-43904-9_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43903-2
Online ISBN: 978-3-031-43904-9
eBook Packages: Computer ScienceComputer Science (R0)