Abstract
Deep learning approaches are state-of-the-art for semantic segmentation of medical images, but unlike many deep learning applications, medical segmentation is characterized by small amounts of annotated training data. Thus, while mainstream deep learning approaches focus on performance in domains with large training sets, researchers in the medical imaging field must apply new methods in creative ways to meet the more constrained requirements of medical datasets. We propose a framework for incrementally fine-tuning a multi-class segmentation of a high-resolution multiplex (multi-channel) immuno-flourescence image of a rat brain section, using a minimal amount of labelling from a human expert. Our framework begins with a modified Swin-UNet architecture that treats each biomarker in the multiplex image separately and learns an initial “global” segmentation (pre-training). This is followed by incremental learning and refinement of each class using a very limited amount of additional labeled data provided by a human expert for each region and its surroundings. This incremental learning utilizes the multi-class weights as an initialization and uses the additional labels to steer the network and optimize it for each region in the image. In this way, an expert can identify errors in the multi-class segmentation and rapidly correct them by supplying the model with additional annotations hand-picked from the region. In addition to increasing the speed of annotation and reducing the amount of labelling, we show that our proposed method outperforms a traditional multi-class segmentation by a large margin.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Dataset was collected following the procedure described in this paper: https://www.nature.com/articles/s41467-021-21735-x The dataset is available at the following link: https://figshare.com/articles/dataset/Whole-brain_tissue_mapping_toolkit_using_large-scale_highly_multiplexed_immunofluorescence_imaging_and_deep_neural_networks_Data_/13731585/1.
References
Azad, R., Aghdam, E. K., Rauland, A., Jia, Y., Avval, A. H., Bozorgpour, A., Karimijafarbigloo, S., Cohen, J. P., Adeli, E., & Merhof, D. (2022). Medical image segmentation review: the success of U-Net. arXiv. arXiv:2211.14830 [cs, eess]. http://arxiv.org/abs/2211.14830. Accessed September 6, 2023.
Beucher, S. (1992). The watershed transformation applied to image segmentation.
Caicedo, J. C., Goodman, A., Karhohs, K. W., Cimini, B. A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., Rohban, M., Singh, S., & Carpenter, A. E. (2019). Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature Methods, 16(12), 1247–1253. https://doi.org/10.1038/s41592-019-0612-7. Number: 12 Publisher: Nature Publishing Group. Accessed September 28, 2023.
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023). Swin-Unet: Unet-like pure transformer for medical image segmentation. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds.) Computer Vision – ECCV 2022 Workshops. Lecture Notes in Computer Science, pp. 205–218. Springer, Cham. https://doi.org/10.1007/978-3-031-25066-8_9
Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). GCNet: Non-local networks meet squeeze-excitation networks and beyond. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1971–1980. https://doi.org/10.1109/ICCVW.2019.00246. ISSN: 2473-9944. Accessed September 30, 2023, from https://ieeexplore.ieee.org/document/9022134
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021). TransUNet: Transformers make strong encoders for medical image segmentation. arXiv. arXiv:2102.04306 [cs]. http://arxiv.org/abs/2102.04306. Accessed September 6, 2023.
Cherukuri, V., Ssenyonga, P., Warf, B. C., Kulkarni, A. V., Monga, V., & Schiff, S. J. (2018). Learning based segmentation of CT brain images: Application to postoperative hydrocephalic scans. IEEE Transactions on Biomedical Engineering, 65(8), 1871–1884. https://doi.org/10.1109/TBME.2017.2783305. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 16, 2023.
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2023). Conditional positional encodings for vision transformers. arXiv. arXiv:2102.10882 [cs]. https://doi.org/10.48550/arXiv.2102.10882. http://arxiv.org/abs/2102.10882. Accessed September 30, 2023.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. Accessed September 29, 2023, from https://openreview.net/forum?id=YicbFdNTTy
Gao, Z.-J., He, Y., & Li, Y. (2023). A novel lightweight Swin-Unet network for semantic segmentation of COVID-19 lesion in CT images. IEEE Access, 11, 950–962. https://doi.org/10.1109/ACCESS.2022.3232721. Conference Name: IEEE Access.
Hu, H., Zheng, Y., Zhou, Q., Xiao, J., Chen, S., & Guan, Q. (2019). MC-Unet: Multi-scale convolution Unet for bladder cancer cell segmentation in phase-contrast microscopy images. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1197–1199. https://doi.org/10.1109/BIBM47256.2019.8983121. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/8983121
Im, K., Mareninov, S., Diaz, M. F. P., & Yong, W. H. (2019). An introduction to performing immunofluorescence staining. In: Yong, W.H. (ed.) Biobanking: Methods and Protocols. Methods in Molecular Biology, pp. 299–311. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-8935-5_26. Accessed September 29, 2023.
Jia, X., Sayed, S. B., Hasan, N. I., Gomez, L. J., Huang, G.-B., & Yucel, A. C. (2023). DeeptDCS: Deep learning-based estimation of currents induced during transcranial direct current stimulation. IEEE Transactions on Biomedical Engineering, 70(4), 1231–1241. https://doi.org/10.1109/TBME.2022.3213266. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 11, 2023.
Kromp, F., Fischer, L., Bozsaky, E., Ambros, I. M., Dorr, W., Beiske, K., Ambros, P. F., Hanbury, A., & Taschner-Mandl, S. (2021). Evaluation of deep learning architectures for complex immunofluorescence nuclear image segmentation. IEEE Transactions on Medical Imaging, 40(7), 1934–1949. https://doi.org/10.1109/TMI.2021.3069558. Accessed September 26, 2023.
Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.-W., & Heng, P.-A. (2018). H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Transactions on Medical Imaging, 37(12), 2663–2674. https://doi.org/10.1109/TMI.2018.2845918. Accessed September 29, 2023.
Li, C., Xi, Z., Jin, G., Jiang, W., Wang, B., Cai, X., & Wang, X. (2023). Deep-learning-enabled microwave-induced thermoacoustic tomography based on ResAttU-Net for transcranial brain hemorrhage detection. IEEE Transactions on Biomedical Engineering, 70(8), 2350–2361. https://doi.org/10.1109/TBME.2023.3243491. Conference Name: IEEE Transactions on Biomedical Engineering, Accessed November 16, 2023.
Lin, D., Cheng, Y., Li, Y., Prasad, S., & Guo, A. (2022). MLSA-UNet: End-to-End multi-level spatial attention guided UNet for industrial defect segmentation. In: 2022 IEEE International Conference on Image Processing (ICIP), pp. 441–445. https://doi.org/10.1109/ICIP46576.2022.9897416. ISSN: 2381-8549. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9897416
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows, pp. 9992–10002. IEEE Computer Society. https://doi.org/10.1109/ICCV48922.2021.00986. Accessed September 29, 2023, from https://www.computer.org/csdl/proceedings-article/iccv/2021/281200j992/1BmGKZoEzug
Mandal, D., Vahadane, A., Sharma, S., & Majumdar, S. (2021). Blur-robust nuclei segmentation for immunofluorescence images. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3475–3478 (2021). https://doi.org/10.1109/EMBC46164.2021.9629787. ISSN: 2694-0604. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9629787
Maric, D., Jahanipour, J., Li, X. R., Singh, A., Mobiny, A., Van Nguyen, H., Sedlock, A., Grama, K., & Roysam, B. (2021). Whole-brain tissue mapping toolkit using large-scale highly multiplexed immunofluorescence imaging and deep neural networks. Nature Communications, 12, 1550 (2021). https://doi.org/10.1038/s41467-021-21735-x. Accessed September 28, 2023.
napari. (2023). A fast, interactive viewer for multi-dimensional images in Python napari. Accessed September 30, 2023, from https://napari.org/stable/
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., Glocker, B., & Rueckert, D. (2022). Attention U-Net: learning where to look for the pancreas. Accessed September 29, 2023, from https://openreview.net/forum?id=Skft7cijM
Paxinos, G., & Watson, C. (2013). The rat brain in stereotaxic coordinates. Academic Press. Google-Books-ID: RiHLCQAAQBAJ.
Phellan, R., Lindner, T., Helle, M., Falco, A. X., & Forkert, N. D. (2018). Automatic temporal segmentation of vessels of the brain using 4D ASL MRA Images. IEEE Transactions on Biomedical Engineering, 65(7), 1486–1494. https://doi.org/10.1109/TBME.2017.2759730. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 16, 2023.
Rahimpour, M., Bertels, J., Radwan, A., Vandermeulen, H., Sunaert, S., Vandermeulen, D., Maes, F., Goffin, K., & Koole, M. (2022). Cross-modal distillation to improve MRI-based brain tumor segmentation with missing MRI sequences. IEEE Transactions on Biomedical Engineering, 69(7), 2153–2164. https://doi.org/10.1109/TBME.2021.3137561. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 16, 2023.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, pp. 234–241. Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28
Stringer, C., Wang, T., Michaelos, M., & Pachitariu, M. (2021). Cellpose: a generalist algorithm for cellular segmentation. Nature Methods, 18(1), 100–106. https://doi.org/10.1038/s41592-020-01018-x. Number: 1 Publisher: Nature Publishing Group. Accessed September 28, 2023.
Tang, X., Wang, X., Yan, N., Fu, S., Xiong, W., & Liao, Q. (2022). A new ore image segmentation method based on Swin-Unet. In: 2022 China Automation Congress (CAC), pp. 1681–1686. https://doi.org/10.1109/CAC57257.2022.10055952. ISSN: 2688-0938. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/10055952
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jegou, H. (2021). Training data-efficient image transformers & distillation through attention. In: Proceedings of the 38th International Conference on Machine Learning, pp. 10347–10357. PMLR. ISSN: 2640-3498. Accessed September 30, 2023, from https://proceedings.mlr.press/v139/touvron21a.html
Valverde, J. M., Shatillo, A., De Feo, R., & Tohka, J. (2023). Automatic cerebral hemisphere segmentation in Rat MRI with ischemic lesions via attention-based convolutional neural networks. Neuroinformatics, 21(1), 57–70. https://doi.org/10.1007/s12021-022-09607-1. Accessed December 19, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, U., & Polosukhin, I. (2017). Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. Accessed September 29, 2023, from https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local Neural Networks. arXiv. arXiv:1711.07971 [cs]. https://doi.org/10.48550/arXiv.1711.07971. http://arxiv.org/abs/1711.07971. Accessed September 30, 2023.
Wang, Y., Gu, L., Jiang, T., & Gao, F. (2023). MDE-UNet: A multitask deformable UNet combined enhancement network for farmland boundary segmentation. IEEE Geoscience and Remote Sensing Letters, 20, 1–5. https://doi.org/10.1109/LGRS.2023.3252048. Conference Name: IEEE Geoscience and Remote Sensing Letters. Accessed September 29, 2023.
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 548–558. https://doi.org/10.1109/ICCV48922.2021.00061. ISSN: 2380-7504. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9711179
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022). PVT v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3), 415–424. https://doi.org/10.1007/s41095-022-0274-8. arXiv:2106.13797 [cs]. Accessed January 1, 2024.
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018). Unified perceptual parsing for scene understanding. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. Lecture Notes in Computer Science, pp. 432–448. Springer, Cham. https://doi.org/10.1007/978-3-030-01228-1_26
Yin, M., Yao, Z., Cao, Y., Li, X., Zhang, Z., Lin, S., & Hu, H. (2020). Disentangled non-local neural networks. In: Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV, pp. 191–207. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-030-58555-6_12. Accessed September 29, 2023.
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F.E., Feng, J., & Yan, S. (2021). Tokens-to-Token ViT: Training vision transformers from scratch on ImageNet. arXiv. arXiv:2101.11986 [cs]. https://doi.org/10.48550/arXiv.2101.11986. http://arxiv.org/abs/2101.11986. Accessed September 30, 2023.
Zhang, Y., Liu, H., & Hu, Q. (2021). TransFuse: Fusing transformers and CNNs for medical image segmentation. In: Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. Lecture Notes in Computer Science, pp. 14–24. Springer, Cham. https://doi.org/10.1007/978-3-030-87193-2_2
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., & Zhang, L. (2021). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6877–6886. https://doi.org/10.1109/CVPR46437.2021.00681. ISSN: 2575-7075. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9578646
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018). UNet++: A nested U-Net architecture for medical image segmentation. In: Stoyanov, D., Taylor, Z., Carneiro, G., Syeda-Mahmood, T., Martel, A., Maier-Hein, L., Tavares, J.M.R.S., Bradley, A., Papa, J.P., Belagiannis, V., Nascimento, J.C., Lu, Z., Conjeti, S., Moradi, M., Greenspan, H., Madabhushi, A. (eds.) Deep learning in medical image analysis and multimodal learning for clinical decision support. Lecture Notes in Computer Science, pp. 3–11. Springer, Cham. https://doi.org/10.1007/978-3-030-00889-5_1
Funding
National Institutes of Health, Grant No. 5R01NS109118.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception. R.F. conducted the experiments, generated the results and wrote the manuscript. S.P. supervised the research. B.R. and D.M. provided context on the neuroscience tasks. All authors read and approved the final version of the manuscript.
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Faulkenberry, R., Prasad, S., Maric, D. et al. Visual Prompting Based Incremental Learning for Semantic Segmentation of Multiplex Immuno-Flourescence Microscopy Imagery. Neuroinform 22, 147–162 (2024). https://doi.org/10.1007/s12021-024-09651-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-024-09651-z