Skip to main content

Image Segmentation in 3D Brachytherapy Using Convolutional LSTM



The accuracy of the segmentation of the target lesion and at-risk surrounding organs is important for cervical cancer patients treated with three-dimensional (3D) brachytherapy. However, the nature of brachytherapy, organ deformities, metal induced artifacts caused by applicators, and limited operating time make this process a challenge. Deep learning segmentation has recently emerged as an approach to this problem. The basic concept proposed in this paper is accurate segmentation of a pelvic computed tomography image dataset such that patients can obtain more accurate radiotherapy.


In this work, we propose a solution based on 3D U-Net and Long Short-Term Memory (LSTM). The model was trained and tested on a computed tomography pelvic image dataset comprising 51 patients who underwent 3D brachytherapy. The organs that required segmentation included the bladder, bowels, sigmoid colon, rectum, and uterus. We also used self-ensemble to improve segmentation accuracy. The Dice coefficient was used as the evaluation metric to determine the segmentation results for all of the organs under consideration.


The proposed model was evaluated using organ segmentation obtained from 10 patients with newly diagnosed stage I-IVA cervical cancer. The test results showed that segmentation conducted using the proposed model resulted in the following Dice coefficient values (mean, ± standard deviation): 87 (± 6.3) %, 72 (± 9.1) %, 72 (± 7.8) %, 95 (± 3.5) %, 86 (± 4.9) %, 77 (± 8.4) %, 73 (± 10.2) %, and 93 (± 3.5) % for the HRCTV, GTV, bowels, foley, bladder, rectum, sigmoid colon, and uterus, respectively.


This method shows that the combination of 3D U-Net and LSTM and self-ensemble post-processing has high potential for segmentation of a pelvic computed tomography dataset.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11


  1. 1.

    International Agency for Research on Cancer. (2012). Cervical cancer estimated incidence, mortality and prevalence worldwide in 2012. World Health Organization.

  2. 2.

    Eifel, P. J., Thoms, W. W., Jr., Smith, T. L., Morris, M., & Oswald, M. J. (1994). The relationship between brachytherapy dose and outcome in patients with bulky endocervical tumors treated with radiation alone. International Journal of Radiation Oncology Biology Physics, 28(1), 113–118.

    CAS  Article  Google Scholar 

  3. 3.

    Aghili, M., Andalib, B., Moghaddam, Z. K., Safaie, A. M., Hashemi, F. A., & Darzikolaie, N. M. (2018). Concurrent chemo-radiobrachytherapy with cisplatin and medium dose rate intra-cavitary brachytherapy for locally advanced uterine cervical cancer. Asian Pacific journal of cancer prevention: APJCP, 19(10), 2745.

  4. 4.

    Sorbe, B. G., Horvath, G., Andersson, H., Boman, K., Lundgren, C., & Pettersson, B. (2012). External pelvic and vaginal irradiation versus vaginal irradiation alone as postoperative therapy in medium-risk endometrial carcinoma: a prospective, randomized study—quality-of-life analysis. International Journal of Gynecologic Cancer, 22(7), 1281–1288.

    Article  Google Scholar 

  5. 5.

    Chargari, C., Deutsch, E., Blanchard, P., Gouy, S., Martelli, H., Guérin, F., & Haie-Meder, C. (2019). Brachytherapy: An overview for clinicians. CA: A Cancer Journal for Clinicians, 69(5), 386–401.

    Article  Google Scholar 

  6. 6.

    Derks, K., Steenhuijsen, J. L., van den Berg, H. A., Houterman, S., Cnossen, J., van Haaren, P., & De Jaeger, K. (2018). Impact of brachytherapy technique (2D versus 3D) on outcome following radiotherapy of cervical cancer. Journal of contemporary brachytherapy, 10(1), 17.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Tuntipumiamorn, L., Lohasammakul, S., Dankulchai, P., & Nakkrasae, P. (2018). Comparison of impact of target delineation of computed tomography-and magnetic resonance imaging-guided brachytherapy on dose distribution in cervical cancer. Journal of Contemporary Brachytherapy, 10(5), 418.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Harkenrider, M. M., Alite, F., Silva, S. R., & Small, W., Jr. (2015). Image-based brachytherapy for the treatment of cervical cancer. International Journal of Radiation Oncology Biology Physics, 92(4), 921–934.

    Article  Google Scholar 

  9. 9.

    Ohno, T., Noda, S. E., Okonogi, N., Murata, K., Shibuya, K., Kiyohara, H., & Wakatsuki, M. (2017). In-room computed tomography–based brachytherapy for uterine cervical cancer: results of a 5-year retrospective study. Journal of Radiation Research, 58(4), 543–551.

    Article  PubMed  Google Scholar 

  10. 10.

    Rijkmans, E. C., Nout, R. A., Rutten, I. H. H. M., Ketelaars, M., Neelis, K. J., Laman, M. S., & Creutzberg, C. L. (2014). Improved survival of patients with cervical cancer treated with image-guided brachytherapy compared with conventional brachytherapy. Gynecologic Oncology, 135(2), 231–238.

    CAS  Article  PubMed  Google Scholar 

  11. 11.

    Toita, T., Ohno, T., Ikushima, H., Nishimura, T., Uno, T., Ogawa, K., & Working Group of the Japanese Group of Brachytherapy/Japan Society for Radiation Oncology (JGB/JASTRO). (2018). National survey of intracavitary brachytherapy for intact uterine cervical cancer in Japan. Journal of Radiation Research, 59(4), 469–476.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Kumar, A., Kim, J., Lyndon, D., Fulham, M., & Feng, D. (2016). An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE Journal of Biomedical and Health Informatics, 21(1), 31–40.

    Article  PubMed  Google Scholar 

  13. 13.

    Anwar, S. M., Majid, M., Qayyum, A., Awais, M., Alnowami, M., & Khan, M. K. (2018). Medical image analysis using convolutional neural networks: a review. Journal of Medical Systems, 42(11), 226.

    Article  PubMed  Google Scholar 

  14. 14.

    Cardenas, C. E., Anderson, B. M., Aristophanous, M., Yang, J., Rhee, D. J., McCarroll, R. E., & Fuller, C. D. (2018). Auto-delineation of oropharyngeal clinical target volumes using 3D convolutional neural networks. Physics in Medicine & Biology, 63(21), 215026.

    Article  Google Scholar 

  15. 15.

    Ye, Y., Cai, Z., Huang, B., He, Y., Zeng, P., Zou, G., & Huang, B. (2020). Fully-Automated segmentation of nasopharyngeal carcinoma on dual-sequence MRI using convolutional neural networks. Frontiers in Oncology, 10, 166.

    Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Li, W. (2015). Automatic segmentation of liver tumor in CT images with deep convolutional neural networks. Journal of Computer and Communications, 3(11), 146.

    Article  Google Scholar 

  17. 17.

    Hesamian, M. H., Jia, W., He, X., & Kennedy, P. (2019). Deep learning techniques for medical image segmentation: Achievements and challenges. Journal of Digital Imaging, 32(4), 582–596.

    Article  PubMed  PubMed Central  Google Scholar 

  18. 18.

    Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241). Springer, Cham.

  19. 19.

    Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).

  20. 20.

    Chen, C., Liu, X., Ding, M., Zheng, J., & Li, J. (2019, October). 3D dilated multi-fiber network for real-time brain tumor segmentation in MRI. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 184–192). Springer, Cham.

  21. 21.

    Wong, K. C., & Moradi, M. (2019). SegNAS3D: Network Architecture Search with Derivative-Free Global Optimization for 3D Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 393–401). Springer, Cham.

  22. 22.

    Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., & Glocker, B. (2018). Attention u-net: Learning where to look for the pancreas. arXiv preprint. arXiv:1804.03999.

  23. 23.

    Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016, October). 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention (pp. 424–432). Springer, Cham.

  24. 24.

    Zhang, Z., Liu, Q., & Wang, Y. (2018). Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters, 15(5), 749–753.

    CAS  Article  Google Scholar 

  25. 25.

    Milletari, F., Navab, N., & Ahmadi, S. A. (2016, October). V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) (pp. 565–571). IEEE.

  26. 26.

    Huang, C., Han, H., Yao, Q., Zhu, S., & Zhou, S. K. (2019, October). 3D U2-Net: A 3D Universal U-Net for Multi-domain Medical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 291–299). Springer, Cham.

  27. 27.

    Xingjian, S. H. I., Chen, Z., Wang, H., Yeung, D. Y., Wong, W. K., & Woo, W. C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems (pp. 802–810).

  28. 28.

    Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint. arXiv:1502.03167.

  29. 29.

    Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980–2988).

  30. 30.

    Berman, M., Rannen Triki, A., & Blaschko, M. B. (2018). The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4413–4421).

  31. 31.

    Timofte, R., Rothe, R., & Van Gool, L. (2016). Seven ways to improve example-based single image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1865–1873).

Download references


We thank the anonymous reviewers for their constructive comments. This research work was supported in part by the Ministry of Science and Technology of Taiwan, ROC. (MOST 109-2634-F-006-014 and MOST 108-2221-E-006 -173 -MY2)

Author information



Corresponding author

Correspondence to Pau-Choo Chung.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chang, JH., Lin, KH., Wang, TH. et al. Image Segmentation in 3D Brachytherapy Using Convolutional LSTM. J. Med. Biol. Eng. 41, 636–651 (2021).

Download citation


  • Cervical cancer
  • Brachytherapy
  • Pelvic computed tomography image
  • 3D U-net
  • LSTM
  • Segmentation
  • Self-ensemble