Skip to main content

Accurate Esophageal Gross Tumor Volume Segmentation in PET/CT Using Two-Stream Chained 3D Deep Network Fusion

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 (MICCAI 2019)


Gross tumor volume (GTV) segmentation is a critical step in esophageal cancer radiotherapy treatment planning. Inconsistencies across oncologists and prohibitive labor costs motivate automated approaches for this task. However, leading approaches are only applied to radiotherapy computed tomography (RTCT) images taken prior to treatment. This limits the performance as RTCT suffers from low contrast between the esophagus, tumor, and surrounding tissues. In this paper, we aim to exploit both RTCT and positron emission tomography (PET) imaging modalities to facilitate more accurate GTV segmentation. By utilizing PET, we emulate medical professionals who frequently delineate GTV boundaries through observation of the RTCT images obtained after prescribing radiotherapy and PET/CT images acquired earlier for cancer staging. To take advantage of both modalities, we present a two-stream chained segmentation approach that effectively fuses the CT and PET modalities via early and late 3D deep-network-based fusion. Furthermore, to effect the fusion and segmentation we propose a simple yet effective progressive semantically nested network (PSNN) model that outperforms more complicated models. Extensive 5-fold cross-validation on 110 esophageal cancer patients, the largest analysis to date, demonstrates that both the proposed two-stream chained segmentation pipeline and the PSNN model can significantly improve the quantitative performance over the previous state-of-the-art work by \(11\%\) in absolute Dice score (DSC) (from \(0.654\, \pm \, 0.210\) to \(0.764\, \pm \, 0.134\)) and, at the same time, reducing the Hausdorff distance from \(129\pm 73\) mm to \(47 \pm 56\) mm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. Bray, F., Ferlay, J., et al.: Global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer J. Clin. 68(6), 394–424 (2018)

    Google Scholar 

  2. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016).

    Chapter  Google Scholar 

  3. Harrison, A.P., Xu, Z., George, K., Lu, L., Summers, R.M., Mollura, D.J.: Progressive and multi-path holistically nested neural networks for pathological lung segmentation from CT images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 621–629. Springer, Cham (2017).

    Chapter  Google Scholar 

  4. Holger, R., Lu, L., Lay, N., et al.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Med. Image Anal. 45, 94–107 (2018)

    Article  Google Scholar 

  5. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE CVPR, pp. 2261–2269 (2017)

    Google Scholar 

  6. Iyer, R., Dubrow, R.: Imaging of esophageal cancer. Cancer Imaging 4(2), 125 (2004)

    Article  Google Scholar 

  7. Jin, D., Guo, D., Ho, T.Y., et al.: Deep esophageal clinical target volume delineation using encoded 3D spatial context of tumor, lymph nodes, and organs at risk. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. xx–yy. Springer, Heidelberg (2019)

    Google Scholar 

  8. Jin, D., Xu, Z., Harrison, A.P., George, K., Mollura, D.J.: 3D convolutional neural networks with graph refinement for airway segmentation using incomplete data labels. In: Wang, Q., Shi, Y., Suk, H.-I., Suzuki, K. (eds.) MLMI 2017. LNCS, vol. 10541, pp. 141–149. Springer, Cham (2017).

    Chapter  Google Scholar 

  9. Jin, D., Xu, Z., Tang, Y., Harrison, A.P., Mollura, D.J.: CT-realistic lung nodule simulation from 3D conditional generative adversarial networks for robust lung segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 732–740. Springer, Cham (2018).

    Chapter  Google Scholar 

  10. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  11. Leong, T., Everitt, C., et al.: A prospective study to evaluate the impact of FDG-PET on CT-based radiotherapy treatment planning for oesophageal cancer. Radiother. Oncol. 78(3), 254–261 (2006)

    Article  Google Scholar 

  12. Muijs, C., Schreurs, L., et al.: Consequences of additional use of pet information for target volume delineation and radiotherapy dose distribution for esophageal cancer. Radiother. Oncol. 93(3), 447–453 (2009)

    Article  Google Scholar 

  13. Rueckert, D., Sonoda, L.I., Hayes, C.J., et al.: Nonrigid registration using free-form deformations: application to breast MR images. IEEE TMI 18(8), 712–721 (1999)

    Google Scholar 

  14. Tai, P., Van Dyk, J., Yu, E., et al.: Variability of target volume delineation in cervical esophageal cancer. Int. J. Radiat. Oncol. Biol. Phys. 42(2), 277–288 (1998)

    Article  Google Scholar 

  15. Yousefi, S., et al.: Esophageal gross tumor volume segmentation using a 3D convolutional neural network. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 343–351. Springer, Cham (2018).

    Chapter  Google Scholar 

  16. Hao, Z., Liu, J., Liu, J.: Esophagus tumor segmentation using fully convolutional neural network and graph cut. In: Jia, Y., Du, J., Zhang, W. (eds.) CISC 2017. LNEE, vol. 460, pp. 413–420. Springer, Singapore (2018).

    Chapter  Google Scholar 

  17. Zhao, X., Li, L., et al.: Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys. Med. Biol. 64(1), 015011 (2019)

    Article  Google Scholar 

  18. Zhong, Z., Kim, Y., et al.: Simultaneous cosegmentation of tumors in PET-CT images using deep fully convolutional networks. Med. Phys. 46(2), 619–633 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Dakai Jin or Tsung-Ying Ho .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jin, D. et al. (2019). Accurate Esophageal Gross Tumor Volume Segmentation in PET/CT Using Two-Stream Chained 3D Deep Network Fusion. In: Shen, D., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science(), vol 11765. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32244-1

  • Online ISBN: 978-3-030-32245-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics