Skip to main content

Joint Segmentation of Multiple Thoracic Organs in CT Images with Two Collaborative Deep Architectures

Part of the Lecture Notes in Computer Science book series (LNIP,volume 10553)


Computed Tomography (CT) is the standard imaging technique for radiotherapy planning. The delineation of Organs at Risk (OAR) in thoracic CT images is a necessary step before radiotherapy, for preventing irradiation of healthy organs. However, due to low contrast, multi-organ segmentation is a challenge. In this paper, we focus on developing a novel framework for automatic delineation of OARs. Different from previous works in OAR segmentation where each organ is segmented separately, we propose two collaborative deep architectures to jointly segment all organs, including esophagus, heart, aorta and trachea. Since most of the organ borders are ill-defined, we believe spatial relationships must be taken into account to overcome the lack of contrast. The aim of combining two networks is to learn anatomical constraints with the first network, which will be used in the second network, when each OAR is segmented in turn. Specifically, we use the first deep architecture, a deep SharpMask architecture, for providing an effective combination of low-level representations with deep high-level features, and then take into account the spatial relationships between organs by the use of Conditional Random Fields (CRF). Next, the second deep architecture is employed to refine the segmentation of each organ by using the maps obtained on the first deep architecture to learn anatomical constraints for guiding and refining the segmentations. Experimental results show superior performance on 30 CT scans, comparing with other state-of-the-art methods.


  • Anatomical constraints
  • CT segmentation
  • Fully Convolutional Networks (FCN)
  • CRF
  • CRFasRNN
  • Auto-context model

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-67558-9_3
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-67558-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. Chen, J., Yang, L., Zhang, Y., Alber, M., Chen, D.Z.: Combining fully convolutional and recurrent neural networks for 3d biomedical image segmentation. In: NIPS, pp. 3036–3044 (2016)

    Google Scholar 

  2. Dou, Q., Chen, H., Jin, Y., Yu, L., Qin, J., Heng, P.: 3d deeply supervised network for automatic liver segmentation from CT volumes. CoRR abs/1607.00582 (2016)

    Google Scholar 

  3. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS (2010)

    Google Scholar 

  4. Han, M., Ma, J., Li, Y., Li, M., Song, Y., Li, Q.: Segmentation of organs at risk in CT volumes of head, thorax, abdomen, and pelvis. In: Proceedings of SPIE, vol. 9413 (2015). Id: 94133J-6

    Google Scholar 

  5. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)

    Google Scholar 

  6. Milletari, F., Navab, N., Ahmadi, S.: V-net: fully convolutional neural networks for volumetric medical image segmentation. CoRR abs/1606.04797 (2016)

    Google Scholar 

  7. Nie, D., Wang, L., Gao, Y., Shen, D.: Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: ISBI, pp. 1342–1345 (2016)

    Google Scholar 

  8. Pinheiro, P.H.O., Lin, T., Collobert, R., Dollár, P.: Learning to refine object segments. CoRR abs/1603.08695 (2016)

    Google Scholar 

  9. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi:10.1007/978-3-319-24574-4_28

    CrossRef  Google Scholar 

  10. Ta, V.-T., Giraud, R., Collins, D.L., Coupé, P.: Optimized PatchMatch for near real time and accurate label fusion. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8675, pp. 105–112. Springer, Cham (2014). doi:10.1007/978-3-319-10443-0_14

    Google Scholar 

  11. Tong, T., et al.: Discriminative dictionary learning for abdominal multi-organ segmentation. Med. Image Anal. 23, 92–104 (2015)

    CrossRef  Google Scholar 

  12. Trullo, R., Petitjean, C., Ruan, S., Dubray, B., Nie, D., Shen, D.: Segmentation of organs at risk in thoracic CT images using a sharpmask architecture and conditional random fields. In: ISBI (2017)

    Google Scholar 

  13. Tu, Z., Bai, X.: Auto-context and its application to high-level vision tasks and 3d brain image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1744–1757 (2010)

    CrossRef  Google Scholar 

  14. Wang, L., et al.: Links: learning-based multi-source integration framework for segmentation of infant brain images. NeuroImage 108, 160–172 (2015)

    CrossRef  Google Scholar 

  15. Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: ICCV (2015)

    Google Scholar 

Download references


This work is co-financed by the European Union with the European regional development fund (ERDF, HN0002137) and by the Normandie Regional Council via the M2NUM project.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Roger Trullo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Trullo, R., Petitjean, C., Nie, D., Shen, D., Ruan, S. (2017). Joint Segmentation of Multiple Thoracic Organs in CT Images with Two Collaborative Deep Architectures. In: , et al. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support . DLMIA ML-CDS 2017 2017. Lecture Notes in Computer Science(), vol 10553. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67557-2

  • Online ISBN: 978-3-319-67558-9

  • eBook Packages: Computer ScienceComputer Science (R0)