Matwo-CapsNet: A Multi-label Semantic Segmentation Capsules Network

  • Savinien BonheurEmail author
  • Darko Štern
  • Christian Payer
  • Michael Pienn
  • Horst Olschewski
  • Martin Urschler
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


Despite some design limitations, CNNs have been largely adopted by the computer vision community due to their efficacy and versatility. Introduced by Sabour et al. to circumvent some limitations of CNNs, capsules replace scalars with vectors to encode appearance feature representation, allowing better preservation of spatial relationships between whole objects and its parts. They also introduced the dynamic routing mechanism, which allows to weight the contributions of parts to a whole object differently at each inference step. Recently, Hinton et al. have proposed to solely encode pose information to model such part-whole relationships. Additionally, they used a matrix instead of a vector encoding in the capsules framework. In this work, we introduce several improvements to the capsules framework, allowing it to be applied for multi-label semantic segmentation. More specifically, we combine pose and appearance information encoded as matrices into a new type of capsule, i.e. Matwo-Caps. Additionally, we propose a novel routing mechanism, i.e. Dual Routing, which effectively combines these two kinds of information. We evaluate our resulting Matwo-CapsNet on the JSRT chest X-ray dataset by comparing it to SegCaps, a capsule based network for binary segmentation, as well as to other CNN based state-of-the-art segmentation methods, where we show that our Matwo-CapsNet achieves competitive results, while requiring only a fraction of the parameters of other previously proposed methods.


Capsules network Convolutional neural network Chest X-ray Multi-label Semantic segmentation 


  1. 1.
    van Ginneken, B., Stegmann, M.B., Loog, M.: Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database. Med. Image Anal. 10(1), 19–40 (2006)CrossRefGoogle Scholar
  2. 2.
    Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 44–51. Springer, Heidelberg (2011). Scholar
  3. 3.
    Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: International Conference on Learning Representations (ICLR) (2018)Google Scholar
  4. 4.
    LaLonde, R., Bagci, U.: Capsules for Object Segmentation. In: International Conference on Medical Imaging with Deep Learning (MIDL) (2018)Google Scholar
  5. 5.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  6. 6.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar
  7. 7.
    Novikov, A.A., Lenis, D., Major, D., Hladuvka, J., Wimmer, M., Bühler, K.: Fully convolutional architectures for multiclass segmentation in chest radiographs. IEEE Trans. Med. Imaging 37(8), 1865–1876 (2018)CrossRefGoogle Scholar
  8. 8.
    Payer, C., Štern, D., Bischof, H., Urschler, M.: Multi-label whole heart segmentation using cnns and anatomical label configurations. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 190–198. Springer, Cham (2018). Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  10. 10.
    Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Neural Information Processing Systems (NIPS) (2017)Google Scholar
  11. 11.
    Shiraishi, J., et al.: Development of a digital image database for chest radiographs with and without a lung nodule. Am. J. Roentgenol. 174(1), 71–74 (2000)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Savinien Bonheur
    • 1
    Email author
  • Darko Štern
    • 2
    • 3
  • Christian Payer
    • 2
    • 3
  • Michael Pienn
    • 1
  • Horst Olschewski
    • 1
    • 4
  • Martin Urschler
    • 1
    • 5
  1. 1.Ludwig Boltzmann Institute for Lung Vascular ResearchGrazAustria
  2. 2.Ludwig Boltzmann Institute for Clinical Forensic ImagingGrazAustria
  3. 3.Institute of Computer Graphics and VisionGraz University of TechnologyGrazAustria
  4. 4.Department of Internal MedicineMedical University of GrazGrazAustria
  5. 5.School of Computer ScienceUniversity of AucklandAucklandNew Zealand

Personalised recommendations