Skip to main content

Select, Attend, and Transfer: Light, Learnable Skip Connections

Part of the Lecture Notes in Computer Science book series (LNIP,volume 11861)

Abstract

Skip connections in deep networks have improved both segmentation and classification performance by facilitating the training of deeper network architectures and reducing the risks for vanishing gradients. The skip connections equip encoder-decoder like networks with richer feature representations, but at the cost of higher memory usage, computation, and possibly resulting in transferring non-discriminative feature maps. In this paper, we focus on improving the skip connections used in segmentation networks. We propose light, learnable skip connections which learn to first select the most discriminative channels, and then aggregate the selected ones as single channel attending to the most discriminative regions of input. We evaluate the proposed method on 3 different 2D and volumetric datasets and demonstrate that the proposed skip connections can outperform the traditional heavy skip connections of 4 different models in terms of segmentation accuracy (2% Dice), memory usage (at least 50%), and the number of network parameters (up to 70%).

Keywords

  • Deep neural networks
  • Skip connections
  • Image segmentation

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-32692-0_48
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-32692-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.

References

  1. Chu, X., et al.: Multi-context attention for human pose estimation. arXiv preprint arXiv:1702.074321(2) (2017)

  2. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_49

    CrossRef  Google Scholar 

  3. Codella, N.C., et al.: Skin lesion analysis toward melanoma detection: ISBI, hosted by ISIC. arXiv preprint arXiv:1710.05006 (2017)

  4. Fedorov, A., et al.: DICOM for quantitative imaging biomarker development: a standards based approach to sharing clinical data and structured PET/CT analysis results in head and neck cancer research. PeerJ 4, e2057 (2016). https://doi.org/10.7717/peerj.2057

    CrossRef  Google Scholar 

  5. Geert, L., et al.: Prostatex challenge data. The cancer imaging archive (2017)

    Google Scholar 

  6. Han, S., et al.: EIE: efficient inference engine on compressed deep neural network. In: ISCA, ACM/IEEE, pp. 243–254. IEEE (2016)

    Google Scholar 

  7. He, K., et al.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  8. He, Y., et al.: Channel pruning for accelerating very deep neural networks. In: ICCV, vol. 2, p. 6 (2017)

    Google Scholar 

  9. Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  10. Hu, J., et al.: Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507 (2017)

  11. Huang, G., et al.: Densely connected convolutional networks. In: CVPR (2017)

    Google Scholar 

  12. Jégou, S., et al.: The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: CVPRW (2017)

    Google Scholar 

  13. Kim, Y.D.: Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 (2015)

  14. Larsson, G., et al.: Fractalnet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)

  15. Lee, C.Y., et al.: Deeply-supervised nets. In: AIS, pp. 562–570 (2015)

    Google Scholar 

  16. Leroux, S., et al.: IamNN: iterative and adaptive mobile neural network for efficient image classification. arXiv preprint arXiv:1804.10123 (2018)

  17. Lin, M., et al.: Network in network. arXiv preprint arXiv:1312.4400 (2013)

  18. Milletari, F., et al.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 3DV, 2016, pp. 565–571. IEEE (2016)

    Google Scholar 

  19. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    CrossRef  Google Scholar 

  20. Springenberg, J.T., et al.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

  21. Srivastava, R.K., et al.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)

  22. Wang, F., et al.: Residual attention network for image classification. arXiv preprint arXiv:1704.06904 (2017)

  23. Wen, W., et al.: Learning structured sparsity in deep neural networks. In: NIPS, pp. 2074–2082 (2016)

    Google Scholar 

  24. Yang, D., et al.: Automatic liver segmentation using an adversarial image-to-image network. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 507–515. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_58

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Saeid Asgari Taghanaki .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Taghanaki, S.A. et al. (2019). Select, Attend, and Transfer: Light, Learnable Skip Connections. In: Suk, HI., Liu, M., Yan, P., Lian, C. (eds) Machine Learning in Medical Imaging. MLMI 2019. Lecture Notes in Computer Science(), vol 11861. Springer, Cham. https://doi.org/10.1007/978-3-030-32692-0_48

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32692-0_48

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32691-3

  • Online ISBN: 978-3-030-32692-0

  • eBook Packages: Computer ScienceComputer Science (R0)