Skip to main content

Transformer Based Multi-view Network for Mammographic Image Classification

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 (MICCAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13433))


Most of the existing multi-view mammographic image analysis methods adopt a simple fusion strategy: features concatenation, which is widely used in many features fusion methods. However, concatenation based methods can’t extract cross view information very effectively because different views are likely to be unaligned. Recently, many researchers have attempted to introduce attention mechanism related methods into the field of multi-view mammography analysis. But these attention mechanism based methods still partly rely on convolution, so they can’t take full advantages of attention mechanism. To take full advantage of multi-view information, we propose a novel pure transformer based multi-view network to solve the question of mammographic image classification. In our primary network, we use a transformer based backbone network to extract image features, a “cross view attention block” structure to fuse multi-view information, and a “classification token” to gather all useful information to make the final prediction. Besides, we compare the performance when fusing multi-view information at different stages of the backbone network using a novel designed “(shifted) window based cross view attention block” structure and compare the results when fusing different views’ information. The results on DDSM dataset show that our networks can effectively use multi-view information to make judgments and outperform the concatenation and convolution based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. Sun, L., Wang, J., Hu, Z., Xu, Y., Cui, Z.: Multi-view convolutional neural networks for mammographic image classification. IEEE Access 7, 126273–126282 (2019)

    Article  Google Scholar 

  2. Nasir Khan, H., Shahid, A.R., Raza, B., Dar, A.H., Alquhayz, H.: Multi-view feature fusion based four views model for mammogram classification using convolutional neural network. IEEE Access 7, 165724–165733 (2019)

    Article  Google Scholar 

  3. Wu, N., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39, 1184–1194 (2020)

    Article  Google Scholar 

  4. Li, C., et al.: Multi-view mammographic density classification by dilated and attention-guided residual learning. IEEE/ACM Trans. Comput. Biol. Bioinf. 18, 1003–1013 (2021)

    Article  Google Scholar 

  5. Zhao, X., Yu, L., Wang, X.: Cross-view attention network for breast cancer screening from multi-view mammograms. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020, pp. 1050–1054. IEEE, Barcelona (2020)

    Google Scholar 

  6. Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3588–3597. IEEE, Salt Lake City (2018)

    Google Scholar 

  7. Ma, J., Li, X., Li, H., Wang, R., Menze, B., Zheng, W.-S.: Cross-view relation networks for mammogram mass detection. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8632–8638. IEEE, Milan (2021)

    Google Scholar 

  8. Yang, Z., et al.: MommiNet-v2: mammographic multi-view mass identification networks. Med. Image Anal. 73, 102204 (2021)

    Article  Google Scholar 

  9. van Tulder, G., Tong, Y., Marchiori, E.: Multi-view analysis of unregistered medical images using cross-view transformers. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 104–113. Springer, Cham (2021).

    Chapter  Google Scholar 

  10. Liu, Y., et al.: Compare and contrast: detecting mammographic soft-tissue lesions with C2 -Net. Med. Image Anal. 71, 101999 (2021)

    Article  Google Scholar 

  11. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30, pp. 5998–6008 (2017)

    Google Scholar 

  12. Dosovitskiy, A., et al.: An image is worth 16 × 16 words: transformers for image recognition at scale. In: ICLR (2021)

    Google Scholar 

  13. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV (2021)

    Google Scholar 

  14. Tsai, Y.-H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., Salakhutdinov, R.: Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558–6569. Association for Computational Linguistics, Florence (2019)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas (2016)

    Google Scholar 

  16. Heath, M., Bowyer, K., Kopans, D., Kegelmeyer, P., Moore, R., Chang, K.: Current status of the digital database for screening mammography. In: Karssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (eds.) Digital Mammography. Computational Imaging and Vision, vol. 13, pp. 457–460. Springer, Dordrecht (1998).

  17. Yan, Y., Conze, P.-H., Lamard, M., Quellec, G., Cochener, B., Coatrieux, G.: Towards improved breast mass detection using dual-view mammogram matching. Med. Image Anal. 71, 102083 (2021)

    Article  Google Scholar 

Download references


This research is supported by Zhengzhou collaborative innovation major special project (20XTZX11020).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Huiqin Jiang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sun, Z., Jiang, H., Ma, L., Yu, Z., Xu, H. (2022). Transformer Based Multi-view Network for Mammographic Image Classification. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16436-1

  • Online ISBN: 978-3-031-16437-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics