Learning from Longitudinal Mammography Studies

  • Shaked PerekEmail author
  • Lior Ness
  • Mika Amit
  • Ella Barkan
  • Guy Amit
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11769)


When reading imaging studies, radiologists often compare the acquired images to one or more prior studies of the patient. Machine learning algorithms that assist in identifying abnormalities in medical images usually do not analyze prior images. This work describes a deep-learning classification framework for mammography studies, which incorporates prior image information using four approaches: (1) late fusion of prediction scores; (2) early fusion of input layers; (3) feature fusion combining a convolutional neural network (CNN) and gradient boosting trees; and (4) feature fusion using CNN and long-short term memory (LSTM) architecture. We demonstrate the advantages and limitations of each approach and compare their performance in identifying biopsy-proven malignancies in mammography screening studies. On an evaluation cohort of 439 patients, adding prior studies to the analysis improved the diagnostic performance of the classification framework. The CNN-LSTM architecture achieved the highest area under the ROC curve of 0.88, with sensitivity and specificity of 0.87 and 0.78, respectively. The methods that were trained using information from prior studies achieved better results than the baseline classifier, with up to 45% reduction in false-positive rate at the same sensitivity. The major advantage of the CNN-LSTM approach is in its flexibility and scalability; it allows to use the same network to classify sequences of multiple priors with variable length. The study demonstrates that longitudinal analysis of images can potentially improve the ability of machine learning algorithms to accurately and reliably interpret imaging studies, thus providing value to the radiology community.


Deep learning Longitudinal analysis Breast imaging 


  1. 1.
    Austin, C., Rijken, T., et al.: The role of deep learning in breast screening. Curr. Breast Cancer Rep. 11, 17–22 (2019)CrossRefGoogle Scholar
  2. 2.
    Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. J. Assoc. Phys. India 42(8), 785–794 (2016)Google Scholar
  3. 3.
    Geras, K.J., Wolfson, S., et al.: High-resolution breast cancer screening with multi-view deep convolutional neural networks, pp. 1–9 (2017)Google Scholar
  4. 4.
    Hayward, J.H., Ray, K.M., et al.: Improving screening mammography outcomes through comparison with multiple prior mammograms. AJR Am. J. Roentgenol. 207(4), 918–924 (2016)CrossRefGoogle Scholar
  5. 5.
    Hochreiter, S., Urgen Schmidhuber, J.: LTSM. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  6. 6.
    Kooi, T., Karssemeijer, N.: Classifying symmetrical differences and temporal change in mammography using deep neural networks, pp. 1–18 (2017)Google Scholar
  7. 7.
    Kyono, T., Gilbert, F.J., van der Schaar, M.: MAMMO: a deep learning solution for facilitating radiologist-machine collaboration in breast cancer diagnosis (2018)Google Scholar
  8. 8.
    Lehman, C.D., Arao, R.F., et al.: National performance benchmarks for modern screening digital mammography: update from the breast cancer surveillance consortium. Radiology 283(1), 49–58 (2017)CrossRefGoogle Scholar
  9. 9.
    McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157 (1947)CrossRefGoogle Scholar
  10. 10.
    Myers, E.R., Moorman, P., et al.: Benefits and harms of breast cancer screening. JAMA 314(15), 1615 (2015)CrossRefGoogle Scholar
  11. 11.
    Perek, S., Hazan, A., Barkan, E., Akselrod-Ballin, A.: Siamese network for dual-view mammography mass matching. In: Stoyanov, D., et al. (eds.) RAMBO/BIA/TIA -2018. LNCS, vol. 11040, pp. 55–63. Springer, Cham (2018). Scholar
  12. 12.
    Ribli, D., Horváth, A., et al.: Detecting and classifying lesions in mammograms with Deep Learning. Sci. Rep. 8(1), 4165 (2018)CrossRefGoogle Scholar
  13. 13.
    Roelofs, A.A.J., Karssemeijer, N., et al.: Importance of comparison of current and prior mammograms in breast cancer screening. Radiology 242(1), 70–77 (2007)CrossRefGoogle Scholar
  14. 14.
    Santeramo, R., Withey, S., Montana, G.: Longitudinal detection of radiological abnormalities with time-modulated LSTM. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 326–333. Springer, Cham (2018). Scholar
  15. 15.
    Sickles, E.A., d’Orsi, C.J., Bassett, L.W., et al.: ACR BI-RADS® mammography. In: ACR BI-RADS® Atlas, Breast Imaging Reporting and Data System. American College of Radiology, Reston (2013)Google Scholar
  16. 16.
    Szegedy, C., Ioffe, S., et al.: Inception-v4, Inception-ResNet and the impact of residual connections on learning (2016)Google Scholar
  17. 17.
    Wu, N., Phang, J., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Shaked Perek
    • 1
    Email author
  • Lior Ness
    • 1
  • Mika Amit
    • 1
  • Ella Barkan
    • 1
  • Guy Amit
    • 1
  1. 1.IBM ResearchHaifa UniversityHaifaIsrael

Personalised recommendations