Assessment of knee pain from MR imaging using a convolutional Siamese network

A Correction to this article is available

This article has been updated



It remains difficult to characterize the source of pain in knee joints either using radiographs or magnetic resonance imaging (MRI). We sought to determine if advanced machine learning methods such as deep neural networks could distinguish knees with pain from those without it and identify the structural features that are associated with knee pain.


We constructed a convolutional Siamese network to associate MRI scans obtained on subjects from the Osteoarthritis Initiative (OAI) with frequent unilateral knee pain comparing the knee with frequent pain to the contralateral knee without pain. The Siamese network architecture enabled pairwise learning of information from two-dimensional (2D) sagittal intermediate-weighted turbo spin echo slices obtained from similar locations on both knees. Class activation mapping (CAM) was utilized to create saliency maps, which highlighted the regions most associated with knee pain. The MRI scans and the CAMs of each subject were reviewed by an expert radiologist to identify the presence of abnormalities within the model-predicted regions of high association.


Using 10-fold cross-validation, our model achieved an area under curve (AUC) value of 0.808. When individuals whose knee WOMAC pain scores were not discordant were excluded, model performance increased to 0.853. The radiologist review revealed that about 86% of the cases that were predicted correctly had effusion-synovitis within the regions that were most associated with pain.


This study demonstrates a proof of principle that deep learning can be applied to assess knee pain from MRI scans.

Key Points

• Our article is the first to leverage a deep learning framework to associate MR images of the knee with knee pain.

• We developed a convolutional Siamese network that had the ability to fuse information from multiple two-dimensional (2D) MRI slices from the knee with pain and the contralateral knee of the same individual without pain to predict unilateral knee pain.

• Our model achieved an area under curve (AUC) value of 0.808. When individuals who had WOMAC pain scores that were not discordant for knees (pain discordance < 3) were excluded, model performance increased to 0.853.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Change history

  • 22 July 2020

    The original version of this article, published on 13 February 2020, unfortunately contained a mistake.



Area under curve


Body mass index


Bone marrow lesion


Class activation mapping


Convolutional neural network


Graphical processing unit




Machine learning


MRI Osteoarthritis Knee Score


Magnetic resonance imaging




Osteoarthritis initiative


Receiver operating characteristic


Western Ontario and McMaster Universities Osteoarthritis Index


  1. 1.

    Cross M, Smith E, Hoy D et al (2014) The global burden of hip and knee osteoarthritis: estimates from the global burden of disease 2010 study. Ann Rheum Dis 73:1323–1330

    Article  Google Scholar 

  2. 2.

    Neogi T (2013) The epidemiology and impact of pain in osteoarthritis. Osteoarthritis Cartilage 21:1145–1153

  3. 3.

    Hunter DJ, Guermazi A, Roemer F, Zhang Y, Neogi T (2013) Structural correlates of pain in joints with osteoarthritis. Osteoarthritis Cartilage 21:1170–1178

  4. 4.

    Felson DT, Chaisson CE, Hill CL et al (2001) The association of bone marrow lesions with pain in knee osteoarthritis. Ann Intern Med 134:541–549

    CAS  Article  Google Scholar 

  5. 5.

    Felson DT, Niu J, Guermazi A et al (2007) Correlation of the development of knee pain with enlarging bone marrow lesions on magnetic resonance imaging. Arthritis Rheum 56:2986–2992

    Article  Google Scholar 

  6. 6.

    Hill CL, Gale DG, Chaisson CE et al (2001) Knee effusions, popliteal cysts, and synovial thickening: association with knee pain in osteoarthritis. J Rheumatol 28:1330–1337

    CAS  PubMed  Google Scholar 

  7. 7.

    Hill CL, Hunter DJ, Niu J et al (2007) Synovitis detected on magnetic resonance imaging and its relation to pain and cartilage loss in knee osteoarthritis. Ann Rheum Dis 66:1599–1603

    Article  Google Scholar 

  8. 8.

    Dimitroulas T, Duarte RV, Behura A, Kitas GD, Raphael JH (2014) Neuropathic pain in osteoarthritis: a review of pathophysiological mechanisms and implications for treatment. Semin Arthritis Rheum 44:145–154

    Article  Google Scholar 

  9. 9.

    Finan PH, Buenaver LF, Bounds SC et al (2013) Discordance between pain and radiographic severity in knee osteoarthritis: findings from quantitative sensory testing of central sensitization. Arthritis Rheum 65:363–372

    Article  Google Scholar 

  10. 10.

    Clauw DJ, Witter J (2009) Pain and rheumatology: thinking outside the joint. Arthritis Rheum 60:321–324

    Article  Google Scholar 

  11. 11.

    O'Neill TW, Felson DT (2018) Mechanisms of osteoarthritis (OA) pain. Curr Osteoporos Rep 16:611–616

    Article  Google Scholar 

  12. 12.

    Bedson J, Croft PR (2008) The discordance between clinical and radiographic knee osteoarthritis: a systematic search and summary of the literature. BMC Musculoskelet Disord 9:116

    Article  Google Scholar 

  13. 13.

    Guermazi A, Zaim S, Taouli B, Miaux Y, Peterfy CG, Genant HG (2003) MR findings in knee osteoarthritis. Eur Radiol 13:1370–1386

    Article  Google Scholar 

  14. 14.

    Sowers MF, Hayes C, Jamadar D et al (2003) Magnetic resonance-detected subchondral bone marrow and cartilage defect characteristics associated with pain and X-ray-defined knee osteoarthritis. Osteoarthritis Cartilage 11:387–393

  15. 15.

    Yusuf E, Kortekaas MC, Watt I, Huizinga TW, Kloppenburg M (2011) Do knee abnormalities visualised on MRI explain knee pain in knee osteoarthritis? A systematic review. Ann Rheum Dis 70:60–67

    Article  Google Scholar 

  16. 16.

    Wenham CY, Conaghan PG (2009) Imaging the painful osteoarthritic knee joint: what have we learned? Nat Clin Pract Rheumatol 5:149–158

    PubMed  Google Scholar 

  17. 17.

    Litjens G, Kooi T, Bejnordi BE et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  18. 18.

    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    CAS  Article  Google Scholar 

  19. 19.

    Hamm CA, Wang CJ, Savic LJ et al (2019) Deep learning for liver tumor diagnosis part I: development of a convolutional neural network classifier for multi-phasic MRI. Eur Radiol 29:3338–3347

    Article  Google Scholar 

  20. 20.

    Kiryu S, Yasaka K, Akai H et al (2019) Deep learning to differentiate parkinsonian disorders separately using single midsagittal MR imaging: a proof of concept study. Eur Radiol 29:6891–6899

    Article  Google Scholar 

  21. 21.

    Laukamp KR, Thiele F, Shakirin G et al (2019) Fully automated detection and segmentation of meningiomas using deep learning on routine multiparametric MRI. Eur Radiol 29:124–132

    Article  Google Scholar 

  22. 22.

    Vreemann S, Dalmis MU, Bult P et al (2019) Amount of fibroglandular tissue FGT and background parenchymal enhancement BPE in relation to breast cancer risk and false positives in a breast MRI screening program : a retrospective cohort study. Eur Radiol 29:4678–4690

    Article  Google Scholar 

  23. 23.

    Wang CJ, Hamm CA, Savic LJ et al (2019) Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features. Eur Radiol 29:3348–3357

    Article  Google Scholar 

  24. 24.

    Norman B, Pedoia V, Majumdar S (2018) Use of 2D U-net convolutional neural networks for automated cartilage and meniscus segmentation of knee MR imaging data to determine Relaxometry and Morphometry. Radiology 288:177–185

    Article  Google Scholar 

  25. 25.

    Ambellan F, Tack A, Ehlke M, Zachow S (2019) Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the osteoarthritis initiative. Med Image Anal 52:109–118

    Article  Google Scholar 

  26. 26.

    Gaj S, Yang M, Nakamura K, Li X (2019) Automated cartilage and meniscus segmentation of knee MRI with conditional generative adversarial networks. Magn Reson Med.

  27. 27.

    Byra M, Wu M, Zhang X et al (2020) Knee menisci segmentation and relaxometry of 3D ultrashort echo time cones MR imaging using attention U-net with transfer learning. Magn Reson Med 83:1109–1122

    Article  Google Scholar 

  28. 28.

    Tack A, Mukhopadhyay A, Zachow S (2018) Knee menisci segmentation using convolutional neural networks: data from the osteoarthritis initiative. Osteoarthritis Cartilage 26:680–688

  29. 29.

    Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med Image Comput Comput Assist Interv 16:246–253

    PubMed  Google Scholar 

  30. 30.

    Bien N, Rajpurkar P, Ball RL et al (2018) Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med 15:e1002699

    Article  Google Scholar 

  31. 31.

    Englund M, Niu J, Guermazi A et al (2007) Effect of meniscal damage on the development of frequent knee pain, aching, or stiffness. Arthritis Rheum 56:4048–4054

  32. 32.

    Eckstein F, Benichou O, Wirth W et al (2009) Magnetic resonance imaging-based cartilage loss in painful contralateral knees with and without radiographic joint space narrowing: data from the osteoarthritis initiative. Arthritis Rheum 61:1218–1225

  33. 33.

    Cibere J, Sayre EC, Guermazi A et al (2011) Natural history of cartilage damage and osteoarthritis progression on magnetic resonance imaging in a population-based cohort with knee pain. Osteoarthritis Cartilage 19:683–688

  34. 34.

    Kim HA, Kim I, Song YW et al (2011) The association between meniscal and cruciate ligament damage and knee pain in community residents. Osteoarthritis Cartilage 19:1422–1428

  35. 35.

    Peterfy CG, Schneider E, Nevitt M (2008) The osteoarthritis initiative: report on the design rationale for the magnetic resonance imaging protocol for the knee. Osteoarthritis Cartilage 16:1433–1441

  36. 36.

    Fawaz-Estrup F (2004) The osteoarthritis initiative: an overview. Med Health R I 87:169–171

    PubMed  Google Scholar 

  37. 37.

    Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. Courant Institute of Mathematical Sciences

  38. 38.

    Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. 2016 Ieee Conference on Computer Vision and Pattern Recognition (Cvpr).

  39. 39.

    Bedson J, Croft PR (2008) The discordance between clinical and radiographic knee osteoarthritis: a systematic search and summary of the literature. BMC Musculoskelet Disord 9:467–411

    Article  Google Scholar 

  40. 40.

    Wenham CY, Conaghan PG (2009) Imaging the painful osteoarthritic knee joint: what have we learned? Nat Clin Pract Rheumatol 5:149–158

  41. 41.

    Sayre EC, Guermazi A, Esdaile JM et al (2017) Associations between MRI features versus knee pain severity and progression: data from the Vancouver Longitudinal Study of Early Knee Osteoarthritis. PLoS One 12:e0176833–e0176812

    Article  Google Scholar 

  42. 42.

    Neogi T, Felson D, Niu J et al (2009) Association between radiographic features of knee osteoarthritis and pain: results from two cohort studies. BMJ 339:b2844–b2844

    Article  Google Scholar 

  43. 43.

    Javaid MK, Kiran A, Guermazi A et al (2012) Individual magnetic resonance imaging and radiographic features of knee osteoarthritis in subjects with unilateral knee pain: the health, aging, and body composition study. Arthritis Rheum 64:3246–3255

  44. 44.

    Minciullo L, Parkes MJ, Felson DT, Cootes TF (2018) Comparing image analysis approaches versus expert readers: the relation of knee radiograph features to knee pain. Ann Rheum Dis 77:1606–1609

    Article  Google Scholar 

  45. 45.

    Dolz J, Desrosiers C, Ben Ayed I (2018) 3D fully convolutional networks for subcortical segmentation in MRI: a large-scale study. Neuroimage 170:456–470

Download references


This work was supported in part by the National Center for Advancing Translational Sciences, National Institutes of Health, through BU-CTSI Grant (1UL1TR001430), a Scientist Development Grant (17SDG33670323) from the American Heart Association, and a Research Award from the Hariri Institute for Computing and Computational Science and Engineering and Digital Health Initiative at Boston University, and NIH grants to VBK, DTF, and TDC (5U01AG-018820 and 1R01AR070139 and supported by the NIHR Manchester Biomedical Research Centre).

Author information



Corresponding author

Correspondence to Vijaya B. Kolachalama.

Ethics declarations


The scientific guarantor of this publication is Dr. David Felson (

Conflict of interest

Ali Guermazi is shareholder of BICL and consultant to Pfizer, AstraZeneca, TissueGene, Roche, Galapagos, and MerckSerono.

Statistics and biometry

No complex statistical methods were necessary for this paper. Additionally, Dr. Kolachalama has needed statistical expertise.

Informed consent

Written informed consent was not required for this study because de-identified data is publicly available (Osteoarthritis Initiative: At the time of enrollment, informed consent and ethical committee approvals were obtained by the OAI investigators.

Ethical approval

Institutional Review Board approval was not required because the data was obtained from a public database (Osteoarthritis Initiative:

Study subjects or cohorts overlap

The cohort has been reported in


• This is a case-control study.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chang, G.H., Felson, D.T., Qiu, S. et al. Assessment of knee pain from MR imaging using a convolutional Siamese network. Eur Radiol 30, 3538–3548 (2020).

Download citation


  • Knee
  • Osteoarthritis
  • Pain
  • Magnetic resonance imaging
  • Machine learning