Skip to main content
SpringerLink
Log in

Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The most common form of arthritis is osteoarthritis (OA) which often affects the knee joint. Recent studies are utilizing convolution neural networks (CNNs) to automatically classify OA severity. These deep learning models are designed to analyze either X-ray images or sequences of images from magnetic resonance imaging (MRI). For the first time, we propose a fusion model that combines three different modalities (X-ray, MRI, and the patient’s clinical information) into one network to improve the accuracy over the models being used individually. First, we construct the fusion architecture using two models from a previous work that were trained on a small dataset (dataset 1). This includes a classic CNN for X-ray with an accuracy of 50%, and a custom 3D model for MRI with an accuracy of 54%. When combining these two models with the clinical information, our fusion architecture increased performance to 62%. To further test the fusion architecture, we created a custom X-ray model and trained it on a larger dataset (dataset 2) which achieved an accuracy of 70% on the testing set from dataset 1. When combining the MRI (54%) model with the new X-ray model (70%) and the clinical information, the fusion model increased performance to 76%. In addition to the 5-category KL classification, the fusion model also improves the 2-category OA and non-OA classification to AUC of 0.964. The results show the proposed fusion architecture can be generalized to combine different individual models and a holistic multimodal approach can further boost OA classification performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The datasets analyzed during this study are available in the OAI repository: https://nda.nih.gov/oai.

References

  1. Lawrence RC, Felson DT, Helmick CG, Arnold LM, Choi H, Deyo RA, National Arthritis Data Workgroup (2008) Estimates of the prevalence of arthritis and other rheumatic conditions in the United States: Part II. Arthritis Rheum 58(1):26–35

    Article  Google Scholar 

  2. Van Saase JL, Van Romunde LK, Cats ARNOLD, Vandenbroucke JP, Valkenburg HA (1989) Epidemiology of osteoarthritis: zoetermeer survey. Comparison of radiological osteoarthritis in a Dutch population with that in 10 other populations. Ann Rheum Dis 48(4):271–280

    Article  Google Scholar 

  3. King LK, March L, Anandacoomarasamy A (2013) Obesity & osteoarthritis. Indian J Med Res 138(2):185

    Google Scholar 

  4. Verbrugge LM, Gates DM, Ike RW (1991) Risk factors for disability among US adults with arthritis. J Clin Epidemiol 44(2):167–182

    Article  Google Scholar 

  5. Palazzo C, Ravaud JF, Papelard A, Ravaud P, Poiraudeau S (2014) The burden of musculoskeletal conditions. PLoS ONE 9(3):e90633

    Article  Google Scholar 

  6. Gupta S, Hawker GA, Laporte A, Croxford R, Coyte PC (2005) The economic burden of disabling hip and knee osteoarthritis (OA) from the perspective of individuals living with this condition. Rheumatology 44(12):1531–1537

    Article  Google Scholar 

  7. Anandacoomarasamy A, March L (2010) Current evidence for osteoarthritis treatments. Ther Adv Musculoskelet Dis 2(1):17–28

    Article  Google Scholar 

  8. Kellgren JH, Lawrence J (1957) Radiological assessment of osteo-arthrosis. Ann Rheum Dis 16(4):494

    Article  Google Scholar 

  9. Vignon E, Conrozier T, Piperno M, Richard S, Carrillon Y, Fantino O (1999) Radiographic assessment of hip and knee osteoarthritis. Recommendations: recommended guidelines. Osteoarthr Cartil 7(4):434–436

    Article  Google Scholar 

  10. Kornaat PR, Ceulemans RY, Kroon HM, Riyazi N, Kloppenburg M, Carter WO et al (2005) MRI assessment of knee osteoarthritis: knee osteoarthritis scoring system (KOSS)—inter-observer and intra-observer reproducibility of a compartment-based scoring system. Skelet Radiol 34(2):95–102

    Article  Google Scholar 

  11. Pessis E, Drape JL, Ravaud P, Chevrot A, Dougados M, Ayral X (2003) Assessment of progression in knee osteoarthritis: results of a 1 year study comparing arthroscopy and MRI. Osteoarthr Cartil 11(5):361–369

    Article  Google Scholar 

  12. Eckstein F, Cicuttini F, Raynauld JP, Waterton JC, Peterfy C (2006) Magnetic resonance imaging (MRI) of articular cartilage in knee osteoarthritis (OA): morphological assessment. Osteoarthr Cartil 14:46–75

    Article  Google Scholar 

  13. Oka H, Muraki S, Akune T, Mabuchi A, Suzuki T, Yoshida H et al (2008) Fully automatic quantification of knee osteoarthritis severity on plain radiographs. Osteoarthr Cartil 16(11):1300–1306

    Article  Google Scholar 

  14. Tiulpin A, Saarakkala S (2020) Automatic grading of individual knee osteoarthritis features in plain radiographs using deep convolutional neural networks. Diagnostics 10(11):932

    Article  Google Scholar 

  15. Gajre SS, Anand S, Singh U, Saxena RK (2006) Novel method of using dynamic electrical impedance signals for noninvasive diagnosis of knee osteoarthritis. In: 2006 International conference of the IEEE engineering in medicine and biology society. IEEE, pp 2207–2210

  16. Thomson J, O’Neill T, Felson D, Cootes T (2015) Automated shape and texture analysis for detection of osteoarthritis from radiographs of the knee. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 127–134

  17. Alkan A (2011) Analysis of knee osteoarthritis by using fuzzy c-means clustering and SVM classification. Sci Res Essays 6(20):4213–4219

    Article  Google Scholar 

  18. LeCun Y, Kavukcuoglu K, Farabet C (2010) Convolutional networks and applications in vision. In: Proceedings of 2010 IEEE international symposium on circuits and systems. IEEE, pp 253–256

  19. Górriz M, Antony J, McGuinness K, Giró-i-Nieto X, O’Connor NE (2019) Assessing knee OA severity with CNN attention-based end-to-end architectures. In: International conference on medical imaging with deep learning. PMLR, pp 197–214

  20. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  21. Thomas KA, Kidziński Ł, Halilaj E, Fleming SL, Venkataraman GR, Oei EH et al (2020) Automated classification of radiographic knee osteoarthritis severity using deep neural networks. Radiol Artif Intell 2(2):e190065

    Article  Google Scholar 

  22. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  23. Ramachandram D, Taylor GW (2017) Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Mag 34(6):96–108

    Article  Google Scholar 

  24. Sharma A, Kumar D (2020) Classification with 2-D convolutional neural networks for breast cancer diagnosis. arXiv preprint arXiv:2007.03218

  25. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, 27

  26. Aderghal K, Benois-Pineau J, Afdel K (2017) Classification of sMRI for Alzheimer's disease diagnosis with CNN: single Siamese networks with 2D+? Approach and fusion on ADNI. In: Proceedings of the 2017 ACM on international conference on multimedia retrieval, pp 494–498

  27. Hu D, Wang C, Nie F, Li X (2019) Dense multimodal fusion for hierarchically joint representation. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 3941–3945

  28. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

  29. U.S. Department of Health and Human Services. (n.d.). OAI. National Institutes of Health. Retrieved 24 Apr 2022, from https://nda.nih.gov/oai

  30. Guida C, Zhang M, Shan J (2021) Knee osteoarthritis classification using 3D CNN and MRI. Appl Sci 11(11):5196

    Article  Google Scholar 

  31. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  32. Lindner C, Thiagarajah S, Wilkinson JM, Wallis GA, Cootes TF, arcOGEN Consortium (2013) Fully automatic segmentation of the proximal femur using random forest regression voting. IEEE Trans Med Imaging 32(8):1462–1472

    Article  Google Scholar 

  33. Zhang B, Tan J, Cho K, Chang G, Deniz CM (2020) Attention-based cnn for kl grade classification: data from the osteoarthritis initiative. In: 2020 IEEE 17th international symposium on biomedical imaging (ISBI). IEEE, pp 731–735

  34. Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV) , pp 3–19

  35. Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Garcia-Rodriguez J (2017) A review on deep learning techniques applied to semantic segmentation. arXiv preprint arXiv:1704.06857

  36. Hayashi D, Roemer FW, Jarraya M, Guermazi A (2013) Imaging of osteoarthritis. In: Geriatric imaging, pp 93-121

  37. Moustakidis S, Papandrianos NI, Christodolou E, Papageorgiou E, Tsaopoulos D (2023) Dense neural networks in knee osteoarthritis classification: a study on accuracy and fairness. Neural Comput Appl 35:21–33

    Article  Google Scholar 

Download references

Funding

This research was funded by National Science Foundation, Grant Number NSF-1723420 and NSF-1723429.

Author information

Authors and Affiliations

  1. Department of CSIS, Pace University, New York, NY, USA

    Carmine Guida & Juan Shan

  2. Department of Computer Science, Boston University, Boston, MA, USA

    Ming Zhang

Authors
  1. Carmine Guida
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juan Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Shan.

Ethics declarations

Conflict of interest

None.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guida, C., Zhang, M. & Shan, J. Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information. Neural Comput & Applic 35, 9763–9772 (2023). https://doi.org/10.1007/s00521-023-08214-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08214-8

Keywords

Search

Navigation