Abstract
Purpose
Our purposes were (1) to explore the methodologic quality of the studies on the deep learning in knee imaging with CLAIM criterion and (2) to offer our vision for the development of CLAIM to assure high-quality reports about the application of AI to medical imaging in knee joint.
Materials and methods
A Checklist for Artificial Intelligence in Medical Imaging systematic review was conducted from January 1, 2015, to June 1, 2020, using PubMed, EMBASE, and Web of Science databases. A total of 36 articles discussing deep learning applications in knee joint imaging were identified, divided by imaging modality, and characterized by imaging task, data source, algorithm type, and outcome metrics.
Results
A total of 36 studies were identified and divided into: X-ray (44.44%) and MRI (55.56%). The mean CLAIM score of the 36 studies was 27.94 (standard deviation, 4.26), which was 66.53% of the ideal score of 42.00. The CLAIM items achieved an average good inter-rater agreement (ICC 0.815, 95% CI 0.660–0.902). In total, 32 studies performed internal cross-validation on the data set, while only 4 studies conducted external validation of the data set.
Conclusions
The overall scientific quality of deep learning in knee imaging is insufficient; however, deep learning remains a promising technology for diagnostic or predictive purpose. Improvements in study design, validation, and open science need to be made to demonstrate the generalizability of findings and to achieve clinical applications. Widespread application, pre-trained scoring procedure, and modification of CLAIM in response to clinical needs are necessary in the future.
Key Points
• Limited deep learning studies were established in knee imaging with mean score of 27.94, which was 66.53% of the ideal score of 42.00, commonly due to invalidated results, retrospective study design, and absence of a clear definition of the CLAIM items in detail.
• A previous trained data extraction instrument allowed reaching moderate inter-rater agreement in the application of the CLAIM, while CLAIM still needs improvement in scoring items and result reporting to become a wide adaptive tool in reviews of deep learning studies.
Similar content being viewed by others
Abbreviations
- AI:
-
Artificial intelligence
- AUC:
-
Area under the curve
- CNN:
-
Convolutional neural network
- CLAIM:
-
Checklist for Artificial Intelligence in Medical Imaging
- DL:
-
Deep learning
- DNN:
-
Deep neural network
- ML:
-
Machine learning
- PRISMA:
-
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
References
Prieto-Alhambra D, Judge A, Javaid MK, Cooper C, Diez-Perez A, Arden NK (2014) Incidence and risk factors for clinically diagnosed knee, hip and hand osteoarthritis: influences of age, gender and osteoarthritis affecting other joints. Ann Rheum Dis 73:1659–1664
Turkiewicz A, Petersson IF, Bjork J et al (2014) Current and future impact of osteoarthritis on health care: a population-based study with projections to year 2032. Osteoarthritis Cartilage 22:1826–1832
Roemer FW, Demehri S, Omoumi P et al (2020) State of the art: imaging of osteoarthritis-revisited 2020. Radiology 296(1):5–21
Dunn R, Greenhouse J, James D, Ohlssen D, Mesenbrink P (2020) Risk scoring for time to end-stage knee osteoarthritis: data from the Osteoarthritis Initiative. Osteoarthritis Cartilage 28(8):1020–1029
Zhai G, Sun X, Randel E et al (2021) Phenylalanine is a novel marker for radiographic knee osteoarthritis progression: the MOST study. J Rheumatol 48(1):123–128
Peterfy CG, Guermazi A, Zaim S et al (2004) WORMS of the knee in osteoarthritis. Osteoarthritis Cartilage 12(3):177–190
Hunter DJ, Guermazi A, Lo GH et al (2011) Evolution of semi-quantitative whole joint assessment of knee OA: MOAKS. Osteoarthritis Cartilage 19(8):990–1002
Hunter DJ, Lo GH, Gale D et al (2008) The reliability of a new scoring system for knee osteoarthritis MRI and the validity of bone marrow lesion assessment: BLOKS. Ann Rheum Dis 67(2):206–211
Peterfy CG, van Dijke CF, Janzen DL et al (1994) Quantification of articular cartilage in the knee with pulsed saturation transfer subtraction and fat-suppressed MR imaging: optimization and validation. Radiology 192:485–491
Eckstein F, Le Graver MP, Charles HC et al (2011) Clinical, radiographic, molecular and MRI-based predictors of cartilage loss in knee osteoarthritis. Ann Rheum Dis 70:1223–1230
Eckstein F, Wirth W, Guermazi A, Maschek S, Aydemir A (2015) Intra-articular sprifermin not only increases cartilage thickness, but also reduces cartilage loss: location-independent post hoc analysis using MR imaging. Arthritis Rheumatol 67(11):2916–2922
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444
Leung K, Zhang B, Tan J et al (2020) Prediction of total knee replacement and diagnosis of osteoarthritis by using deep learning on knee radiographs: data from the Osteoarthritis Initiative. Radiology 296(3):584–593
Bien N, Rajpurkar P, Ball RL et al (2018) Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med 15(11):e1002699
Tiulpin A, Klein S, Bierma-Zeinstra SMA et al (2019) Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Sci Rep 9(1):20038
Mongan J, Moy L, Kahn CE Jr (2020) Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2(2):e200029
McInnes MDF, Moher D, Thombs BD et al (2018) Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: The PRISMA-DTA statement. JAMA 319(4):388–396
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. BMJ 339:b2535
Moher D, Shamseer L, Clarke M et al (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4:1
Zhong J, Hu Y, Si L et al (2021) A systematic review of radiomics in osteosarcoma: utilizing radiomics quality score as a tool promoting clinical translation. Eur Radiol 31(3):1526–1535
Cohen JA (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46
Koo TK, Li MY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15:155–163
Morales Martinez A, Caliva F, Flament I et al (2020) Learning osteoarthritis imaging biomarkers from bone surface spherical encoding. Magn Reson Med 84(4):2190–2203
Roblot V, Giret Y, Bou Antoun M et al (2019) Artificial intelligence to diagnose meniscus tears on MRI. Diagn Interv Imaging 100(4):243–249
Das A, Rad P (2020) Opportunities and challenges in explainable artificial intelligence (XAI): a survey. https://arxiv.org/abs/2006.11371
Norman B, Pedoia V, Noworolski A, Link TM, Majumdar S (2019) Applying densely connected convolutional neural networks for staging osteoarthritis severity from plain radiographs. J Digit Imaging 32(3):471–477
Thomas KA, Kidziński Ł, Halilaj E et al (2020) Automated classification of radiographic knee osteoarthritis severity using deep neural networks. Radiol Artif Intell 2(2):e190065
Chang PD, Wong TT, Rasiej MJ (2019) Deep learning for detection of complete anterior cruciate ligament tear. J Digit Imaging 32(6):980–986
Richardson ML (2021) MR protocol optimization with deep learning: a proof of concept. Curr Probl Diagn Radiol 50(2):168–174
Górriz M, Antony J, Mcguinness K, Giró-i-Nieto X, O’Connor NE (2019) Assessing knee OA severity with CNN attention-based end-to-end architectures. In: International Conference on Medical Imaging with Deep Learning. PMLR, pp 197–214
Armanious K, Abdulatif S, Bhaktharaguttu AR et al (2021) Organ-based chronological age estimation based on 3D MRI scans. In: 2020 28th European Signal Processing Conference (EUSIPCO). IEEE, pp 1225–1228
Guan B, Liu F, Haj-Mirzaian A et al (2020) Deep learning risk assessment models for predicting progression of radiographic medial joint space loss over a 48-month follow-up period. Osteoarthritis Cartilage 28(4):428–437
Sanduleanu S, Woodruff HC, de Jong EEC et al (2018) Tracking tumor biology with radiomics: a systematic review utilizing a radiomics quality score. Radiother Oncol 127(3):349–360
Granzier RWY, van Nijnatten TJA, Woodruff HC et al (2019) Exploring breast cancer response prediction to neoadjuvant systemic therapy using MRI-based radiomics: a systematic review. Eur J Radiol 121:108736
Acknowledgements
The authors would like to express their gratitude to Prof. Huan Zhang and Prof. Qian Wang for their constructive discussion and suggestions.
Funding
This study has received funding by the National Natural Science Foundation of China (81771790) and the Medicine and Engineering Combination Project of Shanghai Jiao Tong University (YG2019ZDB09).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Guarantor
The scientific guarantor of this publication is Prof. Weiwu Yao.
Conflict of interest
The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.
Statistics and Biometry
No complex statistical methods were necessary for this paper, but one of the authors has significant statistical expertise.
Informed Consent
Written informed consent was not required for this study because of the nature of our study, which was a systematic review.
Ethical Approval
Institutional Review Board approval was not required because of the nature of our study, which was a systematic review.
Methodology
• retrospective
• diagnostic or prognostic study
• multicenter study
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(DOCX 566 kb)
Rights and permissions
About this article
Cite this article
Si, L., Zhong, J., Huo, J. et al. Deep learning in knee imaging: a systematic review utilizing a Checklist for Artificial Intelligence in Medical Imaging (CLAIM). Eur Radiol 32, 1353–1361 (2022). https://doi.org/10.1007/s00330-021-08190-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-021-08190-4