Skip to main content

Advertisement

SpringerLink
Log in

Transfer learning-based ensemble convolutional neural network for accelerated diagnosis of foot fractures

  • Scientific Paper
  • Published:
Physical and Engineering Sciences in Medicine Aims and scope Submit manuscript

Abstract

The complex shape of the foot, consisting of 26 bones, variable ligaments, tendons, and muscles leads to misdiagnosis of foot fractures. Despite the introduction of artificial intelligence (AI) to diagnose fractures, the accuracy of foot fracture diagnosis is lower than that of conventional methods. We developed an AI assistant system that assists with consistent diagnosis and helps interns or non-experts improve their diagnosis of foot fractures, and compared the effectiveness of the AI assistance on various groups with different proficiency. Contrast-limited adaptive histogram equalization was used to improve the visibility of original radiographs and data augmentation was applied to prevent overfitting. Preprocessed radiographs were fed to an ensemble model of a transfer learning-based convolutional neural network (CNN) that was developed for foot fracture detection with three models: InceptionResNetV2, MobilenetV1, and ResNet152V2. After training the model, score class activation mapping was applied to visualize the fracture based on the model prediction. The prediction result was evaluated by the receiver operating characteristic (ROC) curve and its area under the curve (AUC), and the F1-Score. Regarding the test set, the ensemble model exhibited better classification ability (F1-Score: 0.837, AUC: 0.95, Accuracy: 86.1%) than other single models that showed an accuracy of 82.4%. With AI assistance for the orthopedic fellow, resident, intern, and student group, the accuracy of each group improved by 3.75%, 7.25%, 6.25%, and 7% respectively and diagnosis time was reduced by 21.9%, 14.7%, 24.4%, and 34.6% respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Naser A, Osama A (2016) A proposed expert system for foot diseases diagnosis. Am J Innov Res Appl Sci 2:160–173

  2. Judd DB, Kim DH (2002) Foot fractures frequently misdiagnosed as ankle sprains. Am Fam Physician 66:785–794

    PubMed  Google Scholar 

  3. Wei CJ, Tsai WC, Tiu CM, Wu HT, Chiou HJ, Chang CY (2006) Systematic analysis of missed extremity fractures in emergency radiology. Acta Radiol 47:710–717. https://doi.org/10.1080/02841850600806340

    Article  PubMed  Google Scholar 

  4. Bruno MA, Walker EA, Abujudeh HH (2015) Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction. Radiographics 35:1668–1676. https://doi.org/10.1148/rg.2015150023

    Article  PubMed  Google Scholar 

  5. Alhasan M, Hasaneen M (2021) Digital imaging, technologies and artificial intelligence applications during COVID-19 pandemic. Comput Med Imaging Graph 91:101933. https://doi.org/10.1016/j.compmedimag.2021.101933

    Article  PubMed  PubMed Central  Google Scholar 

  6. Lalehzarian SP, Gowd AK, Liu JN (2021) Machine learning in orthopaedic surgery. World J Orthop 12:685–699. https://doi.org/10.5312/wjo.v12.i9.685

    Article  PubMed  PubMed Central  Google Scholar 

  7. Zhang C, Lu Y (2021) Study on artificial intelligence: the state of the art and future prospects. J Ind Inf Integr 23:100224. https://doi.org/10.1016/j.jii.2021.100224

    Article  Google Scholar 

  8. Gupta A, Anpalagan A, Guan L, Khwaja AS (2021) Deep learning for object detection and scene perception in self-driving cars: survey, challenges, and open issues. Array 10:100057. https://doi.org/10.1016/j.array.2021.100057

    Article  Google Scholar 

  9. Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51:6400–6429. https://doi.org/10.1007/s10489-021-02293-7

    Article  Google Scholar 

  10. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL (2018) Artificial intelligence in radiology. Nat Rev Cancer 18:500–510. https://doi.org/10.1038/s41568-018-0016-5

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Thian YL, Li Y, Jagmohan P, Sia D, Chan VEY, Tan RT (2019) Convolutional neural networks for automated fracture detection and localization on wrist radiographs. Radiol Artif Intell 1:e180001. https://doi.org/10.1148/ryai.2019180001

    Article  PubMed  PubMed Central  Google Scholar 

  12. Raghavendra U, Bhat NS, Gudigar A, Acharya UR (2018) Automated system for the detection of thoracolumbar fractures using a CNN architecture. Futur Gener Comput Syst 85:184–189. https://doi.org/10.1016/j.future.2018.03.023

    Article  Google Scholar 

  13. Kuang Z, Deng X, Yu L, Zhang H, Lin X, Ma H (2020) Skull R-CNN: a CNN-based network for the skull fracture detection. Med Imaging Deep Learn PMLR 382–392

  14. Kim JH, Mo YC, Choi SM, Hyun Y, Lee JW (2021) Detecting ankle fractures in plain radiographs using deep learning with accurately labeled datasets aided by computed tomography: a retrospective observational study. Appl Sci 11. https://doi.org/10.3390/app11198791

  15. Shelmerdine SC, White RD, Liu H, Arthurs OJ, Sebire NJ (2022) Artificial intelligence for radiological paediatric fracture assessment: a systematic review. Insights Imaging. https://doi.org/10.1186/s13244-022-01234-3

    Article  PubMed  PubMed Central  Google Scholar 

  16. Dupuis M, Delbos L, Veil R, Adamsbaum C (2022) External validation of a commercially available deep learning algorithm for fracture detection in children. Diagn Interv Imaging 103:151–159. https://doi.org/10.1016/j.diii.2021.10.007

    Article  PubMed  Google Scholar 

  17. Guermazi A, Tannoury C, Kompel AJ, Murakami AM, Ducarouge A, Gillibert A, Li X, Tournier A, Lahoud Y, Jarraya M, Lacave E, Rahimi H, Pourchot A, Parisien RL, Merritt AC, Comeau D, Regnard NE, Hayashi D (2022) Improving radiographic fracture recognition performance and efficiency using artificial intelligence. Radiology 302:627–636. https://doi.org/10.1148/radiol.210937

    Article  PubMed  Google Scholar 

  18. Jones RM, Sharma A, Hotchkiss R, Sperling JW, Hamburger J, Ledig C, O’Toole R, Gardner M, Venkatesh S, Roberts MM, Sauvestre R, Shatkhin M, Gupta A, Chopra S, Kumaravel M, Daluiski A, Plogger W, Nascone J, Potter HG, Lindsey RV (2020) Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs. npj Digit Med 3:1–6. https://doi.org/10.1038/s41746-020-00352-w

    Article  Google Scholar 

  19. Lindsey R, Daluiski A, Chopra S, Lachapelle A, Mozer M, Sicular S, Hanel D, Gardner M, Gupta A, Hotchkiss R, Potter H (2018) Deep neural network improves fracture detection by clinicians. Proc Natl Acad Sci USA 115:11591–11596. https://doi.org/10.1073/pnas.1806905115

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Alizadehsani R, Roshanzamir M, Hussain S, Khosravi A, Koohestani A, Zangooei MH, Abdar M, Beykikhoshk A, Shoeibi A, Zare A, Panahiazar M, Nahavandi S, Srinivasan D, Atiya AF, Acharya UR (2021) Handling of uncertainty in medical data using machine learning and probability theory techniques: a review of 30 years (1991–2020). Springer US

  21. Varma M, Lu M, Gardner R, Dunnmon J, Khandwala N, Rajpurkar P, Long J, Beaulieu C, Shpanskaya K, Fei-Fei L, Lungren MP, Patel BN (2019) Automated abnormality detection in lower extremity radiographs using deep learning. Nat Mach Intell 1:578–583. https://doi.org/10.1038/s42256-019-0126-0

    Article  Google Scholar 

  22. Lin YJ, Chung IF (2019) Medical data augmentation using generative adversarial networks: x-ray image generation for transfer learning of hip fracture detection. In: Proceedings—2019 international conference on technologies and applications of artificial intelligence (TAAI). pp 14–18. https://doi.org/10.1109/TAAI48200.2019.8959908

  23. Kitamura G, Chung CY, Moore BE (2019) Ankle fracture detection utilizing a convolutional neural network ensemble implemented with a small sample, de novo training, and multiview incorporation. J Digit Imaging 32:672–677. https://doi.org/10.1007/s10278-018-0167-7

    Article  PubMed  PubMed Central  Google Scholar 

  24. Hardalaç F, Uysal F, Peker O, Çiçeklidağ M, Tolunay T, Tokgöz N, Kutbay U, Demirciler B, Mert F (2022) Fracture detection in wrist X-ray images using deep learning-based object detection models. Sensors. https://doi.org/10.3390/s22031285

    Article  PubMed  PubMed Central  Google Scholar 

  25. Kandel I, Castelli M, Popovič A (2021) Comparing stacking ensemble techniques to improve musculoskeletal fracture image classification. J Imaging. https://doi.org/10.3390/JIMAGING7060100

    Article  PubMed Central  Google Scholar 

  26. Kim JY, Kim JM (2020) Bearing fault diagnosis using grad-CAM and acoustic emission signals. Appl Sci. https://doi.org/10.3390/app10062050

    Article  PubMed  Google Scholar 

  27. Menikdiwela M, Nguyen C, Li H, Shaw M (2018) CNN-based small object detection and visualization with feature activation mapping. In: International conference on image and vision computing New Zealand 2017-December. pp 1–5. https://doi.org/10.1109/IVCNZ.2017.8402455

  28. Zhang Y, Hong D, McClement D, Oladosu O, Pridham G, Slaney G (2021) Grad-CAM helps interpret the deep learning models trained to classify multiple sclerosis types using clinical brain magnetic resonance imaging. J Neurosci Methods 353:109098. https://doi.org/10.1016/j.jneumeth.2021.109098

    Article  PubMed  Google Scholar 

  29. Cheng CT, Chen CC, Fu CY, Chaou CH, Wu YT, Hsu CP, Chang CC, Chung IF, Hsieh CH, Hsieh MJ, Liao CH (2020) Artificial intelligence-based education assists medical students’ interpretation of hip fracture. Insights Imaging. https://doi.org/10.1186/s13244-020-00932-0

    Article  PubMed  PubMed Central  Google Scholar 

  30. Meng XH, Wu DJ, Wang Z, Ma XL, Dong XM, Liu AE, Chen L (2021) A fully automated rib fracture detection system on chest CT images and its impact on radiologist performance. Skelet Radiol 50:1821–1828. https://doi.org/10.1007/s00256-021-03709-8

    Article  Google Scholar 

  31. Ren Y, Zhang L, Suganthan PN (2016) Ensemble classification and regression-recent developments, applications and future directions [review article]. IEEE Comput Intell Mag 11:41–53. https://doi.org/10.1109/MCI.2015.2471235

    Article  Google Scholar 

  32. Wang H, Wang Z, Du M, Yang F, Zhang Z, Ding S, Mardziel P, Hu X (2020) Score-CAM: score-weighted visual explanations for convolutional neural networks. In: IEEE computer society conference on computer vision and pattern recognition work 2020-June. pp 111–119. https://doi.org/10.1109/CVPRW50498.2020.00020

  33. Zuiderveld K (1994) VIII.5.—contrast limited adaptive histogram equalization. In: Heckbert PS (ed) Graphics gems. Academic Press, San Diego, pp 474–485

    Chapter  Google Scholar 

  34. Chollet F (2015) Keras: deep learning for humans. In: Keras. https://github.com/fchollet/keras

  35. Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database. pp 248–255. https://doi.org/10.1109/cvprw.2009.5206848

  36. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: 31st AAAI conference on artificial intelligence (AAAI). pp 4278–4284

  37. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  38. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition 2016-December. pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  39. Huertas-Tato J, Martín A, Camacho D (2020) Cloud type identification using data fusion and ensemble learning. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) 12490 LNCS. pp 137–147. https://doi.org/10.1007/978-3-030-62365-4_13

  40. Rafik HD (2022) Classification and detection of covid-19 in human respiratory lungs using convolutional neural network architectures. pp 1–10. https://doi.org/10.1109/ai-csp52968.2021.9671158

  41. Rezaeijo SM, Ghorvei M, Mofid B (2021) Predicting breast cancer response to neoadjuvant chemotherapy using ensemble deep transfer learning based on CT images. J Xray Sci Technol 29:835–850. https://doi.org/10.3233/XST-210910

    Article  CAS  PubMed  Google Scholar 

  42. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: 3rd international conference on learning representations ICLR 2015—conference track proceedings. pp 1–15

  43. Ju C, Bibaut A, van der Laan M (2018) The relative performance of ensemble methods with deep convolutional neural networks for image classification. J Appl Stat 45:2800–2818. https://doi.org/10.1080/02664763.2018.1441383

    Article  PubMed  Google Scholar 

  44. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings—30th IEEE conference on computer vision pattern recognition (CVPR) 2017-January. pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  45. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings—30th IEEE conference on computer vision pattern recognition (CVPR) 2017-January. pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  46. Zhang X, Yang Y, Shen YW, Zhang KR, Jiang Z, kun, Ma LT, Ding C, Wang BY, Meng Y, Liu H (2022) Diagnostic accuracy and potential covariates of artificial intelligence for diagnosing orthopedic fractures: a systematic literature review and meta-analysis. Eur Radiol 32:7196–7216. https://doi.org/10.1007/s00330-022-08956-4

    Article  PubMed  Google Scholar 

  47. Choi J, Hui JZ, Spain D, Su YS, Cheng CT, Liao CH (2021) Practical computer vision application to detect hip fractures on pelvic X-rays: a bi-institutional study. Trauma Surg Acute Care Open 6:1–4. https://doi.org/10.1136/tsaco-2021-000705

    Article  Google Scholar 

  48. Hendrix N, Scholten E, Vernhout B, Bruijnen S, Maresch B, de Jong M, Diepstraten S, Bollen S, Schalekamp S, de Rooij M, Scholtens A, Hendrix W, Samson T, Ong LLS, Postma E, van Ginneken B, Rutten M (2021) Development and validation of a convolutional neural network for automated detection of scaphoid fractures on conventional radiographs. Radiol Artif Intell. https://doi.org/10.1148/ryai.2021200260

    Article  PubMed  PubMed Central  Google Scholar 

  49. Avrahami D, Pajaczkowski JA (2012) Femoral neck stress fracture in a female athlete: a case report. J Chiropr Med 11:273–279. https://doi.org/10.1016/j.jcm.2012.05.010

    Article  PubMed  PubMed Central  Google Scholar 

  50. Varoquaux G, Cheplygina V (2022) Machine learning for medical imaging: methodological failures and recommendations for the future. npj Digit Med. https://doi.org/10.1038/s41746-022-00592-y

    Article  PubMed  PubMed Central  Google Scholar 

  51. Roberts M, Driggs D, Thorpe M, Gilbey J, Yeung M, Ursprung S, Aviles-Rivero AI, Etmann C, McCague C, Beer L, Weir-McCall JR, Teng Z, Gkrania-Klotsas E, Ruggiero A, Korhonen A, Jefferson E, Ako E, Langs G, Gozaliasl G, Yang G, Prosch H, Preller J, Stanczuk J, Tang J, Hofmanninger J, Babar J, Sánchez LE, Thillai M, Gonzalez PM, Teare P, Zhu X, Patel M, Cafolla C, Azadbakht H, Jacob J, Lowe J, Zhang K, Bradley K, Wassin M, Holzer M, Ji K, Ortet MD, Ai T, Walton N, Lio P, Stranks S, Shadbahr T, Lin W, Zha Y, Niu Z, Rudd JHF, Sala E, Schönlieb CB (2021) Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat Mach Intell 3:199–217. https://doi.org/10.1038/s42256-021-00307-0

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Promoting AI Collaboration Project Research Fund (1.210129.01) of Ulsan National Institute of Science & Technology (UNIST) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (Grant Nos. 2021M2D2A1A01050059 and 2021R1F1A1046079) and Biomedical Research Institute Grant (20210017), Pusan National University Hospital.

Funding

This work was supported by Ulsan National Institute of Science and Technology (Grant No. 1.210129.01), Ministry of Science and ICT, South Korea (Grant Nos. 2021M2D2A1A01050059, 2021R1F1A1046079), Pusan National University Hospital (Grant No. 20210017), Ministry of SMEs and Startups, South Korea (Grant No. S3248116), Research Institute of Industrial Science and Technology (Grant No. 2.220971.01), and Ministry of Trade, Industry and Energy, South Korea (Grant No. 20017502).

Author information

Author notes

  1. Taekyeong Kim and Tae Sik Goh have contributed equally to this work.

Authors and Affiliations

  1. Department of Mechanical Engineering, Ulsan National Institute of Science and Technology, Ulsan, 44919, Republic of Korea

    Taekyeong Kim, Hayeol Kim & Im Doo Jung

  2. Department of Orthopaedic Surgery, Biomedical Research Institute, Pusan National University Hospital, Pusan National University School of Medicine, Busan, 49241, Republic of Korea

    Tae Sik Goh & Jung Sub Lee

  3. Health Insurance Review & Assessment Service, Wonju, 26465, Republic of Korea

    Ji Hyun Lee

Authors
  1. Taekyeong Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tae Sik Goh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jung Sub Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ji Hyun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hayeol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Im Doo Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TK: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing—Original Draft, Writing—Review & Editing. TSG: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Resources, Data curation, Writing—Original Draft, Writing—Review & Editing. JSL: Formal analysis. JHL: Resources, Data Curation. HK: Writing—Original Draft, Writing—Review & Editing. IDJ: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing—Original Draft, Writing—Review & Editing, Supervision.

Corresponding author

Correspondence to Im Doo Jung.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 374 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, T., Goh, T.S., Lee, J.S. et al. Transfer learning-based ensemble convolutional neural network for accelerated diagnosis of foot fractures. Phys Eng Sci Med 46, 265–277 (2023). https://doi.org/10.1007/s13246-023-01215-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13246-023-01215-w

Keywords

Search

Navigation