Skip to main content

Advertisement

SpringerLink
Log in

A convolutional neural network with transfer learning for automatic discrimination between low and high-grade synovitis: a pilot study

  • IM - ORIGINAL
  • Published:
Internal and Emergency Medicine Aims and scope Submit manuscript

Abstract

Ultrasound-guided synovial tissue biopsy (USSB) may allow personalizing the treatment for patients with inflammatory arthritis. To this end, the quantification of tissue inflammation in synovial specimens can be crucial to adopt proper therapeutic strategies. This study aimed at investigating whether computer vision may be of aid in discriminating the grade of synovitis in patients undergoing USSB. We used a database of 150 photomicrographs of synovium from patients who underwent USSB. For each hematoxylin and eosin (H&E)-stained slide, Krenn’s score was calculated. After proper data pre-processing and fine-tuning, transfer learning on a ResNet34 convolutional neural network (CNN) was employed to discriminate between low and high-grade synovitis (Krenn’s score < 5 or ≥ 5). We computed test phase metrics, accuracy, precision (true positive/actual results), and recall (true positive/predicted results). The Grad-Cam algorithm was used to highlight the regions in the image used by the model for prediction. We analyzed photomicrographs of specimens from 12 patients with arthritis. The training dataset included n.90 images (n.42 with high-grade synovitis). Validation and test datasets included n.30 (n.14 high-grade synovitis) and n.30 items (n.16 with high-grade synovitis). An accuracy of 100% (precision = 1, recall = 1) was scored in the test phase. Cellularity in the synovial lining and sublining layers was the salient determinant of CNN prediction. This study provides a proof of concept that computer vision with transfer learning is suitable for scoring synovitis. Integrating CNN-based approach into real-life patient management may improve the workflow between rheumatologists and pathologists.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Availability of data and materials

Image and patient database not available due to local ethical committee privacy issues.

Code availability

Model available upon request.

Abbreviations

AI:

Artificial Intelligence

bDMARDs:

Biological disease-modifying anti-rheumatic drugs

CNN:

Convolutional Neural Network

csDMARDs:

Conventional synthetic disease-modifying anti-rheumatic drugs

DAPSA:

Disease Activity in Psoriatic Arthritis

DAS28-ESR:

Disease Activity Score on 28 joints with Erythrocyte Sedimentation Rate

MLP:

Multi-layer Perceptron

MTX:

Methotrexate

PDN:

Prednisone

TNFi:

Tumour Necrosis Factor inhibitors

SSZ:

Sulfasalazine

USSB:

Ultrasound-guided synovial tissue biopsy

References

  1. Najm A, Le Goff B, Orr C, Thurlings R, Canete JD, Humby F, Alivernini S, Manzo A, Just SA, Romao VC, Krenn V, Muller-Ladner U, Addimanda O, Tas SW, Stoenoiu M, Meric de Bellefon L, Durez P, Strand V, Wechalekar MD, Fonseca JE, Lauwerys B, Fearon U, Veale DJ, Group ESS, Group OSTSI (2018) Standardisation of synovial biopsy analyses in rheumatic diseases: a consensus of the EULAR synovitis and OMERACT synovial tissue biopsy groups. Arthritis Res Ther 20(1):265. https://doi.org/10.1186/s13075-018-1762-1

    Article  Google Scholar 

  2. Venerito V, Cazzato G, Lopalco G, Fornaro M, Righetti G, Urso L, Cimmino A, Iannone F (2019) Histopathologic features of fibrotic knee synovitis in a young adult with seronegative rheumatoid arthritis. J Clin Rheumatol. https://doi.org/10.1097/RHU.0000000000001247

    Article  Google Scholar 

  3. Humby F, Buch M, Durez P, Lewis M, Bombardieri M, Rizvi H, Kelly S, Fosatti L, Hands R, Giorli G, Mahto A, Montecucco C, Lauwerys B, Romao V, Pratt A, Bugatti S, Ng N, Rivellese F, Ho P, Bellan M, Congia M, Verschueren P, Sainaghi P, Gendi N, Dasgupta B, Cauli A, Reynolds P, Cañete J, Moots R, Taylor P, Edwards C, Isaacs J, Sasieni P, Eurico Fonseca J, Choy E, Pitzalis C (2019) A Randomised, Open Labelled Clinical Trial to Investigate Synovial Mechanisms Determining Response – Resistance to Rituximab versus Tocilizumab in Rheumatoid Arthritis Patients Failing TNF Inhibitor Therapy. Arthritis Rheumatol 71 (suppl 10). https://acrabstracts.org/abstract/a-randomised-open-labelled-clinical-trial-to-investigate-synovial-mechanisms-determining-response-resistance-to-rituximab-versus-tocilizumab-in-rheumatoid-arthritis-patients-failing-tnf-inhibitor-t/. Accessed 4 Dec 2020

  4. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115–118. https://doi.org/10.1038/nature21056

    Article  CAS  PubMed  Google Scholar 

  5. Srinidhi CL, Ciga O, Martel AL (2019) Deep neural network models for computational histopathology: A survey.

  6. Gertych A, Swiderska-Chadaj Z, Ma Z, Ing N, Markiewicz T, Cierniak S, Salemi H, Guzman S, Walts AE, Knudsen BS (2019) Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Scientific Reports 9(1):1483. https://doi.org/10.1038/s41598-018-37638-9

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Pandit A, Radstake T (2020) Machine learning in rheumatology approaches the clinic. Nat Rev Rheumatol 16(2):69–70. https://doi.org/10.1038/s41584-019-0361-0

    Article  PubMed  Google Scholar 

  8. Morawietz L, Schaeper F, Schroeder JH, Gansukh T, Baasanjav N, Krukemeyer MG, Gehrke T, Krenn V (2008) Computer-assisted validation of the synovitis score. Virchows Arch 452(6):667–673. https://doi.org/10.1007/s00428-008-0587-8

    Article  PubMed  Google Scholar 

  9. Rooney T, Bresnihan B, Andersson U, Gogarty M, Kraan M, Schumacher HR, Ulfgren AK, Veale DJ, Youssef PP, Tak PP (2007) Microscopic measurement of inflammation in synovial tissue: inter-observer agreement for manual quantitative, semiquantitative and computerised digital image analysis. Ann Rheum Dis 66(12):1656–1660. https://doi.org/10.1136/ard.2006.061143

    Article  PubMed  PubMed Central  Google Scholar 

  10. Krenn V, Morawietz L, Burmester GR, Kinne RW, Mueller-Ladner U, Muller B, Haupl T (2006) Synovitis score: discrimination between chronic low-grade and high-grade synovitis. Histopathology 49(4):358–364. https://doi.org/10.1111/j.1365-2559.2006.02508.x

    Article  CAS  PubMed  Google Scholar 

  11. Wechalekar MD, Najm A, Veale DJ, Strand V (2019) The 2018 OMERACT synovial tissue biopsy special interest group report on standardization of synovial biopsy analysis. J Rheumatol. https://doi.org/10.3899/jrheum.181062

    Article  PubMed  Google Scholar 

  12. Shahin M, Maier H, Jaksa M (2004) Data division for developing neural networks applied to geotechnical engineering. J Comput Civil Eng. https://doi.org/10.1061/(ASCE)0887-3801(2004)18:2(105)

    Article  Google Scholar 

  13. Goodfellow I, Bengio Y, Courville A (2016) Deep Learning. MIT Press

  14. Brunese L, Mercaldo F, Reginelli A, Santone A (2020) Explainable deep learning for pulmonary disease and coronavirus COVID-19 detection from X-rays. Comput Methods Programs Biomed 196:105608. https://doi.org/10.1016/j.cmpb.2020.105608

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kensert A, Harrison PJ, Spjuth O (2019) Transfer learning with deep convolutional neural networks for classifying cellular morphological changes. SLAS Discov 24(4):466–475. https://doi.org/10.1177/2472555218818756

    Article  PubMed  PubMed Central  Google Scholar 

  16. Vu CC, Siddiqui ZA, Zamdborg L, Thompson AB, Quinn TJ, Castillo E, Guerrero TM (2020) Deep convolutional neural networks for automatic segmentation of thoracic organs-at-risk in radiation oncology—use of non-domain transfer learning. J Appl Clin Med Phys 21(6):108–113. https://doi.org/10.1002/acm2.12871

    Article  PubMed  PubMed Central  Google Scholar 

  17. He K, Zhang X, Ren S, Sun J (2015) Deep Residual Learning for Image Recognition.

  18. Howard J, Gugger S (2020) fastai: A Layered API for Deep Learning.

  19. Deng J, Dong W, Socher R, Li L-J, Li K, Li FF (2009) ImageNet: a large-scale hierarchical image database. IEEE. https://doi.org/10.1109/CVPR.2009.5206848

    Article  Google Scholar 

  20. Kaur T, Gandhi TK (2020) Deep convolutional neural networks with transfer learning for automated brain image classification. Mach Vis Appl 31(3):20. https://doi.org/10.1007/s00138-020-01069-2

    Article  Google Scholar 

  21. Monshi M, Poon J, Chung V (2019) Convolutional neural network to detect thorax diseases from multi-view chest X-rays. In: Gedeon T, Wong K, Lee M (eds) Neural Information Processing. ICONIP 2019. Communications in computer and information science. Springer, Cham, pp 148–158. https://doi.org/10.1007/978-3-030-36808-1_17

    Chapter  Google Scholar 

  22. Smith LN (2018) A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay.

  23. He T, Guo J, Chen N, Xu X, Wang Z, Fu K, Liu L, Yi Z (2020) MediMLP: using Grad-CAM to extract crucial variables for lung cancer postoperative complication prediction. IEEE J Biomed Health Inform 24(6):1762–1771. https://doi.org/10.1109/JBHI.2019.2949601

    Article  PubMed  Google Scholar 

  24. Iannone F, Nivuori M, Fornaro M, Venerito V, Cacciapaglia F, Lopalco G (2019) Comorbid fibromyalgia impairs the effectiveness of biologic drugs in patients with psoriatic arthritis. Rheumatology (Oxford). https://doi.org/10.1093/rheumatology/kez505

    Article  PubMed Central  Google Scholar 

  25. Lopalco G, Venerito V, Cantarini L, Emmi G, Salaffi F, Di Carlo M, Tafuri S, Gentileschi S, Di Scala G, Nivuori M, Cacciapaglia F, Galeazzi M, Lapadula G, Iannone F (2019) Different drug survival of first line tumour necrosis factor inhibitors in radiographic and non-radiographic axial spondyloarthritis: a multicentre retrospective survey. Clin Exp Rheumatol 37(5):762–767

    PubMed  Google Scholar 

  26. Venerito V, Lopalco G, Cacciapaglia F, Fornaro M, Iannone F (2019) A Bayesian mixed treatment comparison of efficacy of biologics and small molecules in early rheumatoid arthritis. Clin Rheumatol 38(5):1309–1317. https://doi.org/10.1007/s10067-018-04406-z

    Article  PubMed  Google Scholar 

  27. Orr C, Vieira-Sousa E, Boyle DL, Buch MH, Buckley CD, Canete JD, Catrina AI, Choy EHS, Emery P, Fearon U, Filer A, Gerlag D, Humby F, Isaacs JD, Just SA, Lauwerys BR, Le Goff B, Manzo A, McGarry T, McInnes IB, Najm A, Pitzalis C, Pratt A, Smith M, Tak PP, Thurlings R, Fonseca JE, Veale DJ (2017) Synovial tissue research: a state-of-the-art review. Nat Rev Rheumatol 13(10):630. https://doi.org/10.1038/nrrheum.2017.161

    Article  PubMed  Google Scholar 

  28. Humby F, Lewis M, Ramamoorthi N, Hackney JA, Barnes MR, Bombardieri M, Setiadi AF, Kelly S, Bene F, DiCicco M, Riahi S, Rocher V, Ng N, Lazarou I, Hands R, van der Heijde D, Landewe RBM, van der Helm-van MA, Cauli A, McInnes I, Buckley CD, Choy EH, Taylor PC, Townsend MJ, Pitzalis C (2019) Synovial cellular and molecular signatures stratify clinical response to csDMARD therapy and predict radiographic progression in early rheumatoid arthritis patients. Ann Rheum Dis 78(6):761–772. https://doi.org/10.1136/annrheumdis-2018-214539

    Article  CAS  PubMed  Google Scholar 

Download references

Funding

Authors declare no financial supports for this work.

Author information

Author notes

  1. Vincenzo Venerito and Orazio Angelini joint first authors.

Authors and Affiliations

  1. Department of Emergency and Organ Transplantations-Rheumatology Unit, University of Bari “Aldo Moro”, Bari, Italy

    Vincenzo Venerito, Giuseppe Lopalco & Florenzo Iannone

  2. King’s College London, London, UK

    Orazio Angelini

  3. Amazon Research Cambridge, Cambridge, UK

    Orazio Angelini

  4. Department of Emergency and Organ Transplantations-Pathology Unit, University of Bari “Aldo Moro”, Bari, Italy

    Gerardo Cazzato, Eugenio Maiorano & Antonietta Cimmino

Authors
  1. Vincenzo Venerito
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Orazio Angelini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerardo Cazzato
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giuseppe Lopalco
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eugenio Maiorano
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Antonietta Cimmino
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Florenzo Iannone
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

VV: Conceptualization; VV, GC, GL, AC: Data curation; VV, OA: Formal analysis; No Funding acquisition; VV, OA, EM, FI: Investigation; VV, OA: Methodology; VV: Project administration; VV: Resources; VV: Software; EM, FI; Supervision; VV, OA: Validation; VV, OA: Visualization; VV, OA, FI: Roles/Writing—original draft; VV, OA, FI: Writing—review and editing. VV and OA are joint first authors. This work is not related to OA’s Amazon employment.

Corresponding author

Correspondence to Giuseppe Lopalco.

Ethics declarations

Competing interests

The authors declare that they have no conflict of interest.

Statement of human and animal rights

This study was performed according to the Declaration of Helsinki and study reached the approval by the ethical committee at the University of Bari, Italy as part of the Biopure registry (IRB approval n 5277/2017). The procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation as required by the applicable law.

Informed consent

Patients signed informed consent regarding publishing their data and histopathological imagess.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 28 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Venerito, V., Angelini, O., Cazzato, G. et al. A convolutional neural network with transfer learning for automatic discrimination between low and high-grade synovitis: a pilot study. Intern Emerg Med 16, 1457–1465 (2021). https://doi.org/10.1007/s11739-020-02583-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11739-020-02583-x

Keyword

Search

Navigation