Deep convolutional neural network applied to the liver imaging reporting and data system (LI-RADS) version 2014 category classification: a pilot study

  • Rikiya Yamashita
  • Amber Mittendorf
  • Zhe Zhu
  • Kathryn J. Fowler
  • Cynthia S. Santillan
  • Claude B. Sirlin
  • Mustafa R. Bashir
  • Richard K. G. DoEmail author



To develop a deep convolutional neural network (CNN) model to categorize multiphase CT and MRI liver observations using the liver imaging reporting and data system (LI-RADS) (version 2014).


A pre-existing dataset comprising 314 hepatic observations (163 CT, 151 MRI) with corresponding diameters and LI-RADS categories (LR-1–5) assigned in consensus by two LI-RADS steering committee members was used to develop two CNNs: pre-trained network with an input of triple-phase images (training with transfer learning) and custom-made network with an input of quadruple-phase images (training from scratch). The dataset was randomly split into training, validation, and internal test sets (70:15:15 split). The overall accuracy and area under receiver operating characteristic curve (AUROC) were assessed for categorizing LR-1/2, LR-3, LR-4, and LR-5. External validation was performed for the model with the better performance on the internal test set using two external datasets (EXT-CT and EXT-MR: 68 and 44 observations, respectively).


The transfer learning model outperformed the custom-made model: overall accuracy of 60.4% and AUROCs of 0.85, 0.90, 0.63, 0.82 for LR-1/2, LR-3, LR-4, LR-5, respectively. On EXT-CT, the model had an overall accuracy of 41.2% and AUROCs of 0.70, 0.66, 0.60, 0.76 for LR-1/2, LR-3, LR-4, LR-5, respectively. On EXT-MR, the model had an overall accuracy of 47.7% and AUROCs of 0.88, 0.74, 0.69, 0.79 for LR-1/2, LR-3, LR-4, LR-5, respectively.


Our study shows the feasibility of CNN for assigning LI-RADS categories from a relatively small dataset but highlights the challenges of model development and validation.


Hepatocellular carcinoma Deep learning X-ray computed tomography Magnetic resonance imaging 



Liver imaging reporting and data system


Hepatocellular carcinoma


Convolutional neural network


Region of interest


Area under receiver operating characteristic curve



We thank Joanne Chin for editorial assistance.


Supported by JSPS Overseas Research Fellowships (R.Y.) (Japan Society for the Promotion of Science (JSPS/OT/290125)) and the National Institutes of Health/National Cancer Institute Cancer Center Support Grant P30 CA008748 (R.Y. and R.K.G.D.).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

For this type of study, formal consent is not required.

Supplementary material

261_2019_2306_MOESM1_ESM.pdf (64 kb)
Supplementary material 1 (PDF 63 kb)
261_2019_2306_MOESM2_ESM.pdf (37 kb)
Supplementary material 2 (PDF 36 kb)


  1. 1.
    American College of Radiology ACR LI-RADS v2014. Accessed December 11, 2018.
  2. 2.
    Fowler KJ, Tang A, Santillan C, Bhargavan-Chatfield M, Heiken J, Jha RC, Weinreb J, Hussain H, Mitchell DG, Bashir MR, Costa EAC, Cunha GM, Coombs L, Wolfson T, Gamst AC, Brancatelli G, Yeh B, Sirlin CB (2018) Interreader Reliability of LI-RADS Version 2014 Algorithm and Imaging Features for Diagnosis of Hepatocellular Carcinoma: A Large International Multireader Study. Radiology 286 (1):173-185. CrossRefPubMedGoogle Scholar
  3. 3.
    Schellhaas B, Hammon M, Strobel D, Pfeifer L, Kielisch C, Goertz RS, Cavallaro A, Janka R, Neurath MF, Uder M, Seuss H (2018) Interobserver and intermodality agreement of standardized algorithms for non-invasive diagnosis of hepatocellular carcinoma in high-risk patients: CEUS-LI-RADS versus MRI-LI-RADS. European radiology 28 (10):4254-4264. CrossRefPubMedGoogle Scholar
  4. 4.
    Barth BK, Donati OF, Fischer MA, Ulbrich EJ, Karlo CA, Becker A, Seifert B, Reiner CS (2016) Reliability, Validity, and Reader Acceptance of LI-RADS-An In-depth Analysis. Academic radiology 23 (9):1145-1153. CrossRefPubMedGoogle Scholar
  5. 5.
    Davenport MS, Khalatbari S, Liu PS, Maturen KE, Kaza RK, Wasnik AP, Al-Hawary MM, Glazer DI, Stein EB, Patel J, Somashekar DK, Viglianti BL, Hussain HK (2014) Repeatability of diagnostic features and scoring systems for hepatocellular carcinoma by using MR imaging. Radiology 272 (1):132-142. CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR (2016) Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Jama 316 (22):2402-2410. CrossRefPubMedGoogle Scholar
  7. 7.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542 (7639):115-118. CrossRefPubMedGoogle Scholar
  8. 8.
    Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J, Hermsen M, Manson QF, Balkenhol M, Geessink O, Stathonikos N, van Dijk MC, Bult P, Beca F, Beck AH, Wang D, Khosla A, Gargeya R, Irshad H, Zhong A, Dou Q, Li Q, Chen H, Lin HJ, Heng PA, Hass C, Bruni E, Wong Q, Halici U, Oner MU, Cetin-Atalay R, Berseth M, Khvatkov V, Vylegzhanin A, Kraus O, Shaban M, Rajpoot N, Awan R, Sirinukunwattana K, Qaiser T, Tsang YW, Tellez D, Annuscheit J, Hufnagl P, Valkonen M, Kartasalo K, Latonen L, Ruusuvuori P, Liimatainen K, Albarqouni S, Mungal B, George A, Demirci S, Navab N, Watanabe S, Seno S, Takenaka Y, Matsuda H, Ahmady Phoulady H, Kovalev V, Kalinovsky A, Liauchuk V, Bueno G, Fernandez-Carrobles MM, Serrano I, Deniz O, Racoceanu D, Venancio R (2017) Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. Jama 318 (22):2199-2210. CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, Mahajan V, Rao P, Warier P (2018) Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet (London, England) 392 (10162):2388-2396. CrossRefGoogle Scholar
  10. 10.
    De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, Askham H, Glorot X, O’Donoghue B, Visentin D, van den Driessche G, Lakshminarayanan B, Meyer C, Mackinder F, Bouton S, Ayoub K, Chopra R, King D, Karthikesalingam A, Hughes CO, Raine R, Hughes J, Sim DA, Egan C, Tufail A, Montgomery H, Hassabis D, Rees G, Back T, Khaw PT, Suleyman M, Cornebise J, Keane PA, Ronneberger O (2018) Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine 24 (9):1342-1350. CrossRefPubMedGoogle Scholar
  11. 11.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv. vol 1409.1556.
  12. 12.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115 (3):211-252. CrossRefGoogle Scholar
  13. 13.
    Yamashita R, Nishio M, Do RKG, Togashi K (2018) Convolutional neural networks: an overview and application in radiology. Insights into imaging 9 (4):611-629. CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Kluyver T, Ragan-Kelley B, Pérez F, Granger BE, Bussonnier M, Frederic J, Kelley K, Hamrick JB, Grout J, Corlay S (2016) Jupyter Notebooks-a publishing format for reproducible computational workflows. In: Loizides F, Scmidt B (eds) Positioning and Power in Academic Publishing: Players, Agents and Agendas. IOS Press, pp 87-90.
  15. 15.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33 (1):159-174. CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Jones E, Oliphant T, Peterson P, et al SciPy: open source scientific tools for Python, 2001-, Accessed on December 10, 2018.
  17. 17.
    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V (2011) Scikit-learn: Machine learning in Python. Journal of machine learning research 12:2825-2830. doi:Not availableGoogle Scholar
  18. 18.
    R Development Core Team (2008) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, AustriaGoogle Scholar
  19. 19.
    Yasaka K, Akai H, Abe O, Kiryu S (2018) Deep Learning with Convolutional Neural Network for Differentiation of Liver Masses at Dynamic Contrast-enhanced CT: A Preliminary Study. Radiology 286 (3):887-896. CrossRefPubMedGoogle Scholar
  20. 20.
    Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK (2018) Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS medicine 15 (11):e1002683. CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Park SH (2019) Diagnostic Case-Control versus Diagnostic Cohort Studies for Clinical Validation of Artificial Intelligence Algorithm Performance. Radiology 290 (1):272-273. CrossRefPubMedGoogle Scholar
  22. 22.
    Jha RC, Mitchell DG, Weinreb JC, Santillan CS, Yeh BM, Francois R, Sirlin CB (2014) LI-RADS categorization of benign and likely benign findings in patients at risk of hepatocellular carcinoma: a pictorial atlas. AJR American journal of roentgenology 203 (1):W48-69. CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Radiology, Body Imaging ServiceMemorial Sloan Kettering Cancer CenterNew YorkUSA
  2. 2.Department of Radiology, Center for Advanced Magnetic Resonance DevelopmentDuke University Medical CenterDurhamUSA
  3. 3.Liver Imaging Group, Department of RadiologyUniversity of California San DiegoSan DiegoUSA
  4. 4.Center for Advanced Magnetic Resonance DevelopmentDuke University Medical CenterDurhamUSA
  5. 5.Division of Gastroenterology, Department of MedicineDuke University Medical CenterDurhamUSA

Personalised recommendations