Skip to main content

Advertisement

SpringerLink
Log in

External validation of a machine learning based algorithm to differentiate hepatic mucinous cystic neoplasms from benign hepatic cysts

  • Hepatobiliary
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

To externally validate an algorithm for non-invasive differentiation of hepatic mucinous cystic neoplasms (MCN) from benign hepatic cysts (BHC), which differ in management.

Methods

Patients with cystic liver lesions pathologically confirmed as MCN or BHC between January 2005 and March 2022 from multiple institutions were retrospectively included. Five readers (2 radiologists, 3 non-radiologist physicians) independently reviewed contrast-enhanced CT or MRI examinations before tissue sampling and applied the 3-feature classification algorithm described by Hardie et al. to differentiate between MCN and BHC, which had a reported accuracy of 93.5%. The classification was then compared to the pathology results. Interreader agreement between readers across different levels of experience was evaluated with Fleiss’ Kappa.

Results

The final cohort included 159 patients, median age of 62 years (IQR [52.0, 70.0]), 66.7% female (106). Of all patients, 89.3% (142) had BHC, and the remaining 10.7% (17) had MCN on pathology. Agreement for class designation between the radiologists was almost perfect (Fleiss’ Kappa 0.840, p < 0.001). The algorithm had an accuracy of 98.1% (95% CI [94.6%, 99.6%]), a positive predictive value of 100.0% (95% CI [76.8%, 100.0%]), a negative predictive value of 97.9% (95% CI [94.1%, 99.6%]), and an area under the receiver operator characteristic curve (AUC) of 0.911 (95% CI [0.818, 1.000]).

Conclusion

The evaluated algorithm showed similarly high diagnostic accuracy in our external, multi-institutional validation cohort. This 3-feature algorithm is easily and rapidly applied and its features are reproducible among radiologists, showing promise as a clinical decision support tool.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The final de-identified dataset and the code used for the analyses reported in this study are available from the corresponding author upon reasonable request.

References

  1. Soni S, Pareek P, Narayan S, Varshney V (2021) Mucinous cystic neoplasm of the liver (MCN-L): a rare presentation and review of the literature. Med Pharm Rep 94:366–371. https://doi.org/10.15386/mpr-1543

  2. Mavilia MG, Pakala T, Molina M, Wu GY (2018) Differentiating cystic liver lesions: A review of imaging modalities, diagnosis and management. J Clin Transl Hepatol 6:1–9. https://doi.org/10.14218/jcth.2017.00069

  3. Marrero JA, Ahn J, Rajender Reddy K, Americal College of Gastroenterology (2014) ACG clinical guideline: the diagnosis and management of focal liver lesions. Am J Gastroenterol 109:1328–47; quiz 1348. https://doi.org/10.1038/ajg.2014.213

  4. Lantinga MA, Gevers TJG, Drenth JPH (2013) Evaluation of hepatic cystic lesions. World J Gastroenterol 19:3543–3554. https://doi.org/https://doi.org/10.3748/wjg.v19.i23.3543

    Article  PubMed  PubMed Central  Google Scholar 

  5. Vachha B, Sun MRM, Siewert B, Eisenberg RL (2011) Cystic lesions of the liver. AJR Am J Roentgenol 196:W355-66. https://doi.org/https://doi.org/10.2214/AJR.10.5292

    Article  PubMed  Google Scholar 

  6. Hardie AD, Chamberlin JH, Boyum JH, Sharbidre KG, Petrocelli R, Flemming BP, Zahid M, Venkatesh SK, Mruthyunjayappa S, Hajdu CH, Kovacs MD (2022) Multi-center follow-up study to develop a classification system which differentiates mucinous cystic neoplasm of the liver and benign hepatic cyst using machine learning. Acad Radiol 29:1149–1156. https://doi.org/https://doi.org/10.1016/j.acra.2021.08.025

    Article  PubMed  Google Scholar 

  7. Dua MM, Gerry J, Salles A, Tran TB, Triadafilopoulos G, Visser BC (2016) Biliary cystadenoma: A suggested “cystamatic” approach? Dig Dis Sci 61:1835–1838. https://doi.org/https://doi.org/10.1007/s10620-015-3943-y

    Article  PubMed  Google Scholar 

  8. Xu H-X, Lu M-D, Liu L-N, Zhang Y-F, Guo L-H, Liu C, Wang S (2012) Imaging features of intrahepatic biliary cystadenoma and cystadenocarcinoma on B-mode and contrast-enhanced ultrasound. Ultraschall Med 33:E241–E249. https://doi.org/https://doi.org/10.1055/s-0031-1299276

    Article  PubMed  Google Scholar 

  9. Dong Y, Wang W-P, Mao F, Fan M, Ignee A, Serra C, Sparchez Z, Sporea I, Braden B, Dietrich CF (2017) Contrast enhanced ultrasound features of hepatic cystadenoma and hepatic cystadenocarcinoma. Scand J Gastroenterol 52:365–372. https://doi.org/https://doi.org/10.1080/00365521.2016.1259652

    Article  PubMed  Google Scholar 

  10. Park HJ, Kim SY, Kim HJ, Lee SS, Hong GS, Byun JH, Hong S-M, Lee M-G (2018) Intraductal papillary neoplasm of the bile duct: Clinical, imaging, and pathologic features. AJR Am J Roentgenol 211:67–75. https://doi.org/https://doi.org/10.2214/AJR.17.19261

    Article  PubMed  Google Scholar 

  11. Le DK, Agarwal A (2019) Intraductal papillary neoplasm of the bile duct. Proc (Bayl Univ Med Cent) 32:124–125. https://doi.org/https://doi.org/10.1080/08998280.2018.1520623

    Article  PubMed  Google Scholar 

  12. Bakoyiannis A, Delis S, Triantopoulou C, Dervenis C (2013) Rare cystic liver lesions: a diagnostic and managing challenge. World J Gastroenterol 19:7603–7619. https://doi.org/https://doi.org/10.3748/wjg.v19.i43.7603

    Article  PubMed  PubMed Central  Google Scholar 

  13. Rodriguez RM, Barrio M, Parker ML, Saeed O, Sherman S, Ceppa EP (2019) Mucinous cystic neoplasms of the liver: presence of biliary communication. J Surg Case Rep 2019:rjz364. https://doi.org/10.1093/jscr/rjz364

  14. Chi H, Hansen BE, Tang WY, Schouten JNL, Sprengers D, Taimr P, Janssen HLA, de Knegt RJ (2017) Multiple biopsy passes and the risk of complications of percutaneous liver biopsy. Eur J Gastroenterol Hepatol 29:36–41. https://doi.org/https://doi.org/10.1097/MEG.0000000000000731

    Article  PubMed  Google Scholar 

  15. Myers RP, Fong A, Shaheen AAM (2008) Utilization rates, complications and costs of percutaneous liver biopsy: a population-based study including 4275 biopsies. Liver Int 28:705–712. https://doi.org/https://doi.org/10.1111/j.1478-3231.2008.01691.x

    Article  PubMed  Google Scholar 

  16. Boyum JH, Atwell TD, Schmit GD, Poterucha JJ, Schleck CD, Harmsen WS, Kamath PS (2016) Incidence and risk factors for adverse events related to image-guided liver biopsy. Mayo Clin Proc 91:329–335. https://doi.org/https://doi.org/10.1016/j.mayocp.2015.11.015

    Article  PubMed  Google Scholar 

  17. Anderson MA, Bhati CS, Ganeshan D, Itani M (2021) Hepatobiliary mucinous cystic neoplasms and mimics. Abdom Radiol (NY). https://doi.org/https://doi.org/10.1007/s00261-021-03303-5

    Article  PubMed  Google Scholar 

  18. Kim HJ, Yu ES, Byun JH, Hong S-M, Kim KW, Lee JS, Kim SY (2014) CT differentiation of mucin-producing cystic neoplasms of the liver from solitary bile duct cysts. AJR Am J Roentgenol 202:83–91. https://doi.org/https://doi.org/10.2214/AJR.12.9170

    Article  PubMed  Google Scholar 

  19. Anderson MA, Dhami RS, Fadzen CM, Molina G, Taylor MS, Deshpande V, Qadan M, Catalano OA, Ferrone CR, Mojtahed A (2021) CT and MRI features differentiating mucinous cystic neoplasms of the liver from pathologically simple cysts. Clin Imaging 76:46–52. https://doi.org/https://doi.org/10.1016/j.clinimag.2021.01.036

    Article  PubMed  Google Scholar 

  20. Arnaoutakis DJ, Kim Y, Pulitano C, Zaydfudim V, Squires MH, Kooby D, Groeschl R, Alexandrescu S, Bauer TW, Bloomston M, Soares K, Marques H, Gamblin TC, Popescu I, Adams R, Nagorney D, Barroso E, Maithel SK, Crawford M, Sandroussi C, Marsh W, Pawlik TM (2015) Management of biliary cystic tumors: a multi-institutional analysis of a rare liver tumor. Ann Surg 261:361–367. https://doi.org/https://doi.org/10.1097/SLA.0000000000000543

    Article  PubMed  Google Scholar 

  21. Choi HK, Lee JK, Lee KH, Lee KT, Rhee JC, Kim KH, Jang K-T, Kim SH, Park Y (2010) Differential diagnosis for intrahepatic biliary cystadenoma and hepatic simple cyst: significance of cystic fluid analysis and radiologic findings. J Clin Gastroenterol 44:289–293. https://doi.org/https://doi.org/10.1097/MCG.0b013e3181b5c789

    Article  PubMed  Google Scholar 

  22. Seo JK, Kim SH, Lee SH, Park JK, Woo SM, Jeong JB, Hwang J-H, Ryu JK, Kim J-W, Jeong S-H, Kim Y-T, Yoon YB, Lee KU, Kim SH, Kim MA (2010) Appropriate diagnosis of biliary cystic tumors: comparison with atypical hepatic simple cysts. Eur J Gastroenterol Hepatol 22:989–996. https://doi.org/https://doi.org/10.1097/MEG.0b013e328337c971

    Article  PubMed  Google Scholar 

  23. Kim JY, Kim SH, Eun HW, Lee MW, Lee JY, Han JK, Choi BI (2010) Differentiation between biliary cystic neoplasms and simple cysts of the liver: accuracy of CT. AJR Am J Roentgenol 195:1142–1148. https://doi.org/https://doi.org/10.2214/AJR.09.4026

    Article  PubMed  Google Scholar 

  24. Kovacs MD, Sheafor DH, Burchett PF, Picard MM, Hardie AD (2018) Differentiating biliary cystadenomas from benign hepatic cysts: Preliminary analysis of new predictive imaging features. Clin Imaging 49:44–47. https://doi.org/https://doi.org/10.1016/j.clinimag.2017.10.022

    Article  PubMed  Google Scholar 

  25. Boyum JH, Sheedy SP, Graham RP, Olson JT, Babcock AT, Bolan CW, Menias CO, Venkatesh SK (2021) Hepatic mucinous cystic neoplasm versus simple biliary cyst: Assessment of distinguishing imaging features using CT and MRI. AJR Am J Roentgenol 216:403–411. https://doi.org/https://doi.org/10.2214/AJR.20.22768

    Article  PubMed  Google Scholar 

  26. Bleeker SE, Moll HA, Steyerberg EW, Donders ART, Derksen-Lubsen G, Grobbee DE, Moons KGM (2003) External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56:826–832. https://doi.org/https://doi.org/10.1016/s0895-4356(03)00207-5

    Article  CAS  PubMed  Google Scholar 

  27. Austin PC, Steyerberg EW (2017) Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat Methods Med Res 26:796–808. https://doi.org/https://doi.org/10.1177/0962280214558972

    Article  PubMed  Google Scholar 

  28. Steyerberg EW, Harrell FE Jr (2016) Prediction models need appropriate internal, internal–external, and external validation. J Clin Epidemiol 69:245–247. https://doi.org/https://doi.org/10.1016/j.jclinepi.2015.04.005

    Article  PubMed  Google Scholar 

  29. Steyerberg EW, Harrell FE Jr, Borsboom GJJM, Eijkemans MJC, Vergouwe Y, Habbema JDF (2001) Internal validation of predictive models. J Clin Epidemiol 54:774–781. https://doi.org/https://doi.org/10.1016/s0895-4356(01)00341-9

    Article  CAS  PubMed  Google Scholar 

  30. McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med 22:276–282

    Article  Google Scholar 

  31. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, Müller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77. https://doi.org/https://doi.org/10.1186/1471-2105-12-77

    Article  PubMed  PubMed Central  Google Scholar 

  32. R Core Team (2022) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing

  33. RStudio Team (2022) RStudio: Integrated Development Environment for R. RStudio, PBC

  34. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174. https://doi.org/https://doi.org/10.2307/2529310

    Article  CAS  PubMed  Google Scholar 

  35. Yan W, Huang L, Xia L, Gu S, Yan F, Wang Y, Tao Q (2020) MRI manufacturer shift and adaptation: Increasing the generalizability of deep learning segmentation for MR images acquired with different scanners. Radiol Artif Intell 2:e190195. https://doi.org/10.1148/ryai.2020190195

  36. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK (2018) Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Med 15:e1002683. https://doi.org/10.1371/journal.pmed.1002683

  37. Shah S, Mishra R, Szczurowska A, Guziński M (2021) Non-invasive multi-channel deep learning convolutional neural networks for localization and classification of common hepatic lesions. Pol J Radiol 86:e440–e448. https://doi.org/https://doi.org/10.5114/pjr.2021.108257

    Article  PubMed  PubMed Central  Google Scholar 

  38. Yin J, Qiu J-J, Qian W, Ji L, Yang D, Jiang J-W, Wang J-R, Lan L (2020) A radiomics signature to identify malignant and benign liver tumors on plain CT images. J Xray Sci Technol 28:683–694. https://doi.org/https://doi.org/10.3233/XST-200675

    Article  PubMed  Google Scholar 

  39. Halimu C, Kasem A, Newaz SHS (2019) Empirical comparison of area under ROC curve (AUC) and Mathew correlation coefficient (MCC) for evaluating machine learning algorithms on imbalanced datasets for binary classification. In: Proceedings of the 3rd International Conference on Machine Learning and Soft Computing - ICMLSC 2019. ACM Press, New York, New York, USA

Download references

Acknowledgements

The authors thank Shawn Murphy, Henry Chueh, and the Mass General Brigham Health Care Research Patient Data Registry group for facilitating the use of their database.

Funding

No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, 02114, USA

    Felipe S. Furtado, Álvaro Badenes-Romero, Mina Hesami, Leila Mostafavi, Zahra Najmi, Amirkasra Mojtahed, Mark A. Anderson & Onofrio A. Catalano

  2. Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, 02129, USA

    Felipe S. Furtado, Álvaro Badenes-Romero, Mina Hesami, Zahra Najmi, Amirkasra Mojtahed & Onofrio A. Catalano

  3. Department of Nuclear Medicine, Hospital Universitario de Tarragona Juan XXIII, Tarragona, Spain

    Álvaro Badenes-Romero

  4. Hospital das Clínicas HCFMUSP, Sao Paulo, SP, Brazil

    Marcelo Queiroz

Authors
  1. Felipe S. Furtado
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Álvaro Badenes-Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mina Hesami
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leila Mostafavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zahra Najmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marcelo Queiroz
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Amirkasra Mojtahed
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mark A. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Onofrio A. Catalano
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by FSF, ÁBR, MH, and LM. The first draft of the manuscript was written by FSF, ÁBR, MAA, and OAC. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Onofrio A. Catalano.

Ethics declarations

Competing interests

The authors have no financial or proprietary interests in any material discussed in this article.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This study was approved by the institutional review board.

Informed consent

Informed consent was waived due to the retrospective nature of the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Furtado, F.S., Badenes-Romero, Á., Hesami, M. et al. External validation of a machine learning based algorithm to differentiate hepatic mucinous cystic neoplasms from benign hepatic cysts. Abdom Radiol 48, 2311–2320 (2023). https://doi.org/10.1007/s00261-023-03907-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-023-03907-z

Keywords

Search

Navigation