Skip to main content

Advertisement

SpringerLink
Log in

Assessing the robustness of a machine-learning model for early detection of pancreatic adenocarcinoma (PDA): evaluating resilience to variations in image acquisition and radiomics workflow using image perturbation methods

  • Pancreas
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

To evaluate robustness of a radiomics-based support vector machine (SVM) model for detection of visually occult PDA on pre-diagnostic CTs by simulating common variations in image acquisition and radiomics workflow using image perturbation methods.

Methods

Eighteen algorithmically generated-perturbations, which simulated variations in image noise levels (σ, 2σ, 3σ, 5σ), image rotation [both CT image and the corresponding pancreas segmentation mask by 45° and 90° in axial plane], voxel resampling (isotropic and anisotropic), gray-level discretization [bin width (BW) 32 and 64)], and pancreas segmentation (sequential erosions by 3, 4, 6, and 8 pixels and dilations by 3, 4, and 6 pixels from the boundary), were introduced to the original (unperturbed) test subset (n = 128; 45 pre-diagnostic CTs, 83 control CTs with normal pancreas). Radiomic features were extracted from pancreas masks of these additional test subsets, and the model's performance was compared vis-a-vis the unperturbed test subset.

Results

The model correctly classified 43 out of 45 pre-diagnostic CTs and 75 out of 83 control CTs in the unperturbed test subset, achieving 92.2% accuracy and 0.98 AUC. Model's performance was unaffected by a three-fold increase in noise level except for sensitivity declining to 80% at 3σ (p = 0.02). Performance remained comparable vis-a-vis the unperturbed test subset despite variations in image rotation (p = 0.99), voxel resampling (p = 0.25–0.31), change in gray-level BW to 32 (p = 0.31–0.99), and erosions/dilations up to 4 pixels from the pancreas boundary (p = 0.12–0.34).

Conclusion

The model’s high performance for detection of visually occult PDA was robust within a broad range of clinically relevant variations in image acquisition and radiomics workflow.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Singh DP, Sheedy S, Goenka AH, Wells M, Lee NJ, Barlow J, et al. (2020) Computerized tomography scan in pre-diagnostic pancreatic ductal adenocarcinoma: Stages of progression and potential benefits of early intervention: A retrospective study. Pancreatology 20(7):1495-501 https://doi.org/10.1016/j.pan.2020.07.410.

    Article  PubMed  Google Scholar 

  2. Chari ST, Kelly K, Hollingsworth MA, Thayer SP, Ahlquist DA, Andersen DK, et al. (2015) Early detection of sporadic pancreatic cancer: summative review. Pancreas 44(5):693-712 https://doi.org/10.1097/MPA.0000000000000368.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Chhoda A, Vodusek Z, Wattamwar K, Mukherjee E, Gunderson C, Grimshaw A, et al. (2022) Late-Stage Pancreatic Cancer Detected During High-Risk Individual Surveillance: A Systematic Review and Meta-Analysis. Gastroenterology 162(3):786-98 https://doi.org/10.1053/j.gastro.2021.11.021.

    Article  PubMed  CAS  Google Scholar 

  4. Klatte DCF, Boekestijn B, Onnekink AM, Dekker FW, van der Geest LG, Wasser M, et al. (2023) Surveillance for Pancreatic Cancer in High-Risk Individuals Leads to Improved Outcomes: A Propensity Score-Matched Analysis. Gastroenterology 164(7):1223-31 e4 https://doi.org/10.1053/j.gastro.2023.02.032.

    Article  PubMed  Google Scholar 

  5. Overbeek KA, Goggins MG, Dbouk M, Levink IJM, Koopmann BDM, Chuidian M, et al. (2022) Timeline of Development of Pancreatic Cancer and Implications for Successful Early Detection in High-Risk Individuals. Gastroenterology 162(3):772-85 e4 https://doi.org/10.1053/j.gastro.2021.10.014.

    Article  PubMed  Google Scholar 

  6. Kurita Y, Kuwahara T, Hara K, Mizuno N, Okuno N, Matsumoto S, et al. (2019) Diagnostic ability of artificial intelligence using deep learning analysis of cyst fluid in differentiating malignant from benign pancreatic cystic lesions. Sci Rep 9(1):6893 https://doi.org/10.1038/s41598-019-43314-3.

    Article  ADS  PubMed  PubMed Central  CAS  Google Scholar 

  7. Kang J, Clarke SE, Abdolell M, Ramjeesingh R, Payne J, Costa AF (2021) The implications of missed or misinterpreted cases of pancreatic ductal adenocarcinoma on imaging: a multi-centered population-based study. Eur Radiol 31(1):212-21 https://doi.org/10.1007/s00330-020-07120-0.

    Article  PubMed  Google Scholar 

  8. Kang JD, Clarke SE, Costa AF (2021) Factors associated with missed and misinterpreted cases of pancreatic ductal adenocarcinoma. Eur Radiol 31(4):2422-32 https://doi.org/10.1007/s00330-020-07307-5.

    Article  PubMed  CAS  Google Scholar 

  9. Dewitt J, Devereaux BM, Lehman GA, Sherman S, Imperiale TF (2006) Comparison of endoscopic ultrasound and computed tomography for the preoperative evaluation of pancreatic cancer: a systematic review. Clin Gastroenterol Hepatol 4(6):717-25; quiz 664 https://doi.org/10.1016/j.cgh.2006.02.020.

    Article  PubMed  Google Scholar 

  10. Toshima F, Watanabe R, Inoue D, Yoneda N, Yamamoto T, Sasahira N, et al. (2021) CT Abnormalities of the Pancreas Associated With the Subsequent Diagnosis of Clinical Stage I Pancreatic Ductal Adenocarcinoma More Than 1 Year Later: A Case-Control Study. AJR Am J Roentgenol:1-12 https://doi.org/10.2214/AJR.21.26014.

    Article  PubMed  Google Scholar 

  11. Mukherjee S, Patra A, Khasawneh H, Korfiatis P, Rajamohan N, Suman G, et al. (2022) Radiomics-based Machine-learning models can detect pancreatic cancer on prediagnostic computed tomography scans at a substantial lead time before clinical diagnosis. Gastroenterology 163(5):1435-46 e3 https://doi.org/10.1053/j.gastro.2022.06.066.

    Article  PubMed  Google Scholar 

  12. Zwanenburg A, Leger S, Agolli L, Pilz K, Troost EGC, Richter C, et al. (2019) Assessing robustness of radiomic features by image perturbation. Sci Rep 9(1):614 https://doi.org/10.1038/s41598-018-36938-4.

    Article  ADS  PubMed  PubMed Central  CAS  Google Scholar 

  13. Zhao B (2021) Understanding Sources of Variation to Improve the Reproducibility of Radiomics. Front Oncol 11:633176 https://doi.org/10.3389/fonc.2021.633176.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. (2015) Measuring Computed Tomography Scanner Variability of Radiomics Features. Invest Radiol 50(11):757-65 https://doi.org/10.1097/RLI.0000000000000180.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Yip SS, Aerts HJ (2016) Applications and limitations of radiomics. Phys Med Biol 61(13):R150-66 https://doi.org/10.1088/0031-9155/61/13/R150.

    Article  ADS  PubMed  PubMed Central  CAS  Google Scholar 

  16. Tixier F, Hatt M, Le Rest CC, Le Pogam A, Corcos L, Visvikis D (2012) Reproducibility of tumor uptake heterogeneity characterization through textural feature analysis in 18F-FDG PET. J Nucl Med 53(5):693-700 https://doi.org/10.2967/jnumed.111.099127.

    Article  PubMed  Google Scholar 

  17. Leijenaar RT, Carvalho S, Velazquez ER, van Elmpt WJ, Parmar C, Hoekstra OS, et al. (2013) Stability of FDG-PET Radiomics features: an integrated analysis of test-retest and inter- observer variability. Acta Oncol 52(7):1391-7 https://doi.org/10.3109/0284186X.2013.812798.

    Article  PubMed  CAS  Google Scholar 

  18. Teng X, Zhang J, Zwanenburg A, Sun J, Huang Y, Lam S, et al. (2022) Building reliable radiomic models using image perturbation. Sci Rep 12(1):10035 https://doi.org/10.1038/s41598-022-14178-x.

    Article  ADS  PubMed  PubMed Central  CAS  Google Scholar 

  19. van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. (2017) Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res 77(21):e104-e7 https://doi.org/10.1158/0008-5472.CAN-17-0339.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  20. Tibshirani R (1997) The lasso method for variable selection in the Cox model. Stat Med 16(4):385-95 https://doi.org/10.1002/(sici)1097-0258(19970228)16:4<385::aid-sim380>3.0.co;2-3.

    Article  PubMed  CAS  Google Scholar 

  21. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837-45

    Article  PubMed  CAS  Google Scholar 

  22. Qureshi TA, Gaddam S, Wachsman AM, Wang L, Azab L, Asadpour V, et al. (2022) Predicting pancreatic ductal adenocarcinoma using artificial intelligence analysis of pre- diagnostic computed tomography images. Cancer Biomark 33(2):211-7 https://doi.org/10.3233/CBM-210273.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  23. Oliver JA, Budzevich M, Hunt D, Moros EG, Latifi K, Dilling TJ, et al. (2017) Sensitivity of Image Features to Noise in Conventional and Respiratory-Gated PET/CT Images of Lung Cancer: Uncorrelated Noise Effects. Technol Cancer Res Treat 16(5):595-608 https://doi.org/10.1177/1533034616661852.

    Article  PubMed  Google Scholar 

  24. Suman G, Patra A, Mukherjee S, Korffiatis P, Goenka AH (2022) Radiomics for Detection of Pancreas Adenocarcinoma on CT Scans: Impact of Biliary Stents. Radiol Imaging Cancer 4(1):e210081 https://doi.org/10.1148/rycan.210081.

    Article  PubMed  Google Scholar 

  25. Huang K, Rhee DJ, Ger R, Layman R, Yang J, Cardenas CE, et al. (2021) Impact of slice thickness, pixel size, and CT dose on the performance of automatic contouring algorithms. J Appl Clin Med Phys 22(5):168-74 https://doi.org/10.1002/acm2.13207.

    Article  PubMed  PubMed Central  Google Scholar 

  26. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B (2020) Radiomics in medical imaging-"how-to" guide and critical reflection. Insights Imaging 11(1):91 https://doi.org/10.1186/s13244-020-00887-2.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Haarburger C, Muller-Franzes G, Weninger L, Kuhl C, Truhn D, Merhof D (2020) Radiomics feature reproducibility under inter-rater variability in segmentations of CT images. Sci Rep 10(1):12688 https://doi.org/10.1038/s41598-020-69534-6.

    Article  ADS  PubMed  PubMed Central  CAS  Google Scholar 

  28. Wright DE, Mukherjee S, Patra A, Khasawneh H, Korfiatis P, Suman G, et al. (2022) Radiomics-based machine learning (ML) classifier for detection of type 2 diabetes on standard- of-care abdomen CTs: a proof-of-concept study. Abdom Radiol (NY) 47(11):3806-16 https://doi.org/10.1007/s00261-022-03668-1.

    Article  PubMed  Google Scholar 

  29. Panda A, Korfiatis P, Suman G, Garg SK, Polley EC, Singh DP, et al. (2021) Two-stage deep learning model for fully automated pancreas segmentation on computed tomography: Comparison with intra-reader and inter-reader reliability at full and reduced radiation dose on an external dataset. Med Phys 48(5):2468-81 https://doi.org/10.1002/mp.14782.

    Article  PubMed  Google Scholar 

  30. Khasawneh H, Patra A, Rajamohan N, Suman G, Klug J, Majumder S, et al. (2022) Volumetric Pancreas Segmentation on Computed Tomography: Accuracy and Efficiency of a Convolutional Neural Network Versus Manual Segmentation in 3D Slicer in the Context of Interreader Variability of Expert Radiologists. J Comput Assist Tomogr 46(6):841-7 https://doi.org/10.1097/RCT.0000000000001374.

    Article  PubMed  Google Scholar 

  31. Korfiatis P, Suman G, Patnam NG, Trivedi KH, Karbhari A, Mukherjee S, et al. (2023) Automated Artificial Intelligence Model Trained on a Large Data Set Can Detect Pancreas Cancer on Diagnostic Computed Tomography Scans As Well As Visually Occult Preinvasive Cancer on Prediagnostic Computed Tomography Scans. Gastroenterology https://doi.org/10.1053/j.gastro.2023.08.034.

    Article  PubMed  Google Scholar 

  32. Teng X, Zhang J, Ma Z, Zhang Y, Lam S, Li W, et al. (2022) Improving radiomic model reliability using robust features from perturbations for head-and-neck carcinoma. Front Oncol 12:974467 https://doi.org/10.3389/fonc.2022.974467.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Chari ST, Maitra A, Matrisian LM, Shrader EE, Wu BU, Kambadakone A, et al. (2022) Early Detection Initiative: A randomized controlled trial of algorithm-based screening in patients with new onset hyperglycemia and diabetes for early detection of pancreatic ductal adenocarcinoma. Contemp Clin Trials 113:106659 https://doi.org/10.1016/j.cct.2021.106659.

    Article  PubMed  Google Scholar 

Download references

Funding

Dr. Goenka acknowledges grants from non-profit entities such as the Champions for Hope Pancreatic Cancer Research Program of the Funk Zitiello Foundation, the Centene Charitable Foundation, and the Advance the Practice Award from the Department of Radiology, Mayo Clinic, Rochester, Minnesota. Unrelated to this work (Dr. Goenka): CA190188, Department of Defense (DoD), Office of the Congressionally Directed Medical Research Programs (CDMRP); R01CA256969, National Cancer Institute (NCI) of the National Institutes of Health (NIH); R01CA272628- 01, National Cancer Institute (NCI) of the National Institutes of Health (NIH); Institutional research grant from Sofie Biosciences and Clovis Oncology; Advisory Board (ad hoc), BlueStar Genomics; Consultant, Bayer Healthcare, LLC; Consultant, Candel Therapeutics; Consultant, UWorld.

Author information

Authors and Affiliations

  1. Divisions of Abdominal and Nuclear Imaging, Nuclear Radiology Fellowship, Nuclear Radiology Research Operations, Enterprise PET/MR Research and Development, Department of Radiology, Mayo Clinic, 200 First St SW, Charlton 1, Rochester, MN, 55905, USA

    Sovanlal Mukherjee, Panagiotis Korfiatis, Nandakumar G. Patnam, Kamaxi H. Trivedi, Aashna Karbhari, Garima Suman, Joel G. Fletcher & Ajit H. Goenka

Authors
  1. Sovanlal Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Panagiotis Korfiatis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nandakumar G. Patnam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kamaxi H. Trivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aashna Karbhari
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Garima Suman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Joel G. Fletcher
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ajit H. Goenka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ajit H. Goenka.

Ethics declarations

Competing interest

Authors have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mukherjee, S., Korfiatis, P., Patnam, N.G. et al. Assessing the robustness of a machine-learning model for early detection of pancreatic adenocarcinoma (PDA): evaluating resilience to variations in image acquisition and radiomics workflow using image perturbation methods. Abdom Radiol 49, 964–974 (2024). https://doi.org/10.1007/s00261-023-04127-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-023-04127-1

Keywords

Search

Navigation