Skip to main content
SpringerLink
Log in

The influence of manual segmentation strategies and different phases selection on machine learning-based computed tomography in renal tumors: a systematic review and meta-analysis

  • Computed Tomography
  • Published:
La radiologia medica Aims and scope Submit manuscript

Abstract

Background

Delineating the region/volume of interest (ROI/VOI) and selecting the phases are of importance in developing machine learning (ML). The results will change when choosing different methods of drawing the ROI/VOI and selecting different phases. However, there is no related standard for delineating the ROI/VOI and selecting the phases in renal tumors to develop ML based on computed tomography (CT).

Methods

The PubMed and Web of Science were searched for related studies published until March 1, 2023. Inclusion criteria were studies that developed ML models in renal tumors from CT images. And the binary diagnostic accuracy data were extracted to obtain the outcomes, such as sensitivity (SE), specificity (SP), accuracy (ACC), and area under the curve (AUC).

Results

Twenty-three papers were included in the meta-analysis with a pooled SE of 87% (95% CI 85–88%), SP of 82% (95% CI 79–85%), and AUC of 91% (95% CI 89–93%) in phases; a pooled SE of 82% (95% CI 80–84%), SP of 85% (95% CI 83–86%), and AUC of 90% (95% CI 88–93%) in phases combined with delineating strategies, respectively. In all different combinations, the contour-focused and single phase produce the highest AUC of 93% (95% CI 90–95%). In subgroup analyses (sample size, year of publication, and geographical distribution), the performance was acceptable on phases and phases combined strategies.

Conclusions

To explore the effect of manual segmentation strategies and different phases selection on ML-based CT, we find that the method of single phase (CMP or NP) combined with contour-focused was considered a better strategy compared to the other strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F (2021) Global Cancer Statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 71(3):209–249. https://doi.org/10.3322/caac.21660

    Article  CAS  PubMed  Google Scholar 

  2. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A (2018) Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 68(6):394–424. https://doi.org/10.3322/caac.21492

    Article  PubMed  Google Scholar 

  3. Volpe A, Panzarella T, Rendon RA, Haider MA, Kondylis FI, Jewett MA (2004) The natural history of incidentally detected small renal masses. Cancer 100(4):738–745. https://doi.org/10.1002/cncr.20025

    Article  PubMed  Google Scholar 

  4. Gill IS, Aron M, Gervais DA, Jewett MA (2010) Clinical practice: small renal mass. N Engl J Med 362(7):624–634. https://doi.org/10.1056/NEJMcp0910041

    Article  CAS  PubMed  Google Scholar 

  5. Bhandari A, Ibrahim M, Sharma C, Liong R, Gustafson S, Prior M (2021) CT-based radiomics for differentiating renal tumours: a systematic review. Abdom Radiol 46(5):2052–2063. https://doi.org/10.1007/s00261-020-02832-9

    Article  Google Scholar 

  6. Takagi T, Kondo T, Tanabe K (2011) Impact of the tumor enhancement pattern in computed tomography for the differential diagnosis of renal cell carcinoma and benign renal tumor. Int J Urol 18(12):866–867. https://doi.org/10.1111/j.1442-2042.2011.02872.x

    Article  PubMed  Google Scholar 

  7. Sheir KZ, El-Azab M, Mosbah A, El-Baz M, Shaaban AA (2005) Differentiation of renal cell carcinoma subtypes by multislice computerized tomography. J Urol 174(2):451–455. https://doi.org/10.1097/01.ju.0000165341.08396.a9. (discussion 455)

    Article  PubMed  Google Scholar 

  8. Zhang YY, Luo S, Liu Y, Xu RT (2013) Angiomyolipoma with minimal fat: differentiation from papillary renal cell carcinoma by helical CT. Clin Radiol 68(4):365–370. https://doi.org/10.1016/j.crad.2012.08.028

    Article  PubMed  Google Scholar 

  9. Luo S, Wei R, Lu S, Lai S, Wu J, Wu Z, Pang X, Wei X, Jiang X, Zhen X, Yang R (2022) Fuhrman nuclear grade prediction of clear cell renal cell carcinoma: influence of volume of interest delineation strategies on machine learning-based dynamic enhanced CT radiomics analysis. Eur Radiol 32(4):2340–2350. https://doi.org/10.1007/s00330-021-08322-w

    Article  PubMed  Google Scholar 

  10. Miskin N, Qin L, Silverman SG, Shinagare AB (2023) Differentiating benign from malignant cystic renal masses: a feasibility study of computed tomography texture-based machine learning algorithms. J Comput Assist Tomogr 47(3):376–381. https://doi.org/10.1097/rct.0000000000001433

    Article  PubMed  Google Scholar 

  11. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B (2020) Radiomics in medical imaging-"how-to" guide and critical reflection. Insights Imaging 11(1):91. https://doi.org/10.1186/s13244-020-00887-2

    Article  PubMed  PubMed Central  Google Scholar 

  12. Xu HL, Gong TT, Liu FH, Chen HY, Xiao Q, Hou Y, Huang Y, Sun HZ, Shi Y, Gao S, Lou Y, Chang Q, Zhao YH, Gao QL, Wu QJ (2022) Artificial intelligence performance in image-based ovarian cancer identification: a systematic review and meta-analysis. EClinicalMedicine 53:101662. https://doi.org/10.1016/j.eclinm.2022.101662

    Article  PubMed  PubMed Central  Google Scholar 

  13. Gharaibeh M, Alzu’bi D, Abdullah M, Hmeidi I, Al Nasar MR, Abualigah L, Gandomi AH (2022) Radiology imaging scans for early diagnosis of kidney tumors: a review of data analytics-based machine learning and deep learning approaches. Big Data Cogn Comput 6(1):29

    Article  Google Scholar 

  14. Kocak B, Kus EA, Yardimci AH, Bektas CT, Kilickesmez O (2020) Machine learning in radiomic renal mass characterization: fundamentals, applications, challenges, and future directions. AJR Am J Roentgenol 215(4):920–928. https://doi.org/10.2214/ajr.19.22608

    Article  PubMed  Google Scholar 

  15. de Leon AD, Kapur P, Pedrosa I (2019) Radiomics in kidney cancer: MR imaging. Magn Reson Imaging Clin N Am 27(1):1–13. https://doi.org/10.1016/j.mric.2018.08.005

    Article  PubMed  Google Scholar 

  16. Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278(2):563–577. https://doi.org/10.1148/radiol.2015151169

    Article  PubMed  Google Scholar 

  17. Krishna S, Murray CA, McInnes MD, Chatelain R, Siddaiah M, Al-Dandan O, Narayanasamy S, Schieda N (2017) CT imaging of solid renal masses: pitfalls and solutions. Clin Radiol 72(9):708–721. https://doi.org/10.1016/j.crad.2017.05.003

    Article  CAS  PubMed  Google Scholar 

  18. Fleuren LM, Thoral P, Shillan D, Ercole A, Elbers PWG (2020) Machine learning in intensive care medicine: ready for take-off? Intensive Care Med 46(7):1486–1488. https://doi.org/10.1007/s00134-020-06045-y

    Article  PubMed  Google Scholar 

  19. Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, Swart EL, Girbes ARJ, Thoral P, Ercole A, Hoogendoorn M, Elbers PWG (2020) Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med 46(3):383–400. https://doi.org/10.1007/s00134-019-05872-y

    Article  PubMed  PubMed Central  Google Scholar 

  20. Zhou T, Guan J, Feng B, Xue H, Cui J, Kuang Q, Chen Y, Xu K, Lin F, Cui E, Long W (2023) Distinguishing common renal cell carcinomas from benign renal tumors based on machine learning: comparing various CT imaging phases, slices, tumor sizes, and ROI segmentation strategies. Eur Radiol. https://doi.org/10.1007/s00330-022-09384-0

    Article  PubMed  PubMed Central  Google Scholar 

  21. Lubner MG (2020) Radiomics and artificial intelligence for renal mass characterization. Radiol Clin North Am 58(5):995–1008. https://doi.org/10.1016/j.rcl.2020.06.001

    Article  PubMed  Google Scholar 

  22. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71

    Article  PubMed  PubMed Central  Google Scholar 

  23. Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS (2014) Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med 11(10):e1001744. https://doi.org/10.1371/journal.pmed.1001744

    Article  PubMed  PubMed Central  Google Scholar 

  24. Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R, Kahn CE Jr, Esteva A, Karthikesalingam A, Mateen B, Webster D, Milea D, Ting D, Treanor D, Cushnan D, King D, McPherson D, Glocker B, Greaves F, Harling L, Ordish J, Cohen JF, Deeks J, Leeflang M, Diamond M, McInnes MDF, McCradden M, Abràmoff MD, Normahani P, Markar SR, Chang S, Liu X, Mallett S, Shetty S, Denniston A, Collins GS, Moher D, Whiting P, Bossuyt PM, Darzi A (2021) A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med 27(10):1663–1665. https://doi.org/10.1038/s41591-021-01517-0

    Article  CAS  PubMed  Google Scholar 

  25. Kocak B, Durmaz ES, Kaya OK, Kilickesmez O (2020) Machine learning-based unenhanced CT texture analysis for predicting BAP1 mutation status of clear cell renal cell carcinomas. Acta Radiol 61(6):856–864. https://doi.org/10.1177/0284185119881742

    Article  PubMed  Google Scholar 

  26. Schieda N, Lim RS, Krishna S, McInnes MDF, Flood TA, Thornhill RE (2018) Diagnostic accuracy of unenhanced CT analysis to differentiate low-grade from high-grade chromophobe renal cell carcinoma. AJR Am J Roentgenol 210(5):1079–1087. https://doi.org/10.2214/ajr.17.18874

    Article  PubMed  Google Scholar 

  27. Yap FY, Varghese BA, Cen SY, Hwang DH, Lei X, Desai B, Lau C, Yang LL, Fullenkamp AJ, Hajian S, Rivas M, Gupta MN, Quinn BD, Aron M, Desai MM, Aron M, Oberai AA, Gill IS, Duddalwar VA (2021) Shape and texture-based radiomics signature on CT effectively discriminates benign from malignant renal masses. Eur Radiol 31(2):1011–1021. https://doi.org/10.1007/s00330-020-07158-0

    Article  PubMed  Google Scholar 

  28. Varghese B, Cen S, Zahoor H, Siddiqui I, Aron M, Sali A, Rhie S, Lei X, Rivas M, Liu D, Hwang D, Quinn D, Desai M, Vaishampayan U, Gill I, Duddalwar V (2022) Feasibility of using CT radiomic signatures for predicting CD8-T cell infiltration and PD-L1 expression in renal cell carcinoma. Eur J Radiol Open 9:100440. https://doi.org/10.1016/j.ejro.2022.100440

    Article  PubMed  PubMed Central  Google Scholar 

  29. Demirjian NL, Varghese BA, Cen SY, Hwang DH, Aron M, Siddiqui I, Fields BKK, Lei X, Yap FY, Rivas M, Reddy SS, Zahoor H, Liu DH, Desai M, Rhie SK, Gill IS, Duddalwar V (2022) CT-based radiomics stratification of tumor grade and TNM stage of clear cell renal cell carcinoma. Eur Radiol 32(4):2552–2563. https://doi.org/10.1007/s00330-021-08344-4

    Article  PubMed  Google Scholar 

  30. Nassiri N, Maas M, Cacciamani G, Varghese B, Hwang D, Lei X, Aron M, Desai M, Oberai AA, Cen SY, Gill IS, Duddalwar VA (2022) A radiomic-based machine learning algorithm to reliably differentiate benign renal masses from renal cell carcinoma. Eur Urol Focus 8(4):988–994. https://doi.org/10.1016/j.euf.2021.09.004

    Article  PubMed  Google Scholar 

  31. Kocak B, Ates E, Durmaz ES, Ulusan MB, Kilickesmez O (2019) Influence of segmentation margin on machine learning-based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas. Eur Radiol 29(9):4765–4775. https://doi.org/10.1007/s00330-019-6003-8

    Article  PubMed  Google Scholar 

  32. Yang L, Gao L, Arefan D, Tan Y, Dan H, Zhang J (2022) A CT-based radiomics model for predicting renal capsule invasion in renal cell carcinoma. BMC Med Imaging 22(1):15. https://doi.org/10.1186/s12880-022-00741-5

    Article  PubMed  PubMed Central  Google Scholar 

  33. Kocak B, Durmaz ES, Ates E, Kaya OK, Kilickesmez O (2019) Unenhanced CT texture analysis of clear cell renal cell carcinomas: a machine learning-based study for predicting histopathologic nuclear grade. AJR Am J Roentgenol 212(6):W132-w139. https://doi.org/10.2214/ajr.18.20742

    Article  PubMed  Google Scholar 

  34. Shu J, Wen D, Xi Y, Xia Y, Cai Z, Xu W, Meng X, Liu B, Yin H (2019) Clear cell renal cell carcinoma: machine learning-based computed tomography radiomics analysis for the prediction of WHO/ISUP grade. Eur J Radiol 121:108738. https://doi.org/10.1016/j.ejrad.2019.108738

    Article  PubMed  Google Scholar 

  35. Xv Y, Lv F, Guo H, Zhou X, Tan H, Xiao M, Zheng Y (2021) Machine learning-based CT radiomics approach for predicting WHO/ISUP nuclear grade of clear cell renal cell carcinoma: an exploratory and comparative study. Insights Imaging 12(1):170. https://doi.org/10.1186/s13244-021-01107-1

    Article  PubMed  PubMed Central  Google Scholar 

  36. Li Y, Huang X, Xia Y, Long L (2020) Value of radiomics in differential diagnosis of chromophobe renal cell carcinoma and renal oncocytoma. Abdom Radiol 45(10):3193–3201. https://doi.org/10.1007/s00261-019-02269-9

    Article  Google Scholar 

  37. Yang G, Gong A, Nie P, Yan L, Miao W, Zhao Y, Wu J, Cui J, Jia Y, Wang Z (2019) Contrast-enhanced CT texture analysis for distinguishing fat-poor renal angiomyolipoma from chromophobe renal cell carcinoma. Mol Imaging 18:1536012119883161. https://doi.org/10.1177/1536012119883161

    Article  PubMed  PubMed Central  Google Scholar 

  38. Yang R, Wu J, Sun L, Lai S, Xu Y, Liu X, Ma Y, Zhen X (2020) Radiomics of small renal masses on multiphasic CT: accuracy of machine learning-based classification models for the differentiation of renal cell carcinoma and angiomyolipoma without visible fat. Eur Radiol 30(2):1254–1263. https://doi.org/10.1007/s00330-019-06384-5

    Article  PubMed  Google Scholar 

  39. Feng Z, Rong P, Cao P, Zhou Q, Zhu W, Yan Z, Liu Q, Wang W (2018) Machine learning-based quantitative texture analysis of CT images of small renal masses: differentiation of angiomyolipoma without visible fat from renal cell carcinoma. Eur Radiol 28(4):1625–1633. https://doi.org/10.1007/s00330-017-5118-z

    Article  PubMed  Google Scholar 

  40. Cui EM, Lin F, Li Q, Li RG, Chen XM, Liu ZS, Long WS (2019) Differentiation of renal angiomyolipoma without visible fat from renal cell carcinoma by machine learning based on whole-tumor computed tomography texture features. Acta Radiol 60(11):1543–1552. https://doi.org/10.1177/0284185119830282

    Article  PubMed  Google Scholar 

  41. Erdim C, Yardimci AH, Bektas CT, Kocak B, Koca SB, Demir H, Kilickesmez O (2020) Prediction of benign and malignant solid renal masses: machine learning-based CT texture analysis. Acad Radiol 27(10):1422–1429. https://doi.org/10.1016/j.acra.2019.12.015

    Article  PubMed  Google Scholar 

  42. Sun XY, Feng QX, Xu X, Zhang J, Zhu FP, Yang YH, Zhang YD (2020) Radiologic-radiomic machine learning models for differentiation of benign and malignant solid renal masses: comparison with expert-level radiologists. AJR Am J Roentgenol 214(1):W44-w54. https://doi.org/10.2214/ajr.19.21617

    Article  PubMed  Google Scholar 

  43. Budai BK, Stollmayer R, Rónaszéki AD, Körmendy B, Zsombor Z, Palotás L, Fejér B, Szendrõi A, Székely E, Maurovich-Horvat P, Kaposi PN (2022) Radiomics analysis of contrast-enhanced CT scans can distinguish between clear cell and non-clear cell renal cell carcinoma in different imaging protocols. Front Med 9:974485. https://doi.org/10.3389/fmed.2022.974485

    Article  Google Scholar 

  44. Miskin N, Qin L, Matalon SA, Tirumani SH, Alessandrino F, Silverman SG, Shinagare AB (2021) Stratification of cystic renal masses into benign and potentially malignant: applying machine learning to the bosniak classification. Abdom Radiol 46(1):311–318. https://doi.org/10.1007/s00261-020-02629-w

    Article  Google Scholar 

  45. He QH, Tan H, Liao FT, Zheng YN, Lv FJ, Jiang Q, Xiao MZ (2022) Stratification of malignant renal neoplasms from cystic renal lesions using deep learning and radiomics features based on a stacking ensemble CT machine learning algorithm. Front Oncol 12:1028577. https://doi.org/10.3389/fonc.2022.1028577

    Article  PubMed  PubMed Central  Google Scholar 

  46. Zhang GM, Shi B, Xue HD, Ganeshan B, Sun H, Jin ZY (2019) Can quantitative CT texture analysis be used to differentiate subtypes of renal cell carcinoma? Clin Radiol 74(4):287–294. https://doi.org/10.1016/j.crad.2018.11.009

    Article  PubMed  Google Scholar 

  47. Zhang H, Yin F, Chen M, Yang L, Qi A, Cui W, Yang S, Wen G (2021) Development and validation of a CT-based radiomics nomogram for predicting postoperative progression-free survival in stage I-III renal cell carcinoma. Front Oncol 11:742547. https://doi.org/10.3389/fonc.2021.742547

    Article  PubMed  Google Scholar 

  48. Feng Z, Zhang L, Qi Z, Shen Q, Hu Z, Chen F (2020) Identifying BAP1 mutations in clear-cell renal cell carcinoma by CT radiomics: preliminary findings. Front Oncol 10:279. https://doi.org/10.3389/fonc.2020.00279

    Article  PubMed  PubMed Central  Google Scholar 

  49. Chen X, Zhou Z, Hannan R, Thomas K, Pedrosa I, Kapur P, Brugarolas J, Mou X, Wang J (2018) Reliable gene mutation prediction in clear cell renal cell carcinoma through multi-classifier multi-objective radiogenomics model. Phys Med Biol 63(21):215008. https://doi.org/10.1088/1361-6560/aae5cd

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Kocak B, Durmaz ES, Ates E, Ulusan MB (2019) Radiogenomics in clear cell renal cell carcinoma: machine learning-based high-dimensional quantitative CT texture analysis in predicting PBRM1 mutation status. AJR Am J Roentgenol 212(3):W55-w63. https://doi.org/10.2214/ajr.18.20443

    Article  PubMed  Google Scholar 

  51. Cheng D, Abudikeranmu Y, Tuerdi B (2022) Differentiation of clear cell and non-clear-cell renal cell carcinoma through CT-based Radiomics models and nomogram. Curr Med Imaging. https://doi.org/10.2174/1573405619666221121164235

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work was supported by grants from the Youth Science Foundation of Shandong First Medical University (202201-066), the Science and Technology Development Plan Project of Shandong Province, China (2018GSF118209), and the Shandong Provincial Natural Science Foundation (ZR2017MH091, ZR2022MH095).

Author information

Author notes

  1. Honghao Song and Xiaoqing Wang have contributed equally to this work.

Authors and Affiliations

  1. Department of Pediatric Surgery, Shandong Provincial Hospital, Shandong University, Jinan, 250021, Shandong, People’s Republic of China

    Honghao Song

  2. Department of Pediatric Surgery, Shandong Provincial Hospital Affiliated to Shandong First Medical University, 324 Jingwu Street, Jinan, 250021, Shandong, People’s Republic of China

    Xiaoqing Wang, Rongde Wu & Wei Liu

  3. Post-doctoral Research Station of Clinical Medicine, Liaocheng People’s Hospital, Liaocheng, 252004, Shandong, People’s Republic of China

    Xiaoqing Wang

Authors
  1. Honghao Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rongde Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HHS and XQW put forward the design of the study, searched papers, extracted data, and make data analysis. WL and RDW dealt with the disagreements. All authors contributed to the revision and approved this manuscript. HHS and XQW contributed equally to this work.

Corresponding author

Correspondence to Wei Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics approval and consent to participate

No applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 32375 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, H., Wang, X., Wu, R. et al. The influence of manual segmentation strategies and different phases selection on machine learning-based computed tomography in renal tumors: a systematic review and meta-analysis. Radiol med (2024). https://doi.org/10.1007/s11547-024-01825-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11547-024-01825-8

Keywords

Search

Navigation