Skip to main content
SpringerLink
Log in

Intra-rater variability in low-grade glioma segmentation

  • Clinical Study
  • Published:
Journal of Neuro-Oncology Aims and scope Submit manuscript

Abstract

Assessment of size and growth are key radiological factors in low-grade gliomas (LGGs), both for prognostication and treatment evaluation, but the reliability of LGG-segmentation is scarcely studied. With a diffuse and invasive growth pattern, usually without contrast enhancement, these tumors can be difficult to delineate. The aim of this study was to investigate the intra-observer variability in LGG-segmentation for a radiologist without prior segmentation experience. Pre-operative 3D FLAIR images of 23 LGGs were segmented three times in the software 3D Slicer. Tumor volumes were calculated, together with the absolute and relative difference between the segmentations. To quantify the intra-rater variability, we used the Jaccard coefficient comparing both two (J2) and three (J3) segmentations as well as the Hausdorff Distance (HD). The variability measured with J2 improved significantly between the two last segmentations compared to the two first, going from 0.87 to 0.90 (p = 0.04). Between the last two segmentations, larger tumors showed a tendency towards smaller relative volume difference (p = 0.07), while tumors with well-defined borders had significantly less variability measured with both J2 (p = 0.04) and HD (p < 0.01). We found no significant relationship between variability and histological sub-types or Apparent Diffusion Coefficients (ADC). We found that the intra-rater variability can be considerable in serial LGG-segmentation, but the variability seems to decrease with experience and higher grade of border conspicuity. Our findings highlight that some criteria defining tumor borders and progression in 3D volumetric segmentation is needed, if moving from 2D to 3D assessment of size and growth of LGGs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Sanai N, Chang S, Berger MS (2011) Low-grade gliomas in adults. J Neurosurg. doi:10.3171/2011.7.jns10238

    Google Scholar 

  2. Buckner JC, Shaw EG, Pugh SL, Chakravarti A, Gilbert MR, Barger GR, Coons S, Ricci P, Bullard D, Brown PD, Stelzer K, Brachman D, Suh JH, Schultz CJ, Bahary JP, Fisher BJ, Kim H, Murtha AD, Bell EH, Won M, Mehta MP, Curran WJ Jr (2016) Radiation plus Procarbazine, CCNU, and Vincristine in Low-Grade Glioma. N Engl J Med 374(14):1344–1355. doi:10.1056/NEJMoa1500925

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Pallud J, Capelle L, Taillandier L, Badoual M, Duffau H, Mandonnet E (2013) The silent phase of diffuse low-grade gliomas. Is it when we missed the action? Acta Neurochir (Wien) 155(12):2237–2242. doi:10.1007/s00701-013-1886-7

    Article  Google Scholar 

  4. Pallud J, Fontaine D, Duffau H, Mandonnet E, Sanai N, Taillandier L, Peruzzi P, Guillevin R, Bauchet L, Bernier V, Baron MH, Guyotat J, Capelle L (2010) Natural history of incidental World Health Organization grade II gliomas. Ann Neurol 68(5):727–733. doi:10.1002/ana.22106

    Article  PubMed  Google Scholar 

  5. Lima GL, Zanello M, Mandonnet E, Taillandier L, Pallud J, Duffau H (2015) Incidental diffuse low-grade gliomas: from early detection to preventive neuro-oncological surgery. Neurosurg Rev. doi:10.1007/s10143-015-0675-6

    PubMed  Google Scholar 

  6. Jakola AS, Myrmel KS, Kloster R, Torp SH, Lindal S, Unsgard G, Solheim O (2012) Comparison of a strategy favoring early surgical resection vs a strategy favoring watchful waiting in low-grade gliomas. JAMA 308(18):1881–1888. doi:10.1001/jama.2012.12807

    Article  CAS  PubMed  Google Scholar 

  7. Duffau H, Taillandier L (2015) New concepts in the management of diffuse low-grade glioma: Proposal of a multistage and individualized therapeutic approach. Neuro. Oncol 17(3):332–342. doi:10.1093/neuonc/nou153

    Google Scholar 

  8. Chang EF, Smith JS, Chang SM, Lamborn KR, Prados MD, Butowski N, Barbaro NM, Parsa AT, Berger MS, McDermott MM (2008) Preoperative prognostic classification system for hemispheric low-grade gliomas in adults. J Neurosurg 109(5):817–824. doi:10.3171/JNS/2008/109/11/0817

    Article  PubMed  Google Scholar 

  9. Pignatti F, van den Bent M, Curran D, Debruyne C, Sylvester R, Therasse P, Afra D, Cornu P, Bolla M, Vecht C, Karim AB, European Organization for R, Treatment of Cancer Brain Tumor Cooperative G, European Organization for R, Treatment of Cancer Radiotherapy Cooperative G (2002) Prognostic factors for survival in adult patients with cerebral low-grade glioma. J Clin Oncol 20(8):2076–2084

    Article  PubMed  Google Scholar 

  10. Ahmadi R, Dictus C, Hartmann C, Zurn O, Edler L, Hartmann M, Combs S, Herold-Mende C, Wirtz CR, Unterberg A (2009) Long-term outcome and survival of surgically treated supratentorial low-grade glioma in adult patients. Acta Neurochir (Wien) 151(11):1359–1365. doi:10.1007/s00701-009-0435-x

    Article  Google Scholar 

  11. Capelle L, Fontaine D, Mandonnet E, Taillandier L, Golmard JL, Bauchet L, Pallud J, Peruzzi P, Baron MH, Kujas M, Guyotat J, Guillevin R, Frenay M, Taillibert S, Colin P, Rigau V, Vandenbos F, Pinelli C, Duffau H, French Reseau d’Etude des G (2013) Spontaneous and therapeutic prognostic factors in adult hemispheric World Health Organization Grade II gliomas: a series of 1097 cases: clinical article. J Neurosurg 118(6):1157–1168. doi:10.3171/2013.1.JNS121

    Article  PubMed  Google Scholar 

  12. Chaichana KL, McGirt MJ, Laterra J, Olivi A, Quinones-Hinojosa A (2010) Recurrence and malignant degeneration after resection of adult hemispheric low-grade gliomas. J Neurosurg 112(1):10–17. doi:10.3171/2008.10.JNS08608

    Article  PubMed  Google Scholar 

  13. Claus EB, Horlacher A, Hsu L, Schwartz RB, Dello-Iacono D, Talos F, Jolesz FA, Black PM (2005) Survival rates in patients with low-grade glioma after intraoperative magnetic resonance image guidance. Cancer 103(6):1227–1233. doi:10.1002/cncr.20867

    Article  PubMed  Google Scholar 

  14. Ius T, Isola M, Budai R, Pauletto G, Tomasino B, Fadiga L, Skrap M (2012) Low-grade glioma surgery in eloquent areas: volumetric analysis of extent of resection and its impact on overall survival. A single-institution experience in 190 patients: clinical article. J Neurosurg 117(6):1039–1052. doi:10.3171/2012.8.JNS12393

    Article  PubMed  Google Scholar 

  15. McGirt MJ, Chaichana KL, Attenello FJ, Weingart JD, Than K, Burger PC, Olivi A, Brem H, Quinones-Hinojosa A (2008) Extent of surgical resection is independently associated with survival in patients with hemispheric infiltrating low-grade gliomas. Neurosurgery 63(4):700–707. doi:10.1227/01.NEU.0000325729.41085.73

    Article  PubMed  Google Scholar 

  16. Sanai N, Berger MS (2009) Operative techniques for gliomas and the value of extent of resection. Neurotheraphy 6(3):478–486. doi:10.1016/j.nurt.2009.04.005

    Article  Google Scholar 

  17. Smith JS, Chang EF, Lamborn KR, Chang SM, Prados MD, Cha S, Tihan T, Vandenberg S, McDermott MW, Berger MS (2008) Role of extent of resection in the long-term outcome of low-grade hemispheric gliomas. J Clin Oncol 26(8):1338–1345. doi:10.1200/JCO.2007.13.9337

    Article  PubMed  Google Scholar 

  18. Pallud J, Blonski M, Mandonnet E, Audureau E, Fontaine D, Sanai N, Bauchet L, Peruzzi P, Frenay M, Colin P, Guillevin R, Bernier V, Baron MH, Guyotat J, Duffau H, Taillandier L, Capelle L (2013) Velocity of tumor spontaneous expansion predicts long-term outcomes for diffuse low-grade gliomas. Neuro Oncol 15(5):595–606. doi:10.1093/neuonc/nos331

    Article  PubMed  PubMed Central  Google Scholar 

  19. Pallud J, Taillandier L, Capelle L, Fontaine D, Peyre M, Ducray F, Duffau H, Mandonnet E (2012) Quantitative morphological magnetic resonance imaging follow-up of low-grade glioma: a plea for systematic measurement of growth rates. Neurosurgery 71(3):729–739 (discussion 739–740). doi:10.1227/NEU.0b013e31826213de

    Article  PubMed  Google Scholar 

  20. Brasil Caseiras G, Ciccarelli O, Altmann DR, Benton CE, Tozer DJ, Tofts PS, Yousry TA, Rees J, Waldman AD, Jäger HR (2009) Low-grade gliomas: six-month tumor growth predicts patient outcome better than admission tumor volume, relative cerebral blood volume, and apparent diffusion coefficient. Radiology 253(2):505–512. doi:10.1148/radiol.2532081623

    Article  PubMed  Google Scholar 

  21. Rees J, Watt H, Jäger HR, Benton C, Tozer D, Tofts P, Waldman A (2009) Volumes and growth rates of untreated adult low-grade gliomas indicate risk of early malignant transformation. Eur J Radiol 72(1):54–64. doi:10.1016/j.ejrad.2008.06.013

    Article  PubMed  Google Scholar 

  22. Jakola AS, Moen KG, Solheim O, Kvistad KA (2013) “No growth” on serial MRI scans of a low grade glioma? Acta Neurochir (Wien) 155(12):2243–2244. doi:10.1007/s00701-013-1914-7

    Article  Google Scholar 

  23. Mandonnet E, Pallud J, Fontaine D, Taillandier L, Bauchet L, Peruzzi P, Guyotat J, Bernier V, Baron MH, Duffau H, Capelle L (2010) Inter- and intrapatients comparison of WHO grade II glioma kinetics before and after surgical resection. Neurosurg Rev 33(1):91–96. doi:10.1007/s10143-009-0229-x

    Article  PubMed  Google Scholar 

  24. Bauer S, Wiest R, Nolte LP, Reyes M (2013) A survey of MRI-based medical image analysis for brain tumor studies. Phys Med Biol 58(13):R97–R129. doi:10.1088/0031-9155/58/13/R97

    Article  PubMed  Google Scholar 

  25. Porz N, Bauer S, Pica A, Schucht P, Beck J, Verma RK, Slotboom J, Reyes M, Wiest R (2014) Multi-modal glioblastoma segmentation: man versus machine. PLoS One 9(5):e96873. doi:10.1371/journal.pone.0096873

    Article  PubMed  PubMed Central  Google Scholar 

  26. Akkus Z, Sedlar J, Coufalova L, Korfiatis P, Kline TL, Warner JD, Agrawal J, Erickson BJ (2015) Semi-automated segmentation of pre-operative low grade gliomas in magnetic resonance imaging. Cancer Imaging 15:12. doi:10.1186/s40644-015-0047-z

    Article  PubMed  PubMed Central  Google Scholar 

  27. Angelini ED, Delon J, Bah AB, Capelle L, Mandonnet E (2012) Differential MRI analysis for quantification of low grade glioma growth. Med Image Anal 16(1):114–126. doi:10.1016/j.media.2011.05.014

    Article  PubMed  Google Scholar 

  28. Kaus MR, Warfield SK, Nabavi A, Black PM, Jolesz FA, Kikinis R (2001) Automated segmentation of MR images of brain tumors. Radiology 218(2):586–591. doi:10.1148/radiology.218.2.r01fe44586

    Article  CAS  PubMed  Google Scholar 

  29. Mazzara GP, Velthuizen RP, Pearlman JL, Greenberg HM, Wagner H (2004) Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation. Int J Radiat Oncol Biol Phys 59(1):300–312. doi:10.1016/j.ijrobp.2004.01.026

    Article  PubMed  Google Scholar 

  30. Weizman L, Sira LB, Joskowicz L, Rubin DL, Yeom KW, Constantini S, Shofty B, Bashat DB (2014) Semiautomatic segmentation and follow-up of multicomponent low-grade tumors in longitudinal brain MRI studies. Med Phys 41(5):052303. doi:10.1118/1.4871040

    Article  PubMed  PubMed Central  Google Scholar 

  31. Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin JC, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M, Buatti J, Aylward S, Miller JV, Pieper S, Kikinis R (2012) 3D Slicer as an image computing platform for the quantitative imaging network. Magn Reson Imaging 30(9):1323–1341. doi:10.1016/j.mri.2012.05.001

    Article  PubMed  PubMed Central  Google Scholar 

  32. Egger J, Kapur T, Fedorov A, Pieper S, Miller JV, Veeraraghavan H, Freisleben B, Golby AJ, Nimsky C, Kikinis R (2013) GBM volumetry using the 3D Slicer medical image computing platform. Sci Rep 3:1364. doi:10.1038/srep01364

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber MA, Arbel T, Avants BB, Ayache N, Buendia P, Collins DL, Cordier N, Corso JJ, Criminisi A, Das T, Delingette H, Demiralp C, Durst CR, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X, Hamamci A, Iftekharuddin KM, Jena R, John NM, Konukoglu E, Lashkari D, Mariz JA, Meier R, Pereira S, Precup D, Price SJ, Raviv TR, Reza SM, Ryan M, Sarikaya D, Schwartz L, Shin HC, Shotton J, Silva CA, Sousa N, Subbanna NK, Szekely G, Taylor TJ, Thomas OM, Tustison NJ, Unal G, Vasseur F, Wintermark M, Ye DH, Zhao L, Zhao B, Zikic D, Prastawa M, Reyes M, Van Leemput K (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34(10):1993–2024. doi:10.1109/TMI.2014.2377694

    Article  PubMed  Google Scholar 

  34. Zijdenbos AP, Dawant BM, Margolin RA, Palmer AC (1994) Morphometric analysis of white matter lesions in MR images: method and validation. IEEE Trans Med Imaging 13(4):716–724. doi:10.1109/42.363096

    Article  CAS  PubMed  Google Scholar 

  35. Zou KH, Warfield SK, Bharatha A, Tempany CM, Kaus MR, Haker SJ, Wells WM 3rd, Jolesz FA, Kikinis R (2004) Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 11(2):178–189

    Article  PubMed  PubMed Central  Google Scholar 

  36. Crum WR, Camara O, Hill DL (2006) Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans Med Imaging 25(11):1451–1461. doi:10.1109/TMI.2006.880587

    Article  PubMed  Google Scholar 

  37. McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 22(3):276–282

    Article  Google Scholar 

  38. Rosner BA (2011) Fundamentals of biostatistics. 7th edn. Brooks/Cole. Cengage Learning, Boston

    Google Scholar 

  39. Macdonald DR, Cascino TL, Schold SC Jr, Cairncross JG (1990) Response criteria for phase II studies of supratentorial malignant glioma. J Clin Oncol 8(7):1277–1280

    CAS  PubMed  Google Scholar 

  40. van den Bent MJ, Wefel JS, Schiff D, Taphoorn MJ, Jaeckle K, Junck L, Armstrong T, Choucair A, Waldman AD, Gorlia T, Chamberlain M, Baumert BG, Vogelbaum MA, Macdonald DR, Reardon DA, Wen PY, Chang SM, Jacobs AH (2011) Response assessment in neuro-oncology (a report of the RANO group): assessment of outcome in trials of diffuse low-grade gliomas. Lancet Oncol 12(6):583–593. doi:10.1016/S1470-2045(11)70057-2

    Article  PubMed  Google Scholar 

  41. Schmitt P, Mandonnet E, Perdreau A, Angelini ED (2013) Effects of slice thickness and head rotation when measuring glioma sizes on MRI: in support of volume segmentation versus two largest diameters methods. J Neurooncol 112(2):165–172. doi:10.1007/s11060-013-1051-4

    Article  PubMed  Google Scholar 

  42. Sorensen AG, Patel S, Harmath C, Bridges S, Synnott J, Sievers A, Yoon YH, Lee EJ, Yang MC, Lewis RF, Harris GJ, Lev M, Schaefer PW, Buchbinder BR, Barest G, Yamada K, Ponzo J, Kwon HY, Gemmete J, Farkas J, Tievsky AL, Ziegler RB, Salhus MR, Weisskoff R (2001) Comparison of diameter and perimeter methods for tumor volume calculation. J Clin Oncol 19(2):551–557

    CAS  PubMed  Google Scholar 

  43. Zetterling M, Roodakker KR, Berntsson SG, Edqvist PH, Latini F, Landtblom AM, Ponten F, Alafuzoff I, Larsson EM, Smits A (2016) Extension of diffuse low-grade gliomas beyond radiological borders as shown by the coregistration of histopathological and magnetic resonance imaging data. J Neurosurg. doi:10.3171/2015.10.jns15583

    PubMed  Google Scholar 

  44. Chen L, Liu M, Bao J, Xia Y, Zhang J, Zhang L, Huang X, Wang J (2013) The correlation between apparent diffusion coefficient and tumor cellularity in patients: a meta-analysis. PLoS One 8(11):e79008. doi:10.1371/journal.pone.0079008

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Tschampa HJ, Urbach H, Malter M, Surges R, Greschus S, Gieseke J (2015) Magnetic resonance imaging of focal cortical dysplasia: comparison of 3D and 2D fluid attenuated inversion recovery sequences at 3 T. Epilepsy Res 116:8–14. doi:10.1016/j.eplepsyres.2015.07.004

    Article  PubMed  Google Scholar 

  46. Stensjoen AL, Solheim O, Kvistad KA, Haberg AK, Salvesen O, Berntsen EM (2015) Growth dynamics of untreated glioblastomas in vivo. Neuro Oncol 17(10):1402–1411. doi:10.1093/neuonc/nov029

    Article  PubMed  PubMed Central  Google Scholar 

  47. Tselikas L, Souillard-Scemama R, Naggara O, Mellerio C, Varlet P, Dezamis E, Domont J, Dhermain F, Devaux B, Chretien F, Meder JF, Pallud J, Oppenheim C (2015) Imaging of gliomas at 1.5 and 3T—A comparative study. Neuro Onco 17(6):895–900. doi:10.1093/neuonc/nou332

    Article  Google Scholar 

  48. Neema M, Guss ZD, Stankiewicz JM, Arora A, Healy BC, Bakshi R (2009) Normal findings on brain fluid-attenuated inversion recovery MR images at 3T. AJNR Am J Neuroradiol 30(5):911–916. doi:10.3174/ajnr.A1514

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Kamada K, Kakeda S, Ohnari N, Moriya J, Sato T, Korogi Y (2008) Signal intensity of motor and sensory cortices on T2-weighted and FLAIR images: intraindividual comparison of 1.5T and 3T MRI. Eur Radiol 18(12):2949–2955. doi:10.1007/s00330-008-1069-8

    Article  PubMed  Google Scholar 

  50. Guarnaschelli JN, Vagal AS, McKenzie JT, McPherson CM, Warnick RE, Batra V, Breneman JC, Lamba MA (2014) Target definition for malignant gliomas: no difference in radiation treatment volumes between 1.5T and 3T magnetic resonance imaging. Pract Radiat Oncol 4(5):e195–e201. doi:10.1016/j.prro.2013.11.003

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

A.S.J. holds a grant from The Norwegian Cancer Society for glioma research.

Author information

Authors and Affiliations

  1. Department of Radiology and Nuclear Medicine, St. Olavs University Hospital, P.O. Box 3250, Sluppen, 7006, Trondheim, Norway

    Hans Kristian Bø, Kjell-Arne Kvistad & Erik Magnus Berntsen

  2. Department of Circulation and Medical Imaging, Faculty of Medicine, NTNU, Norwegian University of Science and Technology, Trondheim, Norway

    Hans Kristian Bø & Erik Magnus Berntsen

  3. Department of Neurosurgery, St. Olavs University Hospital, Trondheim, Norway

    Ole Solheim & Asgeir Store Jakola

  4. Department of Neuroscience, Faculty of Medicine, NTNU, Norwegian University of Science and Technology, Trondheim, Norway

    Ole Solheim

  5. National Competence Centre for Ultrasound and Image Guided Therapy, St. Olavs University Hospital, Trondheim, Norway

    Ole Solheim & Ingerid Reinertsen

  6. Department of Neurosurgery, Sahlgrenska University Hospital, Gothenburg, Sweden

    Asgeir Store Jakola

  7. Institute of Neuroscience and Physiology, Sahlgrenska Academy, Gothenburg, Sweden

    Asgeir Store Jakola

  8. SINTEF, Technology and Society, Department of Medical Technology, Trondheim, Norway

    Ingerid Reinertsen

Authors
  1. Hans Kristian Bø
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ole Solheim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Asgeir Store Jakola
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kjell-Arne Kvistad
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ingerid Reinertsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Erik Magnus Berntsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hans Kristian Bø.

Ethics declarations

Conflict of interest

O.S. and E.M.B. have fundings from the Regional brain tumor registry of central Norway funded by the liaison committee of St. Olavs University Hospital and NTNU. The authors declare that they have no conflict of interest.

Ethical approval

The study has been approved by the Regional Ethical Committee for Health Region Mid-Norway (ref 2014/1674).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bø, H.K., Solheim, O., Jakola, A.S. et al. Intra-rater variability in low-grade glioma segmentation. J Neurooncol 131, 393–402 (2017). https://doi.org/10.1007/s11060-016-2312-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11060-016-2312-9

Keywords

Search

Navigation