Accuracy of the neurosurgeons estimation of extent of resection in glioblastoma

Background The surgeons’ estimate of the extent of resection (EOR) shows little accuracy in previous literature. Considering the developments in surgical techniques of glioblastoma (GBM) treatment, we hypothesize an improvement in this estimation. This study aims to compare the EOR estimated by the neurosurgeon with the EOR determined using volumetric analysis on the post-operative MR scan. Methods Pre- and post-operative tumor volumes were calculated through semi-automatic volumetric assessment by three observers. Interobserver agreement was measured using intraclass correlation coefficient (ICC). A univariate general linear model was used to study the factors influencing the accuracy of estimation of resection percentage. Results ICC was high for all three measurements: pre-operative tumor volume was 0.980 (0.969–0.987), post-operative tumor volume 0.974 (0.961–0.984), and EOR 0.947 (0.917–0.967). Estimation of EOR by the surgeon showed moderate accuracy and agreement. Multivariable analysis showed a statistically significant effect of operating neurosurgeon (p = 0.01), use of fluorescence (p < 0.001), and resection percentage (p < 0.001) on the accuracy of the EOR estimation. Conclusion All measurements through semi-automatic volumetric analysis show a high interobserver agreement, suggesting this to be a reliable assessment of EOR. We found a moderate reliability of the surgeons’ estimate of EOR. Therefore, (early) post-operative MRI scanning for evaluation of EOR remains paramount. Electronic supplementary material The online version of this article (10.1007/s00701-019-04089-8) contains supplementary material, which is available to authorized users.


Introduction
Glioblastoma (GBM) represents 47.1% of malignant primary brain tumors, making it the most common type of malignant primary brain tumor [1]. Even though survival rates have improved, prognosis of GBM remains poor. Surgery aimed at as much resection as safely possible is the main treatment option.
Using volumetric analysis, it is possible to quantify the EOR using MR imaging. Prior studies using semi-automated methods for this purpose found a high interobserver agreement [4,5]. However, when manual segmentation is used, a low interobserver agreement in the assessment of tumor resection rates on magnetic resonance imaging (MRI) is described. This applies particularly for post-operative tumor volume and residual tumor volume [6].
Before the general use of post-operative scanning, intraoperative estimation by the neurosurgeon was used to determine partial, subtotal, or total tumor resection. The only study that compared this estimation with the presence of residual tumor mass on a MR image, dates back to 1994 [7]. It showed that neurosurgeons often underestimate the presence of residual tumor. Residual tumor mass was three times more often seen on early post-operative MRI than estimated by the neurosurgeon [7].
With the advancement of microsurgical techniques and the introduction of adjuncts such as fluorescence, we hypothesize a high accuracy of the surgeons' estimation of the EOR. Possibly, the post-op MRI scan can be omitted after resection using fluorescence. This study aims to compare the EOR estimated by the neurosurgeon with the EOR determined using volumetric analysis on post-operative MR images.

Participants
Because the study implied no burden for the patients, ethical approval was waived by the ethical committee. In this retrospective study, patient characteristic, operation details, and tumor characteristics were prospectively collected. Therefore, no informed consent was required. Adult patients (≥ 18 years old) who underwent a resection for GBM were included. Exclusion criteria were as follows: (1.) patients without a pre-operative contrast-enhanced MR image (i.e., less than 48 h before surgery) or without an early postoperative MRI (i.e., within 72 h after surgery) and (2.) patients that had prior treatment for the target tumor.

Test methods
All patients were treated according to protocol; neuronavigation was used in all patients and 5-ALA was used for intra-operative fluorescence in 22 patients. The EOR by the neurosurgeon was estimated by the operating surgeon post-operatively (EOR surgeon ), before the post-operative MR scan was made. This estimation was based on intra-operative microscopic visualization and neuronavigation. Post-operative MR imaging was performed within 72 h after surgery.
Pre-and post-operative T1-weighed MR images (slice thickness 1.0 mm) with and without gadolinium contrast were used for semi-automated segmentation of the tumor. Tumor volumes were calculated through semi-automated volumetric analysis using Brainlab software (Brainlab, iPlanNet version 2.3.1.215.1, Munich, Germany) by three observers (a neurosurgeon, a radiologist, and a trained medical student). Previous literature stated the use of this method to be reliable for tumor measurements [4,5]. All observers are skilled in interpreting MR images for brain tumors. Each observer was blinded for the data measured by the other observers and for the surgeons' estimate. The neurosurgeon did not measure the tumors of his own patients. For this reason, there was a second neurosurgeon measuring these patients' MR images.
Our definition of tumor volume was in accordance with previous descriptions [6]: the contrast enhancing mass, including the possibly present necrotic area. All three observers were instructed to include any tissue with a high post-contrast signal in their estimation. The minimal detectable residual would be one voxel, which corresponds with 0.954 mm 3 using our settings. To eliminate hemorrhage inclusion in post-operative tumor volume (Post-opTV) determination, any parts around the resection area with high signal on T1-weighed image without contrast were subtracted from the hyperintense mass on the contrast-enhanced T1-weighed image. Extent of resection (EOR) was calculated as: This EOR calculation was done for the measurements of the surgeon, the radiologist, and the student, respectively, resulting in an EOR obs1 , EOR obs2 , and EOR obs3 . The mean of these three measurements is defined as the EOR MRI .

Power calculation
Previous literature comparing the presence of residual tumor on MR images with the estimation made by the neurosurgeon is scarce. We predicted to find an accuracy of 80%, meaning that the estimation about the presence of residual tumor mass should agree with the findings on MRI in 80% of the cases. Sample size calculation with a 95% confidence probability, resulted in a sample size of 62.

Analysis
All analyses were performed on SPSS software version 25 (IBM Inc., Armonk, New York) for Windows (Microsoft Inc., Redmond, Washington). We have used an altered Bland-Altmann plot to assess agreement between more than two observers [8].
Interobserver agreement for pre-opTV, post-opTV, and EOR was measured using intraclass correlation coefficient (ICC) based on a two-way random model for absolute agreement. Using semi-automated volumetric analysis methods, even the smallest remnants are detected. Therefore, a cut-off value of 99% was used as a surrogate for residual tumor only for accuracy calculation.
A univariate multivariable general linear model was used to study the factors influencing the accuracy of EOR surgeon . Here, we used Δ = EOR surgeon − EOR MRI as the dependent variable. The following variables were included in analysis: Pre-opTV (mean Pre-opTV of the three observers), blood loss during surgery, use of fluorescence, operating neurosurgeon, and EOR MRI . Significance for all analysis was set at a p value < 0.05.

Participants
Between November 2012 and May 2018, 62 patients were included in the study. A flowchart of the selection process can be found in Fig. 1. The characteristics of the study group are summarized in Table 1. The group consisted of 38 male (61%) and 24 female (39%) patients. The median age at diagnosis was 63.5 year (range 27-78 years).
The Bland-Altmann plot for EOR can be found in Fig. 2. The estimated limits of agreement were -12.39 to 12.39. The plot shows no systematic over or underestimation of EOR by any observer. There was an excellent reliability for EOR measurement, with an ICC of 0.947 (0.917-0.967) ( Table 2).

EOR surgeon , EOR MRI
There is a higher agreement between EOR surgeon and EOR MRI with increasing resection percentage and when fluorescence is used (Fig. 3). The cases are equally distributed above and below the identity line, so there was no systematic EOR over or underestimation by the surgeon.
Presence of residual tumor (defined as EOR < 99%) was estimated by the neurosurgeon almost as often as it was seen on the MRI: surgeons expected residual tumor presence in 76% of the patients, whilst residual was seen in 79% of the MR images (Table 3). The accuracy of predicting presence of residual tumor (defined as EOR < 99%) was 77%, meaning that 77% of the estimations about presence of residual tumor was correct.
The ICC for EOR surgeon and EOR MRI was 0.641 (95% CI 0.404-0.784). The latter indicates a moderate reliability of the EOR surgeon [9].
Multivariable analysis showed a statistically significant effect of operating neurosurgeon (p = 0.01), use of fluorescence (p < 0.001), and EOR MRI (p < 0.001) on Δ surgeon -MRI . Blood loss and pre-operative tumor volume were not statically significant factors.

Discussion
Our main purpose in this study was to compare the EOR estimated by the neurosurgeon post-operatively (EOR surgeon ) with the EOR determined using volumetric analysis on the post-operative MR scan (EOR MRI ).
In order to achieve this, we first studied the reliability of semi-automated volumetric analysis for determining EOR. There was a very high interobserver agreement between the three observers for measurements of pre-operative tumor volume, post-operative tumor volume, and extent of resection. Therefore, semi-automatic volumetric measurement of extent resection of glioblastoma is reliable.
Our data shows that residual tumor (defined as EOR < 99%) is seen on MR images as often as suggested by the neurosurgeon. This simple comparison shows an evident result and this way of reporting data is similar to that of the previous study on this subject [7]. However, with the increased precision of statistical analysis in this era, we added a more statistical approach to display our findings. So, we found an accuracy of 77% and a moderate reliability [9].
Higher resection percentage contributed to a higher agreement between EOR surgeon and EOR MRI . The use of fluorescence also contributed to a higher agreement, with a tendency of overestimation of the EOR by the surgeon (marginal means 12.48 95% CI 7.26; 17.71). The reliability of the surgeons' estimate was surgeon dependant.
Using manual segmentation, others have reported high interobserver agreement only for pre-operative tumor volume, but not regarding post-operative volume and EOR [6]. Prior studies using semi-automated methods found a high interobserver agreement for calculation of resection percentage [4,5]. Our results agree, showing that semi-automatic volume assessment in GBM is a reliable method for detecting postoperative residual and determining EOR.
Only one prior study has reported on the reliability of surgeons' estimate of EOR [7]. In this study, data collection was nominal: presence of residual tumor was answered with "yes," "no," or "?." Residual tumor was detected 3 times more often Fig. 2 Bland-Altman plot for three observers [8]. EORmri (x-axis) is plotted against EORmri minus the EOR measured by the individual observer (y-axis). This graph displays per patient the agreement between the three observers. High agreement corresponds with the three colors being close together. The lines represent the limits of agreement. EOR, extent of resection; EORmri, mean EOR of three observers; EORobs1, EOR measured by neurosurgeon; EORobs2, EOR measured by radiologist; EORobs3, EOR measured by student The ICC was calculated using a two-way random model for absolute agreement. Pre-opTV, post-opTV, and EOR all showed a high ICC, indicating an excellent reliability between observers. ICC, intraclass correlation coefficient; Pre-opTV, pre-operative tumor volume; Post-opTV, post-operative tumor volume; EOR, extent of resection on MRI than estimated by the surgeon. Our data shows the findings of residual tumor on MRI to be equal to the estimation of the neurosurgeon. At first hand, this seems as an improvement in reliability of surgeons' estimate as hypothesized, but further analysis shows only moderate reliability. The strength of our study is the use of semi-automatic volumetric assessment of EOR and showing this method of determining EOR is highly reliable. Another strength is the statistical assessment of reliability of EOR surgeon . We found an equal distribution of over and underestimation of EOR by the surgeon, so we did not confirm the systematic overestimation of the surgeon as suggested by others [7]. We did find that accuracy of the surgeons' estimate is surgeon dependant. So, some surgeons estimate the resection percentage better than others. Even though the use of fluorescence resulted in a slight overestimation, it did increase the accuracy of surgeons' estimate.
There were a few limitations in our study. Our study was powered to detect accuracy of 80%, but the low number of subjects limits the factors that can be included in our multivariate analysis. We have calculated accuracy in an indirect way, using a dichotomy with 99% EOR as the cut-off value. Nevertheless, the different statistical approaches in this study all show a comparable result: moderate agreement for surgeons' estimate and volumetric assessment by MRI. Finally, since residual volume is clinically more relevant than percentages, we have plotted surgeons estimating complete resection against residual volume. There was no estimation of complete resection when residual was above 6 mL. With minimal residual volume (< 2 mL, 2-4 mL, and 4-6 mL), 17 to 36% of surgeons estimated a complete resection (Supplemental Digital Content, Fig. 3).

Conclusion
Semi-automatic volumetric analysis showed a high interobserver agreement and should therefore be considered the gold standard for assessment of EOR. The introduction of fluorescence has resulted in better resections [10]. We found it to increase the accuracy of the surgeons' estimate of fluorescence, whilst resulting in a tendency towards overestimation. Even though surgeons' estimate of extent of resection has clearly improved since the report of Albert et al., the reliability of their estimation is statistically moderate. Therefore, early post-operative MRI scanning for evaluation of EOR remains paramount.

Compliance with ethical standards
Conflict of interest The authors declare that they have no conflict of interest.
Ethical approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent For this type of study, formal consent is not required.  Open Access This article is distributed under the terms of the Creative Comm ons Attribution 4.0 International License (http:// creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.