Abstract
Purpose
We investigated whether the reproducibility of standard visual reporting (STD method) in flutemetamol (FMM) PET can be improved using a newly introduced method that uses grey matter edges derived from the perfusion phase (GM-EDGE method).
Methods
Two-phase FMM PET was performed in 121 patients with mild cognitive impairment. Five nuclear medicine physicians blindly and independently evaluated all late-phase scans, initially employing the STD method and later the GM-EDGE method. A five-point scale was used to express the degree of amyloid positivity, and a binary classification (positive/negative) was used in combination with subjective confidence (five-point scale). Multirater Fleiss’ kappa, intraclass correlation coefficient (ICC) and inter-rater reliability (Cohen’s kappa) were determined for the STD and GM-EDGE methods.
Results
The weighted Cohen’s kappa values for the five-point measure of amyloid positivity ranged from 0.63 to 0.73 (median 0.70) for the STD method and from 0.76 to 0.89 (median 0.80) for the GM-EDGE method (ICC 0.84, 95% CI 0.79–0.88, for the STD method; 0.91, 95% CI 0.89–0.94, for the GM-EDGE method). The nonweighted Cohen’s kappa value for the binary classification ranged from 0.73 to 0.93 (median 0.82) for the STD method and 0.90 to 0.97 (median 0.93) for the GM-EDGE method (Fleiss’ kappa 0.82, 95% CI 0.77–0.88, for the STD method; 0.93, 95% CI 0.87–0.99, for the GM-EDGE method). The GM-EDGE method resulted in significantly greater subjective confidence in the readings of four physicians (p < 0.010). The binary classification was concordant among all five physicians in 80.8% of the scans using the STD method and in 91.6% of the scans using the GM-EDGE method (p = 0.016).
Conclusion
The newly introduced GM-EDGE method was associated with significantly higher inter-rater agreement among physicians and higher subjective confidence in the reading. The method is easy to implement in clinical practice, especially when the perfusion phase is utilized clinically.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
18F-Labelled beta-amyloid PET tracers, including florbetaben, florbetapir and flutemetamol (FMM), have successfully crossed the gap between research laboratories and routine clinical practice. Nevertheless, specific training of physicians is required according to the Summary of Product Characteristics (SPC) for tracers authorized for marketing in European countries (Neuraceq, Amyvid, Vizamyl). The aim of training is to guarantee reliable interpretation of PET brain scans in view of the difficulty in distinguishing specific pathological uptake in the grey matter from nonspecific physiological uptake in the white matter. This is especially difficult in the presence of brain atrophy, when the grey matter ribbon is thin and the sulci are flat.
Various approaches have been proposed to facilitate reporting and/or to quantify beta-amyloid load in the brain. They include registration of PET and MRI data, in which regions of interest in the grey matter are defined manually or using various automated approaches [1,2,3]. Other PET-only approaches use spatial normalization to a brain template [4] including adaptive template registration [5, 6]. The value of partial volume correction has also been considered [7] as well as standardizing quantitative amyloid PET using a centiloid scale [8]. The field is open for the application of artificial intelligence as well [9]. All these approaches are complex and they are predominantly intended for use in research. Routine clinical practice still relies on subjective visual scoring defined in the SPC.
Some authors have reported the use of beta-amyloid PET tracers for the assessment of brain perfusion [10,11,12,13]. For this purpose, supplementary data are acquired several minutes after administration of the radiotracer (early phase), when the signal from the brain represents mainly the perfusion of grey matter. In addition to the clinical benefit of the combined evaluation of brain perfusion and amyloid positivity, it is possible to easily extract the borders of the grey matter by thresholding the perfusion images, and then to superimpose these borders presented on the isocontour colour scale onto late-phase images. This facilitates delineation of white/grey matter borders and thus the determination of late-phase FMM uptake in the grey matter, which is related to the beta-amyloid load. We introduced this method [14] as an easy-to-use method, easily applicable in routine clinical practice.
The aim of this work was to assess inter-rater agreement with the standard FMM PET evaluation according to the SPC (STD method) [15] for Vizamyl as well as with the new method of evaluation involving superimposition of the grey matter borders onto late-phase images (GM-EDGE method).
Materials and methods
Patients
Over a period of 20 months (August 2016 to April 2018), 121 FMM PET/CT investigations were performed in 121 consecutive patients for clinical indications fulfilling the following criteria for reimbursement from the national health insurance system: for evaluation of possible Alzheimer disease (AD) not proven by National Institute on Aging-Alzheimer’s Association criteria [16], stable or progressive unexplained mild cognitive impairment or progressive dementia with atypical initial phase, for differential diagnosis (especially frontotemporal dementia), and for definition of dementia type in ambiguous cases when interpretation of clinical evaluation was not possible. Patients with clinically proven AD, patients undergoing assessment of dementia burden, asymptomatic subjects with a positive family history or ApoE genotype, and patients subjectively suffering from memory impairment without positive neuropsychological investigation were not included.
PET/CT acquisition
A gross activity of 206.7 ± 12.7 MBq of FMM (Vizamyl; GE Healthcare) was measured 3.9 ± 3.0 min before dose administration. A noncontrast low-dose CT scan was performed for attenuation correction in the PET/CT scanner (Biograph 40 TrueV HD; Siemens). A PET list-mode acquisition was performed starting at the time of FMM administration. Data were acquired for 8 min and rebinned to dynamic datasets of 4 × 2 min without attenuation correction for motion checking only. The early-phase images were initially iteratively reconstructed to a 168 × 168 matrix, with three iterations, 21 subsets, zoom 2 and Gaussian filter 2 mm using attenuation, scatter and point spread function correction. After an uptake time of 90.6 ± 5.1 min, late-phase data were acquired for 20 min and reconstructed to a 168 × 168 matrix, with the other parameters as given above including rebinning into dynamic sequences for motion checking. After initial experience with the first 25 investigations, we shortened the late-phase acquisition time from 20 min to 10 min due to frequent patient motion in the late phase of the acquisition. We also changed the reconstruction matrix from 168 × 168 to 128 × 128 (with the other parameters as given above) to achieve a low level of noise without apparent loss of diagnostic image quality.
Observers
Five physicians (referred to as A, B, C, D and E) certified in nuclear medicine with 5–14 years of experience in PET/CT retrospectively blindly and independently evaluated all 121 FMM PET scans extracted from the PACS in two independent runs. However, their previous experience with amyloid FMM PET reporting varied: 121, 93, 10, 0 and 0 cases, respectively. All investigators had successfully completed training prescribed in the SPC [15] 1 year previously.
Image analysis
The Siemens syngo.via MM Reading tool was used for image analysis. Patient studies were arranged alphabetically. In the first run, only the late-phase images were evaluated using the STD method reported in the SPC [15] and the colour scale “Spectrum”, i.e. images were visually interpreted by comparing the activity in the cortical grey matter with the activity in the adjacent white matter. A region was considered abnormal if the tracer signal in cortical regions appeared high, i.e. approximately the same or higher than the signal intensity in the adjacent white matter and higher than that in the grey matter-rich regions of the cerebellum. If any one of following regions was clearly abnormal, then the finding was classified as positive for beta-amyloid: frontal lobes and lateral temporal lobes, anterior and posterior cingulate, precuneus, striatum and temporoparietal areas and the insula.
The instructions to start the second read were released 6 days after the start of the first read. In the second run, both early-phase and late-phase reconstructed volumes were mutually registered and oriented using the Siemens syngo.via MM Reading tool. Early-phase images were presented in the “Edges” isocontour colour scale and the threshold was set to delineate a thin superficial brain ribbon that mainly represented grey matter (GM). Late-phase images were presented in the “Spectrum” colour scale and the upper threshold was adjusted to 90% in the pons and cerebellar peduncles. Alpha blending of early-phase edge images with late-phase images was done by manually adjusted the mixing ratio (GM-EDGE method). Beta-amyloid-specific uptake in the delineated grey matter was evaluated in the same way as in the first run, but without knowledge of the previous result. Typical examples of image processing are discussed below.
Each physician reported beta-amyloid positivity using the STD and GM-EDGE methods in two ways. First, a five-point ordinal numerical scale was used: 1 clearly negative, 2 probably negative, 3 ambiguous, 4 probably positive, and 5 clearly positive. Second, the scans were classified in a binary fashion as negative or positive, together with the application of a five-point ordinal numerical scale expressing the subjective level of confidence: 1 very low, 2 low, 3 intermediate, 4 high, and 5 very high.
Statistical analysis
The intraclass correlation coefficient (ICC) as a measure of absolute agreement of the ratings for one typical single rater (single measures) was determined for the five-point scale of beta-amyloid positivity using the STD and GM-EDGE methods. Additionally, inter-rater agreement (linearly weighted Cohen’s kappa) was calculated for the STD and GM-EDGE methods in physician pairs. Fleiss’ fixed-marginal multirater kappa was calculated for the binary classification of beta-amyloid positivity. In addition, inter-rater agreement (nonweighted Cohen’s kappa) was determined for the binary classification of beta-amyloid positivity. A paired t test was used to evaluate any differences in subjective confidence between the STD and GM-EDGE binary classifications for each physician. The chi-squared test was used to test concordance between the STD and GM-EDGE binary decision frequencies for all five physicians. Values of p <0.050 were considered as statistically significant. Statistical analyses were performed using MedCalc, version 18.5, except for the calculation of Fleiss’ kappa [17].
Results
The cohort of 121 consecutive patients consisted of 64 women and 57 men with a median age of 71 years (range 40–90 years). One patient showed significant movement in the late-phase, and the scan in this patient was excluded from analysis of the STD and GM-EDGE methods. In another patient, the early-phase images were too noisy to reliably delineate the brain boundary, and the scan in this patient was therefore excluded from analysis of the GM-EDGE method. In total, the scans from 120 patients were available for analysis of the STD method and from 119 patients for analysis of the GM-EDGE method.
Ten one-to-one physician pairs were generated. Cohen’s kappa values for inter-rater agreement for the five-point beta-amyloid positivity scale for each physician pair are shown in Table 1 for the STD and GM-EDGE methods. The median kappa value for the STD method was 0.70 (range 0.63–0.73), and for the GM-EDGE method was 0.80 (range 0.76–0.89). The ICC was higher for the GM-EDGE method (0.91, 95% CI 0.89–0.94) than for the STD method (0.84, 95% CI 0.79–0.88; non-overlapping 95% CIs for ICC indicate that the inter-rater agreement for the GM-EDGE method was significantly higher). Cohen’s kappa values for inter-rater agreement for the binary beta-amyloid classification for each physician pair are shown in Table 2 for both methods. The median kappa value for the STD method was 0.82 (range 0.73–0.93), and for the GM-EDGE method was 0.93 (range 0.90–0.97). Fleiss’ fixed-marginal multirater kappa value was higher for the GM-EDGE method (0.93, 95% CI 0.87–0.99) than for the STD method (0.82, 95% CI 0.77–0.88).
The arithmetic mean scores on the five-point scale for subjective confidence in the binary classification for individual physicians were 4.51, 4.55, 4.61, 4.11 and 4.38 for the STD method, and 4.88, 4.87, 4.79, 4.35 and 4.41 for the GM-EDGE method (Fig. 1). The subjective confidence in the reading of four physicians was significantly higher with the GM-EDGE method than with the STD method (p < 0.001, p < 0.001, p = 0.006, p < 0.001). The subjective confidence in the reading of one physician without previous experience with amyloid FMM-PET was nonsignificantly higher with the GM-EDGE method (p = 0.717).
Six subgroups of patients were created according to concordance in the binary classification among the five physicians. Observed counts for the STD and GM-EDGE methods are presented in Table 3 and Fig. 2. Using the STD method, all five physicians expressed the same decision in 97 of 120 patients (80.8%; 47 positive and 50 negative). Using the GM-EDGE method, all five physicians expressed the same decision in 109 of 119 patients (91.6%; 51 positive and 58 negative). The difference in frequencies was statistically significant (p = 0.016).
Discussion
Visualization of the border between white and grey matter using the GM-EDGE method enabled the easy distinction between normal nonspecific FMM uptake in the white matter and pathological uptake in the grey matter, consistent with the presence of beta-amyloid. Figures 3, 4 and 5 show FMM PET imaging in three example patients demonstrating the benefits of the use of the GM-EDGE method.
Brain biopsy/necropsy is considered the gold standard for determining the presence of beta-amyloid in the brain, but it is not commonly performed in the clinical setting and thus was not available in our cohort of patients. We were therefore not able to define the diagnostic accuracy of the methods. We did, however, evaluate inter-rater reliability for the GM-EDGE and STD methods in determining the presence of beta-amyloid. The median Cohen’s kappa value among the five physicians was 0.82 for the binary classification using the STD method. This result is consistent with kappa values in the range 0.73–0.82 reported by Harn et al. [18], who evaluated the agreement among three experienced raters in the reading of 55 florbetapir PET scans, and with the kappa values reported by Buckley et al. [19], who found that most values were greater than 0.80. Yamane et al. [20] found a slightly higher kappa value of 0.89 with 11C-based Pittsburgh compound B, which may be explained by the fact that 57 normal controls were included in the test cohort of 162 patients. Patients without cognitive impairment might exhibit lower variability leading to higher kappa values than patients with various levels of cognitive impairment, as those in our cohort.
Inter-rater reliability was significantly higher with the GM-EDGE method than with the STD method. The median Cohen’s kappa values for the binary classification among the five physicians were 0.82 with the STD method and 0.93 with the GM-EDGE method. This excellent inter-rater agreement is comparable with those reported by Harn et al. [18], who improved image reading with the addition of quantitative information extracted with the software MIMneuro (kappa 0.88–0.96). Together with the improvement in kappa values using the GM-EDGE method, there was also a significant decrease in the frequency of discordant reports among the physicians. Therefore, the use of the GM-EDGE method may improve consensus reading without any special software.
We evaluated the reports of five nuclear physicians, two experienced (>90 reports), two with no experience except the required training, and one with minimal experience with ten FMM PET/CT reports before starting the retrospective evaluation. For the binary classification of beta-amyloid positivity, inter-rater reliability was comparable between experienced physicians A and B and between physicians D and E without any experience with the STD method (Cohen’s kappa 0.83 and 0.87, respectively) and with the GM-EDGE method (Cohen’s kappa 0.93 and 0.93, respectively). This is a promising finding with respect to the reliability of beginners in reading of FMM PET/CT.
We also tested the subjective confidence of raters in the binary classification of beta-amyloid positivity. This result should be viewed with caution because possible bias might have been introduced by the subjective expectations of the authors, and by the possible scepticism of some physicians concerning the introduction of a new approach. Nevertheless, the physicians showed a higher degree of confidence in the GM-EDGE method.
At the time that FMM PET was introduced into clinical practice in our department, we did late-phase PET registration with common MRI sequences as recommended in the SPC [15]. The white/grey matter borders were not well demarcated on fused images, and photon spillover from white to grey matter complicated assessment. On the other hand, both early-phase and late-phase PET datasets are based on the same physical principle and therefore exhibit the same spatial resolution and the same intrinsic artefacts including the same photon spillover from grey to white and white to grey matter. Thresholding both datasets can easily compensate for spillover and facilitate assessment. This is a clear advantage over image fusion with MRI.
We assume that the GM-EDGE method would show similar performance in dual-phase PET with other beta-amyloid-seeking tracers. We have also confirmed the feasibility of this method in a group of more than 200 florbetapir PET scans (unpublished data).
Adequate evaluation of beta-amyloid PET data using both the STD and GM-EDGE methods requires the cooperation of patients in keeping their head still during data acquisition. Slight head movement during the late phase blurs white matter activity into the grey matter and may lead to false-positive findings. Similarly, slight head movement during the early phase results in fuzzy, enlarged grey matter. For correct interpretation, a check of movement on the cine display in all three planes and selection of a motion-free period for the final data reconstruction is essential.
Conclusion
We introduce a simple method for beta-amyloid PET evaluation that is easily applicable to routine clinical practice without specialized software. This method exhibits excellent inter-rater agreement (median Cohen’s kappa 0.93) that is significantly better than that for the STD method of evaluation. The increased diagnostic sensitivity, specificity and overall accuracy of the new method needs to be confirmed in multicentre trials based on histologically confirmed two-phase beta-amyloid PET datasets.
References
Zhou L, Salvado O, Dore V, Bourgeat P, Raniga P, Macaulay SL, et al. MR-less surface-based amyloid assessment based on 11C PiB PET. PLoS One. 2014;9(1):e84777 https://doi.org/10.1371/journal.pone.0084777.
Rosario BL, Weissfeld LA, Laymon CM, Mathis CA, Klunk WE, Berginc MD, et al. Inter-rater reliability of manual and automated region-of-interest delineation for PiB PET. Neuroimage. 2011;55(3):933–41. https://doi.org/10.1016/j.neuroimage.2010.12.070.
Knešaurek K, Warnock G, Kostakoglu L, Burger C. Comparison of standardized uptake value ratio calculations in amyloid positron emission tomography brain imaging. World J Nucl Med. 2018;17(1):21–6. https://doi.org/10.4103/wjnm.WJNM_5_17.
Joshi AD, Pontecorvo MJ, Lu M, Skovronsky DM, Mintun MA, Devous MD Sr. A semiautomated method for quantification of F18 florbetapir PET images. J Nucl Med. 2015;56(11):1736–41. https://doi.org/10.2967/jnumed.114.153494.
Lundqvist R, Lilja J, Thomas BA, Lötjönen J, Villemagne VL, Rowe CC, et al. Implementation and validation of an adaptive template registration method for 18F-flutemetamol imaging data. J Nucl Med. 2013;54(8):1472–8. https://doi.org/10.2967/jnumed.112.115006.
Thurfjell L, Lilja J, Lundqvist R, Buckley C, Smith A, Vandenberghe R, et al. Automated quantification of 18F-flutemetamol PET activity for categorizing scans as negative or positive for brain amyloid: concordance with visual image reads. J Nucl Med. 2014;55(10):1623–8. https://doi.org/10.2967/jnumed.114.142109.
Su Y, Blazey TM, Snyder AZ, Raichle ME, Marcus DS, Ances BM, et al. Partial volume correction in quantitative amyloid imaging. Neuroimage. 2015;107:55–64. https://doi.org/10.1016/j.neuroimage.2014.11.058.
Klunk WE, Koeppe RA, Price JC, Benzinger TL, Devous MD Sr, Jagust WJ, et al. The Centiloid project: standardizing quantitative amyloid plaque estimation by PET. Alzheimers Dement. 2015;11(1):1-15.e1–4 https://doi.org/10.1016/j.jalz.2014.07.003.
Vandenberghe R, Nelissen N, Salmon E, Ivanoiu A, Hasselbalch S, Andersen A, et al. Binary classification of 18F-flutemetamol PET using machine learning: comparison with visual reads and structural MRI. Neuroimage. 2013;64:517–25 https://doi.org/10.1016/j.neuroimage.2012.09.015.
Blomquist G, Engler H, Nordberg A, Ringheim A, Wall A, Forsberg A, et al. Unidirectional influx and net accumulation of PIB. Open Neuroimag J. 2008;2:114–25. https://doi.org/10.2174/1874440000802010114.
Hsiao IT, Huang CC, Hsieh CJ, Hsu WC, Wey SP, Yen TC, et al. Correlation of early phase 18F-florbetapir (AV-45/Amyvid) PET images to FDG images: preliminary studies. Eur J Nucl Med Mol Imaging. 2012;39(4):613–20. https://doi.org/10.1007/s00259-011-2051-2.
Lin KJ, Hsiao IT, Hsu JL, Huang CC, Huang KL, Hsieh CJ, et al. Imaging characteristic of dual-phase (18)F-florbetapir (AV-45/Amyvid) PET for the concomitant detection of perfusion deficits and beta-amyloid deposition in Alzheimer’s disease and mild cognitive impairment. Eur J Nucl Med Mol Imaging. 2016;43(7):1304–14. https://doi.org/10.1007/s00259-016-3359-8.
Valentina G, Silvia M, Marco P. Dual-phase amyloid PET: hitting two birds with one stone. Eur J Nucl Med Mol Imaging. 2016;43(7):1300–3. https://doi.org/10.1007/s00259-016-3393-6. Erratum in: Eur J Nucl Med Mol Imaging. 2016;43(9):1747. 10.1007/s00259-016-3426-1.
Belohlavek O, Jaruskova M. An easy way to increase confidence in beta-amyloid PET evaluation. Nucl Med Rev. 2017;20(2):107–9. https://doi.org/10.5603/NMR.2017.0019.
GE Healthcare. Vizamyl. Summary of product characteristics. European Medicines Agency; 2014. http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Product_Information/human/002557/WC500172950.pdf. Accessed 23 Aug 2018.
McKhann GM, Knopman DS, Chertkow H, Hyman BT, Jack CR Jr, Kawas CH, et al. The diagnosis of dementia due to Alzheimer’s disease: recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement. 2011;7(3):263–9. https://doi.org/10.1016/j.jalz.2011.03.005.
Randolph JJ. Online kappa calculator. 2008. http://justusrandolph.net/kappa. Accessed 23 Aug 2018.
Harn NR, Hunt SL, Hill J, Vidoni E, Perry M, Burns JM. Augmenting amyloid PET interpretations with quantitative information improves consistency of early amyloid detection. Clin Nucl Med. 2017;42(8):577–81. https://doi.org/10.1097/RLU.0000000000001693.
Buckley CJ, Sherwin PF, Smith AP, Wolber J, Weick SM, Brooks DJ. Validation of an electronic image reader training programme for interpretation of [18F] flutemetamol β-amyloid PET brain images. Nucl Med Commun. 2017;38(3):234–41. https://doi.org/10.1097/MNM.0000000000000633.
Yamane T, Ishii K, Sakata M, Ikari Y, Nishio T, Ishii K, et al. Inter-rater variability of visual interpretation and comparison with quantitative evaluation of 11C-PiB PET amyloid images of the Japanese Alzheimer’s Disease Neuroimaging Initiative (J-ADNI) multicenter study. Eur J Nucl Med Mol Imaging. 2017;44(5):850–7. https://doi.org/10.1007/s00259-016-3591-2.
Funding
This work was supported by the Ministry of Health, Czech Republic – Conceptual Development of Research Organization (Nemocnice Na Homolce – NNH, 00023884), IG184401.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
None.
Research involving human participants and/or animals
For this type of study formal consent is not required. Nevertheless, the local institutional ethics committee approved retrospective re-evaluation of images archived at our institution, without any impact on further patient management. This article does not describe any studies with animals performed by any of the authors.
Informed consent
All patients and/or their guardians provided signed, informed consent in the course of routine investigation according to the local standard of care.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Belohlavek, O., Jaruskova, M., Skopalova, M. et al. Improved beta-amyloid PET reproducibility using two-phase acquisition and grey matter delineation. Eur J Nucl Med Mol Imaging 46, 297–303 (2019). https://doi.org/10.1007/s00259-018-4140-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00259-018-4140-y