Interpretations of diffusion-weighted MR imaging by radiology residents in the emergency department: is diagnostic performance influenced by the level of residency training?

Background To evaluate the diagnostic performance of radiology residents’ interpretations for diffusion-weighted MR imaging (DWI) in the emergency department at different levels of residency training. Method and materials A total of 160 patients who underwent DWI with acute neurologic symptoms were included in this retrospective study with an institutional review board approval. Four radiology residents with different training years and one attending neuroradiologist independently assessed the DWI results. Discordances between the results of residents and attending neuroradiologist were classified as follows: false positive (FP) and false negative (FN). We also evaluated the diagnostic performance of four residents according to the reference standard. Results Overall, the concordance rate was 84.8 %, with a 15.2 % overall discordance rate. There were 83 FN results. The most common misses were small vessel disease (n = 55), acute focal infarction (n = 10), diffuse axonal injury (n = 6), solitary mass (n = 5), extraaxial hemorrhages (n = 3), posterior reversible encephalopathy syndrome (n = 2), and postictal change (n = 2). Fourteen FP results were interpreted as hemorrhage and acute infarction. The 4th year resident exhibited the highest diagnostic performance, and the level of training had a significant influence on the rates of concordance (P < 0.05). Interobserver reliability was good between the interpretations of the residents and the final interpretations of the attending neuroradiologists. Conclusion The level of resident training had a significant effect on their diagnostic performance, and good interobserver reliability was noted between the results of the residents and attending neuroradiologist.


Conclusion
The level of resident training had a significant effect on their diagnostic performance, and good interobserver reliability was noted between the results of the residents and attending neuroradiologist.
Keywords Diffusion-weighted MR imaging · Radiology · Residency training · Diagnostic performance

Background
In the emergency department (ED), patients with acute neurologic deficits are carefully evaluated for a timely diagnosis of intracranial abnormalities by performing neuroimaging studies, such as computed tomography (CT) and magnetic resonance (MR) imaging. Numerous institutions are increasingly performing diffusion-weighted imaging (DWI) for patients with sudden neurologic deficits in the ED to save time and make a correct diagnosis. Furthermore, radiology residents commonly provide preliminary interpretations of neuroimaging studies ordered by the ED at most academic medical centers [1]. Several previous studies reported agreement or discrepancy rates of radiologic examinations [1][2][3][4][5]. However, to date, no objective study has shown a comparison of diagnostic performances with statistical significance for interpreting DWI in radiologic residents with different training years. We hypothesized that the relative inexperience of junior residents may lead to increased discordances of their interpretations and that the level of residency training may be related to the discordance rate. Therefore, the purpose of this study was to retrospectively assess the rates of diagnostic discordances for DWI in ED between the interpretations of radiology residents and the final interpretations of an attending neuroradiologist. We also sought to evaluate the diagnostic performance of radiology residents at different levels of residency training.

Study population
A review of the database of our institution identified 297 consecutive patients who underwent DWI in the emergency department between September 2015 and December 2015. We then selected 213 of these 297 patients with acute neurologic symptoms using electronic medical charts and picture archiving and communicating system (PACS). Of these 213 patients, 53 were excluded due to inadequate medical records (n = 24); poor image quality, including motion artifacts or susceptibility artifacts (n = 19); and inadequate diagnosis by only DWI (n = 10). The final 160 patients who were included in this study comprised 84 males and 76 females (age range, 28-86 years; mean age, 63.4 years). Retrospective data collection and analysis were performed according to our local institutional review board (IRB) guidelines after its approval, and the IRB determined that patient approval and informed consent were not required for reviewing images and records.

Imaging analyses and reference standard
In our institution, we had a total eight radiologic residents in the radiologic department. Of these eight residents, four residents could not join in this study, because of a secondment for outreach education of interventional radiology, personal reason, and training schedule. Finally, four radiology residents at different training year levels interpreted the DWI of all patients. At the time of this study, the 1st year resident completed one-half of her 1st year of training, had 2 months of neuroradiology experience interpreting both CT and MR imaging, and participated in neuroradiology teaching conferences, including staff lectures and interesting case presentations. All residents evaluated images and recorded the following information: the presence of abnormalities, location of detected abnormalities, and presumed diagnosis. An attending neuroradiologist (H.J.B. with 6 years of experience in brain, head, and neck imaging) also interpreted the same images independently, and her interpretations were used as the reference standard. All reviewers were blinded to patient clinical data, except the reason for the examination. Discordance between the resident's and staff's interpretations was classified as either false positive (FP; e.g., misinterpreting normal images as abnormal) or false negative (FN; e.g., failure to diagnose an abnormality). In addition, concordance was classified as true positive (e.g., agreement of results between resident and staff) or true negative (e.g., negative finding).

Statistical analysis
Data were analyzed using Fisher's exact test for evaluating discordance rates. The diagnostic indices (sensitivity, specificity, positive and negative predictive values, and accuracy) of each resident were also calculated. A receiver-operating characteristic (ROC) curve was constructed to evaluate the diagnostic performance of each resident with the largest Az value. Interobserver agreement between residents and attending neuroradiologist was assessed by kappa (κ) statistics. The κ statistics results were interpreted as follows: κ values ranging from 0.21 to 0.40 indicated fair agreement; 0.41 to 0.60 indicated moderate agreement; 0.61 to 0.80 indicated good agreement; and 0.80 to 1.00 indicated very good agreement. All statistical analyses were performed with statistical software (SPSS, version 19.0, SPSS, Chicago, IL, USA; MedCalc, version 9.0, MedCalc Software, Mariakerke, Belgium), and P values less than 0.05 were considered statistically significant.
Among residents with different years of training, the 4th year resident exhibited the highest diagnostic performance with the largest area under the ROC curve   Fig. 1. A good degree of interobserver reliability was noted between all residents and attending neuroradiologist (P < 0.0001 and Table 4).

Discussion
In ED, patients with acute neurologic deficits are carefully evaluated for the early diagnosis of intracranial abnormalities with neuroradiologic examinations, including CT and MR imaging. However, routine MR imaging can be a time-consuming assessment for these patients. The previous studies demonstrated that DWI is an effective imaging modality that has reliable sensitivity and specificity in patients with acute neurologic problems [5,6]. Therefore, many institutions have used DWI in combination with FLAIR and T2*-weighted GRE as a timesaving substitute for routine brain MR imaging to make a timely diagnosis. At most academic medical centers, physicians in the ED request urgent DWI studies for patients with acute neurologic deficits and radiology residents are often responsible for providing preliminary interpretations of those studies before the final interpretations of the neuroradiologist become available [1]. However, the physicians' need for rapid and accurate diagnoses of neuroimaging studies can conflict with the need for radiologic residents to acquire clinical experience and confidence [3]. To maintain proper resident training, meticulous analyses of residents' misinterpretations and discordances between residents' and final interpretations are mandatory, because residents' interpretations may impact patient management and treatment planning in the ED. In this study, we retrospectively assessed the rates of diagnostic concordances and discordances for DWI in ED between the interpretations of radiology residents and the final interpretations of attending neuroradiologist. We also evaluated the diagnostic performance of radiology residents at different levels of residency training.
In this study, the overall rate of concordance was 84.8 % with 15.2 % of overall discordance rate, and most of discordances were insignificant. Although the discordance rate between the initial interpretations of head CT scans by ED physicians and the final interpretations by radiologists has been found to be nearly 39 % [7], the discordance  rate of residents is much lower in this study. This discordance rate is higher than that reported by investigators who examined radiology residents' interpretations of head CT scan or brain MR imaging studies below 5 % [1,2,[8][9][10][11][12][13].
The discordance rate of our study is better than that previously reported for imaging modalities of other body sections, where disagreement rates as high as 26 % were reported for chest radiography [14]. These differences may be produced by the relatively small number of enrolled patients and participating radiology residents as well as the selected imaging modality for the study. Our institution is a medium-sized academic medical center with 2 radiology residents at each level of training. Of a total of 8 residents, only 4 residents participated in this study due to their training schedule. Consistent with the previous results [1,3], we found that the discordance rate for the 1st year resident was significantly greater than those of 2nd, 3rd, and 4th year residents. Similarly, the diagnostic performance of high-level residents was also significantly greater for DWI interpretations. Our results suggest that clinical and educational experiences may play a role in interpreting imaging studies. Although individual differences exist, confident interpretation and decision-making is one of the most important educational and clinical experiences for radiology residents [1,14].
Of 160 cases, 14 FP findings with misinterpretations of acute focal infarction and focal hemorrhage were noted. All of the 14 FP lesions were small in size, and these errors were related to artifacts intrinsic to DWI, such as physiologic hyperintensity by anisotropy or T2 shine-through effect (Figs. 2, 3). Interpretations can be made easier if radiology residents keep these errors in mind.
Among FN cases, grade I small vessel disease was the most common missed diagnosis followed by acute focal infarctions (Figs. 4, 5). Fortunately, these FN cases were not significant. All of these lesions were relatively small in size and number; thus, a more careful imaging evaluation may improve the diagnostic accuracy of resident's interpretations. In the case of acute focal infarction, the meticulous evaluation of the apparent diffusion coefficient map can be helpful to make an accurate diagnosis.
In addition, our study showed good interobserver reliability between the interpretations of residents and attending neuroradiologist. This result suggests the possibility that the resident's interpretations of DWI may be reliable in the patient with acute neurologic deficits who visit the ED before the final interpretations of the subspecialized neuroradiologist become available. Interestingly, κ values for interobserver reliability between residents and attending neuroradiologist tended to increase as the level of the compared resident increased. These results may indicate that clinical experiences during residency training can be an important factor for imaging interpretation.
There are several limitations of this study. First, a relatively small number of enrolled patients and participating residents at each level of residency training were included in this study. Therefore, our study had a weakness for generalization. Second, we did not investigate the clinical outcome during patient's total hospital stay, because this study was retrospective. Thus, we could not analyze the final effect of residents' interpretations on patients' clinical outcomes. Third, insufficient evaluations were available for infratentorial lesions, because DWI examinations performed at our institution did not focus on the posterior fossa with thin-section slices. Finally, we used the final interpretation of only one attending neuroradiologist as the reference standard. Ultimately, we could not evaluate the possibility of FP and FN results made by the attending radiologist. However, this method was used successfully in the previous studies [1,3,8]. To valid our result, further studies with additional attending neuroradiologists This finding is characteristic of acute lateral medullary infarction. However, all residents missed this lesion and interpreted the image as normal. c, d Two days after admission, the lesion increases in size with more conspicuous ADC change or more experienced senior attending neuroradiologists are required.

Conclusion
In conclusion, high-level residents exhibited a better diagnostic accuracy for interpreting DWI ordered from the ED compared with junior residents, and the level of resident training had a significant effect on their diagnostic performances. Good interobserver reliability was noted between the interpretations of residents and attending neuroradiologist. Therefore, radiology residents can safely provide interpretations of DWI requested by the ED, and efforts to focus on detecting small lesions can be helpful to reduce residents' errors. Consent for publication Written informed consent was obtained from the patient for publication of this study and accompanying images. A copy of the written consent is available for review by the Editor-in-Chief of this journal on request.
Ethical approval and consent to participate This was purely an observational retrospective study. The patients' management and outcome were unaltered using retrospective analyses of their imaging studies. Therefore, no ethical approval was required for this study. Retrospective data collection and analysis for this study were performed according to our Institutional Review Board (IRB) guidelines after its approval. In addition, the IRB determined that patient' approval and informed consent were not required for retrospectively reviewing images and records. A copy of the regarding IRB document about reason for exemption is available for review by the Editor-in-Chief of this journal on request.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.