Introduction

Nationwide population-based analysis is now widely used to evaluate the prevalence of a specific disease and to discover the relationship between new risk factors and clinical outcomes [1, 2]. Moreover, as the analysis targets the whole population, this approach could give a real-world evidence, especially in patients who were not included in well-designed randomized controlled trials [3]. The National Health Insurance Service (NHIS) is a single-insurer system, which covers the whole population of Korea [4, 5]. The NHIS provides various medical information of the Korean population over the past decades, including inpatient and outpatient records of diagnostic codes, procedures, examinations including imaging studies, medication prescriptions, and demographic information of patients. Recently, public access to the NHIS database has been available, which enabled many researchers to apply the population-based analysis in various study fields including epidemiology, economics, environmental policy and industry and public health and medicine [6,7,8,9,10]. In a population-based analysis, diagnostic codes of the disease based on the International Classification of Diseases (ICD) code is used to identify clinical outcomes. However, in Korea, as the record of the diagnosis is cumulatively stored in the claims database, it is hard to differentiate true clinical events from previous records based on diagnostic codes. In this regard, most of the population-based studies in Korea had focused on primary prevention of the disease by excluding patients with previous diagnosis of the study outcome [6,7,8,9,10]. Previously, several studies had attempted to evaluate the validity of primary diagnostic codes in nationwide claims based on its higher association with true clinical events [11, 12]. However, there is a paucity of reports assessing the validity of the primary diagnostic codes in the NHIS database in Korea. Therefore, we aimed to validate the primary diagnostic codes at discharge in patients with major clinical outcomes including acute myocardial infarction (AMI), gastrointestinal bleeding (GIB), stroke, and intracranial hemorrhage (ICH).

Methods

Data source and study population

From the database of a single tertiary medical center in Korea (Seoul National University Hospital), patients who had primary diagnostic codes of AMI, GIB, stroke, or ICH at discharge from 2016 to 2017 were randomly selected (200 patients for each diagnosis). The diagnostic codes of clinical outcomes were based on the International Classification of Diseases code, tenth version (ICD-10 code, Table 1). The study protocol of the current study was approved waived for informed consent by the Seoul National University Hospital Institutional Review Board (No. 1805-133-948).

Table 1 Diagnostic codes (ICD-10) for clinical outcomes
Full size table

The validation of diagnosis for acute myocardial infarction and gastrointestinal bleeding

Among the 200 patients with primary discharge diagnostic codes of AMI (ICD-10 code I21, I22) at discharge, true AMI events were identified based on the universal definition of AMI [13] by reviewing medical records including discharge records electrocardiograms, laboratory results such as cardiac enzymes, procedure records including percutaneous coronary intervention or coronary artery bypass graft surgery and prescriptions of medications (antiplatelet agents, statin or beta-blockers).

Primary discharge diagnostic codes representing upper and lower GIB were selected (Table 1). Patients with a true GIB event were identified through reviewing discharge records, procedure records including gastroduodenoscopy or colonoscopy with endoscopic hemostasis or transcatheter arterial embolization, the initial serum hemoglobin level or the existence of blood transfusions of red blood cells. The true GIB events were defined as following: (1) definite visual detection of bleeding or clearly documented medical records of bleeding described by physicians; or (2) identification of bleeding sources by endoscopic or imaging studies, which were accordant to the previous study [14]. Moreover, the history of red blood cell (RBC) transfusion was used as the inclusion criteria to evaluate improvement in diagnostic accuracy.

The validation of diagnosis for stroke and intracranial hemorrhage

True cases of stroke were verified based on medical records, including discharge records, a duration of hospitalization, brain imaging studies including a computed tomography (CT) scan or a magnetic resonance imaging (MRI), and prescription records of antithrombotic agents. From the previous studies, the median duration of hospital stay was reported for 7.4 days (interquartile range 5.1–11.7) [15]. Therefore, the duration of hospitalization more than 3 days was analyzed whether it improves the accuracy of diagnosis. For antithrombotic agents, any prescription record of the medication at discharge was considered for inclusion criteria in patients with stroke diagnoses.

For the confirmation of patients with true ICH events, we reviewed medical records, including discharge records, a duration of hospitalization, brain CT or MRI, and prescription records of antithrombotic agents. To improve the accuracy, we added no prescriptions of antithrombotic agents at discharge as one of the diagnostic criteria of ICH.

Statistical analysis

The two investigators (JSP and KSI) reviewed the medical records independently to discriminate true clinical outcomes. In the cases with discordance, agreement by discussion was used to make a final decision. Positive predictive value (PPV), sensitivity, and specificity were evaluated for each diagnostic code of the clinical outcomes. The PPV was calculated using the following formula: the numbers of patients with the true event of study outcome confirmed by medical record review/the total numbers of patients with ICD-10 codes × 100 (%). The final diagnoses for false positive cases in each clinical outcome were summarized to overview the factors affecting the accuracy. Statistical analyses were performed using SPSS, version 21.0 (SPSS Inc., Windows, Chicago, IL, USA).

Results

Acute myocardial infarction

The validation result of AMI is given in Table 2. Among the 200 cases, 184 cases (92%) were identified to have true AMI events who had admitted to the hospital for. The rest of 16 (8.0%) cases were those who were admitted for follow-up elective coronary angiography (n = 11, 5.5%), presented with angina (n = 2, 1.0%) or had previous history of AMI (n = 3, 1.5%) (Additional file 1: Table S1). The PPV of the diagnostic code was 92.0%.

Table 2 Validation of diagnostic codes for myocardial infarction
Full size table

Gastrointestinal bleeding

Among the 200 randomly selected cases, 184 patients (92%) were confirmed to be true GIB events (Table 3). The major proportion of false cases were those who had a previous history of GIB (n = 9, 4.5%) followed by those with no clinical evidence of GIB (n = 4, 2.0%) and bleeding event other than GI tract (3, 1.5%) (Additional file 1: Table S1). When RBC transfusion was combined with diagnostic code, 44 cases (22.0%), with 41 true events, were excluded, and there was no improvement in PPV (91.7%) with sensitivity and specificity of 77.7% and 18.8%, respectively.

Table 3 Validation of diagnostic codes for gastrointestinal bleeding
Full size table

Stroke

Various combinations of diagnostic criteria for stroke are given in Table 4. The combination of primary diagnostic code with brain imaging studies showed PPV and sensitivity of 92.2% and 91.2%, respectively. By combining prescription records of antithrombotic agents with primary diagnostic codes, PPV and sensitivity were 93.4% and 93.4%, respectively. Among various inclusion criteria, the highest PPV, sensitivity, and specificity were achieved by combining primary diagnostic code with hospitalization over 3 days and prescription records of antithrombotic therapy (PPV 94.3%, sensitivity 90.1%, and specificity 47.4%).

Table 4 Validation of diagnostic codes for stroke
Full size table

Intracranial hemorrhage

Table 5 shows the validation results of various inclusion criteria for ICH events. The combination of primary diagnostic code with hospitalization and brain imaging studies showed PPV, sensitivity, and specificity of 81.4%, 95.1%, and 45.6%, respectively. When combining the primary diagnostic code with hospitalization and prescription of antithrombotic therapy, we found that PPV, sensitivity, and specificity were 72.1%, 90.2%, and 12.3%, respectively. By considering the condition for the duration of hospitalization (over 3 days), PPV and specificity showed mild improvement in both criteria.

Table 5 Validation of diagnostic codes for intracranial hemorrhage
Full size table

Discussion

We have validated that the primary diagnostic codes at discharge of major clinical outcomes comprising AMI, GIB, stroke, and ICH were favorable and reliable, which could be used in nationwide population-based studies based on the NHIS database.

Validation of primary diagnostic code in identifying acute myocardial infarction

In our study results, a reliable diagnosis for AMI was shown with PPV 92% based on primary diagnosis codes at discharge. The PPV of AMI in our result is comparable to those previously reported, which ranges from 93 to 96.9%, though the diagnostic codes in these studies were based on ICD-9 [11, 12]. Cheng et al. had reviewed medical records of 281 patients with the primary diagnostic code of AMI and showed PPV of 93% [11]. Also, they compared multiple inclusion criteria by combining medical records of antithrombotic agents or procedures for coronary revascularization. The highest PPV (95%) was found in that comprising primary diagnosis for AMI and percutaneous coronary intervention, although it showed the lowest sensitivity (62%). In the current study, we did not include information of coronary revascularization, such as percutaneous coronary intervention or bypass surgery. As a substantial proportion of patients with AMI would not require coronary revascularization [16], considering procedure information with diagnostic code in defining AMI may increase false negative rates. Petersen et al. had reported PPV of 96.9% of AMI among 4712 patients with primary diagnostic code for AMI at admission [12]. In his report, of 147 false positive cases, 54% (n = 82) of them were those only with a previous history of AMI, which were similar to our results. Previously, Kimm et al. had reported validation study for AMI using diagnostic codes based on ICD-10, which showed somewhat a lower diagnostic accuracy of 71.4% compared to our result [17]. In this study, the authors selected patients with any diagnostic code of I21 (not I22) from 1994 to 2007 and investigated medical information of the patients by survey with questionnaires from multiple study centers. Therefore, those with insufficient survey records would be excluded, and higher numbers of false positive cases could be included compared to previous studies using primary diagnostic code, which may result in the relatively lower rate of PPV. However, due to development in electronic medical record system and high sensitivity diagnostic modalities, the accuracy of the diagnostic codes for AMI has been improved recently, which is supported by the recent study from the same research group reporting improved PPV of 85.6% among 164 patients from 2009 to 2011 [18].

Validation in gastrointestinal bleeding

In our study results, the primary discharge diagnostic code showed PPV of 82% for true GIB event. Cunningham et al. had reported a validation of bleeding events among 186 warfarin users by using primary diagnostic code during hospitalization [19]. Of the 120 patients who had a primary diagnostic code for GI bleeding, PPV was 85.8%. And, 73.8% of those patients were definite bleeding events. Another study validated bleeding codes comprising intracranial, gastrointestinal, genitourinary, and respiratory bleeding from ICD-9 in 361 warfarin users reporting PPV of 91% [14]. In this study, patients who were hospitalized due to the bleeding event, accompanied with a decrease in hemoglobin level > 2 g/L or had undergone ≥ 2 units of RBC transfusion were defined as those with major bleeding events, which showed PPV of 87%. Although hemoglobin level is not available in the NHIS database, history of RBC transfusion could be analyzed in those with GIB events. However, we found that combining information of RBC transfusion with primary diagnosis at discharge did not improve the PPV and resulted in lower sensitivity. Although RBC transfusion related to bleeding events was considered as a major or severe bleeding condition in the previous classification system [20], it could exclude patients with evident GIB events but without indication for blood transfusion. In our result, after combining RBC transfusion with primary diagnostic code and hospitalization, 44 patients were excluded whom 93.2% of them were true events. Also, as most of the false positive cases had RBC transfusion based on a low level of hemoglobin without clear gastrointestinal bleeding focus, information on RBC transfusion cannot discriminate these patients. However, considering that all patients with GIB in our study had hospital admission, the primary diagnostic code of GIB with hospitalization might be sufficient to identify patients with true major GIB.

Validation of diagnostic code in stroke and intracranial hemorrhage

Previous number of studies had validated the diagnostic codes for stroke, and ICH with a wide range of PPV depends on different criteria for outcome confirmation [21, 22]. Golomb et al. had validated ICD-9 codes of stroke among 663 children from 1999 to 2004 with outcome confirmation based on radiologic evidence of stroke. In this report, compared to diagnostic code in any position, those in the primary position showed better PPV (88% to 92% for code 436 and 37% to 53% for code 342). Park et al. had reported a validation study among 626 patients with a primary diagnosis for cerebrovascular disease (ICD-10, I60–I68) in medical claims from 1993 to 1997 [23]. The accuracy rate of the diagnostic code for stroke was 83.4% for I63 and 71.4% for I64–I68. Although the medical information of the patients was retrieved by trained record administrator using survey methods from study centers, 18.2% (n = 114) of patients were excluded for analysis due to incomplete or unavailable medical information, which may result in relatively lower accuracy than our result.

In the current study, we tried to improve the accuracy of the primary diagnostic code by combining relevant clinical information. In both stroke and ICH, a combination of primary discharge diagnostic code, hospitalization at least 3 days, and conditions for prescription of antithrombotic agents showed the highest PPV. However, the specificity was low ranged less than 60% regardless of combination with brain imaging studies or conditions for prescription of antithrombotic agents (26.3% to 47.4% for stroke and 12.3% to 57.9% for ICH). As for the false positive cases in stroke event, most of them were those with a previous history of stroke or suspected for symptoms which show no evidence of stroke in clinical or any diagnostic modalities. Majority of false positive cases in ICH were those with previous history of ICH and those admitted for unrelated medical issues such as a routine follow-up examination for cerebral angiography. Previously, Park et al. had investigated factors associated with the inaccuracy of diagnostic codes in medical insurance claims [23]. Among them, the most frequent reason of the inaccuracy was to meet review criteria of medical insurance benefits (n = 45 of 85 cases, 52.9%) followed by incorrect coding habits (n = 13, 15.3%) and diagnostic error (n = 9, 10.6%), which would explain the relatively higher false positive rate in diagnostic codes for stroke and ICH.

Limitations

Several limitations of our study must be clarified. First, as for MI validation, we did not include the prescription records in validation compared to the previous report [11]. However, we concluded that it is sufficient to discriminate the true MI events based on primary diagnosis, and there could be false negative cases excluded by applying strict inclusion criteria. Second, as for GIB, laboratory data of hemoglobin level or hemodynamic condition of the patient such as systolic blood pressure or pulse rate could not be considered as inclusion criteria which were not available in the NHIS database. Third, the specificity of the inclusion criteria in GIB, stroke, and ICH was relatively low. For GIB and stroke, the primary diagnostic code demonstrated favorable performance in PPV (92.0% and 90.5% for GIB and stroke, respectively), and the number of false positive cases was small (16 and 19 for GIB and stroke, respectively). As most of those cases were included to the additional criteria (i.e., RBC transfusion, brain imaging studies, or prescription records of antithrombotic therapy) irrelevant to the true diagnosis, the specificity was low. For ICH, however, the primary diagnostic code showed moderate accuracy (PPV 71.5%). Therefore, the additional inclusion criteria improved both PPV and specificity. These results suggest that primary diagnostic code would be sufficient to identify true patients correctly, and supplementary clinical information could be considered in case higher PPV is pursued. Fourth, this study was performed in a tertiary medical center, and the definition was not validated in primary and secondary hospitals, so further study including primary and secondary hospital would be needed. Finally, this study could not evaluate the patients with false negative. We only analyzed the patients with the primary diagnostic code of clinical outcomes, but it is also important to assess the patients without primary diagnostic code but actual clinical events.

Conclusion

Primary discharge diagnostic code showed favorable reliability in AMI and GIB which showed high PPV in those with primary diagnostic code at discharge and hospitalization. Stroke and ICH showed the highest accuracy in a combination of primary diagnostic code, hospitalization, and prescription. Primary discharge diagnostic code of AMI, GIB, stroke, and ICH could be used for outcome study using the NHIS database.