Background

Over the last 30 years, electronic medical records (EMR) are becoming a standard tool of care in both in ambulatory and inpatient settings. Error and data loss reduction, patient-care efficiency increase, decision-making assistance and facilitation of event surveillance are some of the many processes that EMRs help improve [1, 2]. In addition, they show a lot of promise in terms of data collection to facilitate observational epidemiological studies and their use for this purpose has increased significantly over the recent years [3,4,5].

Even though the quantity and availability of the data for research are clearly improved thanks to EMRs, still, the problem of data-quality remains. This is especially important when attempting to determine if an event has actually occurred or not. In case of having a problem-based EMR, the determination of the events can be performed through the selection of combinations of coded terms (diagnoses, procedures, etc.), and also through the development of algorithms which use the data contained in free-text clinical notes [3, 6]. This can be implemented towards determining the occurrence of events [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22], but also in order to define exposure to risk factors [23,24,25,31] (i.e., diabetes, smoking status, etc.), medication use [32,33,34,35,36,37] and monitoring adverse events [38,39,40]. This process is usually referred to as electronic phenotyping.

The Hospital Italiano de Buenos Aires (HIBA, Italian Hospital of Buenos Aires) has developed and implemented an in-house EMR since the year 2000, and currently allows registration of outpatient, home-care, inpatient and emergency-room episodes. The EMR’s terminology system is based on the systematized nomenclature in medicine (SNOMED-CT). It is a flexible system that allows for standardized and also customized terms to be added. These terms are then linked to the clinical notes that are stored in free text format.

Several studies have described algorithms to detect events such as acute myocardial infarction (MI) [13, 41,42,43,44], stroke [17, 45], or chronic problems, such as congestive heart failure (CHF) [13, 22, 43, 46,47,48,49,50] or peripheral vascular disease (PVD) [11]. Nevertheless, because of both heterogeneity in medical practice and language barriers, these algorithms are dependent on the context they were developed on and usually need local adaptations in order to achieve high classification accuracy. Thus, we decided to locally develop and validate an algorithm for the detection of clinically relevant CVD, based on diagnostic codes from the EMR at HIBA in any clinical setting (inpatient, outpatient and emergency room visits).

Objectives

To assess the sensitivity, specificity, and agreement level of a rule-based algorithm for the detection of cardiovascular and cerebrovascular disease, using structures data from electronic medical records.

Methods

Algorithm development

Three family physicians from the research group (SE, RIR, MRT) selected clinically relevant Cardiovascular disease (CaVD) and Cerebrovascular disease (CeVD) terms from the international classification of primary care, Second Edition (ICPC-2), the ICD 10 version 2015[§] and SNOMED-CT 2015 Edition. These terms included both signs, symptoms, diagnoses and procedures associated with CaVD and CeVD. Using these terms as reference, we run a query in the HIBA’s EMR database[††]. This yielded a list of codes that was revised and agreed upon by the three reviewers. Terms not related to CaVD or CeVD symptoms, signs, diagnoses or procedures were excluded. Terms describing incidental findings without clinical relevance, were also excluded. Discrepancies were resolved in favor of maximizing the algorithm’s sensitivity. A list of the SNOMED-CT codes included in the final algorithm can be found in Additional file 1: Appendix. The algorithm yielded a positive result if the patient had at least one of the selected terms in their medical records, as long as it was not recorded as an error. Else, if no terms were found, the patient was classified as negative.

The algorithm was applied to a randomly selected sample of the active patients within the hospital’s HMO by 1/1/2005 that were 40–79 years old, had at least one year of seniority in the HMO and at least one clinical encounter. Thus, patients were classified into four groups: (1) Negative patients (2) Patients with CaVD but without CeVD; (3) Patients with CeVD but without disease CaVD; (4) Patients with both diseases. To facilitate the validation process, a stratified sample was taken so that each of the groups represented approximately 25% of the sample.

Manual review of medical records

Three family physicians (SE, RIR and MRT) participated in this process. One-third of the patients were assigned randomly to each reviewer. Both coded and un-coded (free text) sections of the EMR were reviewed. This was done from the first clinical found note in the patients’ records to the last one registered prior to 1/1/2005.

Manual classification criteria

We considered a case as positive, if positive mentions regarding CaVD or CeVD medical history, signs, symptoms, procedures or treatments were found in their records (inpatient, outpatient or emergency room clinical notes). Regarding results from diagnostic tests or procedures, in order to focus only on clinically relevant pathology, we only classified as positive those patients in whom positive test results fulfilled one of the following two conditions: (1) the result was preceded by some CaVD or CeVD compatible clinical context; (2) that, even though the result detected an incidental finding (i.e. electrocardiographic findings of an MI without symptoms), treatment was prescribed which would mean that the treating physician attributed some degree of clinical relevance to it. Also, we classified as positive, codes registered prior to the implementation of the EMR, assuming that it was any part of the patient’s past medical history.

On the other hand, we considered that the patient did not have CaVD or CeVD if: (1) There was no mention in the clinical notes of CaVD or CeVD; (2) A health problem related to CaVD or CeVD was mentioned but later on dismissed (even if this rejection was made after 1/01/2005); (3) A diagnostic code related to CaVD or CeVD was recorded, but there was no record of it in the free text of the clinical notes. An example of this are diagnostic codes that were recorded in the patient’s EMR, but did not represent a clinical entity but rather an incidental finding (i.e., vascular lesions detected in a brain CT scan that had not been ordered in the context of neurological symptoms). Also, we classified the finding as incidental if it did not trigger any preventive or therapeutic measures.

Unclear cases were subject to discussion between the three reviewers. The level of agreement between reviewers was estimated from a sample of 50 records using the Cohen’s κ statistic. The yielded value was 0.91[‡‡].

Quantitative analysis

The operational characteristics (sensitivity, specificity, agreement level with 95% confidence intervals) of the algorithm were calculated based on 2 by 2 tables, using the manual review of the EMR as reference the test. Each result of the detection algorithm was classified as true positive, false positive, true negative or false negative, both for CaVD and CeVD pathology separately, as well as a combined outcome. Analyses were carried out using SAS University Edition (SAS Institute Inc, USA).

Qualitative analysis of errors

We qualitatively analyzed false positives and false negatives. The reviewers described the reasons that led them to classify each case as an error. These errors were then grouped and quantified in different subcategories.

Results

Quantitative analysis

The sample was composed of 1137 patients. 31 did not fulfill the inclusion criteria. 1106 were included in the final analysis. See Fig. 1.

Fig. 1
figure 1

Flowchart

Full size image

Table 1 shows the results of comparing the classification by the algorithm to the manual review. We can see that the kappa statistic, sensitivity and specificity show more than acceptable values (Table 2).

Table 1 Performance of the algorithm compared to manual review
Full size table
Table 2 Stratified sample of 1106 patients as classified by the algorithm
Full size table

Qualitative analysis of errors

Erroneous results of the detection algorithm were classified into false positive (FP) and false negative (FN) results. They are described in Table 3. Table 4 describes the distribution of the erroneous results classified by type of clinical problem.

Table 3 Distribution of erroneous outcomes of the detection algorithm of cardiovascular disease or stroke through the records of an electronic medical record
Full size table
Table 4 Distribution of erroneous outcomes, classified by type of clinical problem
Full size table

Discussion

This algorithm was developed in the context of constructing a retrospective cohort of patients without CeVD or CaVD, emulating the exclusion criteria of the cohorts used for the development of several cardiovascular risk calculators [51, 52]. For this purpose, the sensitivity and specificity achieved seem more than appropriate. However, it is important to note that when using the combined algorithm to exclude patients with history of CaVD and CeVD, its lower specificity might create a cohort that is slightly healthier than the average population of the HIBA.

The qualitative analysis of errors provides relevant information regarding the algorithm itself, the EMR and also about the physicians’ EMR usage habits. The high sensitivity of the algorithm is probably due to the relevance of the selected problems; we did not expect to find underreporting or “undercoding”Footnote 1 of events such as MI, CHF or stroke that are both clinically relevant and usually easier to define than other subtler entities. Nevertheless, problems that tend to be harder to define like CaVD symptoms or PVD represented an important fraction of the total FPs.

The lower specificity found required an analysis of the FPs. The most common type of error was the dismissal of a diagnostic code. This was usually related to not deleting or registering the code as an error once further testing had dismissed the initial diagnosis. The EMR of the HIBA allows assigning each code, a status (active, passive or error). Active codes or problems are those that the treating physicians deems as a current confirmed problem that requires an intervention. Passive problems are those that although confirmed, do not require an intervention at the moment (i.e. stable, well controlled hypertension). The status ‘Error’ is assigned to those codes that prove to be false, usually after further testing. The results of our study seem to indicate that many diagnoses initially coded as active, once they are later on dismissed (mainly observed in cardiovascular and PVP symptoms), their status is not changed to error. This could be improved through interventions on how physicians interact with the EMR or, from the algorithm point of view, by implementing natural language processing (NLP) algorithms to obtain information from the free text. However, this would add a new level of complexity to the algorithm that might not be cost-effective given the costly process involved in training NLP tools.

The second most frequent cause of FP are codes that do not translate to clinically relevant events, for example the incidental finding of lacunar lesions in a brain CT scan that was requested not due to acute neurological symptoms, but rather due to head trauma or cognitive impairment. This could also be improved by implementing the strategies mentioned in the previous paragraph.

It is difficult to find an explanation regarding the coded terms for which no information was found in the free text. They could simply be recording errors. Nevertheless, given the fact that some of these were very relevant terms, like stroke or MI, we believe that there must be some alterative explanation that is eluding us.

One of our main goals was to develop a parsimonious algorithm that required the least amount of information to be able to correctly classify patients. In this sense, other authors have also used coded terms to develop algorithms for detection of cases with good results. Regarding CHF, Kottke et al. developed an algorithm with excellent sensitivity and specificity only based on ICD codes [43]. The results are consistent with the review carried out by Saczynski and cols [50]. In MI and stroke/TIA similar results were found [13, 17, 44]. More complex algorithms, which for example include laboratory results or imaging reports, did not perform superiorly than those only based on codes, at least for these pathologies [13, 44]. Nevertheless, as shown by Fan and cols [11], algorithms that include more complex and data-driven techniques, like NLP or regression techniques, seem to perform better. On the other hand, algorithms based only on data driven processes can be less generalizable over time even within a single institution, given the usual changes in language throughout time.

Detecting many pathologies using a single algorithm is useful for the design of studies requiring broad exclusion/inclusion criteria. In this respect, we found only a few previous attempts to do so in the literature. Tu et al. describe an algorithm to detect cases of stroke/TIA combined; Kottke et al. do the same for MI and CHF [13]. Our algorithm could contribute to fill this information gap.

One of the limitations of this study is that by not limiting the algorithm to standardized codes, it might be too specific for our hospital. However, this could also be seen as a strength: perhaps the algorithms that have better performance are precisely those that are developed locally by people who have a profound knowledge of the idiosyncrasies of the EMR users and thus incorporate nonstandard local terms.

On the other hand, it is important to emphasize that this algorithm sought to emulate the medical criteria for the detection of cases and not the detection of true cases, since no objective criteria were used in the algorithm or the manual review process. Therefore, if the physician made a mistake in the diagnosis process, this would be not detected by the algorithm.

Finally, these results may not be generalizable within our hospital given that we only evaluated charts until 2005 and not later.

Conclusion

We developed a simple algorithm, using only standardized and non-standardized coded terms within an EMR that can properly detect clinically relevant cases of CaVD and CeVD with excellent sensitivity. If used to rule in cases, manual review of the detected cases might be granted because of the algorithm’s lower specificity. Nevertheless, we believe that the specificity level is difficult to improve without altering the excellent sensitivity attained, since much of the deficit of specificity is due how users interact with the EMR. We believe that this would hardly be rectified by the inclusion or exclusion of terms in the algorithm, but perhaps combining it with an analysis of the free text using an NLP approach would yield better results.