Background

The term "evidence-based medicine" has become a catchphrase for the twenty-first century. Physicians are being called upon to justify their treatment decisions with valid, up-to-date evidence. There have been attempts within many areas of medicine to quantify the evidence that is available to, and used by, the physicians of that discipline. The first such study, conducted by Ellis et al [1] in 1995, found that 82% of treatments in internal medicine were evidence-based. This study has been replicated among a variety of disciplines, including family medicine [2] (81% evidence-based interventions), hematology [3] (70% evidence-based interventions), and surgery [4] (45% interventions based on randomized control trial [RCT] evidence or better).

Within pediatrics, there have been three such studies. Moyer et al [5] examined pediatric inpatients and found that 75% of primary interventions were supported by evidence. Rudolf et al [6] found 40% of clinical actions in a community-based pediatric practice were supported by RCT evidence and 7% were supported by convincing non-experimental evidence. In a study of pediatric surgical patients, Baraldini et al [7] found that only 29% of their treatments were based on level I or II evidence, as defined by Ellis et al (described below).

There is a need to investigate whether and to what extent the treatment of pediatric emergency patients is based on evidence. The wide variety of ailments seen within this area should offer a more comprehensive assessment of the overall state of evidence within pediatrics – encompassing surgical, outpatient and inpatient medicine, across all ages. At the same time, we need to gain an understanding of the quality and quantity of evidence in existence to support pediatric clinical decisions. How often is adult evidence generalized to the treatment of children? This is an important issue given that children differ from adults in terms of their physiology, biology, and developmental processes. There is limited health information available from research in children and, as a result, health care providers repeatedly turn to other sources of evidence, such as adult studies, to assist with managing health care issues in children [8].

In the present study, we sought to identify the extent to which evidence exists for use within the unique environment of the pediatric emergency department. A secondary objective was to determine the amount of evidence, applied to pediatrics, that is derived from adult studies. We hypothesized, a priori, that this was a common practice within pediatrics. Our hope was to identify areas within the body of evidence for pediatric emergency medicine that require future study.

Methods

This study was approved by the Health Research Ethics Board at our institution.

We conducted a retrospective chart review of randomly selected patient charts based on patients seen in a Pediatric Emergency Department at a tertiary care facility during the 2002 year (January 1 to December 31). The sampling frame was created using the Emergency Department Information System (EDIS) database. The following variables: hospital record number, patient hospital number, patient visit number, date, time, and age made up the sampling frame. A random sample of patient visits were then generated using random-number computer software (S-plus 6.0). Any individual patient was included only once in the sample. A sample size calculation of 295 showed that a minimum of 5.6% precision would be achieved for any given estimate of percentage. The primary investigator (PI) retrospectively reviewed the sampled records and, using a standardized abstraction form, identified a primary diagnosis and primary intervention for each based on the definition set out by Ellis et al. That is, the primary diagnosis was defined as "the disease, syndrome or condition entirely or, if there were several diagnoses, [the diagnosis] most responsible for the patient's [visit to the pediatric emergency department]" [1]. The PI used information from the emergency physician's notes, lab and radiographic results, and any other relevant documentation from the emergency room visit to establish this. If conflicting information was present on the chart, the final diagnosis, as described at the bottom of the emergency record form, was used.

The primary intervention was taken to be "the treatment or other maneuver that represented our most important attempt to cure, alleviate, or care for the patient [with respect to] his or her primary diagnosis" [1]. This was also determined by the PI based on the patient's chart.

A thorough literature search was then undertaken with respect to the primary interventions. Current (at the time of searching) MEDLINE, EMBASE and the Cochrane Library databases were searched comprehensively to determine if any evidence existed with respect to the individual primary interventions. Abstracts and references of relevant scientific papers were also searched. Local experts were contacted and asked if they were aware of RCTs in existence that had not yet been published or that were not found through the search strategy (this step yielded no additional information).

The primary interventions were then divided into the following categories that have been used in previous research [1–4].

Level I evidence

The abstract or paper demonstrated that this intervention had been evaluated in an RCT, systematic review or meta-analysis. No attempt was made to rate the quality of the study or its findings as this did not relate directly to our goal of determining the quantity of evidence available to PED physicians.

Level II evidence

No abstract or paper could be found that investigated this intervention in an RCT, systematic review or meta-analysis. However, a committee determined there to be convincing non-RCT evidence for this intervention. That is, the face validity of such an intervention was considered to be so great that randomized trials were unanimously judged to be unnecessary or, if a placebo were to be involved, potentially harmful (i.e., giving a blood transfusion to a patient who was exsanguinating [1]). The committee for this study was made up of five pediatric emergency physicians with extensive experience in pediatric emergency medicine and evidence-based medicine, one of whom was a co-investigator. The remainder of the committee had no other involvement in the study and was provided with only the definitions of the levels of evidence and a brief synopsis of the patient visit. Only unanimous committee decisions were considered sufficient for the inclusion of the intervention as level II evidence.

Level III evidence

No abstract or paper could be found that involved this intervention in an RCT, systematic review or meta-analysis and the committee did not unanimously agree that there was convincing non-RCT evidence to support it.

Level I evidence was further subdivided into those studies that included a majority (i.e., more than 50%) of pediatric patients and those that did not. Systematic reviews and meta-analyses were included under the "pediatric studies" grouping if they included one RCT of predominantly pediatric patients. Every attempt was made to search out pediatric studies, even if an adult RCT was found first for any given intervention.

A further category was included of those patients who received no primary intervention. This category included patients who were discharged from the pediatric emergency department after receiving only reassurance and/or advice. The percentage of primary interventions which were included in each category was calculated as was the percentage of primary interventions deemed to be from pediatric and adult studies, along with their exact 95% confidence intervals [9].

Results

From the 18,855 patient visits to the PED during the year January 1, 2002 and December 31, 2002, a random sample of 295 patient charts was computer generated, representing approximately 1% of patient visits. Any individual patient was only included once in the sample. Of these 295 charts, 272 (92.2%) were available to be viewed. The other 23 were missing from the chart area. Of the 272, 10 were discarded because of insufficient information to determine a primary diagnosis and intervention (i.e. the emergency record was missing from the chart). Ninety-four patients received no intervention, representing 35.9% of our sample (95% CI 29.9 to 41.9).

Table 1 lists the interventions that qualified as level I evidence, based on the finding of at least one RCT or SR. In total, 135 patients were found to have received evidence-based interventions (level I evidence), representing 80.4% (CI 73.4 to 85.9) of patients who received interventions and 51.5% of our sample.

Table 1 Diagnoses and Interventions seen in the PED for which there is Level I Evidence
Full size table

The most common interventions were analgesics for musculoskeletal pain (17/135 = 12.6%) and inhaled beta-agonist for asthma exacerbations (12/135 = 8.9%); both were supported by pediatric RCTs. Out of the 135 patients who received interventions that were based on level I evidence, 113 patients' interventions (83.7%; CI 76.1 to 89.3) were supported by pediatric RCTs.

The interventions unanimously determined, by the five person committee of pediatric emergency department (PED) physicians, to be worthy of level II evidence status are listed in Table 2. Overall, 7.1% (CI 3.9 to 12.4) of interventions, representing 4.6% of the sample, had face validity worthy of level II evidence.

Table 2 Diagnoses and Interventions seen in the PED for which there is Level II Evidence
Full size table

Including level I and II evidence, 87.5% (CI 81.3 to 91.9) of interventions can be considered evidence-based.

Those interventions which were not considered to be evidence-based (level III evidence) are listed in Table 3. Twelve point five percent (CI 8.1 to 18.7) of interventions (8.0% of the sample) fit into this category.

Table 3 Diagnoses and Interventions seen in the PED for which there is Level III Evidence
Full size table

Discussion

The results of this study are important for several reasons. First, we have shown that a substantial proportion of the children seen in the PED do not receive any treatment or intervention. In this sample of 1.4% of all PED visits over one year, 35.9% of children received only reassurance or advice. One of the most common reasons that children were seen in the PED without intervention was upper respiratory tract infection (26%) (data not shown), usually associated with fever. This leaves one wondering whether parental education about benign fever and supportive care for upper respiratory tract infection could help shorten PED wait times. Studies on the effect of education in this area could have very important beneficial effects in terms of costs to the healthcare system.

Our study has revealed that a great majority of interventions in the PED are based on pediatric studies (83.7%). Although this is a positive finding, there are several areas of pediatric medicine, such as migraine treatment and fracture management, for which treatment continues to be based on adult studies. Ideally, we would like to see 100% of pediatric interventions based on pediatric studies.

A comparison of our results to similar studies is shown in Table 4. Interestingly, our total "evidence-based" interventions (level I and II) at 56.1% is substantially less than that found in similar studies in inpatient general medicine [1] (82%) and inpatient pediatrics [5] (75%). However, a closer look reveals that the discrepancy between this and previous studies is not in the number of interventions which involved an RCT (51.5% vs. 53% for Ellis and 31% for Moyer) but in the number of interventions meeting the criteria for level II evidence (4.6% vs. 29% for Ellis and 44% for Moyer).

If one compares what was categorized as level II evidence by Moyer's study with our study, there are important differences. For example, interventions such as IV rehydration and antibiotics for urinary tract infection, categorized as level II evidence in the Moyer study, were classified as level I evidence in our study. This may be a reflection of the four year difference between studies and the accumulation of research evidence during that time. Alternatively, our study involved not only inpatients, but also patients discharged directly from the PED. Therefore, interventions which have been better studied such as antibiotics for otitis media [13] and glucocorticoids for croup [19] were included in our study. There were also differences with respect to those interventions that were considered to be worthy of level II evidence status by Dr Moyer's group and which our group relegated to level III evidence. This may simply be due to the degree of specificity involved in labeling interventions. Antibiotics for pneumonia and cellulitis were two such examples. Although our group refused to agree that intravenous antibiotics were required for these diagnoses, they certainly agreed treating a bacterial infection with some form of antibiotics was completely valid. It is interesting to note that something as simple as inserting the label "IV" or "PO" into the intervention could affect the level of evidence that treatment attained.

Table 4 Levels of evidence for interventions across different disciplines
Full size table

The limitations of this study are similar to those found with the studies previously carried out in other disciplines [1–5]. Certainly, there was the potential for the committee to be biased when assigning the level of evidence based on personal knowledge of the cases or personal treatment preferences. This situation was minimized by employing a committee of five physicians and requiring group unanimity for placing an intervention in the level II category. One might argue that all members of the committee had an inclination to prefer to categorize interventions as level II, simply based on their own positions as PED physicians. However, this was not proven true as we had substantially fewer interventions placed in the level II category compared to previous studies.

Another potential limitation is derived from the inclusion of interventions under the category of level I evidence without interpreting the results of the RCT. This was done in order to compare our results to the original study by Ellis which, likewise, did not comment on the content of level I evidence.

Finally, while this was a random sample, it represented only 1% of all PED patients seen over the course of a year, therefore it provides an estimate of the proportion of PED treatments that are evidence-based but may not be representative of the full spectrum of different presenting conditions.

There were many strengths of our study when compared to previous studies. The relative inflexibility of our expert panel in committing interventions to the level II evidence group is one positive feature. Although there must be subjectivity in a process such as this, our team was quite rigorous in applying the criteria for this category and limited what must be assigned to this group accordingly. This is reflected in the lower percentage of interventions ranked as level II evidence by our committee (5% compared to 44% in the Moyer study [5]) than those previous. When consensus surrounding the validity of a treatment without RCT evidence is sufficient to warrant the label "level II evidence", certainly there should be a very limited number of interventions that will fit into this category. Another strength was that our study sample was derived from a full year of PED visits which would account for seasonal variation of presenting diagnoses. This made for a more accurate representation of the most common diagnoses and interventions seen within the pediatric emergency department. We would expect the results of our 1% sample to be relatively generalizable to any tertiary care pediatric centre. Finally, the search for evidence was exhaustive, making it unlikely that relevant RCTs, which had been published at the time of this study, were simply not found during the literature search.

Conclusion

We have shown that a surprisingly large proportion of patients seen in the pediatric emergency department (35.9%) receive no intervention, while approximately half of patients in this setting receive interventions that are based on evidence from randomized controlled trials (level I evidence). The vast majority of these are derived from randomized control trials containing a majority of pediatric patients. Although our study speaks positively of the breadth of knowledge within the unique environment of the pediatric emergency department, we have also identifed numerous important areas for future study within pediatric research.