Background

Gaps between evidence-based ideal clinical practice and what doctors actually do can be responsible for substantial harm [1, 2]. For example, beta-blockers, angiotensin-converting enzyme inhibitors, and lipid-lowering drugs have been shown to reduce re-hospitalizations and mortality following an acute myocardial infarction (AMI), yet these drugs are underprescribed [3]. Printed educational messages (PEMs) directed to doctors may be one way to address such gaps. PEMs have the considerable advantage of low cost and easy dissemination, so even small improvements may be efficiently obtained [4]. However, the evidence on their effectiveness is contradictory. Freemantle et al. conducted a systematic review of the effects of PEMs [5]. None of the small number of included trials found statistically significant improvements in practice, leading to the conclusion that PEMs were ineffective. When Farmer et al. updated this review [6] using a superior analytic approach (summarising observed effect sizes instead of vote counting studies with statistically significant effects), they observed a median absolute improvement of care of 4.9% (range −8.0% to +9.6%), on par with other much more expensive interventions like audit and feedback, or academic outreach [7]. A further updated review by Giguère et al.[8] observed smaller improvements in care (median absolute improvement of care of 2.0%, range 0 to +11.0%), and argued for the need for additional primary research and further investigation of different characteristics of PEMS in head-to-head comparisons. These contradictory findings leave policy makers uncertain about the role of PEMs, at a time when closing evidence-to-practice gaps may offer a more cost-effective investment of health system resources than developing new interventions [9].

Reviews cannot overcome limitations of the primary evidence. There are few trials of PEMs, often small in size with methodological weaknesses (for example unit of analysis errors in cluster randomized trials). To clarify the usefulness of PEMs, large, pragmatic [7, 10], and well-designed randomized controlled trials on the effect of PEMs on guideline adherence, conducted in real world settings, amongst typical practitioners are needed. The Ontario Printed Education Materials (OPEM) trial on retinopathy screening was such a trial: a large pragmatic cluster randomised trial powered to be able to detect small effects.

In Ontario, diabetes mellitus (diabetes) affects over 15% of women and 19% of men aged 65 years and older. Diabetic retinopathy is a common complication, affecting about 70% of persons with type 1 diabetes and 40% of persons with type 2 diabetes, and is the leading cause of preventable blindness in Canadians aged 30 to 69 [11]. Most vision loss related to diabetic retinopathy can be prevented through retinal screening examinations and appropriate treatment. Early treatment has been shown to decrease the risk of severe vision loss from proliferative diabetic retinopathy by 90% and the risk of vision loss from macular oedema by 50% [12]. Evidence-based practice guidelines in Canada call for screening for retinopathy at the time of diagnosis and every two years thereafter for everyone with type 2 diabetes [13].

Under the Ontario Health Insurance Plan (OHIP) Ontario residents have universal public health insurance, including free at point of service access to family practitioner (FP) care. Eye examinations (which include retinal screening), performed by an optometrist, FP, or ophthalmologist, are also an insured service, and thus are free at point of service to patients without any out-of-pocket payment, for all adults aged 65 years and older and also for those under age 65 who have been diagnosed with diabetes.

Yet in Ontario, the observed screening rates within the first year of diabetes diagnosis falls far below recommended levels (43% and 50% for diabetes patients aged 30 to 49 and 50 to 64 respectively).

If the impact on screening uptake of PEMs was indeed 5% improvement, then at the trivial cost of a letter to each FP, Ontario’s rate of screening and treatment, especially among younger patients, could be substantially increased, averting blindness in more than 100 Ontarians per annum [14].

We report here a trial of PEMs, evaluating their impact on screening rates for diabetic retinopathy. The objective of the study was to determine whether sending a PEM to FPs (and the format of the PEM) would affect the likelihood that a patient visiting their doctor would receive an eye exam within 90 days of their visit.

Methods

These are presented in detail in our published protocol [15].

Study design and randomisation

The study was a pragmatic, factorial, cluster-randomized controlled trial (Table 1). The study intervention was mailed on 1 April 2005 and the follow up period extended from 1 July 2005 to 30 June 2006. To prevent contamination (sharing of information among doctors in group practice) we randomised at the level of the practice. FPs were placed into practices on the basis of a shared street address. Practices were randomly assigned to an intervention group by the study statistician, using computer-generated random numbers.

Table 1 Study design and number of practice groups/number of physicians
Full size table

Study population

The study participants were all FPs with an active practice in Ontario in 2003/2004. ‘Active’ practice was defined as a total billing volume for the year of at least $50,000 and writing prescriptions for at least 100 different patients, with at least one prescription in at least 10 of the 12 months.

The Ontario Diabetes Database was used to identify Ontario residents diagnosed with diabetes on or before 31 March 2005. Individuals were included if they were at least 30 years old on 31 March 2005, were alive at the end of the follow-up period (30 June 2006), and visited one of the target FPs within one year of the intervention mail-out.

Patients who had an eye exam in the preceding nine months were excluded from the study.

The printed educational materials interventions

Informed was a free, peer-reviewed, evidence-based primary care practice synopsis, written and produced by clinical and research staff from the Institute for Clinical Evaluative Sciences (ICES, http://www.ices.on.ca), and mailed to nearly 15,000 health care providers in Ontario from 1994 to 2007.

We designed two types of PEM to address the identified evidence-practice gap: a short, directive, evidence-based PEM on a postcard-sized card stapled to the front page of informed (the ‘outsert’); and a two-page insert, indistinguishable from the rest of informed in size and style (the ‘insert’). The insert excluded the directive statements of the outsert but included more background, a summarised evidence-based guideline, and references. Both were developed over three months of meetings with input from five ICES staff with knowledge translation experience (an internist, two family physicians, and two knowledge translation researchers), the writing team of the informed newsletter and a communications consultant.

We also issued a pad of take-home reminders (aimed at patients, to remind them to make an appointment for an eye exam), to be given to patients by their FP. Because it was not clear whether the reminder would be any more effective than the doctor’s verbal advice, doctors randomised to receive the outsert were also randomized to receive or not receive a pad of patient-aimed reminder slips.

Insert, outsert, and patient reminder were included with the 1 April 2005 edition of informed.

Practices were randomly assigned to one of four intervention groups. The two intervention groups selected to receive an outsert were further randomly divided into two subgroups, one of which received the patient reminder notepad.

Figure 1 shows the outsert and patient reminder. The full insert is included as Additional file 1.

Figure 1
figure 1

Interventions: the outsert and patient memo.

Full size image

Data sources

The following administrative data sources were used [16]. These data sets were held securely in a linked, de-identified form and analyzed at the Institute for Clinical Evaluative Sciences.

  1. 1.

    The OHIP Claim History Database details payments to health care professionals, including an encrypted provider number unique to each health care professional, an anonymous, encrypted patient identifier unique to each patient, the service provided, and the service date.

  2. 2.

    The Registered Persons Database contains basic demographic information for each insured person.

  3. 3.

    The Corporate Provider Database contains limited demographic and practice information for each practitioner.

Finally, we used the Ontario Diabetes Database, a validated registry of Ontario residents diagnosed with diabetes [17].

Records from these databases were linked, using the patient and provider identifiers, to determine which individuals with diabetes visited one of the target physicians in the year following the intervention, and whether that individual subsequently received an eye examination.

Outcomes

The primary outcome measure was whether or not an eligible trial patient received an eye exam within 90 days of their first FP visit during the one-year period following the mail-out. Because the PEMs specifically addressed the issue of insured eye exams for patients with diabetes younger than 65 years, a secondary analysis examined the impact of patient age on the uptake of eye exams.

Power

Based on Monte Carlo simulations, assuming an intra-cluster correlation coefficient of 0.1 [18], three patients with diabetes per physician, and a baseline rate of screening of 36%, a trial with 1,250 practices per arm would provide over 97% power to detect an absolute increase of 5% in the screening rate, and over 98% power to distinguish between the effects of the combined intervention and either alone, assuming the effect of each to be 5% and the combined effect to be additive.

Statistical analysis

Analysis was on an intention-to-treat basis. Logistic regression models, estimated using generalized estimating equations (GEE) methods to account for the clustering of patients within physician practices, were used to estimate the effect of the intervention on the likelihood of patient screening. Two logistic regression models were fitted: one including only the interventions (insert and/or outsert and the patient reminders), and a second also adjusting for patient- and physician-level covariates. All analyses were performed using SAS version 9.1 (SAS Institute, Cary, North Carolina). Two-tailed p-values less than or equal to 0.05 were considered to be significant.

Ethics

This study was approved by the Research Ethics Board at Sunnybrook Health Sciences Centre.

Results

Physician and patient selection

Figure 2 shows the number of physicians and patients included in the study. Three-quarters of patients with diabetes who met the inclusion criteria visited one of the physicians targeted by this study during the follow up period. However, two-thirds of these individuals had already had an eye examination in the nine months immediately prior to the office visit and were excluded (Figure 2).

Figure 2
figure 2

Patient and physician selection: consort diagram.

Full size image

Baseline characteristics

There were small, clinically unimportant, differences between the demographics of patients with diabetes who paid a visit to a study physician and those who did not, and between those who were and were not included in the analysis (Table 2). Patients aged 65 years and older were significantly more likely to visit a primary care doctor, and also more likely to have received a recent eye exam (and hence to be excluded from the analysis).

Table 2 Characteristics of individuals with diabetes
Full size table

There were no meaningful physician differences among the intervention groups (Table 3).

Table 3 Physician characteristics, by intervention group
Full size table

Analysis of intervention effects

Intervention effects are shown in Table 4. Neither the unadjusted nor the adjusted results show any evidence that the interventions (alone or in combination), were effective in increasing rates of eye examination among eligible trial patients. The widest confidence interval reported in the table, an odds ratio between 0.94 and 1.04 for the insert + outsert + reminder notepad, corresponds to a true absolute effect of the intervention lying between a decrease in retinal screening of 1.3% and an increase of 1.1%. Thus, the estimate of effect for the presumably strongest ‘dose’ of intervention is of trivial clinical value, is not statistically significantly superior compared no intervention, and the confidence interval does not contain values of clinical importance.

Table 4 Results
Full size table

While the probability of having an eye examination depended on patient age and rises sharply at age 65, there was no indication that intervention effectiveness varied with patient age.

The intracluster correlation coefficient was 0.024 (95% confidence interval 0.022 to 0.025).

Discussion

In the face of the previous uncertainty about the effectiveness of printed educational materials in changing physician behaviour, this study, with sufficient power to detect even a small effect, failed to change practitioner behaviour, as measured by retinal screening rates among patients with diabetes. The intervention was a faithful operationalisation of the sort of printed educational materials that are routinely used and explored both long and short formats; this can be regarded as an evaluation of a realistic intervention.

The effectiveness of printed educational materials depends on the message being received by the physicians at whom the PEMs were aimed, read, judged credible, transmitted to (and acted on by) patients who might benefit from screening. The first three steps appear likely to have occurred: The reliability of the postal services in Canada are such that we can be confident that the materials were successfully delivered. They are also likely to have been read and judged credible: In 1997, The Strategic Council Inc., a market research organisation, contacted 500 Ontario physicians by phone to determine readership and recall of informed. They found that 71% of the respondents recalled receiving informed and that of these, 89% found it useful or very useful and 53% read most or every issue (internal report, personal communication, Dianne Kelsall, editor of informed).

It is possible that the physicians did recommend screening but did not make a referral or that patients did not keep their ophthalmology appointments. However, given that eye examinations are free for Ontario patients with diabetes and for all people aged 65 and over (in which group eye examination among patients with diabetes rises to 89% coverage within one year of diagnosis), it seems unlikely that patients with diabetes would fail to take up their physician’s recommendation simply because they are not yet 65. Thus patient failure to respond to the physician recommendation seems to be an unlikely explanation for the lack of impact of our intervention. We believe that the message was likely received, read, and judged credible by physicians, and would not have been ignored by patients. Yet PEMs were not successful at improving patient screening rates.

This suggests that PEMs simply failed to improve physician knowledge regarding the availability of free tests for patients with diabetes who were aged under 65; alternatively the PEMs failed to either influence the transmission of this message to patients or to lead to referral.

The content of our PEMs was not based on a prior assessment of barriers to screening and so it is possible that these are not the relevant barriers. Perhaps the actual barriers to improved screening lie elsewhere, with possibilities including the organisation of screening or patient factors such as motivation or geographical access. These potential barriers are indicated by the process study conducted in parallel with this RCT on a subsample of practitioners, and reported alongside [19].

This study is reliable and was conducted under realistic conditions to mimic typical programme delivery. The use of administrative data as the source of our outcome measure allowed us to include nearly the complete population of family practitioners and all of their eligible patients, to be able to provide a definitive result. This trial is one of the largest published randomised trials of knowledge translation or implementation to date, and given its power a false negative result is unlikely.

Conclusions

This real-world study supports the conclusion that PEMs, whether long and discursive or short and directive, and with or without patient reminder notes, do not change physician behaviour as measured by adherence to evidence based screening recommendations for diabetic retinopathy. These results strongly suggest that the use of PEMs alone is not a useful strategy for closing this evidence-practice gap, and underline the importance of continuing to explore complementary or alternative behaviour change interventions. We have replicated these results for two prescribing behaviours (in preparation) and further replication for diagnostic and other therapeutic behaviours would be informative.

Authors’ information

Retired: Pamela M. Slaughter.