Background

The use of Complementary and Alternative Medicine (CAM) in the North American population has significantly increased in the last decade [1]. The majority of decision-making in CAM is based on experience, observation and traditional healing manuscripts. In contrast allopathic medicine has moved away from these methods of decision-making towards a more evidence-based approach. The term Evidence-Based Medicine (EBM) has been used to describe this new paradigm for clinical decision-making. EBM refers to the conscientious, explicit and judicious use of current evidence in making decisions about the care of individual patients [2].

There is a growing call for CAM to also adopt evidence-based medicine concepts as its use spreads amongst the general population. One of the most effective ways of ensuring that EBM is optimally used in CAM is to teach future CAM practitioners critical appraisal skills which are important tools in EBM. Many studies have examined the effectiveness of teaching critical appraisal skills in undergraduate medical and residency programmes [3]. However, no studies have thus far examined the effectiveness of teaching critical appraisal within CAM education. We aimed to determine whether a critical appraisal educational intervention had an impact on naturopathic students comfort in reading academic research articles, their ability to critically appraise these articles and their attitudes towards EBM.

Methods

Study population

Our study population was the final year students at The Canadian College of Naturopathic Medicine, the largest CAM training centre in Canada. We invited 83 students in total to participate. Students were allocated to one of two groups (Group A: n = 47 and Group B: n = 36). The group to which students were allocated was based on their availability to ensure no conflict with their clinical rotation schedules. Students in the two groups had an identical education over the previous three years in CAM modalities and sciences. All students received a short course in research methods in year 1 of their education, but none had been trained formally in critical appraisal.

Study design

Our study consisted of three separate phases (See Figure 1). A baseline assessment of attitudes towards critical appraisal and understanding of critical appraisal concepts was conducted in both groups. Group A was then exposed to the intervention, a 3 ½ hour workshop on the concepts of EBCAM and critical appraisal skills. Group B received a control intervention, a 3 ½ hour workshop on Bioethics and philosophy. Students' abilities and attitudes towards critical appraisal were then reassessed. We then crossed over the two groups so that Group B received the workshop on critical appraisal and Group A the control intervention. This cross-over ensured that all students received the critical appraisal workshop. We conducted a final assessment of abilities and attitudes towards critical appraisal following this stage. We recognized the potential of diffusion of information amongst students' in Group A and Group B and requested that students' not discuss the sessions with their colleagues.

figure 1

Figure 1

Educational intervention

The 3 ½ hour intervention training sessions aimed to teach question formulation and levels of evidence. We taught Critical appraisal skills didactically while also allowing for question and answers. No handouts were provided. The training program of both intervention and control were delivered by the same instructor (#1) to avoid bias.

The primary objectives of the intervention session were as follows:

  • How to structure a question

  • How to derive search terms

  • Understanding levels of evidence

  • How to choose correct Critical Appraisal Tools

  • How to critically appraise an article

Detailed intervention session activities

  • We taught the students to convert common clinical scenarios into three and four part answerable clinical questions.

  • We discussed databases and search strategies, recognising sources of best evidence in relation to CAM and sciences. Journal quality and biases were discussed.

  • We provided examples of the use of EBCAM in making clinical decisions and using evidence to disprove commonly accepted beliefs.

  • The bulk of the session was spent addressing the hierarchy of evidence, with particular emphasis placed on the concept of controlled (both randomised and non-randomised) trials. The User's Guides to the Medical Literature: How to Use an Article About Therapy or Prevention [4] was used throughout as a template for reading articles about therapy.

Control sessions

  • The students received a 3 ½ didactic lecture on the history and philosophy of Bioethics. No discussion of research methods was allowed.

Assessment of students' attitudes and abilities

Students' self-confidence in their critical appraisal skills was assessed by having students indicate a 10 point Likert scale their response to the statement" I am comfortable reading research ".

Assessment of EBCAM concepts was performed using a written test which included a study to critique, and a set of questions relating to EBCAM concepts and levels of evidence. We specifically examined the students' ability to do the following:

  • Formulate a question and develop a search strategy after having been provided with a clinical scenario

  • Critically appraise a supplied article.

  • Understand EBM concepts.

The same written test was used for Phase 2 and Phase 3. The clinical scenario and EBCAM questions remained unchanged to determine more precisely the level of clinical decision change in the students over the three phases. However, the studies the students were asked to critically appraise were changed [5–7]. The effectiveness of the training intervention could be estimated by comparing the outcomes before and after training in Group A or by comparing the outcomes after initial training of Group A to those of Group B.

Evaluation

Written tests were evaluated out of a score of a possible 23 per written test. A blinded investigator (#2) performed the marking and assessment. Blinding codes were determined and hidden by investigator #3. All investigators had previously determined the correct answers.

Statistical methods

For Groups A and B we calculated means scores, standard deviations and 95% confidence intervals for written tests and level of confidence in reading critical appraisal papers at each Phase of the study.

T-tests were performed (one at each phase of the experiment for each of our two outcome measures) in order to test for performance differences between the two groups. If the workshop was effective, we would expect to see no difference in the baseline scores, a significant difference between the groups in Phase 2 (with higher scores belonging to the group who had already been exposed to the training), and no difference between the groups at Phase 3 (since both had received the training by this point). We would further expect scores for both groups at Phase 3 to be higher than they initially were at baseline.

Results

The flow of students through the trial is displayed in Figure 1 (See Figure 1). Student compliance to the trial is shown in Table 1. A total of 41 % of students did not complete all phases of the trial (See Table 1). However, ample attendance (82%) during the first two phases allows us to observe differences in the groups receiving the different workshops.

Table 1 Attendance/Compliance
Full size table

Scores

Table 2 displays the mean scores obtained by students writing the question formulation and critical appraisal written tests from all phases of the trial by groups. The written tests were scored out of a possible 23 (See Table 2).

Table 2 Scores, out of a possible 23 per phase
Full size table

No significant difference in either test scores (p = 0.45) or self reported confidence scores (p = 0.58) were found during this phase of the experiment, suggesting that no initial differences were present between the two groups (See Table 3).

Table 3 Difference between groups through course of study
Full size table

The 2nd phase of the study revealed no differences in test scores (p = 0.74) or self reported confidence scores (p = 0.36), suggesting that even after one group had received training in EBM, their scores were not found to be significantly higher than those of the subjects who had not yet received any training.

Again at the 3rd Phase of the analysis, we did not find any evidence to suggest a difference in performance between the two groups, either in their test scores (p = 0.77) or their self-reported confidence (p = 0.58).

Student confidence

The Likert scale (1–10) was used to determine student's self-perceived confidence in reading medical research. Table 4 illustrates the mean scores of the groups. Student self-perception of confidence tended to increase slightly during the trial, although not to a level that was statistically significant (See Table 4). Many students verbally expressed interest in ongoing training in EBCAM.

Table 4 Mean student confidence in reading research, on a Likert scale (1–10)
Full size table

Discussion

The present study was undertaken to investigate whether or not the introduction of a single educational intervention such as a workshop would positively influence the critical appraisal skills and confidence in reading scientifically written research papers of students of naturopathy. Our results indicate that a one-time academic intervention in the form of a workshop did not appear to provide any measurable effect in either skill or confidence. There may be several reasons for the lack of effect. Our study design may not have been adequately powered to identify smaller differences in improvement. This lack of power was magnified by a high dropout rate. The high dropout rate may have also introduced important systematic bias as the students who continued on in the trial may have been more motivated learners, recognised the importance of learning EBM skills, or were previously familiar with critical appraisal.

Another important reason exists as to why we observed no effect with our academic intervention. Critical thinking is not a mechanistic change, instead requiring a level of cognition developed over time, akin to a cultural shift. The intense clinical schedule experienced by final year students does not provide the opportunity to reflect and observe new concepts in relation to past studies. Students may also not want to learn something which suggests that the paradigm in which they have invested a great deal of intellectual and personal capital may be wrong or lack sound evidence. Students enter CAM studies with either a "holistic" or "scientific" worldview, and are unlikely to change [8], potentially rendering it difficult to change their clinical behaviours and perhaps forego a CAM modality. It is possible that philosophical values are too strongly ingrained to allow EBCAM in this population. Similarly, it may be that the 4th year students have developed an attitude that they already practice evidence-based medicine using their own understanding of what constitutes best evidence.

There are several complex factors involved in effectively teaching EBCAM to students of naturopathy, despite there being real potential for increasing EBCAM usage amongst students in the aim of affecting clinical practice. The final year is a difficult but important time to teach critical appraisal and evidence skills [9]. Single-intervention strategies are the most commonly used ways to introduce critical appraisal skills in CAM institutions, and educational gains achieved by these approaches may be small and need to be replaced by a multi-staged and multi-factorial approach introduced earlier into their learning experience. On a positive note, all but one of the participants answered in the affirmative when asked whether or not they thought that an evidence-based approach to CAM would benefit their profession. This suggests that students may be receptive to a more comprehensive approach to teaching EBM.

Conclusion

Our study did not find that a single academic intervention consisting of a workshop to teach critical appraisal skills was able to improve student's confidence in evidence-based concepts or their ability to apply these concepts. However, introducing an evidence-based approach to CAM remains an important objective. We recommend that future studies examine the effectiveness of interventions introduced early in the curriculum and that carry on throughout the student's CAM education.