In New directions in medical education related to anesthesiology and perioperative medicine, Bould et al. ask the question, “How do we know what we know (in medical education)?”1 The answer suggesting itself to a scientific community is “through research conducted to advance knowledge in the field”. This makes it important to understand the nature of this research and its directions, foundation, and limitations. The first objective of this narrative review is to examine themes in resident education and acquisition of competence, along with the underlying concepts and contextual factors, to explore how the discussion about critical issues relates to current research in the field. The aim is to use the results of this assessment to propose potentially fruitful areas for future research at the interface of medical education, clinical practice, and the acquisition of competence. The overall aim is to suggest ways to add to the evidence base for physician education that focuses on competency for independent practice.

Faculty, clinical work, and the learning environment are the context for clinical education in anesthesiology. They may facilitate learning or present significant barriers. That organizational context matters was shown by seminal work on differences and conflict in the early 1960s.2 In education, context has been examined for its learning effects, with factors studied encompassing social background, diversity, and differences among schools in class size, programs offered, and teacher qualifications and salaries.3 The role of the contextual frame has also been analyzed by Ericsson and others studying deliberate practice in the attainment of competence.4 A review of the medical education literature, including editorials and commentaries, highlights three factors relevant to residents’ attainment of competence and to the quality of care they provide after completion of training: 1) transitions across the education continuum and into practice and the potential for mismatch between learning and expectations of competencies for independent practice; 2) limits on resident hours that have reduced the hours of clinical learning for most residents with a possible negative effect on competence for independent practice; and 3) work that embeds the competencies in resident teaching and evaluations. While these topics have been discussed in editorials and commentaries, research to date has been limited largely to surveys of residents, program directors, faculty, or organizations hiring physicians or to studies with a narrow focus, limited sample size, and threats to validity. This suggests a need for research that could offer empirically-based direction for education reform and innovation.

These observations are common in much of the research in medical education over the past decade.5-8 A theme across authors and years is that the research has not produced results capable of answering the pressing questions at the interface of education and medicine. Much of the work to date has explored narrow curricular or program intervention, e.g., assessing the effect of an objective skills-based clinical exam on subsequent clinical task performance. The value of this work relates to the importance of the research to holistic physician competence.8 However, many articles being published address relatively narrow topics without seeming impact on the larger problems being discussed. As a possible consequence, there is little empirical evidence for the benefit of conceptually appealing approaches like problem-based learning.9 A reason for the narrow focus of research is that this work often is performed as an adjunct to faculty teaching responsibilities, hence small studies are produced that address local problems5 and lack a theoretic basis7 and a conceptual framework; only one-half of medical education studies mention a framework on which they are based.10 Even when studies produce “significant” findings, there are questions about utility and translation of findings into practice. This creates a gap between editorials calling for change in medical education and the evidence available to provide empirically derived direction. The three themes are discussed in more detail in the remainder of the article, and suggestions are offered for promising work that could contribute to closing the research gap through a better understanding of the research completed to date.

Methods

Searches of editorials and commentaries related to graduate medical education on Medline and Google Scholar produced the three themes. Further search using search terms and Boolean searches to identify research articles relevant to one or more of the themes produced 342 citations and one narrative review.11 Search of the references of these studies identified another 14 citations. With further analysis, articles were selected wherein interventions relevant to one or more of the three themes were studied, resulting in a final selection of 56 citations Twenty-one studies analyzed transitions in, during, and after residency education;12-32 19 studies addressed the educational effects of duty hour limits,33-51 and 16 articles presented research on use of the competencies in teaching and assessing anesthesiology residents.19-27,52-58 For the theme “duty hours and the acquisition of competence”, the only articles included were those that assessed the effect of duty hour limits on measures of physician competence and those that studied changes in patient care experience in anesthesiology that could be attributed to the reduction in resident hours.

Transitions across the medical education continuum

The concept of a medical education continuum suggests seamless progression from undergraduate to postgraduate medical education and into lifelong learning in practice. In contrast, the experience of most learners is marked by conspicuous transitions, from medical students’ primarily learning-focused activities to the resident role where learning is combined with the provision of clinical care and subsequent advancements into independent practice. In the first transition, from medical school to residency, deficiencies have been highlighted in many graduates’ preparedness to provide clinical care under supervision, as expected of first-year residents. While few objective studies have compared the capabilities of current medical school graduates with the demands of current practice, local experience that new residents feel less than optimally prepared has given rise to “academic boot camps” to prepare the residents for the basic elements of supervised clinical practice.12-15 Through the use of simulation, some of these “boot camps” have found deficiencies in basic skills, such as aseptic technique, hand hygiene, cross-cultural communication, obtaining informed consent,12 and basic procedure skills such as laceration management.14 In surgical specialties, transitioning from medical school to residency has focused on the development of basic surgical technique.16-18

If faculty and the organizations hiring them can be believed, new physicians may be less well prepared at the completion of residency and transition into independent practice, when compared to early cohorts. This phenomenon has been attributed, at least partly, to more limited exposure to patients when working under duty hour restrictions. Current residency also do not appear to meet expectations for person-centred, prevention-focused, and cost-effective care despite efforts to identify physician roles and attributes that are important to meet societal needs.59-63 Finally, progression within residency may lead to transitions that leave residents less than optimally prepared for a new, more responsible and independent role in providing care. A recent systematic review supported the existence of a “July effect” with a higher incidence of errors and adverse events in the early months of the academic year, particularly in studies with higher-quality designs and larger samples.64 Studies that have explored the effect of transitions in anesthesiology across the educational continuum are relatively rare, although an Australian study found that the adjusted risk for adverse events from care provided by anesthesiology residents was significantly higher at the start of the academic year.65 Also, a study in the United States showed that more than one-half of anesthesiology residents failed to adhere to the standard of care for preoperative cardiac evaluation.52

Despite commentaries and editorials, there are few studies of interventions to minimize any negative impact of shifts in roles and responsibilities as physicians progress through medical education. In anesthesiology, much of the research has focused on technical skills, such as airway management and dealing with anesthesia crisis management, with frequent use of simulation for teaching and assessment.19-25 The focus of other recent work that explicitly or implicitly mentioned educational transition has been on education in the “business” aspects of medicine26 and leadership training for fellows.27 Promising areas to prepare physicians for the transition into practice include enhanced preparation for providing culturally competent care;28 end-of-life care, particularly for individuals in subspecialty training in pain medicine and critical care;29 and the cost-effective use of medical resources.30 Another focal area is monitored and mentored experiences in a clinical specialty at the completion of residency so as to ease the surgical/technical transition into practice. Studies have focused on surgical specialties where operative experience and skills acquisition may have been negatively affected by duty hour limits,31 yet the concept is equally applicable to specialties like anesthesiology. Evidence suggests that mentoring is helpful even for practicing physicians when they make transitions into new settings.32 This suggests another promising area for study—how to make seamless transitions in medical education and, as a result, enhance the quality and safety of care and the professional development of physicians. Research in this area has the potential to add significantly to evidence-based approaches to reform physician education.

Duty hour limits and the acquisition of competence

Underlying the second theme is a sizable body of research that addressed the effect of work hour limits on the safety and effectiveness of care in settings where residents learn and participate in care.66 This has demonstrated some benefit, though studies have not shown that duty hour limits have resulted in safer care in teaching hospitals,67,68 and in a recent review, studies were found with improved patient care outcomes, with worsening outcomes after duty hour limits, and with a number showing no change.11 In the same review, the educational outcomes of duty hour limits were also analyzed and significant problems were found with study quality. Aside from a small number of studies that assessed performance on standardized tests, there were no data on the impact that the reduction in residents’ hours would have on the acquisition of competence for independent practice,11 potentially because true educational outcomes can be studied only years after the standards have been implemented. As a consequence, much of the research on the effect of duty hour limits uses exposure to clinical training as a proxy for educational outcomes. That shorter hours and reduced patient care and procedural experience may produce graduates who are less well prepared is a concern of faculty and educators, in part because graduates may not reach the volume of patients and/or the number of procedures thought important for the attainment of competence. These numbers abound in United States accreditation requirements, yet they are not empirically based—they result from expert consensus. This suggests that research has yet to be carried out to establish an evidence base from which to assess the educational effect of the duty hour limits on competence for independent practice. Under limited duty hours, United States surgical programs have sought to preserve procedural volume—potentially at the expense of continuity and involvement of residents in postoperative decision-making and patient management.69,70 Informal reports from pediatrics suggest that residents in programs with higher total hours of clinical learning performed better on the board certification exam.Footnote 1 Support for the importance of research into the educational effects of duty hour limits comes from studies showing reduced competence in graduates trained under the more restrictive European standards,33,34 and from Canadian research that surgical graduates lack confidence in their skills at completion of residency.35 A small number of studies, all in surgical specialties, have assessed the educational impact of the limits in the United States. One national study36 and two single institution studies37,38 found improved scores in the in-training exam in surgical specialties. In a national study of orthopedic surgery residents,39 three single institution studies in surgery and otolaryngology,40-42 and an older study of the impact of New York state’s duty hour regulation on obstetrics-gynecology residents40 showed no changes in in-training exam performance. Among the three studies that assessed the effect of the duty hour reduction on board certification in surgical programs, declining performance was found in a national study of surgery residents44 and a single institution study of neurological surgery residents,45 and no change was found in a second single institution study of surgery residents.41

Commentaries have expressed concern about the educational effect of duty hour limits in anesthesiology.71,72 In two studies in the United Kingdom, reductions in clinical exposure and case load were found in anesthesiology after the limits were introduced.46,47 In one study, reduced opportunities for training with faculty and reduced perceived competence were found,34 and no change was shown in four studies.48-51 There is a considerable lack of high-quality studies that focus on the effect of duty hour limits on the competence of anesthesiologists after completion of training. In studies on the effect of the limits on surgical training, the hours in operative and perioperative training totalled only slightly more than 5,000 hours or somewhat more than one-fourth of the 19,200 hours in five years of surgical training with an 80-hour work week.73 Total hours of active practice fell significantly short of the 10,000 mark found in individuals who have attained “international-level performance” in other sectors of competence, such as athletes, professional musicians, and chess players.4,74-77 The absence of research on the hours and the processes involved in the acquisition of clinical skills makes these proxies from other domains of performance appealing, but their applicability to medical education has not been studied formally; at the same time, much of the research cited in Ericsson’s 2004 comprehensive review about the acquisition of physician competence was conducted in the 1960s.77 There has been little recent work in this important area, including work on the benefits of simulation, rehearsal, and “deliberate and guided practice” in medical education.

Embedding the competencies in resident teaching and evaluation

The aim of competency-based education is to take the holistic concept, “competent physician”, and deconstruct it to illuminate behaviours and underlying cognitive and haptic functions to improve teaching and assessment.78,79 Slightly different models have been used to implement the competencies in the United States and Canada, with the United States using the competencies and Canada using the CanMEDS roles, yet the basic aims are comparable.78-82 In the United States, nearly ten years into the initial implementation of the competencies, research has yet to produce evidence for the benefits of the current approach. Innovation in the field currently comes through collaborative work among stakeholders in the clinical specialties to develop the educational milestones.83,84

Competency-based education is curricular innovation. Enhancing education and evaluation adds value for learners, the health care system, and society. By deconstructing the holistic “effective practitioner” into a set of observable teachable behaviours, it enables explicit teaching and observation-based evaluation. At the macro level of the United States health care system, adoption of the aims of the Institute of Medicine promotes attributes and behaviours associated with a high-performance health care system.19 While there is acknowledged conceptual legitimacy of the construct “competencies”,78-80 the key issue is that attempts to apply these to teaching and evaluation may create problems for evaluators for whom these deconstructed elements appear unrelated to the complex and frequently indiscernible ways they are integrated into clinical practice.78,85,86 This is particularly true when the residents under observation have moved past the Dreyfuss’ levels of “novice” and “advanced beginner” to “competent” and “proficient.” At these stages, learners deal simultaneously with multiple activities and accumulated information; they begin to take a holistic view of situations and perceive deviations from the normal pattern.87,88

Better evaluation tools are needed that allow for the scoring of complex integrated cognitive-haptic behaviours essential for clinical practice in the specialty. Findings in certain studies have shown that assessment based on the CanMEDS framework was not optimally transparent to learners and may have misrepresented the framework itself.86 Recent efforts to enhance implementation and application of the competencies have focused on the development of educational milestones, namely, accepted defined aspects of clinical competence that should be attainable at a certain stage of training, and teaching and assessment regarding entrustable professional activities (EPAs).89-91 Entrustable professional activities are “real-life” clinical tasks (e.g., the preoperative anesthesia assessment of a patient undergoing elective surgery) that incorporate a number of discrete elements of clinical performance and with each element being comprised of several competencies. These clinical tasks can be “entrusted” to a resident by faculty with supervision present or remote and with feedback that includes both a holistic focus on the entrusted activity as well as disaggregated suggestions for components such as communication or advocacy. The EPA framework matches the competencies to “essential activities” of physicians, thus creating a clinical context that adds meaning for the learner and the evaluator.84 This attribute of EPAs has the potential to bridge another gap, i.e., the discrepancy between the theory and the practice related to the competencies, and it is viewed by a growing number of educators as the new building block for the assessment of residents’ acquisition of competence.84,89-91 What is largely lacking to date is research to develop and test tools and to establish their relationship to the accepted observable abstractions called the competencies and to the newer concepts of the milestones and EPAs. Also, studies are lacking to validate that teaching and evaluation using the deconstructed concept of the competencies produces better physicians.

While the focus of much of the work in anesthesiology is still on medical knowledge and technical skills,19-25 it is promising that the focus of recent research has been to improve adherence to care guidelines,52 to validate and demonstrate the utility of the CanMEDS framework,53 to contextualize professionalism, systems-based practice, and other non-technical skills for anesthesiologists,54-57 and to put forward a model that will enable anesthesiology residents to integrate the competencies and the aims of the Institute of Medicine for the health care system.58 The focus is on approaches to teach and assess the non-technical competencies, including the development of validated tools. The aim of these new tools is to support faculty expertise in education and evaluation and not to supplant it. This should result in better teaching, additional assessment, and the provision of high-quality feedback.

Scientific theory in medical education research

The review suggested a relative paucity of scientific evidence in these areas and a need for research and the development of new scientific theory. In addition, studies generally treated the themes in isolation, while in actuality they interact to produce optimal as well as suboptimal learning situations. Medical education research is often limited by small samples, brief follow-up, and threats to validity. Overall, this suggests a “research gap” where editorials and commentaries have moved ahead of an evidence base for education. Colliver asserted that even when theory underlies recommendations for significant change in educational practice there is a lack of clarity even when the theory itself is based on scientific evidence.7 Problem-based learning in undergraduate medical education should have a powerful effect, yet 20 years after its implementation there still is no convincing evidence of educational superiority and effectiveness.9 The lack of findings may relate to methodological limitations of much of the research in the field. A review of the literature shows few studies with interventions, fewer with control groups, and fewer yet that span multiple sites and have sufficiently large samples with the power to reject the null hypothesis. Systematic reviews and meta-analyses to overcome the limitations of small underpowered primary studies are rare, and few studies have dealt effectively with the lack of reliable assessment tools and with the threats to validity in field research. Many medical education studies are presented as preliminary and exploratory, yet a worrying finding from a longitudinal view of research literature in the field is that there are no published “full-scale” follow-up studies for the majority of these “exploratory” articles where the preliminary findings are applied in a larger group and over a longer follow-up period. Two types of studies to advance the research on contextual factors are almost completely lacking, i.e., “utility studies” that assess which approaches are most efficacious or cost-effective, and research reports regarding what “did not work” to prevent others from testing approaches that were found to be clearly ineffective.

If the relative lack of progress in medical education research is attributable to the dearth of “good” theory, Bordage provides guidance on how conceptual frameworks can be used to guide research in the field.92 The major barrier to advancement in key areas of medical education research to close the research gap may be the scarcity of reliable valid assessment tools to facilitate comparisons of approaches. Beyond this, experimental and simulation studies and particularly field research retain threats to validity not always considered or well addressed, including effects of history, reactive effects, testing-treatment interactions, and selection-treatment interactions that produce outcomes that are relevant only to the populations studied but not at a national level.93,94

Guiding research to close the gap

A challenge of education on the three themes identified here is their interconnectedness and that of the elements of the clinical learning and care environment in general, which makes it difficult to ascertain cause-effect relationships and to assess the contribution of individual factors. Regehr observed that medical education is an applied field, and research into issues of curriculum and teaching and related innovation and evaluation are a natural consequence of this.8 This results in a large domain of potential topics without a ready means to organize the medical community to select and systematically address the most important questions in the field.8 Eva and Lingard’s recommendations are to aggregate forces and skills in research centres, and to view scientific progress not as the single important study but as the multi-year program of research that allows broader exploration of topics.5 Over multiple years and studies, Tamblyn et al. documented the relationship between scores on the Canadian provincial licensing examination and attributes of physicians’ performance in practice,95-99 and in their seminal work, they have shown that a focused body of high-quality research can create a scientific foundation for medical education and assessment. A future cohesive body of research to close the gap at the interface of medical education and practice could be similar to the research in hematology/oncology being organized around the Cancer and Leukemia Group B as a way to overcome the limitations of small local samples of cancer patients through pooled data and collaborative research for more than four decades.100

The work to be done will benefit from community deliberation and agreement on what would constitute significant educational outcomes to affirm or refute the effectiveness of interventions or the role of factors at the interface of learning and clinical care. Interesting work at the interface of medical education and delivery of ambulatory care suggests a focus on “educationally sensitive” outcomes that may advance the assessment of the effectiveness of educational innovation to domains that truly make them relevant, i.e., the impact on patients and the clinical system in which residents practice.101 There are recent examples that show the value of the competencies in anesthesiology.53-58 Future research could build on this beginning to develop and validate explicit “milestones” of expected learning achievements modelled on the competencies or the CanMEDS roles. Such an effort has the potential to offer enhanced models for teaching and assessment that could bridge the three themes discussed in this article, lead to clearer aims for educational progress across the continuum, enhance the assessment of clinical competence under restricted hours while ensuring its acquisition, and allow society and the profession to articulate expectations for the capabilities and skills of physicians in practice.

This discussion appears to omit the obvious—that a problem in the field of medical education research is the lack of funding. Better financial support could result in larger studies with control groups and better follow-up.

While funding could help support research, the priority is to establish a research agenda that will promote systematic study of contextual factors in resident education. The starting point at an institution, in a discipline, or nationally is dialogue to achieve consensus on some of the most pressing questions in the specialty. Regehr comments on the importance of knowledge building in the community using the analogy of parallel vs interactive play.8 The concept is taken from childhood development.102 Applied to institutions, it compares centres working in parallel on similar topics without communication with centres that communicate, collaborate, and cross-inform each other. This interactive effort is dually important. First, it expedites the generation of ideas and allows centres to learn from each other. Second, it is important in defining the priorities for work to be done at the interface of learning and clinical care. While this article suggests some potentially promising areas for research specific to anesthesiology, exploration and prioritization of areas for future research is ultimately the prerogative and responsibility of the specialty. An ideal venue for these deliberations would be centres that aggregate groups of researchers, possibly with multi-disciplinary engagement of social scientists and education and assessment experts. The researchers could integrate their investigations into theoretical fields, provide coordination to overcome fragmentation, and produce longitudinal programs of research capable of producing findings with real practical significance and benefit for the field. Canadian institutions and their centres of medical education research appear to have progressed further in this area than many sites in the United States.

Key points

  • There is a need for seamless transitions across the continuum of education.

  • The potential negative effect that existing and forthcoming reductions in hours of clinical training will have on competence needs to be addressed in medical curricula.

  • It is important to establish a strong linkage between the holistic construct “competent physician” and the competencies of the CanMEDS roles that illuminate behaviours and underlying cognitive and haptic functions.

  • There is a dearth of scientific evidence in the areas of medical education and evaluation, and there is an apparent need for high quality education research and the development of new scientific theory. Medical education research is often limited by small samples, brief follow-up, and threats to validity. This produces a “research gap” where editorials and commentaries have moved ahead of an evidence base for education.

  • Promising areas for research include preparation for care deemed important by society; work to apply knowledge about the development of expertise in other disciplines to medicine, with this effort spanning the continuum; and ways to embed the competencies in teaching and evaluation more effectively.