Faculty involvement in medical education is intrinsic to working in an academic health centre. This involvement ranges from the role of the “clinician-teacher”, where informal teaching and assessment of learners may predominate, to the role of the “clinician-educator”, where development of curricula, administration of training programs, and scholarship in medical education might be involved. Education has been described as one of the “three legs” of academic medicine along with research and clinical work. Its importance is self-evident and described well by this metaphor: as research is necessary for the creation of new knowledge, effective education is essential to generate new doctors; without it, the three-legged stool would collapse. In this paper, we argue that evolving changes to complex healthcare systems demand innovative approaches to effective medical education.

Although a considerable volume of education-related research is published in the anesthesiology literature, relevant reports of original investigations in education research often appear in many other specialty journals, which can hinder access for many teachers and educators. Furthermore, literature on educational theory and broader contextual issues is most likely published in the education and social sciences literature. Consequently, academic anesthesiologists may be less familiar with current trends in education science than with trends in clinical research.1

The aim of this review is to provide an overview of the literature pertaining to education in anesthesiology and relevant to perioperative medicine and critical care and pain that will be of interest to clinician-teachers and education specialists alike. Our aim is to stimulate discussion rather than to provide definitive answers, as in an overview of such a large topic, only a selected number of issues can be addressed. We will highlight issues of both current and future relevance to perioperative and critical care specialties with an overall goal of linking them to the broader context in medical education. A number of issues discussed in this article will be explored in greater detail in other articles appearing within this special theme issue.

In this review, we examine topics considered to be at the cutting edge of anesthesiology education and “hot topics” in medical education that are under-reported in the anesthesiology literature. Where possible, we have cited high-quality review articles published within the past five years so readers can easily reference further introductory reading. In the first part of this review, we consider relevant contextual issues from outside the medical education literature; we highlight the multi-disciplinary nature of medical education, and we consider the research methodologies that underpin educational theory. In the second part of this review, we address educational theory relating to curricula, teaching and learning, and assessment in anesthesiology. Other articles in this issue elaborate further on these topics. They broadly follow the outline described above and allow further exploration of recurring educational themes, such as the use of simulation-based education, interprofessional education, and the non-medical expert roles as defined by CanMEDS.2 The theme of anesthesiology education in this issue is intended for an international audience, although the focus of the final paper is on current challenges in anesthesiology education within the Canadian context.

Why should we innovate in medical education?

“He that will not apply new remedies must expect new evils; for time is the greatest innovator.” Francis Bacon, Of Innovations, Essays (1625)

All innovations involve risk and there is inevitable resistance to change. Why should we seek to innovate in anesthesiology education? Clinicians may ask, “It was good enough for me, why isn’t it good enough for the current generation of residents?” There is certainly change in all aspects of life, and innumerable factors outside educational theory impact what, how, and why anesthesiology educators do what they do (Table 1). These contextual issues include an increasing international focus on patient safety and risk management initiatives.3 High-profile cases of medical mismanagement have resulted in increased demands for accountability to the public.4,5 Research regarding anesthesia trainees who were taught in a traditional apprenticeship paradigm has identified poor unsupervised practice.6 Another interesting and important finding is an increased incidence of anesthesia-related morbidity and mortality occurring at the beginning of the academic year.7-9 In addition, there is a considerable body of evidence documenting the negative effects of sleep deprivation in both trainees and staff, and many jurisdictions have mandated reduced working hours, which, in turn, has resulted in concerns of decreased training opportunities and lack of continuity of medical care.10,11 One of the largest changes in medical education over the past 25 years has occurred in the United States as a result of the impact of managed care. There has been an actual decrease in the opportunity to learn from each patient’s set of circumstances due to the shift of patient care from the hospital environment to a more outpatient-based model.12 These cultural and societal changes demand a reconsideration of medical education. Other relevant issues include increasing medical commercialism, the continuing exponential increase in medical knowledge and medical technology, the often complex evaluation and accreditation of foreign medical graduates, and society’s varying perceptions and expectations of anesthesiologists as professionals.

Table 1 Evolving contextual issues affecting anesthesiology education
Full size table

The 21st century anesthesiology educator is faced with some challenging questions. As a specialty, how can we ensure that residents are trained adequately to deal with the ever increasing complexity of the modern healthcare system while simultaneously structuring training programs to minimize the fatigue shown to lead to medical error and burn out? Is it possible to improve the efficiency of training programs so that residents learn more in a shorter period of time? Is it possible to reduce the risks to patients inherent in anesthesiology training, especially in the earlier phases of learning? How can we assess the cost-effectiveness and efficacy of new technologies for use in medical education? What is the most effective means to teach residents the knowledge base and skill sets for rare events so they can retain the acquired skills for when they are needed? It is questions such as these that point us in a direction well beyond traditional medical literature. Fortunately, researchers in other fields are beginning to study medical education with different perspectives that can help to answer some of the questions posed above. In the following two sections of this article, we discuss the value of reviewing the literature from a broad range of disciplines and the importance of using a correspondingly broad variety of research methods that are often a significant departure from quantitative biomedical research methods.

The influence of other disciplines on medical education

As medical educators, we argue that we must look outside our own field for innovation. It is useful to consider the related knowledge base in a diverse range of other fields, including psychology, sociology, anthropology, bioethics, kinesiology, and human factors research.13 The recent cross-cutting edge series of articles published in the journal, Medical Education, celebrates this multidisciplinary nature by presenting “cutting edge research from another discipline of study that cuts into the field of medical education”.14 One example from this series that has particular relevance to education in anesthesia is the article by Martimianakis et al. which considers some of the sociological interpretations of professionalism15 (along with two accompanying commentaries).16,17 Although Martimianakis et al. acknowledge the need to identify unacceptable and unprofessional behaviour, they raise the concern about a “checklist conception” of professionalism characterized by defining criteria of competence and behavioural assessment. It is argued that defining professionalism in this manner is inadequate, and broader economic, political, and social dimensions must be incorporated into our understanding. Martimianakis et al. also review the concept of interprofessionalism; if professionalism is not merely an individual’s traits and behaviours but relates to complex interactions amongst different professional groups, patients, and institutional structures, how can we expect to teach or to assess professionalism adequately in professional silos? Reflected in this point is the increasing focus on interprofessional education in the medical education literature. Martimianakis et al. also consider social constructivist ideas of professionalism as “boundary work” that enables professional groups to protect markets, e.g., the creation of a boundary between obstetrics and midwifery from 1900 to 1930 in the United States.18 It could be argued that these issues are pertinent to the current relationships between nurse anesthetists and anesthesiologists in the United States. Cross-cultural differences, e.g., the different roles played by anesthesiologists, nurse anesthetists, anesthesia assistants, and operating department practitioners in Canada, the United States, and the United Kingdom, speak to the idea that any definition of the “profession” of anesthesiology is socially constructed. More contrasting comparisons can be made regarding the status of anesthesiology in the global South where, in many jurisdictions, anesthesia is delivered almost exclusively by non-physicians. It is the view of many that establishing cadres of well-trained non-physician anesthetists is essential to reducing mortality in practice settings where resources are limited, but this view may not always be accepted by local physician anesthesiologists.19 It can be argued that strengthening a physician-based anesthesiology profession is essential for effective advocacy amongst other more established medical specialties.

Cognitive and experimental psychology can provide medical educators with insight into factors of potential clinical importance, such as attention;20 perception and pattern recognition;21-24 the encoding, retrieval, and retention of memory;25-28 clinical reasoning and decision making;20,29-31 the nature of expertise;20,32-34 feedback;35 individual differences in learning;36 self-assessment;37 and the effects of stress and fatigue.38 An area of emerging interest in cognitive psychology is the observation that information retained through repeated testing surpasses that of repeated studying, especially if testing is spaced over time.39 This phenomenon builds on our understanding of the educational concept of formative assessment (assessment to promote learning)40 and has been shown to be clinically relevant in teaching in-hospital resuscitation skills.41 The testing effect has the potential to improve educational outcomes and, therefore, improved management of critical events that are infrequent but require immediate assessment and therapeutic intervention. Examples of important research studies from disciplines outside of medicine that nevertheless have relevance to education in anesthesia are too numerous to be dealt with comprehensively in an overview such as this paper. Herein, we highlight selected important recent research, and other examples are presented in Table 2. Additionally, this theme issue includes a further article in which the influence of psychology and human factors on anesthesiology education is explored.42

Table 2 Studies from disciplines outside medicine that impact anesthesiology education
Full size table

As an important means to foster answers to pressing research questions in anesthesiology education, other disciplines should use a broader range of research methodologies (discussed in the following section) than is found in traditional clinical research.

Medical education: research principles and methodology

As clinicians, it is natural to compare educational research methods to more familiar methods used in clinical research. However, medical education is a very different field to clinical medicine. Perhaps it is best at first to step back and consider the big picture, i.e., what constitutes the nature and limitations of knowledge (epistemology)? For instance, it is often stated that the randomized controlled trial is the optimal design for reducing threats to validity in clinical research, but is this study design optimal, applicable, or even desirable for medical education research? Furthermore, it is difficult to know which, if any, standardized reporting guideline (e.g., the Consolidated Standards of Reporting Trials [CONSORT] at http://www.consort-statement.org) might apply to medical education research.

The randomized controlled trial is considered to be the gold standard in assessing healthcare interventions.43 This methodology has also been applied successfully to educational research, a notable example being Ahlberg et al.’s observation that randomizing residents-in-training to virtual reality simulation training reduces both errors and surgical times for laparoscopic cholecystectomy.44 Such quantitative methods lend themselves well to the evaluation of teaching discrete psychomotor skill sets for which it is relatively straightforward to define objective goals and endpoints and for which subjects have little relevant prior experience. It is challenging to consider how the same methodology could be used to evaluate an intervention intended to teach domains, such as professionalism or health advocacy, for which there is likely to be significant variation in baseline behaviour and no well-established measurement tools.45

Clearly, some research questions in education require solutions that cannot be addressed by the more familiar quantitative methodologies. Qualitative methods employ a wide range of tools, including focus groups,46 research interviews,47 participant observation,48 and textual analysis. However, qualitative methodologies are probably better described by their similarly diverse range of underlying theoretical approaches, such as ethnography,48,49 phenomenology, and grounded theory (Table 3).50 In addition, researchers using qualitative methodologies often apply epistemological viewpoints that are very different from the dominant (positivist) paradigm of science and medicine, which states that knowledge is based on experimentation and measurable evidence.51,52 Qualitative researchers may approach a research question with the perspective that it is not possible to be independent and subjective as a researcher; that the researcher should acknowledge her/his own involvement in the analysis and interpretation of data (reflexivity); that there are no absolute or generalizable truths; or that the role of research includes challenging existing power relations and hierarchies.52 For instance, a key stance of the qualitative approach is the assumption of the uniqueness of every individual’s experience and every event, whether measured or not. The variety of different methodologies and theoretical backgrounds can make it difficult for those with a quantitative background to appraise qualitative research; other reviews provide guidance on this matter.53-55

Table 3 Some recent research questions in anesthesiology education investigated with qualitative methods
Full size table

It is easy then to see why qualitative research can be challenging, and this difference has often produced a qualitative-quantitative divide.56,57 However, it is becoming increasingly apparent that any question as to the absolute validity of quantitative or qualitative methodology is irrelevant. Both methodologies are appropriate for different goals, and one must first decide the nature of the research question before deciding the most appropriate methodology.58 Moreover, the combination of both quantitative and qualitative methodologies is now increasingly seen in one study (mixed methods research),59 though further developments are required to take full advantage of this approach.

Recent editorials in this60 and other anesthesia journals have highlighted qualitative methods.61 In 2009, a fine example of qualitative research in anesthesia education was highlighted in the journal Anesthesiology as one of the most important publications of that year.62 In their study, Waisel et al. 63 investigated challenges faced by anesthesiology residents in obtaining informed consent by performing textual analysis using the principles of grounded theory whereby themes are allowed to develop through an iterative process of coding data. Anesthesiology residents faced complex challenges, including patients’ mistrust towards healthcare professionals in general, questioning of trainees’ competence, and misunderstandings in the consent process that were not fully resolved. It should not be difficult to imagine how related issues could be explored further with this approach. Such examples could include cultural issues and contextual factors that determine the degree of trust that patients place in healthcare professionals as well as issues of accessibility to healthcare in various jurisdictions. In Table 3, there are detailed examples of recent research questions in anesthesia education that have been explored using various qualitative methods and epistemological assumptions.

In several recent articles, Cook et al. adopted a somewhat different approach by classifying educational research according to the purpose of the research rather than according to methodology.64,65 Descriptive research is used to disseminate educational innovations. Justification research aims to compare new educational interventions with existing practice and is more akin to clinical research. Clarification research aims to identify how and when to best use educational interventions. Rather than asking “Does simulation-based education work?” clarification research might ask “How should residents be debriefed following their simulation session to best identify their personal performance deficits?” Currently, findings in most medical education literature are reported as either descriptive research or justification research; there have been calls for more clarification research that is well grounded in educational theory.

Educational curricula

After considering the drivers to change and looking outwards to other relevant fields and research methods, it is essential to consider how this theoretical background affects our performance as teachers and educators. Before discussing teaching, learning, and assessment in anesthesiology, we will consider the application of theory to practice in curricula. Although curriculum is often considered to be the “content” of an educational program,66-69 its broader definition includes the teaching and learning strategies related to that content, the assessment of learners, and evaluation of the program.70 Hence, curricula can be seen as the overall educational plans. A still broader definition includes: a) the informal curriculum, which describes that which is not planned or taught but is nevertheless learned during interpersonal interactions with patients, other learners, and healthcare professionals during anesthesiology training, and b) the hidden curriculum, which refers to wider structural and cultural issues that result in the socialization of medical trainees (Fig. 1).71,72 Table 4 shows key issues in anesthesiology curricula that are discussed further in this section.

Fig. 1
figure 1

The three levels of a curriculum, as described by Prideaux135

Full size image
Table 4 Key issues in anesthesiology curricula
Full size table

Various organizational issues result in considerable international variation in the postgraduate anesthesiology curriculum, e.g., length of training, comparative emphasis on training or service, amount of time spent in the core specialty, and the amount of time “off-service” in other specialties. Historically, there has been considerable debate as to whether critical care medicine is considered a core specialty in itself or an extension of other specialties, such as anesthesiology or respiratory medicine. There is also considerable variation in terms of the timing of certifying examinations related to completion of training as well as the methods of in-training assessment. These variations have as much to do with historical and political factors as they have to do with educational theory.73 Traditionally, the specialties of anesthesiology and critical care medicine have been seen as postgraduate specialties; however, the appreciation that certain skills, such as basic airway management and the recognition of the critically ill patient, should be learned early in one’s clinical training has seen anesthesiology increasingly integrated into undergraduate curricula. Wong recently provided an in-depth review of theoretical principles of curriculum development in anesthesiology with particular reference to the undergraduate curriculum at McMaster University.74

Curricula are necessarily in a constant state of change prompted by the drivers discussed above, including a continuously evolving knowledge base, changing societal needs, and feedback from program evaluation. In addition, current trends in medical education present a number of challenges to anesthesiology curricula. There is constant evolution of simulation technology,75 which is increasingly recognized as offering a unique learning opportunity76 without exposing patients to risk.77 It is an ongoing challenge to determine how simulation and other educational technologies, such as e-learning,78 can best be integrated into existing anesthesiology curricula. Much of professional practice in anesthesiology relates to interactions with other specialties and healthcare professionals. Interprofessional education is currently a key theme in the medical education literature,79 and although the evidence base is still weak in this area, it was suggested in a recent Cochrane review that interprofessional education can improve healthcare processes and outcomes.80 Interprofessional education seems especially suited to the learning of CanMEDS roles such as communicator and collaborator;2 however, postgraduate education remains predominantly in professional silos. Interprofessional education presents the logistic and organizational/cultural issues involved in gathering different professional groups with the ultimate goal of improving quality of care through coordinated curriculum design—one can easily imagine the potential contributions of social psychology or sociology to this problem, as noted above. A related issue is the challenge of assessing a team’s performance in certain skills, e.g., cardiopulmonary resuscitation, for which teamwork is critical to patient safety but for which current assessments are preoccupied with individual performance.81 This is an especially vexing challenge given that the unit of professional certification will remain at the level of the individual for the foreseeable future. Creative solutions in this area are sorely needed to enhance the quality of teamwork, and we will likely need to look outside the disciplines of medicine and medical education, e.g., drawing lessons from the aviation industry. Certainly, a culture of continuous assessment would be beneficial at both the individual and team levels.

Postgraduate anesthesia training has long been based on the Halstedian apprenticeship model.82 Currently, competency-based education is the most influential idea in medical education curricula, i.e., to consider standardized measurable competencies as the central issue on which to base the entire curriculum. Outcome-based curricula have been described as defining a curriculum backwards, i.e., instead of emphasizing the instructional process of the curriculum, the desired competencies determine the content, teaching, and assessment. Competencies have become the “unit” of education83 in many jurisdictions found within competency frameworks, such as the Outcome Project of the Accreditation Council for Graduate Medical Education (ACGME) in the United States84,85 and CanMEDS in Canada and some other countries.2 Recently, Frank et al. reviewed the theoretical basis of competency-based medical education as part of a theme issue in the journal, Medical Teacher.83 Advocates of competency-based education argue that a focus on competency in specified domains will allow greater public accountability and greater responsiveness to societal needs. Competency-based education moves away from a preoccupation with knowledge and places more emphasis on skills and attitudes. It is argued that a curriculum of competencies can engage and empower learners by providing a transparent path to well-defined outcomes.83 In competency-based education, it is possible to tailor the duration of training to individual learning rather than expecting competence to diffuse over a pre-specified amount of time as in the “teabag model”.86 However, competency-based medical education is not without controversy, including ideological differences in opinion and epistemology and doubts that quantitative research methods can answer these “bigger” research questions adequately. Those who argue against competency-based education have the view that reducing medicine to a series of core competencies neglects bigger, important (but less observable) issues, such as medical altruism, the integration of knowledge into a cohesive whole, striving for excellence rather than for a minimum standard, and working as part of a team.87 The results of a recent review of the literature indicated that it may be exceedingly difficult to assess performance in certain domains of the ACGME competency framework, such as “systems-based practice” and “practice-based learning and improvement”.88 The challenges of introducing a competency-based curriculum have resulted in hybrid systems where elements of competency-based medical education have been combined with traditional apprenticeship curricula. There is much research to be done regarding issues such as the unintended consequences of introducing frequent formal competency assessments into training and into the interactions between competency-based education and the culture of medicine.

Teaching and learning in anesthesiology

For the majority of anesthesiologists in academic health centres, the significance of medical education theory will probably be best described by how well it translates into the practice of formal and informal teaching of residents. Teaching and learning in anesthesiology may be achieved through a variety of means, including small-group89,90 or large-group91,92 didactic methods, each with its advantages and disadvantages.74 However, experiential learning is necessary to integrate the “book” knowledge into a practical skill set in anesthesiology93 as well as to learn tacit knowledge that is not (or cannot be) made explicit.94,95 The focus of this section on teaching and learning in anesthesiology is on two types of experiential learning—clinical teaching and simulation—which dominate the literature in anesthesiology education research.

Clinical teaching

Despite the mainstream acceptance of simulation-based education, the majority of teaching and learning in anesthesiology and perioperative care is, and is likely to remain, in clinical settings, such as the operating room, the critical care unit, or the emergency department. Anesthesiology has its own particular challenges and opportunities for the “clinician teacher”. The operating room can be an uncomfortable or threatening experience for a novice. Masks can hinder non-verbal communication.93 Patient safety and comfort must be balanced against a responsibility to the learner.96 The patient is present throughout the case, and although periods of uneventful maintenance of anesthesia may allow in-depth case-based discussions, the primary responsibility must always be to the unconscious patient. More challenging cases may only allow the clinical teacher to demonstrate clinical management with the learner taking a more passive role. Teachers may also have to choose between providing a teaching opportunity and maintaining the efficiency of an operating list,96 and learners may be inhibited if they perceive their teachers (or other operating room staff) to be impatient. Trainees often work with a large number of different clinical teachers, and they can find the variety in practice confusing.97 An operating list of many short cases with rapid turnover makes it difficult to develop any discussion;98 anesthesiologists must then be skilled at identifying and using the “teachable moment”. How can the medical education literature inform how we improve clinical teaching in anesthesiology?

Sutkin et al. recently reported a systematic review asking the question, “What makes a good clinical teacher in medicine?” The most commonly reported themes included the importance of being a content expert and possessing non-cognitive attributes, such as creating a positive learning environment, having excellent listening skills, enthusiasm for medicine, enthusiasm for teaching, and enthusiasm in general.99 Cleave-Hogg interviewed anesthesiologists who had been identified a priori as excellent clinical teachers and asked them what led to their success. They reported similar themes, including willingness to give one’s own time for teaching, enjoyment of one’s own profession, continually enriching one’s own learning, and an ability to establish and maintain interactive professional relationships.98

Many clinicians in teaching hospitals have received no formal training in how to teach81 and base their teaching methodology on intuition and their own learning experiences.1 Although, as described by Sutkin and Cleave-Hogg, many of the attributes of a good clinical teacher relate to personality, clinicians can certainly be taught how to improve their teaching. The value of faculty development to encourage even simple measures, such as establishing learning objectives and discussing these with trainees, cannot be over-emphasized.100,101 Further refinement would include conducting a needs assessment or gap analysis, and developing a plan for evaluation.102 Techniques that can improve clinical teaching in the operating room are detailed in Table 5. Good clinical teachers are characterized by an “inquiry” approach to teaching, i.e., getting residents to think through complex situations using Socratic questioning.98

Table 5 Techniques to improve clinical teaching in the operating room
Full size table

Feedback, described as the “cornerstone of effective clinical teaching”,103 has been found to be particularly challenging for clinical teachers98 and has been the subject of recent reviews.103,104 Having a culture of feedback is important and should be viewed as a normal component of the teacher-learner relationship. In contrast to the simulation debriefing literature, there is a paucity of research on clinical teaching and feedback in the operating room. Some guidance for effective feedback in clinical teaching is summarized in Table 5.

Other factors that have been identified as promoting excellence in clinical teaching include using senior educators as mentors for junior faculty and rewarding excellence in teaching as equivalent to excellence in research.99,100 The opportunity to have teaching evaluated by peers or experts allows valuable feedback; however, this is often not feasible with the practical constraints of clinical teaching.100 Evaluation of anesthesiologists’ teaching with feedback from residents has been shown to improve teaching.105

Simulation-based education

Simulation is a term that covers an increasingly broad range of techniques and technologies75,106-110 that share the characteristics of providing the opportunity for experiential learning without exposing patients to risk.77 Anesthesia was an early leader in simulation for medical education, and the adaptation of Crisis Resource Management from aviation’s Crew Resource Management is well described. There has been a tendency in aviation and other high-stakes high-reliability industries to introduce simulation without researching its advantage compared with other educational methods. David Gaba’s statement that “no industry in which human lives depend on the skilled performance of responsible operators has waited for unequivocal proof of the benefits of simulation before embracing it” is often quoted111; however, this sentiment contrasts with the prevailing emphasis for an evidence-base in clinical medicine. Simulation has become a mainstream enterprise in medical education, and the need to identify the most effective and most cost-effective aspects of simulation has resulted in simulation research dominating the recent literature in anesthesiology education. Perhaps the research that most convinces sceptics of the value of simulation are those “justification” studies65 that show the translation of skills taught in the simulator to clinical practice. Hall et al. reported a study which evaluated paramedics learning tracheal intubation using either a high-fidelity simulator or routine training on anesthetized patients. Later, both groups were evaluated on anesthetized patients, and the simulator group performed with competency similar to that of the control group despite not having previously performed tracheal intubation.112 Similarly, Barsuk et al. identified a reduction in complications and costs from catheter-related septicemia after simulation-based learning,113 and Wayne et al. found that trainees who had undergone simulator-based learning adhered more closely to the American Heart Association guidelines during cardiac arrest.114 Finally, Bruppacher et al. showed that residents who were randomized to simulation-based learning performed better than a control group of residents who had a case-based tutorial related to the complex task of weaning a patient from cardiopulmonary bypass. Performance in cognitive and interpersonal non-technical skills was also superior in the simulation group.115

Issenberg’s group has performed a series of reviews on simulation-based education.76,116 They identified feedback as the most important feature that promotes learning in simulation, but they noted a need for more research on how best to deliver feedback. They also commented that simulation offers the advantage of deliberate practice that is a key part of the development of expertise32,33 and the related concept of mastery-learning where skills are practised until a minimum mastery standard is achieved before moving on to the next educational unit.

Assessment in medical education

In his review of assessment in medical education, Epstein reported the common scenario of an attending physician receiving a form requesting an evaluation of a trainee’s competencies in diverse areas (e.g., procedural skills, professionalism, and system-based practice), and he noted the reaction of the attending physician questioning whether brief interactions with that trainee were sufficient to provide a meaningful assessment.117 Epstein also noted that we might question the validity of all other types of assessment that occur during training, such as multiple choice questions (MCQ), Objective Structured Clinical Examinations (OSCEs), short-answer questions, and oral examinations. Instead, we suggest that we as clinicians often assume that board examinations will cover any deficiencies in workplace-based assessment, and we give little further thought about the way in which these different forms of assessment work together to form a system of evaluation.

The first question to consider is “What is the purpose of our assessment?” Assessment can be either formative or summative. Formative assessment is designed to promote further learning; it identifies deficiencies in knowledge, skills, or attitude and motivates further learning. Tools for formative assessment should therefore be designed to provide rich and timely feedback. Summative assessment aims to judge competence, readiness for a lower level of supervision, or fitness for independent practice. Ideally, both types of assessment should have robust psychometric properties: they should be reliable (i.e., similar results would be produced if a different assessor administered the test) and valid (i.e., the test is really testing the skill/s we want it to test).

Taxonomies of assessment, such as Miller’s pyramid, are useful for identifying different levels of performance that may be assessed with different tools (Fig. 2).118 Written tests, such as MCQs and oral examinations, can assess the “knows” level of factual information. When designed to be rich in clinical context, the same tools probe the integration of that knowledge into clinical judgment and decision-making, i.e., the “knows how” level of assessment.117 It has been suggested that simulation has advantages over the oral examination in the domain of crisis management because it can assess the actions trainees would actually perform (the “shows how” level of performance) rather than what they say.119 Consequently, simulation has been introduced in various forms into the anesthesia board examinations in Israel, the United Kingdom, and more recently, Canada,120 although the use of simulation for high-stakes assessment remains controversial. The “does” level of assessment can be performed only in the workplace by the expert anesthesiologists who are supervising trainees. All of these levels of assessment should be incorporated into a comprehensive system of assessment, and even the most robust board examinations cannot compensate for a failure to assess actual practice in the workplace.

Fig. 2
figure 2

Miller’s pyramid. This diagram illustrates hierarchical levels of performance that can be used to assess learning. The bottom two levels refer to cognition: “Knows” refers to the retention of factual information, and “knows how” refers to the interpretation and integration of this knowledge into a management plan. The top two levels describe behaviour: “Shows” refers to a demonstration of learning, and “does” refers to an assessment of actual clinical practice

Full size image

Therefore, what competencies should be assessed? There has been a recent international move towards the use of explicit frameworks of competencies. The CanMEDS model places the medical expert role at the core of six overlapping roles or domains of competency: scholar, manager, communicator, collaborator, health advocate, and professional. The corresponding model in the United States is the Accreditation Council for Graduate Medical Education (ACGME) which has six domains of competence: medical knowledge, patient care, professionalism, communication and interpersonal skills, practice-based learning and improvement, and systems-based practice.85

A wide variety of different assessment tools are necessary to assess such different domains of performance, and a detailed review of individual tools is outside the scope of this review. A retrospective global rating after a lengthy period of unstructured observation has inappropriate qualities for summative evaluation. van der Vleuten writes, “Research indicates that the human mind is easily led by what we think we have seen, usually based on gross generalizations of a few cues or samples of performance which do not necessarily coincide with reality.”121 Psychometric properties are unsurprisingly stronger for psychomotor skills122 than for more subtle cognitive and interpersonal skills assessment.123 Training improves the inter-rater reliability of assessments, speaking to the value of education-focused faculty development. Some areas of competency, such as professionalism, provide the same challenges discussed previously regarding quantitative research; no psychometrically robust tools have been developed to assess a construct over which there is considerable disagreement regarding its definition.15,45 Few would argue the central importance of professionalism to anesthesia practice. In that case, where does this leave the clinical educator reliant on traditional assessment tools? Qualitative methods of evaluation may be more fruitful than attempting to map this construct onto a numerical scale. Perhaps we should also consider alternative frameworks for judging assessment tools, including those recommended for evaluating qualitative research55,124: Is the method of sampling appropriate? Did the assessor acknowledge his or her own influence in the assessment of behaviour? Is the information triangulated with other sources of information? What is the credibility of the information? Posing such questions might assist us in developing alternative approaches to assess problematic areas, such as teamwork,117 as the question remains concerning the extent to which teamwork can be assessed as an individual competency.

van der Vleuten has noted that the literature on assessment in medical education is dominated by discussions regarding the psychometric properties of individual tools, without consideration as to how they fit into a larger system of assessment.125 In his literature review, he notes that reliability of assessment tools is not intrinsic to that tool, but sampling is far more important; most tools become reliable if used for at least four hours. He notes that subjective assessments, such as the mini-Clinical Evaluation Exercise (mini-CEX), can be as reliable as more objective tools.

Any assessment can have unintended consequences. Summative assessments influence learning even in the absence of feedback, as students “study to the test”. Students may “cram” for examinations, reducing the chance of long-term retention of information, and they may substitute superficial knowledge for reflective learning if this is the content of the test.117 Other research has shown the following results: deterioration in the actual competency of students who were memorizing objective structured clinical examination (OSCE) checklists; an unexpected increase in rote memorization after a switch from MCQs to free response questions; and role conflict when facilitators in small-group teaching were requested to provide summative evaluations.121

So what of the practical dilemma of the clinician asked to assess a trainee after only a few operating lists or interactions in the intensive care unit? Workplace-based training may be formative, and certain principles of providing good feedback apply, including finding time and a space free from distractions, encouraging the trainee to reflect on her/his personal performance, providing detailed feedback on a few key areas of performance, identifying areas of good practice, and identifying practical ways in which gaps in performance can be remedied in the future. Ideally, to make workplace-based assessment a useful part of summative evaluation, it should be performed prospectively for a pre-defined competency by an assessor who has been trained in the use of the assessment tool. Limitations should be acknowledged; every assessment tool has its strengths, weaknesses, and contextual issues.117,125 Certain tools are better for assessing different competencies,126 and multiple assessments over time can compensate for the failings of individual assessment tools from a single clinical encounter.127 Key issues in anesthesiology assessment are outlined in Table 6.

Table 6 Key issues in anesthesiology assessment
Full size table

Other important issues in assessment include post-certification lifelong learning. How valid is the assumption that our self-assessment is an appropriate method for directing continuing professional development? How can we ensure that our profession’s contract with society is fulfilled by the career-long maintenance of competence? These and other related issues are explored further in articles appearing in this issue of the Journal.

Conclusions

In this review, we have attempted to provide a broad overview of current and emerging issues in anesthesiology education. Medical education is vital to anesthesiology. Along with research and clinical service, education represents one of the “three legs” of academic medicine.128 In many ways, however, medical education is separate from the other two legs due to its multidisciplinary nature. Academics in fields, such as psychology, sociology, medical anthropology, and kinesiology, are invested in medical education and make vital contributions to this branch of learning. We argue that embracing the diversity of these other relevant fields is a key aspect of truly innovative work in medical education. These fields often have challenging theoretical backgrounds and research methodologies that can inform curricula, teaching, and evaluation in anesthesiology. The use of research from other disciplines is probably underutilized in anesthesiology. Cross-referencing research from other disciplines and methodologies could be useful to investigate questions such as: How should we provide feedback on a task so the task is better performed the following week rather than better performed only immediately following the feedback? What are the implications of assessing resuscitation training as a team activity? How should we teach health advocacy in anesthesia? Can competency-based education cause medical education to respond more effectively to societal needs? Should simulation be used for high-stakes evaluation?

The purpose for this review is to promote further interest in a large body of peer-reviewed literature that is relevant to enhancing educational opportunities in anesthesiology and related disciplines. Hence, we aimed to highlight relevant questions rather than to answer them. Other articles in this issue will expand on this discussion and further explore some of the key issues in anesthesiology education.