INTRODUCTION

Residency training is a pivotal time in the development of a physician. In their 3 or more years of training, residents are immersed in the culture and experience the behavioral norms of their attending faculty. At the same time, they model behaviors and expectations for the medical students that they supervise. The residency program “culture” is the shared learning of the residents and others, its norms, beliefs, values, behavior patterns, customs, traditions, and stories that are held in common. Culture is customs and rights. Whereas the culture for faculty and the “hidden curriculum” for medical students have been studied, there exists no similar documentation or database for residents, despite calls for attention.1 – 4

Leaders in residency training have suggested the need to consider and assess the role of residency culture, because it is the culture of everyday life that fosters or inhibits residents’ ability to develop into competent and compassionate physicians,1 – 3 and to develop the six core competencies stipulated by the Accreditation Council for Graduate Medical Education (ACGME), the accrediting body for U.S. residency training programs.5

A wealth of studies conducted over the past six decades have documented the harrowing nature and stress of residency training.6 – 8 Suicide and burnout have been established as growing areas of concern in residency training,3 , 7 , 9 with recognition of the tragically high suicide rates among U.S. residents.3 Seelig and colleagues developed a residency program evaluation questionnaire designed to assess emotional distress and satisfaction with the workload and learning environment.10 Other studies have assessed residency programs using the Maslach Burnout Inventory.11 Gruppen and colleagues recently found that satisfaction with the learning environment and workload were closely correlated, and advocated for additional studies to investigate factors such as program culture, the quality of relationships, feedback, and level of competition.12 Studies have also demonstrated that sustained learning is more likely to occur when conditions are supportive.13 A growing body of research on physicians suggests that cultivating meaning and joy in work, and enhancing relationships, empathy, and psychological safety, improve patient outcomes and physician vitality.1 , 14 , 15

In a recent initiative, the ACGME Clinical Learning Environment Review (CLER) recognized the critical role that the culture or clinical learning environment plays in the training of physicians and the provision of safe, high-quality patient care.16 Meeting the goals outlined in the six domains of ACGME’s CLER (patient safety; quality improvement; transitions in care; supervision; duty hours oversight, fatigue management and mitigation; and professionalism) will depend in large part on the institution’s culture. For example, is the culture perceived as trustworthy, such that residents can report a medical error to superiors without fear? Are residents likely to admit when they are too tired to function safely?17 Do residents feel that their ethical and moral values are shared and modeled by their faculty and by the teaching institution? Both the Institute of Medicine’s To Err is Human report18 and the ACGME’s CLER initiative, however, have largely focused on technical components of patient care and safety.

The National Initiative on Gender, Culture and Leadership in Medicine, known as C - Change (for culture change),19 seeks to facilitate change in the culture of academic medicine in order to maximize the contributions of all faculty and trainees and increase diversity in leadership.

This report describes the development and steps to determine the valid use of a C - Change questionnaire to document and measure residents’ perceptions of the clinical learning environment, professionalism, and the culture of residency programs. The study is a foundational step to foster improvement in residency programs. This study of the culture for residents complements similar efforts focused on medical students20 and faculty, and prior studies by researchers addressing residents’ distress and workload.

METHODS

Measures

C - Change Resident Survey Adaptation from a Validated C - Change Faculty Survey

The C - Change Resident Survey (CRS) was developed by adapting the reliable and well-validated use of the C - Change Faculty Survey (CFS), which measures 12 dimensions of the culture of academic medical centers and has been used extensively throughout the United States, as well as in Canada and the UK.21 – 25 The dimensions and items of the survey were derived in large part from themes identified by C - Change qualitative studies of in-depth interviews with male and female faculty in five schools26 – 31 in conjunction with an expansive search of the literature. The survey was fielded to a randomly selected sample of faculty in 26 nationally representative randomly selected schools. Factor analyses with semantic review revealed 12 dimensions of the culture representing various aspects of faculty experience in academic medical centers. The unique analytic contribution of each dimension provided evidence of the C - Change faculty instrument’s construct validity and the instrument’s content validity established by its basis on in-depth qualitative research. The reliability of the instrument was established by the high internal consistency of the 12 dimensions.21 The dimensions have been used repeatedly in statistical models and reported in published papers to describe faculty attrition,21 vitality,22 and mentoring,23and to evaluate the role of minority status24 and gender25 in understanding the culture of academic medicine. Detailed descriptions of our faculty survey study methods have been published previously.21

Survey Development

The 68-item resident survey uses many of the same items as the faculty survey to measure the dimensions of the culture, including vitality and engagement, relationships and feelings of inclusion and trust, self-efficacy in career advancement, perceptions of institutional support, feeling respected, intimidation, equity for women and underrepresented in medicine minority members, institutional values and practices, leadership aspirations, mentoring and work–life integration. An additional dimension focused on residents’ own self-assessed “competencies.” In consultation with a senior ACGME officer, we added six survey items to represent the six core competencies for residency training defined by the ACGME.5 We anticipated that this addition would be useful both for analytic purposes in examining the culture of residency programs and as a metric to assist the leaders of academic programs.

Table 1 describes the 13 dimensions of the culture. Items were scored on a five-point Likert scale (range: 1 = strongly disagree to 5 = strongly agree). After reverse coding negatively stated items, the mean response for all items in the scale provided a summary score on the original five-point Likert scale. Additionally, the CRS collected information from respondents on gender, sexual orientation, race/ethnicity, age, country of origin of their medical degree, presence of children younger than age 18 years at home, and year of post-graduate training.

Table 1 C - Change Dimensions of the Culture for Residents
Full size table

Content Validity

The survey’s content was based on the hypothesis that well-established dimensions important in the culture for faculty would be the same for residents who are practicing and learning in the same culture. Prior to this study, the CRS was first pilot-tested with residents at two major teaching hospitals in 2012 and at a multi-specialty residents’ conference. We made several changes to the survey items based on the pilot studies.

The dimensions of the culture assessed by the CRS were also deemed relevant by considering prior scholarship on measuring educational environments for postgraduate trainees. For example, in studies aiming to enhance patient safety, the most effective responses to fear and error were positive connection between people and the development of trust.32 These prior studies suggest the need to assess relationships and the trustworthiness of learning environments as were included in the C - Change (Relationships/Inclusion/Trust dimension) items. Connor et al. identified self-efficacy and values orientation as qualities of a resilient individual.33 These attributes are assessed by the C - Change dimensions of Self-Efficacy in Career Advancement and Values Alignment. There are calls for more respectful behaviors in academic medical settings34 and in large studies in the UK, bullying was reported by 7% of residents,35 suggesting the appropriate inclusion of the C - Change Respect dimension. Assessment of ethical and moral distress is warranted based on reports of sources of ethical conflict and of poor role models and unethical behaviors among clinical teaching faculty.36 – 39 One study reported that nearly half of clinical students reported witnessing a clinical teacher acting unethically.39 Studies have also documented the existence of gender and racial discrimination in academic medicine40 – 45 and support the need to measure resident perspectives on equity with regard to gender and underrepresented minorities in medicine.

Sample

C - Change partnered with the CIR Policy and Education Initiative to identify a diverse sample of U.S. residency programs, which varied by specialty, geography, program size, population served, and institutional sector (public or private). We included 34 residency programs (all years of training) across three specialties (internal medicine, pediatrics, and general surgery) at 14 academic health centers. At nine of the 14 institutions, all three specialty programs were surveyed; at two institutions, only two specialties were included; and at three institutions, only one specialty program was included (Table 2). Generally, internal medicine programs at the 14 sites were much larger than general surgery or pediatrics programs.

Table 2 Characteristics of 14 Academic Health Systems and 34 Residency Programs Participating in the Culture of Residency Study
Full size table

Data Collection

The CRS was sent electronically to all 1333 internal medicine residents, all 643 pediatric residents, and all 476 general surgery residents who were enrolled in the 34 participating programs. Program directors and resident champions were enlisted to encourage survey participation. An announcement from the program director or other institutional leader preceded survey administration. The survey was administered in three phases from late fall 2014 to spring 2015. Frequent electronic reminders were sent by C - Change staff. All data were stored by C - Change on a secure, password-protected external site. Results were reported in aggregate form only, and no record-level data were provided to schools, in order to ensure the confidentiality of respondents. Institutional review board (IRB) approval was obtained from Brandeis University.

Analytic Overview

Reliability

To assess the reliability (internal consistency) of the dimensions of residency culture, we estimated Cronbach α coefficients. A dimension was considered reliably measured if α exceeded 0.7. We also estimated item-to-scale correlations and assessed whether the α coefficients were stable across specialties.

Additional Evidence of Validity

Response Process

We evaluated several aspects of the response process to assess whether the CRS was a feasible method for measuring the learning environment of residency programs. We focused on the proportion of invited residents who completed the survey (response rate), the frequency of missing responses, and the proportion of surveys with evidence of logically inconsistent responses. Logical inconsistency was evaluated by comparing positively and negatively worded items measuring the same underlying construct to determine the frequency of contradictory responses (both responses strongly agree, or both responses strongly disagree).

Construct-Related Evidence of Validity

To accrue construct-related evidence of validity, we examined whether the dimensions of the culture behaved in ways consistent with our conceptual model and with results of previous research. In addition, we tested whether the instrument could detect differences across programs—in particular, whether differences could be detected across institutions within the same specialty and across specialties, within the same institution. Based on our conceptual model, we tested whether Vitality was strongly correlated with scores on the dimensions of Relationships/Inclusion, Values Alignment and Work–Life Integration, but negatively correlated with Ethical/Moral distress. Our conceptual model also predicted that scores on the Competencies dimension would be positively correlated with scores on Self-Efficacy. Pearson correlation coefficients assessed the associations between the dimensions.

Analyses were conducted using Stata version 14 (StataCorp LP, College Station, TX) and SAS version 13 (SAS Institute Inc., Cary, NC). We accounted for clustering of respondents within program using multilevel models (mixed command in Stata).

RESULTS

The overall response rate for residents in the 34 participating programs was 70% (range: 51% to 87%), with little variation by specialty: 72% for internal medicine, 69% for pediatrics, and 65% for general surgery. In all, 1708 residents responded to the survey: 956 internal medicine, 441 pediatrics, and 311 general surgery. Respondents included 879 (51%) women and 268 (16%) underrepresented in medicine minority residents.

Reliability

Cronbach α coefficients for all 13 dimensions of the culture exceeded 0.7 (range: 0.73–0.90), reflecting strong internal consistency for each dimension and for each specialty (Table 3). In particular, the newly-developed dimension measuring self-assessed Competencies demonstrated high α coefficients for each specialty (Table 3).

Table 3 Estimated Cronbach α Coefficients of Dimensions of the Residency Culture by Specialty
Full size table

Response Process

A good response process was demonstrated by the high response rate of 70%, low incidence of missing data, and very few logically inconsistent responses. Eighty percent of respondents’ surveys had no missing data, and 90% had missing data on six or fewer items. Fewer than 2% of respondents demonstrated logically inconsistent response patterns (for example, strongly agreeing or strongly disagreeing with a pair of items measuring the same construct but where one was worded positively and other negatively).

Construct-Related Evidence of Validity

Figure 1 shows the distribution of mean dimension scores for the 34 programs, aggregated by specialty. There were significant differences in dimension scores across institutions within each specialty (p < 0.001). There were also significant differences in dimension scores across specialties within the same institution (p < 0.001). For example, one institution had the highest mean Vitality score among all 11 pediatric programs, but its general surgery program had the lowest mean Vitality score among the 11 surgery programs.

Figure 1
figure 1

Residents’ perceptions of the clinical learning environment and professionalism in 14 representative academic health centers: distribution of responses in internal medicine, pediatrics, and general surgery residency programs. *Reverse-coded, lower is better. Abbreviations: IM, Internal Medicine; P, Pediatrics; GS, General Surgery; Vi, Vitality; SE, Self-Efficacy in Career Advancement; IS, Institutional Support; RI, Relationships/Inclusion/Trust; VA, Values Alignment; EMD, Ethical/Moral Distress; Re, Respect; LA, Leadership Aspirations; WL, Work–Life Integration; GEq, Gender Equity; URMMEq, Underrepresented in Medicine Minority Equity

Full size image

As predicted based on our conceptual model and previous research, Vitality scores were highly correlated with Relationships/Inclusion/Trust (r = 0.57, p < 0.001), Values Alignment (r = 0.57, p < 0.001), and Work–Life Integration (r = 0.60, p < 0.001), but negatively correlated with Ethical/Moral Distress (r = −0.54, p < 0.001). Female residents scored the dimension of Gender Equity lower than male residents, except in Pediatrics, where this gender difference vanished. We also confirmed the hypothesized strong correlation between Self-Efficacy in Career Advancement and Competencies (r = 0.74, p < 0.001).

DISCUSSION

The CRS can serve as a feasible, reliable instrument for assessing the culture of residency training programs and the clinical learning environment, with promising evidence of validity in the environments evaluated thus far. Our findings regarding the 13 dimensions of the resident culture are consistent with the results of our C - Change faculty and medical student surveys. We demonstrate the instrument’s utility in detecting differences across programs of the same specialty, and even across programs within the same institution. These findings suggest that culture is specific to a particular program within an institution.

We were not able to assess the criterion-related evidence of validity for the instrument, because there is no accepted “gold standard” for measuring the learning environment of residents, but results provide construct-related evidence for the instrument’s validity. The content validity of the CRS is demonstrated by well-written questions that adequately represent the dimensions measured, similar to findings in the parallel validated faculty and medical student surveys.20 , 21 Thus, we are optimistic that the instrument can facilitate the mission of improving resident well-being, of educating fully competent physicians and surgeons, and helping to address the ACGME’s focus on professionalism and the clinical learning environment.

Limitations of the study include the fact that the programs were not selected randomly; however, a number of important attributes of programs were represented in the sample. The data were collected from three major specialties, although not all specialties were represented. Residents self-reported data, but the high response rates and assurance of confidentiality through external collection of data and reporting of aggregated data only would support the likelihood of the authenticity of responses. The data were collected at only one point in time; however, large numbers in multiple cohorts of residents in 34 programs were included, and all years of residency training were surveyed.

The CRS allows for detailed exploration of the nurturing of residents (e.g., trust, relationships, respect, perception of opportunity) as well as identification of negative experiences (e.g., disconnection, ethical distress, burnout, disrespect, and inequity, and difficulty in integrating one’s professional and personal responsibilities). Three of the 13 dimensions focus specifically on humanistic health care that is also the focus of ACGME’s CLER initiative: Relationships (feelings of belonging and trust), Values Alignment, and Ethical/Moral Distress. For many residents, it appears that these dimensions are being adversely affected. It is important to note that these same dimensions are linked to resident vitality. Residents’ experiences of disrespect, humiliation, and intimidation result in avoidance and a lack of psychological safety. As Leape proposes, “In such an environment, workers often do not feel safe about reporting an error—their own or that of another—because of fear of punishment.”34 Other dimensions measured in the survey, such as Mentoring, Self-Efficacy, Vitality, Gender Equity, and equity for Underrepresented in Medicine Minority residents—are also of broad interest to graduate medical education.

We anticipate that documenting the culture of residency training will assist in identifying areas for improvement and intervention, as well as in identifying programs that are “bright spots” deserving of imitation. Examples of questions that a survey of a residency program might answer include: To what degree do residents report trusting relationships and feel included? To what degree are residents’ personal values aligned with institutional values? To what degree do residents exhibit ethical and moral distress, and feel that they are adversely changed by working in that institution? Hoff and colleagues called for the creation of cultural “templates” that serve as blueprints for creating specific work environments for residents.2 The CRS provides such a template and a means of both stimulating action and monitoring progress in efforts to enhance the training environment and well-being of residents.