Journal of General Internal Medicine

, Volume 29, Issue 6, pp 894–910

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

  • Rachel D. A. Havyer
  • Majken T. Wingo
  • Nneka I. Comfere
  • Darlene R. Nelson
  • Andrew J. Halvorsen
  • Furman S. McDonald
  • Darcy A. Reed
Review

DOI: 10.1007/s11606-013-2686-8

Cite this article as:
Havyer, R.D.A., Wingo, M.T., Comfere, N.I. et al. J GEN INTERN MED (2014) 29: 894. doi:10.1007/s11606-013-2686-8

ABSTRACT

OBJECTIVE

Valid teamwork assessment is imperative to determine physician competency and optimize patient outcomes. We systematically reviewed published instruments assessing teamwork in undergraduate, graduate, and continuing medical education in general internal medicine and all medical subspecialties.

DATA SOURCES

We searched MEDLINE, MEDLINE In-process, CINAHL and PsycINFO from January 1979 through October 2012, references of included articles, and abstracts from four professional meetings. Two content experts were queried for additional studies.

STUDY ELIGIBILITY

Included studies described quantitative tools measuring teamwork among medical students, residents, fellows, and practicing physicians on single or multi-professional (interprofessional) teams.

STUDY APPRAISAL AND SYNTHESIS METHODS

Instrument validity and study quality were extracted using established frameworks with existing validity evidence. Two authors independently abstracted 30 % of articles and agreement was calculated.

RESULTS

Of 12,922 citations, 178 articles describing 73 unique teamwork assessment tools met inclusion criteria. Interrater agreement was intraclass correlation coefficient 0.73 (95 % CI 0.63–0.81). Studies involved practicing physicians (142, 80 %), residents/fellows (70, 39 %), and medical students (11, 6 %). The majority (152, 85 %) assessed interprofessional teams. Studies were conducted in inpatient (77, 43 %), outpatient (42, 24 %), simulation (37, 21 %), and classroom (13, 7 %) settings. Validity evidence for the 73 tools included content (54, 74 %), internal structure (51, 70 %), relationships to other variables (25, 34 %), and response process (12, 16 %). Attitudes and opinions were the most frequently assessed outcomes. Relationships between teamwork scores and patient outcomes were directly examined for 13 (18 %) of tools. Scores from the Safety Attitudes Questionnaire and Team Climate Inventory have substantial validity evidence and have been associated with improved patient outcomes.

LIMITATIONS

Review is limited to quantitative assessments of teamwork in internal medicine.

CONCLUSIONS

There is strong validity evidence for several published tools assessing teamwork in internal medicine. However, few teamwork assessments have been directly linked to patient outcomes.

KEY WORDS

teamwork systematic review medical education interprofessional teams 

Effective teamwork among health professionals improves patient safety.1,2 A substantial proportion of preventable errors in United States hospitals are attributable to teamwork and communication failures.3,4 Recognizing this, the Institute of Medicine, the Joint Commission, the Agency for Healthcare Research and Quality (AHRQ), and others have made teamwork a top priority in their recommendations for improving healthcare.59

Teamwork is also prominently positioned within the American Board of Internal Medicine (ABIM) requirements for maintenance of certification for internists,10 as well as the Accreditation Council for Graduate Medical Education’s core competencies,11 milestones,12 and medical student competencies.13 As such, every physician at the undergraduate, graduate, and continuing professional level must demonstrate competency in teamwork.

While there is broad agreement on the imperative to improve teamwork, there is little consensus regarding how to measure it. Internal medicine teams vary substantially in composition, setting, function and charge. The knowledge, skills, and attitudes required for optimal teamwork within an inpatient medical team may differ from those necessary for successful interprofessional collaboration among undergraduate students in a classroom.13,14 Additionally, there are numerous purposes for teamwork assessment, including determining individual physician competence as well as measuring the effectiveness of teams as a whole.15

Given the heterogeneity of healthcare teams within internal medicine, it is logical that no single teamwork measurement tool will suit all clinical and educational situations. Yet, any endeavor to measure teamwork is likely to be most successful if it is grounded in the literature, built upon prior work, reliable and valid.15 Prior reviews have examined teamwork training and interventions, as well as the outcomes of effective teams.1,1624 These reviews have advanced the understanding of ‘what works’ to improve teamwork (i.e. curricula and interventions), but they do not fully answer the critical question of how teamwork is best measured in healthcare.

Therefore, the objective of this systematic review is to provide a synthesis of published instruments that have been used to assess teamwork in internal medicine. Given the breadth and marked heterogeneity of literature on teamwork assessment within healthcare as a whole, this review was limited to a synthesis of teamwork tools used in internal medicine. It encompasses all instruments used in undergraduate, graduate, and continuing medical education in general internal medicine and internal medicine subspecialties. To capture all published validity evidence for each tool, we also included articles from non-internal medicine specialties that reported additional validity evidence. This paper is intended to serve as a resource to help educators, clinicians, and other health professionals identify appropriate teamwork measurement tools to apply to their own internal medicine settings and teams.

METHODS

Although there are no standard reporting guidelines specific to systematic reviews of assessment tools, this review is reported according to applicable sections of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) standards25 and similar reviews of assessment tools in medical education.26

Data Sources and Search Strategy

We searched MEDLINE, MEDLINE In-process, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and PsycINFO for English language studies from January 1, 1979 through October 31, 2012. To identify studies related to teamwork, the term team was exploded to include all Medical Subject Headings (MeSH) and keywords containing “team” (e.g. team, teamwork, teamworking, team behavior, team climate, team culture, team collaboration, team effectiveness). Other MeSH terms and keywords related to teamwork included interprofessional relations, patient care team, cooperative behavior, crew resource, crisis resource and non-technical skills. These terms were combined with measurement terms including: scale, measure, inventory, questionnaire, tool, instrument, assessment, evaluation, profile, indicator, index and survey. Last, terms for teamwork and measurement were combined with terms pertaining to medical education and health professionals, including MeSH students, health occupations, health personnel, education professional, internship and residency, healthcare facilities, manpower and services, and exploded terms doctor, physician, nurse, student, intern, resident, registrar, house officer, medical, surgeon, operating, health, clinic, patient, interdiscliplinary, multidiscliplinary and interprofessional. An expert librarian with experience conducting literature searches for systematic reviews assisted in development and implementation of the search. The exact search strategies for each database are available from the authors.

The reference lists of all included articles were reviewed for additional studies. To identify in-press and unpublished studies, we searched scientific abstracts from national meetings between 2010 and 2012 of the Association of American Medical Colleges (AAMC), Association of Medical Educators of Europe (AMEE), Society of General Internal Medicine (SGIM), and the International Meeting on Simulation in Healthcare (IMSH). Authors of relevant abstracts were contacted for unpublished manuscripts. Finally, two experts who have published prior systematic reviews of teamwork or empiric studies of teamwork assessment reviewed the list of included articles to identify additional studies.

Study Selection

Articles were included if they were original research describing a quantitative tool designed for measuring teamwork within healthcare teams involving physicians and/or trainees in general internal medicine or an internal medicine subspecialty. To provide a comprehensive synthesis, we also included articles from non-internal medicine specialties that reported validity evidence for included tools. Studies of interprofessional teams (defined as two or more professions working together as a team)27,28 were included as long as internal medicine physicians (or medical students, residents, fellows) were one of the professions studied. Tools were considered measures of teamwork, based on authors’ descriptions of tools as measuring teamwork, collaboration, team process or function, team behavior, team effectiveness, team climate/environment, team culture, non-technical skills, or crew/crisis management. We excluded studies that measured just one specific aspect of team function, such as conflict, negotiation, leadership, communication, disruptive behavior and harassment. Studies of patient hand-over were excluded since recent reviews on this topic have been published.2931

Title and Abstract Review

The search yielded 12,922 citations (Fig. 1). Each title/abstract was reviewed and we erred on the side of full article retrieval if titles/abstracts were insufficient to determine eligibility. A total of 892 articles was included for full article review. All uncertainties regarding inclusion were resolved by consensus.
Figure 1

Article search and selection.

Data Extraction

Data were entered into a structured extraction form that included information on articles (study location, design, participants, setting) and tool characteristics (content, validity, and outcomes). Five authors extracted data. These authors met weekly during the study period and uncertainties were resolved by consensus. Thirty percent of articles were independently extracted by two authors to verify consistency in coding and determine interrater agreement using an intra-class correlation coefficient (ICC). The remaining 70 % of articles were extracted by a single reviewer.

We used an established framework to categorize the validity of instruments3234 that has been used in similar evaluations of assessment tools.26 This framework includes five categories of validity evidence: 1) content (the degree to which the tool content reflects the construct being measured); 2) response process (training of raters to use the tool); 3) internal structure (instrument reliability including internal consistency, interrater, intrarater, and test-retest reliability); 4) relationships to other variables (relationship between scores and other variables measuring the same construct); and 5) consequences (outcomes associated with tool scores). Kirkpatrick’s hierarchy was used to categorize outcomes as satisfaction/opinion, knowledge and skills, behaviors, and patient outcomes.35 Patient measures were recorded as outcomes only if the study reported a direct quantitative association between the teamwork assessment score and the patient outcome.

To evaluate the methodological quality of studies, we used criteria from the ten-item Medical Education Research Study Quality Instrument (MERSQI),36 which encompasses basic methodological components (e.g. study design, sampling, analysis). Validity evidence for the MERSQI includes content, interrater, intrarater, and internal consistency reliability, and relationships to other variables, including correlations between instrument items and journal impact factor, 3-year citation rate, and journal editors’ quality ratings,36 as well as predictive validity based on associations with editors’ decisions to accept or reject manuscripts for publication.37 We tallied the number of studies that fully, partially or failed to satisfy each of the ten quality criteria in the MERSQI.

Data Synthesis

Characteristics of studies and teamwork measurement tools were synthesized qualitatively and reported in evidence tables. Articles describing identical tools were grouped to enable examination and presentation of all validity evidence and outcomes for each unique tool. Frequencies and percentages were used to describe study and tool characteristics. Means and standard deviations were used to summarize quality scores. Meta-analysis was not possible nor logical, given that this was a review of assessment tools with obvious heterogeneity among instruments, study designs, and outcomes.

RESULTS

Of the 12, 922 citations, 12,629 were identified through the electronic database searches, 16 from reference lists of included articles, two from expert review, and 275 from relevant meeting abstracts. We identified 98 articles from non-internal medicine specialties that contained validity evidence for included tools. The total number of articles meeting inclusion criteria was 178 (Fig. 1). Interrater agreement for data extraction was very good (ICC = 0.73, 95 % CI: 0.63–0.81).

Table 1 shows the characteristics of the 178 included studies. Approximately half of studies were conducted in the U.S. and one-third in Europe. Most (142, 80 %) of studies included practicing physicians as participants, followed by residents (68, 38 %) and medical students (11, 6 %). The majority (152, 85 %) of studies also assessed non-physician professionals (e.g. nurses, pharmacists, mid-level providers, therapists, social workers, administrators) in interprofessional teams. Although most studies took place in actual inpatient or outpatient practice settings, 37 (21 %) of studies were simulation-based and 13 (7 %) took place in classrooms.
Table 1

Characteristics of 178 Studies Describing 73 Tools for Measuring Teamwork

Characteristics

No. (%)

Location*

 United States

101 (57)

 Europe

58 (33)

 Canada

14 (8)

 Australia

9 (5)

 Asia

1 (1)

 Other

8 (4)

Single/multi-institution

 Single institution

82 (46)

 Multi-institution

96 (54)

Setting*

 Inpatient

77 (43)

 Outpatient

42 (24)

 Operating room

23 (13)

 Emergency room

2 (1)

 Simulation

37 (21)

 Classroom

13 (7)

 Other

5 (3)

Level of participants*

 Medical students

11 (6)

 Residents

68 (38)

 Fellows

10 (6)

 Practicing physicians

142 (80)

Interprofessional team

 Single profession

26 (15)

 Interprofessional team

152 (85)

Study design

 Single group cross sectional

158 (89)

 Single group post-test only

0 (0)

 Single group pre-post test

0 (0)

 Prospective cohort

10 (6)

 Nonrandomized two-group experiment

6 (3)

 Randomized two-group experiment

10 (6)

* More than one location, setting, and level of participants may apply for each study

† Includes Egypt, Israel, New Zealand, Turkey

‡ A healthcare team including professions, in addition to physicians, such as pharmacists, nurses, nurse practitioners, physician assistants, certified nurse specialists, therapists, clinical assistants, administrators, secretaries

Study Quality

Figure 2 shows the proportion of studies satisfying the ten MERSQI quality criteria. Ten (6 %) studies fully satisfied, 59 (33 %) partially satisfied, and 109 (61 %) did not satisfy quality criteria for study design. The most frequent study design was single group cross-sectional (89 %). Ten studies were randomized controlled experiments.3847 A majority (153, 86 %) of studies fully satisfied at least one validity criterion: 122 (69 %) studies reported content validity, 115 (65 %) reported internal structure, and 47 (26 %) described relationships to other variables. Twenty-nine (16 %) studies fully satisfied all three of these validity criteria. Most studies (140, 79 %) relied on subjective assessments by study participants for measuring teamwork.
Figure 2

Methodological quality of 178 studies describing teamwork assessment tools.

Description and Validity Evidence for Teamwork Assessment Tools

The 178 included articles described 73 unique tools designed to measure teamwork (Table 2). Of the 73 tools, 15 (21 %) measured the teamwork of individuals working within teams, 43 (60 %) measured the teamwork of teams as a whole, and 15 (21 %) assessed both individuals and teams.
Table 2

Characteristics and Validity Evidence for 73 Teamwork Measurement Tools

Tool*

Setting

Specialty

Non-Physician Professions

Data

Assesses Individual or Team

Content Validity

Response Process

Internal Structure

Relation- ships to Other Variables

Outcomes

Tools used with Practicing Physicians, Residents/Fellows, and Medical Students

 Anesthetists’ Non-Technical Skills 4144,4757

Simulation NS; OR

Critical care, Anesthesia, Pediatrics, General surgery

Nurse, Technician

Objective

Both

Yes

Yes

Alpha 0.79–0.87 42,49,52,55; Interrater reliability 0.88–9253,56

Yes 43,48,50

Satisfaction/Opinion 48,56; Skill4144,4951,53,56,57

 Attitudes Toward Health Care Teams Scale5867

Classroom; Inpatient; Outpatient

General medicine,

Pain medicine, Geriatrics

Multi-disciplinary students; Nurse, Pharmacist, Social work, Other;

Subjective

Team

Yes

No

Factor analysis58,62,66Alpha 0.61–0.8 58,60,62,66,61,65

Test-retest reliability 0.7165

Yes65

Satisfaction/Opinion

 Jefferson Scale of Attitudes toward Physician-Nurse Collaboration6879

Classroom; Simulation NS; Inpatient; Outpatient; OR

Anesthesia, General surgery, Family medicine, General medicine

Nurse, Midlevel provider§, Nursing students

Subjective

Team

Yes

No

Factor analysis 74,76,77

Alpha 0.7–0.9 68,71,78,70,73,77,79

Interrater reliability 0.02–0.6470

Yes71,72,78

Satisfaction/Opinion

 TeamSTEPPS Team Assessment Questionnaire8082

Simulation, technology-assisted, OR, Inpatient,

General surgery, Anesthesia, Critical care, Ob/Gyn

Midlevel provider§, Nurse, Pharmacist, Technician

Subjective

Both

Yes

No

Alpha 0.9–0.9881,82

No

Satisfaction/Opinion; Behavior

Tools used with Practicing Physicians and Residents/Fellows

 Collaboration & Satisfaction about Care Decisions49,8387

Inpatient; Simulation, technology-assisted, OR

Anesthesia

Pediatrics, Critical care, General surgery

Nurse

Subjective

Team

Yes

No

Factor Analysis49,84, Alpha 0.90–0.9649,8386

No

Satisfaction/Opinion49,8387; Patient Outcome83

 Explicit Professional Oral Communication Measurement88

Inpatient

Critical care

Non-physicians NS

Objective

Individuals

No

Yes

No

No

None

 Group Environment Scale 8991

Inpatient; Outpatient

PM&R, IM specialty

Nurse, Therapist, Psychologist, Social work

Subjective

Team

Yes 89

No

Alpha 0.84–89

No

Satisfaction/Opinion 89,90,91

 ICU Nurse-Physician Questionnaire 9295

Simulation NS; Inpatient; ER

Critical care, EM

Nurse

Subjective

Team

Yes 9295

Yes 92

Factor analysis 95

Alpha 0.62–0.9 93,94,95

Yes92,93,95

Satisfaction/Opinion 9295

Behavior 93,95

 Intensity of Inter-professional Collaboration questionnaire96

Inpatient

IM specialty

Nurse

Subjective

Team

No

No

Alpha 0.81

Yes

Satisfaction/Opinion; Patient Outcome

 Interdisciplinary Collaboration Questionnaire97

Inpatient

Critical care

Nurse, Nursing students

Subjective

Team

Yes

No

Alpha > 0.7

Yes

Knowledge

 Management Attitudes Questionnaire, Operating Room or ICU98104

Simulation NS; Inpatient; OR

General surgery, Surgical specialty, Anesthesia Critical care

Nurse, Technician

Subjective

Individuals, Both 99,104

Yes 98,100104

No

Factor analysis 98,104;

Alpha 0.54–0.82 99101,103,104; Intrarater reliability 0.34 103;

Test-retest reliability 103

Yes 103

Satisfaction/Opinion 98102

 Non-technical Skills for Surgeons40,57,105109

Simulation, technology-assisted; OR

General surgery, Surgical specialty, IM specialty, Anesthesia

None

Objective

Individuals

Yes 40,105109

Yes 105,107109

Factor analysis108; Interrater reliability 0.68–0.94 40,107;

D-study coefficient 0.82 108

No

Satisfaction/Opinion 57,106,108,109 ; Skill 40,106108

 Organizational Culture Inventory 95,110

Inpatient; ER

Critical care, EM

Nurse, Other

Subjective

Team

Yes 95

No

Factor analysis 95,110

Yes110

Satisfaction/Opinion 95,110

 Organizational Management of the ICU Questionnaire111

Inpatient

Hospital Medicine

Nurse

Subjective

Team

Yes

No

Alpha 0.90–0.92

No

Patient Outcome

 Practice Environment Checklist- Short Form112

Outpatient

General medicine

Nurse, Midlevel provider§, Other

Subjective

Team

Yes

No

Factor analysis Alpha 0.8

No

Validity only

 SafeTeam Questionnaire88

Inpatient

Critical care

Non-physicians NS

Subjective

Both

Yes

No

No

No

None

 Safety Attitudes Questionnaire45,46,113137

Classroom; Simulation NS; Inpatient; Outpatient; OR

Pediatrics, Anesthesia Ob/Gyn, General surgery, Surgical specialty, General medicine, Critical care

Nurse, Midlevel provider§ Technician, Therapist, Pharmacist, Social work, Other

Subjective

Team

Yes 46,113,114,116,118,122,123,125,127,129132,134,135

Yes115

Factor analysis 46,113,123,127,129,134;

Alpha 0.54–0.9 116,119,127,123,129,134,117 Interrater reliability 0.78–0.86 129,123,134

Yes127,134,136,123,136

Satisfaction/Opinion45,46,113,114,117,120123,125130,132137; Behavior91,97,106; Skill 131; Patient Outcome116118

 Simulation Team Assessment Tool138

Simulation, technology-assisted

Critical care, Pediatrics

None

Objective

Team

Yes

Yes

Inter-rater reliability 0.81

No

Skill

 SPRinT Questionnaire139

Simulation, technology-assisted

Anesthesia, Critical care, General surgery, Surgery specialty

Nurse, Non-physicians NS

Subjective

Individuals

No

No

No

No

Satisfaction/Opinion

 Teamwork Behavioral Rater39

Simulation, technology-assisted

Critical care

Nurse

Objective

Team

Yes

No

No

No

Skill

 Unnamed140

Inpatient

General medicine, IM specialty

Nurse

Subjective

Team

No

No

No

No

None

 Unnamed141

Inpatient

IM specialty,

Nurse, Therapist, Other

Objective

Individuals

No

No

Factor analysis Alpha 0.9

No

Patient Outcome

 Unnamed142

Inpatient

IM specialty

Nurse, Therapist

Subjective

Team

No

No

No

No

None

 Unnamed143

Classroom

General medicine

None

Objective

Team

No

No

No

No

Knowledge

 Unnamed143

Classroom

General medicine

None

Subjective

Team

No

No

No

No

Satisfaction/Opinion

 Unnamed144

Simulation, technology-assisted

Critical care

Nurse

Objective

Team

Yes

Yes

Factor analysis Alpha 0.89

No

Skill

 Unnamed88

Inpatient

Critical care

Non-physicians NS

Subjective

Team

Yes

No

No

No

None

 Unnamed145

Outpatient

General medicine

Nurse, Therapist, Pharmacist, Psychologist, Social work, Optometrist

Subjective

Team

Yes

No

Factor analysis

No

Satisfaction/Opinion; Behavior

Tools used with Practicing Physicians Only

 Collaboration Skills Simulation Vignette Test92

Simulation, technology-assisted

Critical care

Nurse

Objective

Individuals

Yes

No

Interrater reliability 0.82

Yes

Skill

 Collaboration with Medical Staff Scale146

Inpatient

Critical care

Nurse

Subjective

Team

Yes

No

Alpha 0.8

No

None

 Collaborative Behavior Scale147

Inpatient

Critical care

Nurse

Subjective

Team

No

No

Alpha 0.97

No

Behavior

 Collaborative Practice Scale147152

Inpatient; Outpatient

Critical care, Not specified

Nurse, Mid-level provider§

Subjective

Team

No

No

Factor analysis147; Alpha 0.80–0.83147,152

Yes152

Satisfaction/Opinion

 Doctors’ Opinions on Collaboration153

Outpatient

Multi-specialty

None

Subjective

Individuals

Yes

No

Factor analysis Alpha 0.64–0.9

No

Validity only

 Group Behavior Inventory154

Inpatient

IM specialty, Surgical specialty, Radiology, Pathology

Nurse

Subjective

Team

No

No

No

No

None

 Group Development Questionnaire155

Inpatient

Critical care

Nurse, Other

Subjective

Team

No

No

No

No

Patient Outcome

 Healthcare Team Vitality Instrument156

Inpatient

Hospital medicine

Nurse, Non-physicians NS

Subjective

Both

Yes

No

Factor analysis

Yes

Satisfaction/Opinion

 Human Factors Attitude Survey157

Classroom

General medicine, IM specialty, Pediatrics, General surgery, EM

Nurse, Technician, Other

Subjective

Both

No

No

Alpha 0.8

No

Satisfaction/Opinion

 Inter-professional Collaborative Learning Series158

Inpatient

General medicine

Nurse, Therapist, Pharmacist, Non-physicians NS

Subjective

Both

Yes

No

No

No

Satisfaction/Opinion

 Leader Communication Self-Report of Collaboration92

Inpatient

Critical care

Nurse

Subjective

Both

Yes

No

Alpha 0.7–0.9

No

Satisfaction/Opinion

 Multi-disciplinary Collaboration Instrument159

Inpatient

Hospital medicine

Nurse, Therapist, Pharmacist, Other

Subjective

Team

Yes

No

Alpha 0.8

Yes

Validity only

 Multi-disciplinary Team Performance Assessment Tool160,161

Inpatient; Outpatient

IM specialty, General surgery, Radiology, Pathology

Nurse, Midlevel provider§, Other

Objective

Both

Yes

Yes160

Interrater reliability ≥ 0.7161

Yes161

Validity only

 Nursing Home Survey on Patient Safety Culture162

Outpatient

Geriatrics

Midlevel provider, Nurse, Therapist, AdministratorSocial work

Subjective

Both

No

No

No

No

Satisfaction/Opinion

 Observational Skill-based Clinical Assessment Tool for Resuscitation163

Simulation, technology-assisted

General medicine

Nurse, Therapist

Objective

Individuals

Yes

No

Factor analysis Alpha 0.74–0.96

Interrater reliability 0.65–0.91

No

Validity only

 Physician/ Pharmacist Collaboration Index 38,164170

Outpatient; Inpatient

General medicine, Pediatrics, Family Medicine, Ob/Gyn, Critical care

Pharmacist

Subjective

Team

Yes164,165,169

No

Factor analysis 164,165;

Alpha 0.70–0.90164,165,168,169;

Interrater reliability 0.89167

Yes164,165,167

Satisfaction/Opinion164168; Behavior167,169; Patient Outcome38

 Practice Climate Scale for Practitioners171

Outpatient

IM specialty

Midlevel provider§, Nurse, Other

Subjective

Both

No

No

Alpha 0.90

Yes

Patient Outcome

 Team Climate Inventory172191

Outpatient; Inpatient

Multi-specialty

Nurse, Midlevel provider§, Therapist, Pharmacist, Social work, Other

Subjective

Team

Yes

No

Alpha 0.80–0.90 174,177,180,184186,189,190,192 Interrater reliability 0.60– 0.98175,174,182,189,190,187

Yes172,174,176,178,182,186,187,189,190,192

Satisfaction/Opinion173,174,176183,186189,192; Knowledge 175; Behavior175,176,189; Patient Outcome172175,185,187

 Team Emergency Assessment Measure193

Inpatient, technology-assisted simulation

General medicine, EM

Nursing students

Objective

Both

Yes

No

Factor analysis Alpha 0.89–0.90

Interrater reliability 0.53–0.80

No

Validity only

 Team Functioning Assessment Tool194

Inpatient

Hospital medicine

Non-physicians NS

Objective

Individuals

Yes

Yes

Interrater reliability 0.67–0.99

No

Validity only

 Team Observation Scale 195,196

Outpatient

Geriatrics, Hospice

Nurse, Social work, Other

Objective

Team

Yes 195

Yes 195

No

No

Skill 195

Behavior 196

 TeamSTEPPSTeamwork Attitudes Questionnaire197

Classroom

Critical care, EM

Nurse

Subjective

Both

Yes

No

Alpha 0.70

No

Validity only

 Teamwork Evaluation of Non-technical skills198

Inpatient

Pediatrics, General surgery, Critical care

Nurse, Therapist

Objective

Team

Yes

No

Interrater reliability 0.96

Yes

Behavior; Patient Outcome

 Teamwork Scale199

Inpatient

Hospital medicine

Non-physicians NS

Subjective

Team

Yes

No

Alpha 0.80

Yes

Satisfaction/Opinion

 Trainee Test of Team Dynamics200

Outpatient simulation, SP simulation

Geriatrics

Midlevel provider§, Pharmacist, Social work

Objective

Team

Yes

No

Factor analysis

No

Validity only

 Working as a Team201

Outpatient

General medicine, IM specialty, Surgical specialty

Non-physicians NS

Subjective

Individuals

Yes

No

Factor analysis Alpha 0.71–0.90

No

Behavior

 Unnamed165

Outpatient

General medicine, IM specialty, Family Medicine, Ob/Gyn

Pharmacist

Subjective

Team

Yes

No

Alpha 0.90

Yes

Satisfaction/Opinion

 Unnamed202

Inpatient

IM specialty

Nurse, Therapist

Subjective

Both

Yes

No

No

No

Patient Outcome

 Unnamed203

Inpatient

General medicine, General surgery

Nurse, Other

Subjective

Team

Yes

No

No

No

Satisfaction/Opinion

 Unnamed204

Simulation, technology-assisted

General medicine, General surgery

Midlevel provider§, Nurse, Technician

Subjective

Individuals

Yes

No

Factor analysis

Alpha 0.94

No

Satisfaction/Opinion

 Unnamed205

Outpatient

General medicine

Midlevel provider§, Nurse, Other

Subjective

Team

No

No

Alpha 0.73–0.80

Interrater reliability > 0.4

No

Patient Outcome

 Unnamed206

Inpatient

General medicine, General surgery, Critical care, Anesthesia

Nurse, Other

Subjective

Team

Yes

No

Factor analysis

No

Satisfaction/Opinion

 Unnamed207

Inpatient

General medicine

Nurse, Pharmacist

Objective

Both

No

No

No

No

Satisfaction/Opinion; Patient Outcome

 Unnamed208

Inpatient

General medicine, General surgery, Critical care

Nurse, Pharmacist, Clerical

Subjective

Team

Yes

No

No

No

Satisfaction/Opinion

Tools used with Residents/Fellows and Medical Students

 Team Skills Scale58,209

Classroom

General medicine58

Students of Nursing, Pharmacy, Social work, Other

Subjective

Team

Yes

No

Factor analysis58; Alpha 0.9 58

No

Skill58,209

Tools used with Residents/Fellows Only

 Communication, Collaboration and Critical Thinking for Quality Patient Outcomes Survey79

Inpatient

General medicine

Nurse

Subjective

Team

No

No

Alpha 0.9

No

Satisfaction/Opinion

 Decision About Transfer210

Inpatient

Critical care

Nurse

Subjective

Team

Yes

No

No

Yes

Patient Outcome

 Geriatric Interdisciplinary Team Training Videotape Test of Team Dynamics58

Classroom

General medicine

Students of Nursing, Social work

Objective

Team

Yes

No

No

No

Knowledge

 Ottawa Crisis Resource Management Checklist211

Simulation NS

Multi- specialty

Nurse

Objective

Individuals

Yes

Yes

Interrater reliability 0.63

Yes

Skill

 Ottawa Global Rating Scale 48,211,212

Simulation NS

Multi-specialty

None

Objective

Individuals

Yes 48,211

Yes 211,212

Interrater reliability 0.49–0.8548,211,212

Yes 211

Satisfaction/Opinion 48; Skill 211,212

 Resident Leadership Scale213

Inpatient

General medicine

None

Objective

Individuals

Yes

No

Factor analysis Alpha 0.90

Yes

Skill

 Team Diagnostic Survey214

Inpatient

Critical care

Nurse

Subjective

Team

Yes

No

Factor analysis Alpha 0.72–0.80

No

Satisfaction/Opinion

 Unnamed214

Inpatient

Critical care

Nurse

Subjective

Team

Yes

No

Factor analysis Alpha 0.77–0.80

Yes

Satisfaction/Opinion

 Unnamed114

Inpatient

General medicine

Nurse

Subjective

Individuals

Yes

No

No

No

Satisfaction/Opinion

Tools used with Medical Students Only

 Situation Awareness Global Assessment Technique215

Simulation, Technology-assisted

Critical care

None

Objective

Individuals

No

No

No

No

Knowledge

Abbreviations: ER Emergency Room; EM Emergency Medicine; IM Internal Medicine; NS Not specified; Ob/Gyn Obstetrics and Gynecology; OR Operating Room; PM&R Physical Medicine and Rehabilitation; SP Standardized patient

* Unnamed if tool not named in the study

† May include Students of Nursing, Mid-level provider, Pharmacy, Therapy, Social work, Allied health, Dentistry, Public health, Podiatry, Chiropracty

‡Other may include Chaplain, Clerical, Administration, Dietitian, Clerical, Psychologist, Non-physicians NS

§ Midlevel provider may include Nurse practitioner, Physician assistant, Certified Register Nurse Anesthetist, or Clinical Nurse Specialist

▪ Includes Family Medicine, General medicine, IM specialty, General surgery, Ob/Gyn, Psychiatry, Pathology, Radiology, Not specified

¶ Includes Pediatrics, General medicine, General surgery, Anesthesia, EM, Family medicine, Ob/Gyn, Otolaryngology, Ophthalmology, Orthopedics, Neurology, Psychiatry

Content validity was demonstrated for 54 (74 %) of tools (Table 2) and generally consisted of developing instrument content from expert panels, existing instruments, and literature review. The TeamSTEPPS Teamwork Attitudes Questionnaire80 is an example of an assessment tool with strong content validity designed to assess the teamwork attitudes, knowledge and skills of learners participating in the TeamSTEPPS curriculum. TeamSTEPPS is a training program developed by the United States Department of Defense and the AHRQ that encompasses leadership, situation monitoring, mutual support and communication.7,80,198 The TeamSTEPPS Teamwork Perception Questionnaire is a second instrument associated with this curriculum that measures individuals’ perceptions of organizational teamwork.216

Few tools (12, 16 %) reported response process, which included training raters to correctly use tools. The Multi-disciplinary Team Performance Assessment Tool160,161 is an observational teamwork assessment of cancer teams modified from an established teamwork assessment tool in the surgical literature (Observational Teamwork Assessment in Surgery).217219 Assessors were trained in the use of this tool by an expert psychologist with experience using the tool.160

Reliability of tools was demonstrated by internal consistency (38, 52 %), interrater reliability (16, 22 %), intrarater reliability (1, 1 %), and test-retest reliability (2, 3 %). Reliability estimates for most tools were very good (> 0.7).220 The Physician/Pharmacist Collaboration Index is an example of a tool assessing interactions between internists and pharmacists that has extensive reliability evidence, including factor analysis, internal consistency (Crohnbach alpha 0.70-0.90) and interrater reliability (ICC 0.89).38,164170 This tool measures the pharmacist’s view of collaboration among physicians and other health professionals in both inpatient and outpatient settings.

Relationships between teamwork scores and other variables reflecting the construct of teamwork were reported for 25 (34 %) tools (Table 2). Studies varied widely with regard to the specific variables reported. The Attitudes Toward Health Care Teams Scale (ATHCTS) has been used in ten studies measuring attitudes towards interprofessional collaboration in a variety of settings, most commonly interprofessional education.5867 It consists of three subscales assessing attitudes about team value, team efficiency and the physician’s shared role on the team. ATHCTS scores have been shown to correlate with other measures of team process.65 The Ottawa Global Rating Scale has been used in multi-specialty education as an objective measurement of an individual’s crisis resource management skills in simulated scenarios.211,212 This tool has been shown to differentiate among residents’ level of training when applied in simulated medical crisis scenarios.211

Consequences validity refers to the outcomes associated with scores from teamwork tools. For many tools (35, 48 %), outcomes included satisfaction or opinion of participants (Table 2). Twelve (16 %) tools measured participants’ teamwork skills. Teamwork skills such as leadership, communication and crisis management were assessed through simulation;41,138,209 however, other tools involved direct observation of skills in actual practice settings, such as medical residents’ abilities to lead ward teams213 and palliative care physicians’ communication in team meetings.195 Behaviors of students, residents/fellows, or practicing physicians were reported outcomes for ten (14 %) tools.

Teamwork Tools Associated with Patient Outcomes

Relationships between teamwork scores and patient outcomes have been directly examined for 13 (18 %) of teamwork tools (Table 3). Teamwork tools by Baggs83,210 and Wheelan155 show inverse relationships between positive teamwork and mortality rates.
Table 3

Relationships Between Scores from Teamwork Measurement Tools (n = 13) and Patient Outcomes

Tool*

Participants

Patient Outcomes

Collaboration & Satisfaction about Care Decisions83

Residents, Practicing physicians, Nurse

Nurses’ ratings of collaboration negatively associated with patient death or readmission to the Intensive Care Unit (p = 0.037).

Decision About Transfer210

Residents, Nurse

Nurse ratings of collaboration negatively associated with patient death or readmission to the Intensive Care Unit (p = 0.02). Resident-reported collaboration was not associated with death or readmission.

Group Development Questionnaire155

Practicing physicians, Nurse, Other

Group development (teamwork) inversely associated with mortality rate (r = −0.662, p = 0.004).

Intensity of Interprofessional Collaboration Questionnaire96

Residents, Practicing physicians, Nurse

Patients cared for by teams with high intensity collaboration had higher mean patient satisfaction (0.501, 95 % CI 0.286-0.715) and lower mean uncertainty scores (0.138, 95 % CI 0.002–0.275) compared to teams with low intensity collaboration. High intensity collaboration teams demonstrated better pain management (92.6 %, 95 % CI 87.9–97.3 %) compared to low intensity collaboration teams (82.7 %, 95 % CI 76.3–89.2 %). There were no significant differences in length of hospital stay among teams with high versus low intensity collaboration

Organizational Management of the ICU Questionnaire111

Residents, Practicing physicians, Nurse

Perceptions of increased nurse–physician collaboration were associated with increased length of stay (p < 0.001).

Physician/ Pharmacist Collaboration Index38

Practicing physicians, Pharmacist

Teamwork scores inversely correlated with blood pressure (Spearman = −0.153).

Practice Climate Scale for Practitioners171

Practicing physicians, Midlevel provider, Nurse, Other

Positive perceptions of teamwork practice climate were positively associated with patient ratings of trust in primary care physicians (p = 0.04).

Safety Attitudes Questionnaire (SAQ)116118

Residents, Fellows, Practicing physicians, Nurse, Non-physicians NS

SAQ teamwork and communication scores were inversely correlated with risk-adjusted morbidity (defined as patient having one or more of 21 specific postoperative complications up to 30 days after surgery), r = −0.38, p < 0.0.116

Improvement in SAQ score correlated with lower postoperative complication rate (r = 0.7143, p = 0.0381).117

No association between SAQ scores and hospital reported patent safety events.118

Team Climate Inventory 172175,185,187

Practicing physicians, Midlevel,

Nurse, Therapist, Pharmacist, Clerical, Non-physicians NS

Team Climate scores were positively associated with overall patient satisfaction (regression coefficient 1.35, 95 % CI: 0.43–2.26, p = 0.005) and higher quality diabetes management (2.13, 95 % CI: 0.20–4.05, p = 0.031).172

Team Climate scores were positively associated with patients’ experience of improved patient care (p < 0.02).175

Team Climate scores in the participation subscale were positively associated with patients’ perceptions of continuity [regression coefficient 3.72 (95 % CI 0.56,6.87, p = 0.02)].173

Patient satisfaction increased by 0.042 (95 % CI = 0.047–0.129, p = 0.014) for an increase of one standard deviation in overall Team Climate score.174

No relationships between overall Team Climate score and diabetes care or overall patient satisfaction.174

No relationship between Team Climate score and prevalence of pressure ulcers (OR 1.00, 95 % CI 0.50–2.02).185

Unnamed141

Residents, Practicing physicians, Nurse, Therapist, Other

Physician relational coordination was inversely associated with length of stay (r = −0.46, p < 0.01) and log total costs per stay (r = − 0.08, p = 0.03).

Unnamed207

Practicing physicians, Nurse, Pharmacist

Improved teamwork was associated with patients’ ratings of satisfaction with physicians treating them with respect and nurses listening to them (p = 0.001 and 0.0003, respectively)

Unnamed202

Practicing physicians, Nurse, Therapist

Team organization and task orientation were positively associated with improvement in motor function among stroke rehabilitation patients (both p < 0.05).

Higher team effectiveness was associated with shorter length of stay among stroke rehabilitation patients (p = 0.02)

No association between teamwork scores and patient discharge destination.

Unnamed205

Practicing physicians, Midlevel provider, Nurse, Other

Patient physical function was higher among patients impanelled to teams with higher teamwork scores (p = 0.05)

No association between teamwork scores and patient physical and mental quality of life scores.

Abbreviations: IM Internal Medicine; NS Not specified

* Unnamed if tool not named in the study

† Other may include Assistant, Case manager, Clerical, Non-physicians NS

‡ Midlevel provider may include Nurse practitioner, Physician assistant, Certified Register Nurse Anesthetist, or Clinical Nurse Specialist

Of the tools shown to correlate with patient outcomes, the Safety Attitudes Questionnaire (SAQ)113 has the strongest validity evidence, and has been adapted for use across multiple settings and learner levels. The SAQ contains six domains, one of which is teamwork. Twenty-seven studies have reported validity evidence for the SAQ.45,46,113137 SAQ scores have been correlated with reduced postoperative complications;116,117 however, studies have not shown associations between the SAQ and mortality or patient safety events.116,118

The Team Climate Inventory (TCI) has been used to assess teamwork among inpatient and outpatient interprofessional teams in 21 studies.172192 The TCI has four subscales: vision, participative safety, task orientation, and support for innovation.190 A study by Bower et al. found that ratings on the Team Climate Inventory (TCI) were associated with better diabetes care,172 while another study showed no relationship between the TCI and diabetes management.173

The Intensity of Interprofessional Collaboration Questionnaire is an instrument that measures the nurse–physician collaboration in the inpatient setting. Patients cared for by teams with high intensity collaboration on this scale reported higher satisfaction, lower uncertainty, and better pain management.96 However, there was no relationship between collaboration and patient length of stay.96

DISCUSSION

Assessing teamwork is imperative for determining physician competency11,13,221 and ensuring patient safety.3,5 Valid and reliable measurement of teamwork is necessary to understand connections between teamwork and patient safety, and to maximize gains achieved through teamwork education.

Together, the 178 studies and 73 teamwork tools summarized in this review constitute a resource for internists who wish to apply teamwork assessment tools to their local settings and teams. Although there is considerable validity evidence for many of these teamwork tools, most assessments consisted of participants’ subjective reports of satisfaction, attitude, or opinion. A thorough understanding of attitudes is prerequisite to improving teamwork; yet, tools examining teamwork behaviors in actual practice provide scores that may be more readily linked to important patient safety outcomes.83,155,202 Unfortunately, these assessments often require extensive rater training to achieve adequate reliability,105 which can be time consuming and costly. Implementing existing tools, rather than creating new ones, should reduce the cost of tool development so that these resources can be allocated to rater training and implementation. Furthermore, the trustworthiness of validity information depends upon the methodological quality of studies from which it is derived. Based on MERSQI criteria, further studies should aim to improve rigor of study design and outcome assessment.

Evidence suggests that teamwork training should improve patient safety,1,2 yet our review indicates that most studies examining teamwork in internal medicine do not directly link teamwork measures to reported patient outcomes. Several studies in this review described concurrent changes in patient outcomes and teamwork scores (e.g. pre/post teamwork training), but did not actually examine relationships between outcomes and teamwork scores, thus making it difficult to attribute gains in patient safety to teamwork improvements. To advance the understanding of how to improve safety through collaboration, future studies should not only apply valid teamwork assessments, but should directly examine relationships between these assessments and patient outcomes. Robust teamwork assessments and appropriate conceptual frameworks are essential to meaningful evaluations of relationships between teamwork and patient outcomes.

The majority of teamwork tools in this review were applied to groups of individuals working together to achieve a common goal within traditional team structures (e.g. physically side by side/face to face).23,222 However, the concept of ‘team’ in healthcare is rapidly evolving to include a greater emphasis on interprofessional collaboration,223 as well as new team structures. With the advent of restricted duty hours,224 and frequent hand-offs,29,30 team members are often working in shifts225227 and are also becoming dispersed geographically. The telemedicine intensive care unit is an example in which intensivists and nurses use telemetry and electronic medical records to provide care to patients hospitalized remotely.228 Teams dispersed over distance and/or time face unique teamwork challenges119 that may require new or adapted assessment tools.

There are limitations to this review. First, although our search was comprehensive, we may have failed to capture some non-indexed or unpublished studies. We attempted to limit this possibility by reviewing abstracts from four professional meetings that are likely to include teamwork content, reviewing reference lists of included articles, and by having two content experts examine our reference list. Also, our electronic search included terms such as “registrar” that should have helped capture studies across countries. Second, to make the scope of the review manageable, it was limited to tools published in the field of internal medicine. However, some validity evidence was obtained from studies conducted in other specialties such as surgery and anesthesia. Validity is not a property of an instrument itself; rather it is a property of inferences derived from implementation of the instrument within specific contexts.34 As such, the setting in which tools are applied influences the validity information acquired. When selecting a tool for use in a new setting, it is important to consider the degree to which existing validity evidence may apply to the new context.

Third, this review included only quantitative measurement tools; however, qualitative studies provide valuable frameworks for understanding team behaviors and processes28,229,230 that are essential to the development of meaningful assessment tools. A synthesis of findings from the qualitative literature on teamwork would be a useful next step. Fourth, although we used an extremely broad definition and search strategy for teamwork, we excluded studies that examined just one specific element of interpersonal interaction, such as disruptive behavior and harassment. These behaviors alone do not constitute teamwork; however, they certainly may influence team interactions.231233 Finally, we used established frameworks for abstracting tool validity32,33 and study quality;36 however, these frameworks do not encompass every aspect of validity and/or quality present in studies.

In conclusion, this systematic review provides a synthesis of teamwork assessment tools in internal medicine that may serve as a resource for educators who wish to assess teamwork for various learner levels and settings. Valid teamwork assessment is essential to determine physician competency and to ensure patient safety. Future research should expand the validity evidence for existing tools and further explore relationships between teamwork assessment and important patient safety outcomes.

Acknowledgements

Contributors

Dr. Paul Haidet of Pennsylvania State Hershey College of Medicine and Dr. Chayan Chakraborti, MD FACP of Tulane University School of Medicine conducted expert reviews of the reference list. Karen Larsen of Mayo Clinic assisted with the electronic searches. Kathy Thompson of Mayo Clinic assisted with article retrieval.

Funding /Support

This study was supported in part by the Mayo Clinic Program in Professionalism and Ethics and the Mayo Clinic-Rochester Internal Medicine Residency Office of Educational Innovations as part of the ACGME Educational Innovations Project.

Role of the Sponsor

The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

Prior Presentations

This review was presented in an oral session at the Society of General Internal Medicine Annual Meeting in Denver, Colorado in April 2013.

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Copyright information

© Society of General Internal Medicine 2013

Authors and Affiliations

  • Rachel D. A. Havyer
    • 1
  • Majken T. Wingo
    • 1
  • Nneka I. Comfere
    • 2
  • Darlene R. Nelson
    • 1
  • Andrew J. Halvorsen
    • 1
  • Furman S. McDonald
    • 1
  • Darcy A. Reed
    • 1
  1. 1.Department of MedicineMayo ClinicRochesterUSA
  2. 2.Department of DermatologyMayo ClinicRochesterUSA

Personalised recommendations