A systematic review of performance assessment tools for laparoscopic cholecystectomy

Abstract

Background

Multiple tools are available to assess clinical performance of laparoscopic cholecystectomy (LC), but there are no guidelines on how best to implement and interpret them in educational settings. The purpose of this systematic review was to identify and critically appraise LC assessment tools and their measurement properties, in order to make recommendations for their implementation in surgical training.

Methods

A systematic search (1989–2013) was conducted in MEDLINE, Embase, Scopus, Cochrane, and grey literature sources. Evidence for validity (content, response process, internal structure, relations to other variables, and consequences) and the conditions in which the evidence was obtained were evaluated.

Results

A total of 54 articles were included for qualitative synthesis. Fifteen technical skills and two non-technical skills assessment tools were identified. The 17 tools were used for either: recorded procedures (nine tools, 60 %), direct observation (five tools, 30 %), or both (three tools, 18 %). Fourteen (82 %) tools reported inter-rater reliability and one reported a Generalizability Theory coefficient. Nine (53 %) had evidence for validity based on clinical experience and 11 (65 %) compared scores to other assessments. Consequences of scores, educational impact, applications to residency training, and how raters were trained were not clearly reported. No studies mentioned cost.

Conclusions

The most commonly reported validity evidence was inter-rater reliability and relationships to other known variables. Consequences of assessments and rater training were not clearly reported. These data and the evidence for validity should be taken into consideration when deciding how to select and implement a tool to assess performance of LC, and especially how to interpret the results.

Appendix: Search strategy of MEDLINE

1	exp cholecystectomy/or cholecystectomy, laparoscopic/
2	exp professional competence/or exp clinical competence/
3	(Task Performance and Analysis).mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
4	exp study characteristics/or exp evaluation studies/
5	(Internship and Residency).mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
6	1 and 2 and 3
7	exp Curriculum/ed, mt, sn [Education, Methods, Statistics & Numerical Data]
8	exp validation studies/
9	exp Laparoscopy/ed, mt, st [Education, Methods, Standards]
10	1 and 2 and 3 and 8
11	1 and 2 and 8
12	limit 1 to systematic reviews
13	2 and 12
14	4 and 8
15	1 and 2 and 14
16	3 and 15
17	exp self concept/or self-assessment/
18	exp methods/or exp observation/or exp research design/
19	(decision making and clinical competence$ and skill$).ab.
20	1 and 2 and 18 and 19
21	Educational Measurement/and “Internship and Residency”/
22	1 and 2 and 21
23	1 and 9 and 17
24	1 and 2 and 5
25	evaluation studies/or evaluation studies as topic/or program evaluation/or validation studies as topic/or Intervention Studies/or (effectiveness or (pre- adj5 post-)).ti,ab. or (program* adj3 evaluat*).ti,ab. or intervention*.ti,ab.
26	1 and 2 and 25
27	exp Clinical Trial/or double-blind method/or (clinical trial* or randomized controlled trial or multicenter study).pt. or exp Clinical Trials as Topic/or ((randomi?ed adj7 trial*) or (controlled adj3 trial*) or (clinical adj2 trial*) or ((single or doubl* or tripl* or treb*) and (blind* or mask*))).ti,ab.
28	1 and 2 and 27
29	(exp methods/or exp observation/or exp research design/) and #1.mp. and #9.mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
30	1 and 17 and 29
31	17 not patients.mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
32	1 and 9 and 31
33	(Task Performance and Analysis).tw
34	1 and 2 and 33
35	1 and 2 and 4 and 33
36	1 and 2 and 4 and 5
37	(self concept or self-assessment).tw
38	((self concept or self-assessment) not patients).tw
39	1 and 2 and 4 and 5 and 38
40	1 and 2 and 38
41	1 and 33 and 38
42	1 and 2 and 5 and 7
43	curriculum.tw
44	1 and 5 and 43 (
45	validation studies.tw
46	1 and 2 and 29 and 45
47	1 and 5 and 18
48	1 and 5 and 25
49	1 and 5 and 27
50	1 and 17 and 25
51	1 and 21 and 25
52	1 and 5 and 9 and 21 and 25
53	decision$making.tw
54	(Educational adj2 assessment).tw
55	(General surgery adj2 training).mp
56	(objective adj2 assessment).mp
57	Non$technical skill$.mp
58	((performance adj2 assessment) or (performance adj2 evaluation)).tw
59	(surgical adj2 assessment tool$).mp
60	(surgical adj2 skill$).mp
61	Technical error$.tw
62	(Resident adj2 evaluation).tw
63	(simulator adj2 training).tw
64	((mental adj2 training) and (mental adj2 practice)).tw
65	(motor adj2 skill$).tw
66	(Intraoperative adj2 performance).tw
67	human error$.tw
68	direct observation.tw
69	(acquisition adj2 skil$).tw
70	or/53–69
71	1 and 70
72	feedback.tw
73	expert testimony.tw
74	Confidence Intervals.tw
75	video recording.tw
76	operating rooms.tw
77	simulation.tw
78	or/72–77
79	1 and 70 and 78