Pairwise Comparison-Based Objective Score for Automated Skill Assessment of Segments in a Surgical Task

Abstract

Current methods for manual evaluation of surgical skill yield a global score for the entire task. The global score does not inform surgical trainees about where in the task they need to improve. We developed and evaluated a framework to automatically generate an objective score for assessing skill in maneuvers (circumscribed segments) within a surgical task. We used an existing video and kinematic data set (with manual annotation for maneuvers) of a suturing and knot-tying task performed by 18 surgeons on a bench-top model using a da Vinci^® Surgical System (Intuitive Surgical, Inc., CA). We collected crowd annotations of preferences, for which of the maneuver in a presented pair appeared to have been performed with greater skill and their confidence in the annotation. We trained a classifier to automatically predict preferences using quantitative metrics of time and motion. We generated an objective percentile score for skill assessment by comparing each maneuver sample to all remaining samples in the data set. Accuracy of the classifier for assigning a preference to pairs of maneuvers was at least 80.06% against a single individual (with a larger training data set) and at least 68.0% against each of the seven individuals (with a smaller training data set). Our reliability analyses indicate that automated preference annotations by the classifier are consistent with those by the seven individuals. Trial-level scores computed from maneuver-level scores generated using our framework were moderately correlated with global rating scores assigned by an experienced surgeon (Spearman correlation = 0.47; P-value < 0.0001).