Pairwise Comparison-Based Objective Score for Automated Skill Assessment of Segments in a Surgical Task
Current methods for manual evaluation of surgical skill yield a global score for the entire task. The global score does not inform surgical trainees about where in the task they need to improve. We developed and evaluated a framework to automatically generate an objective score for assessing skill in maneuvers (circumscribed segments) within a surgical task. We used an existing video and kinematic data set (with manual annotation for maneuvers) of a suturing and knot-tying task performed by 18 surgeons on a bench-top model using a da Vinci® Surgical System (Intuitive Surgical, Inc., CA). We collected crowd annotations of preferences, for which of the maneuver in a presented pair appeared to have been performed with greater skill and their confidence in the annotation. We trained a classifier to automatically predict preferences using quantitative metrics of time and motion. We generated an objective percentile score for skill assessment by comparing each maneuver sample to all remaining samples in the data set. Accuracy of the classifier for assigning a preference to pairs of maneuvers was at least 80.06% against a single individual (with a larger training data set) and at least 68.0% against each of the seven individuals (with a smaller training data set). Our reliability analyses indicate that automated preference annotations by the classifier are consistent with those by the seven individuals. Trial-level scores computed from maneuver-level scores generated using our framework were moderately correlated with global rating scores assigned by an experienced surgeon (Spearman correlation = 0.47; P-value < 0.0001).
Unable to display preview. Download preview PDF.
- 1.Wilson, E.B.: The evolution of robotic general surgery. Scandinavian Journal of Surgery 98, 125–129 (2009)Google Scholar
- 7.Cole, S.J., Mackenzie, H., Ha, J., Hanna, G.B., Miskovic, D.: Randomized controlled trial on the effect of coaching in simulated laparoscopic training. Surgical Endoscopy, 1–8 (2013)Google Scholar
- 8.Reiley, C.E., Hager, G.D.: Decomposition of Robotic Surgical Tasks: An Analysis of Subtasks and Their Correlation to Skill. In: Medical Image Computing and Computer-Assisted Intervention M2CAI Workshop (2009)Google Scholar
- 9.Ahmidi, N., Gao, Y., Béjar, B., Vedula, S.S., Khudanpur, S., Vidal, R., Hager, G.D.: String Motif-Based Description of Tool Motion for Detecting Skill and Gestures in Robotic Surgery. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013, Part I. LNCS, vol. 8149, pp. 26–33. Springer, Heidelberg (2013)CrossRefGoogle Scholar
- 10.Kumar, R., Rajan, P., Bejakovic, S., Seshamani, S., Mullin, G., Dassopoulos, T., Hager, G.: Learning disease severity for capsule endoscopy images. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 1314–1317 (2009)Google Scholar
- 11.Dwork, C., Kumar, R., Naor, M., Sivakumar, D.: Rank Aggregation Methods for the Web. In: Proceedings of the 10th International Conference on World Wide Web, pp. 613–622 (2001)Google Scholar
- 12.Yoav, F., Raj, I., Schapire Robert, E., Singer, Y.: An Efficient Boosting Algorithm for Combining Preferences. The Journal of Machine Learning Research 4, 933–969 (2013)Google Scholar
- 13.Kumar, R., Raghavan, P., Rajagopalan, S., Tomkins, A.: Recommendation Systems: A Probabilistic Analysis. In: Proc. IEEE Symp. on Foundations of Computer Science FOCS, pp. 664–673 (1998)Google Scholar
- 14.Curet, M., Dimaio, S.P., Gao, Y., Hager, G.D., Itkowitz, B., Jog, A.S., Kumar, R., Liu, M.: Method and system for analyzing a task trajectory. Patent, WO2012151585 A2 (2012)Google Scholar
- 18.Chen, C., White, L., Kowalewski, T., Aggarwal, R., Lintott, C., Comstock, B., Kuksenok, K., Aragon, C., Holst, D., Lendvay, T.: Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance. Journal of Surgical Research (2013)Google Scholar
- 19.Varadarajan, B.: Learning and inference algorithms for dynamical system models of dextrous motion. Ph.D. Thesis (2011)Google Scholar