Query-by-example surgical activity detection

  • Yixin Gao
  • S. Swaroop Vedula
  • Gyusung I. Lee
  • Mija R. Lee
  • Sanjeev Khudanpur
  • Gregory D. Hager
Original Article

DOI: 10.1007/s11548-016-1386-3

Cite this article as:
Gao, Y., Vedula, S.S., Lee, G.I. et al. Int J CARS (2016) 11: 987. doi:10.1007/s11548-016-1386-3



Easy acquisition of surgical data opens many opportunities to automate skill evaluation and teaching. Current technology to search tool motion data for surgical activity segments of interest is limited by the need for manual pre-processing, which can be prohibitive at scale. We developed a content-based information retrieval method, query-by-example (QBE), to automatically detect activity segments within surgical data recordings of long duration that match a query.


The example segment of interest (query) and the surgical data recording (target trial) are time series of kinematics. Our approach includes an unsupervised feature learning module using a stacked denoising autoencoder (SDAE), two scoring modules based on asymmetric subsequence dynamic time warping (AS-DTW) and template matching, respectively, and a detection module. A distance matrix of the query against the trial is computed using the SDAE features, followed by AS-DTW combined with template scoring, to generate a ranked list of candidate subsequences (substrings). To evaluate the quality of the ranked list against the ground-truth, thresholding conventional DTW distances and bipartite matching are applied. We computed the recall, precision, F1-score, and a Jaccard index-based score on three experimental setups. We evaluated our QBE method using a suture throw maneuver as the query, on two tool motion datasets (JIGSAWS and MISTIC-SL) captured in a training laboratory.


We observed a recall of 93, 90 and 87 % and a precision of 93, 91, and 88 % with same surgeon same trial (SSST), same surgeon different trial (SSDT) and different surgeon (DS) experiment setups on JIGSAWS, and a recall of 87, 81 and 75 % and a precision of 72, 61, and 53 % with SSST, SSDT and DS experiment setups on MISTIC-SL, respectively.


We developed a novel, content-based information retrieval method to automatically detect multiple instances of an activity within long surgical recordings. Our method demonstrated adequate recall across different complexity datasets and experimental conditions.


Query-by-example Stacked denoising autoencoder Asymmetric subsequence dynamic time warping Surgical data indexing Surgical activity detection 

Funding information

Funder NameGrant NumberFunding Note
The Johns Hopkins Science of Learning Institute
    Department of Computer Science, The Johns Hopkins University

      Copyright information

      © CARS 2016

      Authors and Affiliations

      1. 1.Department of Computer Science, Whiting School of EngineeringThe Johns Hopkins UniversityBaltimoreUSA
      2. 2.Department of SurgeryJohns Hopkins University School of MedicineBaltimoreUSA
      3. 3.Department of Electrical and Computer Engineering, Whiting School of EngineeringThe Johns Hopkins UniversityBaltimoreUSA

      Personalised recommendations