Against spatial–temporal discrepancy: contrastive learning-based network for surgical workflow recognition

Abstract

Purpose

Automatic workflow recognition from surgical videos is fundamental and significant for developing context-aware systems in modern operating rooms. Although many approaches have been proposed to tackle challenges in this complex task, there are still many problems such as the fine-grained characteristics and spatial–temporal discrepancies in surgical videos.

Methods

We propose a contrastive learning-based convolutional recurrent network with multi-level prediction to tackle these problems. Specifically, split-attention blocks are employed to extract spatial features. Through a mapping function in the step-phase branch, the current workflow can be predicted on two mutual-boosting levels. Furthermore, a contrastive branch is introduced to learn the spatial–temporal features that eliminate irrelevant changes in the environment.

Results

We evaluate our method on the Cataract-101 dataset. The results show that our method achieves an accuracy of 96.37% with only surgical step labels, which outperforms other state-of-the-art approaches.

Conclusion

The proposed convolutional recurrent network based on step-phase prediction and contrastive learning can leverage fine-grained characteristics and alleviate spatial–temporal discrepancies to improve the performance of surgical workflow recognition.

Appendix: Detailed study on sequence length

Table 3 Average duration of each surgical step in cataract surgery

When selecting the input sequence length, a key factor that should be considered is the average duration of each surgical step. The statistical results of each cataract surgery step were calculated from the Cataract-101 dataset and are presented in Table 3. The table shows that the shortest duration of a surgical step is 13 s. Half of the surgical steps have a duration of approximately 20 s. There is only one longest surgical step with a duration of 145 s. For usage under different occasions, we can select different input sequence lengths to satisfy specific needs. In our experiments, for real-time intra-operative use, the sequence length is set to 10 for safety concerns. For online post-operative use, the sequence length is set to 20 to improve the average recognition results by a trade-off several steps.

Fig. 6

Model performance under different input sequence lengths according to the evaluation metrics of the recall (left), precision (middle), and macro-F1 score (right). The input sequences are set to 3–60 frames, which correspond to 3–60 s in the original videos. S1–S10 represent the 10 surgical steps in cataract surgery, respectively

Table 4 Accuracy and Jaccard performance with the extension of input sequence length

To validate our assumptions, we conduct experiments with different input sequence lengths (3 s, 5 s, 8 s, 10 s, 20 s, 40 s, and 60 s). The results based on recall, precision, accuracy, and F1 score are presented in Fig. 6 and Table 4. Figure 6 shows that, as the input sequence length extends from 3 s to 10 s, our model can exploit more sufficient temporal information between frames and achieve improving performance for most of the surgical steps. As sequence length increases up to 20 s, most surgical steps reach higher performance. However, there is an obvious decline in Step 2, viscous agent injection. This verifies our assumption that this step accounts for the least amount of time in the whole procedure and is very likely to be interrupted when aggregating temporal information from too long ago. When extending the sequence length from 20 to 60 s, more irrelevant information is introduced for most of the surgical steps, therefore decreasing average results, as shown in Table 4. The experimental results demonstrate the soundness of our consideration of sequence length in both intra-operative and post-operative uses.