Active Frame Selection for Label Propagation in Videos

  • Sudheendra Vijayanarasimhan
  • Kristen Grauman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


Manually segmenting and labeling objects in video sequences is quite tedious, yet such annotations are valuable for learning-based approaches to object and activity recognition. While automatic label propagation can help, existing methods simply propagate annotations from arbitrarily selected frames (e.g., the first one) and so may fail to best leverage the human effort invested. We define an active frame selection problem: select k frames for manual labeling, such that automatic pixel-level label propagation can proceed with minimal expected error. We propose a solution that directly ties a joint frame selection criterion to the predicted errors of a flow-based random field propagation model. It selects the set of k frames that together minimize the total mislabeling risk over the entire sequence. We derive an efficient dynamic programming solution to optimize the criterion. Further, we show how to automatically determine how many total frames k should be labeled in order to minimize the total manual effort spent labeling and correcting propagation errors. We demonstrate our method’s clear advantages over several baselines, saving hours of human effort per video.


Dynamic Programming Markov Random Field Foreground Object Label Propagation Video Object 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Agarwala, A., Hertzmann, A., Salesin, D., Seitz, S.: Keyframe-Based Tracking for Rotoscoping and Animation. In: SIGGRAPH (2004)Google Scholar
  2. 2.
    Badrinarayanan, V., Galasso, F., Cipolla, R.: Label Propagation in Video Sequences. In: CVPR (2010)Google Scholar
  3. 3.
    Bai, X., Wang, J., Simons, D., Sapiro, G.: Video SnapCut: Robust Video Object Cutout using Localized Classifiers. In: SIGGRAPH (2009)Google Scholar
  4. 4.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: Interactive Co-segmentation with Intelligent Scribble Guidance. In: CVPR (2010)Google Scholar
  5. 5.
    Boykov, Y., Veksler, O., Zabih, R.: Fast Approximate Energy Minimization via Graph Cuts. TPAMI (2001)Google Scholar
  6. 6.
    Budvytis, I., Badrinarayanan, V., Cipolla, R.: Label Propagation in Complex Video Sequences using Semi-supervised Learning. In: BMVC (2010)Google Scholar
  7. 7.
    Cooper, M., Foote, J.: Discriminative Techniques for Keyframe Selection. In: ICME (2005)Google Scholar
  8. 8.
    Fathi, A., Balcan, M., Ren, X., Rehg, J.: Combining Self Training and Active Learning for Video Segmentation. In: BMVC (2011)Google Scholar
  9. 9.
    Fauqueur, J., Brostow, G., Cipolla, R.: Assisted Video Object Labeling by Joint Tracking of Regions and Keypoints. In: Proc. Int. Workshop on Interactive Computer Vision (2007)Google Scholar
  10. 10.
    Grundmann, M., Kwatra, V., Han, M., Essa, I.: Efficient Hierarchical Graph-based Video Segmentation. In: CVPR (2010)Google Scholar
  11. 11.
    Hoi, S., Jin, R., Zhu, J., Lyu, M.: Semi-supervised SVM Batch Mode Active Learning with Applications to Image Retrieval. ACM Trans. on Info Systems (2009)Google Scholar
  12. 12.
    Liu, C., Yuen, J., Torralba, A.: Nonparametric Scene Parsing: Label Transfer via Dense Scene Alignment. In: CVPR (2009)Google Scholar
  13. 13.
    Liu, T., Kender, J.: Optimization Algorithms for the Selection of Key Frame Sequences of Variable Length. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part IV. LNCS, vol. 2353, pp. 403–417. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  14. 14.
    Patras, I., Hendriks, E., Lagendijk, R.: Semi-automatic Object-based Video Segmentation with Labeling of Color Segments. In: Signal Processing: Image Communication (2003)Google Scholar
  15. 15.
    Price, B.L., Morse, B.S., Cohen, S.: Livecut: Learning-based Interactive Video Segmentation by Evaluation of Multiple Propagated Cues. In: ICCV (2009)Google Scholar
  16. 16.
    Ren, X., Malik, J.: Tracking as Repeated Figure/Ground Segmentation. In: CVPR (2007)Google Scholar
  17. 17.
    Sundaram, N., Brox, T., Keutzer, K.: Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 438–451. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  18. 18.
    Tsai, D., Flagg, M., Rehg, J.M.: Motion Coherent Tracking with Multi-label MRF Optimization. In: BMVC (2010)Google Scholar
  19. 19.
    Vazquez-Reina, A., Avidan, S., Pfister, H., Miller, E.: Multiple Hypothesis Video Segmentation from Superpixel Flows. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 268–281. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  20. 20.
    Vijayanarasimhan, S., Grauman, K.: What’s It Going to Cost You?: Predicting Effort vs. Informativeness for Multi-Label Image Annotations. In: CVPR (2008)Google Scholar
  21. 21.
    Vijayanarasimhan, S., Jain, P., Grauman, K.: Far-Sighted Active Learning on a Budget for Image and Video Recognition. In: CVPR (2010)Google Scholar
  22. 22.
    Vondrick, C., Ramanan, D.: Video Annotation and Tracking with Active Learning. In: NIPS (2011)Google Scholar
  23. 23.
    Vondrick, C., Ramanan, D., Patterson, D.: Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 610–623. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  24. 24.
    Wang, J., Bhat, P., Colburn, R.A., Agrawala, M., Cohen, M.F.: Interactive Video Cutout. In: SIGGRAPH (2005)Google Scholar
  25. 25.
    Wolf, W.: Key Frame Selection by Motion Analysis. In: ICASSP (1996)Google Scholar
  26. 26.
    Yuen, J., Russell, B., Liu, C., Torralba, A.: Labelme Video: Building a Video Database with Human Annotations. In: ICCV (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Sudheendra Vijayanarasimhan
    • 1
  • Kristen Grauman
    • 1
  1. 1.University of Texas at AustinUSA

Personalised recommendations