Finding Actions Using Shape Flows

  • Hao Jiang
  • David R. Martin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5303)


We propose a novel method for action detection based on a new action descriptor called a shape flow that represents both the shape and movement of an object in a holistic and parsimonious manner. We find actions by finding shape flows in a target video that are similar to a template shape flow. Shape flows are largely independent of appearance, and the match cost function that we propose is invariant to scale changes and smooth nonlinear deformation in space and time. The problem of matching shape flows is difficult, however, yielding a large, non-convex, integer program. We propose a novel relaxation method based on successive convexification that converts this hard program into a vastly smaller linear program: By using only those variables that appear on the 4D lower convex hull of the matching cost volume, most of the variables in the linear program may be eliminated. Experiments confirm that the proposed shape flow method can successfully detect complex actions in cluttered video, even with self-occlusion, camera motion, and intra-class variation.


Action Recognition Delaunay Triangulation Camera Motion Background Clutter Joint Trajectory 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Besag, J.: On the Statistical Analysis of Dirty Pictures. Journal of the Royal Statistical Society, Series B 48(3), 259–302 (1986)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R.: Actions as Space-Time Shapes. In: ICCV (2005)Google Scholar
  3. 3.
    Bobick, A., Davis, J.: The Recognition of Human Movement Using Temporal Templates. IEEE Trans. PAMI 23(3), 257–267 (2001)CrossRefGoogle Scholar
  4. 4.
    Efros, A., Berg, A., Mori, G., Malik, J.: Recognizing Action at a Distance. In: ICCV (2003)Google Scholar
  5. 5.
    Interrante, V., Grosch, C.: Visualizing 3D Flow. IEEE Computer Graphics and Applications 18(4), 49–53 (1998)CrossRefGoogle Scholar
  6. 6.
    Jiang, H., Drew, M.S., Li, Z.N.: Matching by Linear Programming and Successive Convexification. IEEE Trans. on PAMI 29(6), 959–975 (2007)CrossRefGoogle Scholar
  7. 7.
    Ke, Y., Sukthankar, R., Hebert, M.: Event Detection in Crowded Videos. In: ICCV (2007)Google Scholar
  8. 8.
    Kenwright, D., Mallinson, G.: A 3-D Streamline Tracking Algorithm Using Dual Stream Functions. Visualization, 62–68 (1992)Google Scholar
  9. 9.
    Laptev, I., Lindeberg, T.: Space-Time Interest Points. In: ICCV (2003)Google Scholar
  10. 10.
    Laptev, I., Prez, P.: Retrieving Actions in Movies. In: ICCV (2007)Google Scholar
  11. 11.
    Mori, G., Ren, X.F., Efros, A., Malik, J.: Recovering Human Body Configurations: Combining Segmentation and Recognition. CVPR, 326–333 (2004)Google Scholar
  12. 12.
    Parameswaran, V., Chellappa, R.: Human Action-Recognition Using Mutual Invariants. CVIU 98(2), 295–325 (2005)Google Scholar
  13. 13.
    Ramanan, D., Forsyth, D.A., Zisserman, A.: Tracking People by Learning Their Appearance. IEEE Trans. on PAMI 29(1), 65–81 (2007)CrossRefGoogle Scholar
  14. 14.
    Schuldt, C., Laptev, I., Caputo, B.: Recognizing Human Actions: A Local SVM Approach. In: ICPR (2004)Google Scholar
  15. 15.
    Scovanner, P., Ali, S., Shah, M.: A 3-Dimensional SIFT Descriptor and its Application to Action Recognition. ACM Multimedia (September 2007)Google Scholar
  16. 16.
    Shechtman, E., Irani, M.: Space-Time Behavior Based Correlation. In: CVPR (2005)Google Scholar
  17. 17.
    Sheikh, Y., Shah, M.: Exploring the Space of an Action for Human Action Recognition. In: ICCV (2005)Google Scholar
  18. 18.
    Weinland, D., Ronfard, R., Boyer, E.: Free Viewpoint Action Recognition using Motion History Volumes. In: CVIU (November/December 2006)Google Scholar
  19. 19.
    Yilmaz, A., Shah, M.: Actions as Objects: A Novel Action Representation. In: CVPR (2005)Google Scholar
  20. 20.
    Yilmaz, A., Shah, M.: Recognizing Human Actions in Videos Acquired by Uncalibrated Moving Cameras. In: ICCV (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Hao Jiang
    • 1
  • David R. Martin
    • 1
  1. 1.Computer Science DepartmentBoston College, Chestnut HillUSA

Personalised recommendations