Ego-Motion Classification for Body-Worn Videos

  • Zhaoyi Meng
  • Javier Sánchez
  • Jean-Michel Morel
  • Andrea L. BertozziEmail author
  • P. Jeffrey Brantingham
Conference paper
Part of the Mathematics and Visualization book series (MATHVISUAL)


Portable cameras record dynamic first-person video footage and these videos contain information on the motion of the individual to whom the camera is mounted, defined as ego. We address the task of discovering ego-motion from the video itself, without other external calibration information. We investigate the use of similarity transformations between successive video frames to extract signals reflecting ego-motions and their frequencies. We use novel graph-based unsupervised and semi-supervised learning algorithms to segment the video frames into different ego-motion categories. Our results show very accurate results on both choreographed test videos and ego-motion videos provided by the Los Angeles Police Department.



The work was supported by the ONR grant N00014-16-1-2119, NSF grant DMS-1737770, NSF grant DMS-1417674, FUI project Plein Phare by BPI-France and NIJ Grant 2014-R2-CX-0101.


  1. 1.
    S. Baker, I. Matthews, Equivalence and efficiency of image alignment algorithms, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2001)Google Scholar
  2. 2.
    S. Baker, I. Matthews, Lucas-Kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56, 221–255 (2004)CrossRefGoogle Scholar
  3. 3.
    H.C. Barrett, P.M. Todd, G.F. Miller, P.W. Blythe, Accurate judgments of intention from motion cues alone: a cross-cultural study. Evol. Hum. Behav. 26(4), 313–331 (2005)CrossRefGoogle Scholar
  4. 4.
    A.L. Bertozzi, A. Flenner, Diffuse interface models on graphs for classification of high dimensional data. Multiscale Model. Simul. 10(3), 1090–1118 (2012)MathSciNetCrossRefGoogle Scholar
  5. 5.
    A.L. Bertozzi, A. Flenner, Diffuse interface models on graphs for classification of high dimensional data. SIAM Rev. 58(2), 293–328 (2016)MathSciNetCrossRefGoogle Scholar
  6. 6.
    P. Bouthemy, M. Gelgon, F. Ganansia, A unified approach to shot change detection and camera motion characterization. IEEE Trans. Circuits Syst. Video Technol. 9(7), 1030–1044 (1999)CrossRefGoogle Scholar
  7. 7.
    F. Chung, Spectral Graph Theory, vol. 92 (American Mathematical Society, Providence, 1997)zbMATHGoogle Scholar
  8. 8.
    C. Fowlkes, S. Belongie, F. Chung, J. Malik, Spectral grouping using the Nyström method. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 214–225 (2004)CrossRefGoogle Scholar
  9. 9.
    C. Garcia-Cardona, E. Merkurjev, A.L. Bertozzi, A. Flenner, A.G. Percus, Multiclass data segmentation using diffuse interface methods on graphs. IEEE Trans. Pattern Anal. Mach. Int. 36(8), 1600–1613 (2014)CrossRefGoogle Scholar
  10. 10.
    H. Hu, J. Sunu, A.L. Bertozzi, Multi-class graph Mumford-Shah model for plume detection using the MBO scheme, in Energy Minimization Methods in Computer Vision and Pattern Recognition (Springer International Publishing, Berlin, 2015)Google Scholar
  11. 11.
    T. Huynh, M. Fritz, B. Schiele, Discovery of activity patterns using topic models, in Proceedings of the 10th International Conference on Ubiquitous Computing (2008), pp. 10–19Google Scholar
  12. 12.
    V. Kiani, H.R. Pourreza, Robust GME in encoded mpeg video, in Proceedings of the 9th International Conference on Advances in Mobile Computing and Multimedia (2011), pp. 147–154Google Scholar
  13. 13.
    J. Kim, H.S. Chang, J. Kim, H. Kim, Efficient camera motion characterization for MPEG video indexing. IEEE Int. Conf. Multimed. Expo 2, 1171–1174 (2000)CrossRefGoogle Scholar
  14. 14.
    K.M. Kitani, T. Okabe, Y. Sato, A. Sugimoto, Fast unsupervised ego-action learning for first-person sports videos, in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011), pp. 3241–3248Google Scholar
  15. 15.
    B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in Proceedings of the 7th International Joint Conference on Artificial intelligence (IJCAI) (1981)Google Scholar
  16. 16.
    X. Luo, A.L. Bertozzi, Convergence analysis of the graph Allen-Cahn scheme. J. Stat. Phys. 167(3), 934–958 (2017)MathSciNetCrossRefGoogle Scholar
  17. 17.
    J. MacQueen, Some methods for classification and analysis of multivariate observations, in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (1967)Google Scholar
  18. 18.
    W.W. Mayol, D.W. Murray, Wearable hand activity recognition for event summarization, in International Symposium on Searable Computers (2005), pp. 122–129Google Scholar
  19. 19.
    Z. Meng, A. Koniges, H. Yun, S. Williams, T. Kurth, B. Cook, J. Deslippe, A.L. Bertozzi: OpenMP parallelization and optimization of graph-based machine learning algorithms, in OpenMP: Memory, Devices, and Tasks, ed. by N. Maruyama, B. de Supinski, M. Wahib. Lecture Notes in Computer Science, vol. 9903 (Springer, Berlin, 2016). IWOMPCrossRefGoogle Scholar
  20. 20.
    Z. Meng, E. Merkurjev, A. Koniges, A.L. Bertozzi, Hyperspectral image classification using graph clustering methods. Image Process. Line 7, 218–245 (2017)MathSciNetCrossRefGoogle Scholar
  21. 21.
    E. Merkurjev, T. Kostic, A.L. Bertozzi, An MBO scheme on graphs for classification and image processing. SIAM J. Imag. Sci. 6(4), 1903–1930 (2013)MathSciNetCrossRefGoogle Scholar
  22. 22.
    E. Merkurjev, E. Bae, A.L. Bertozzi, X.C. Tai, Global binary optimization on graphs for classification of high-dimensional data. J. Math. Imag. Vision 52(3), 414–435 (2015)MathSciNetCrossRefGoogle Scholar
  23. 23.
    H. Pirsiavash, D. Ramanan, Detecting activities of daily living in first-person camera views, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012), pp. 2847–2854Google Scholar
  24. 24.
    X. Ren, C. Gu, Figure-ground segmentation improves handled object recognition in egocentric video. CVPR 2(3), 6 (2010)Google Scholar
  25. 25.
    J. Sánchez, The inverse compositional algorithm for parametric registration. Image Process. Line 6, 212–232 (2016)MathSciNetCrossRefGoogle Scholar
  26. 26.
    J. Sánchez, J.-M. Morel, Motion smoothing strategies for 2D video stabilization. SIAM J. Imag. Sci. 11(1), 219–251 (2018)MathSciNetCrossRefGoogle Scholar
  27. 27.
    E.H. Spriggs, F. de la Torre, M. Hebert, Temporal segmentation and activity classification from first-person sensing, in 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (2009), pp. 17–24Google Scholar
  28. 28.
    T. Starner, B. Schiele, A. Pentland, International Symposium on Wearable Computers (1998), pp. 50–57Google Scholar
  29. 29.
    T. Starner, J. Weaver, A. Pentland, Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1371–1375 (1998)CrossRefGoogle Scholar
  30. 30.
    M. Stoer, F. Wagner, A simple min-cut algorithm. J. ACM 44(4), 585–591 (1997)MathSciNetCrossRefGoogle Scholar
  31. 31.
    L. Sun, U. Klank, M. Beetz, EYEWATCHME—3D hand and object tracking for inside out activity analysis, in 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (2009), pp. 9–16Google Scholar
  32. 32.
    S. Sundaram, W. Cuevas, High level activity recognition using low resolution wearable vision, in 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (2009), pp. 25–32Google Scholar
  33. 33.
    R. Szeliski, Computer Vision: Algorithms and Applications (Springer Science & Business Media, Berlin, 2010)zbMATHGoogle Scholar
  34. 34.
    A. Szlam, X. Bresson, A total variation-based graph clustering algorithm for cheeger ratio cuts. UCLA CAM Report: 09-68 (2009)Google Scholar
  35. 35.
    Y. Van Gennip, A.L. Bertozzi, Gamma-convergence of graph Ginzburg-Landau functionals. Adv. Differ. Equ. 17(11/12), 1115–1180 (2012)zbMATHGoogle Scholar
  36. 36.
    U. Von Luxburg, A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Zhaoyi Meng
    • 1
  • Javier Sánchez
    • 2
  • Jean-Michel Morel
    • 3
  • Andrea L. Bertozzi
    • 1
    Email author
  • P. Jeffrey Brantingham
    • 1
  1. 1.University of California, Los AngelesLos AngelesUSA
  2. 2.Universidad de Las Palmas de Gran CanariaLas Palmas de Gran CanariaSpain
  3. 3.Ecole Normale Supérieure de CachanCachanFrance

Personalised recommendations