Moving Camera Analytics: Computer Vision Applications

  • Chung-Ching Lin
  • Karthikeyan Natesan Ramamurthy
  • Sharathchandra U. Pankanti


To date, billions of cameras have been actively used on moving platform. Video analytics applications for camera are emerging in diverse areas. Among various video analytics applications for moving cameras, we will discuss the application of unmanned aerial vehicles (UAVs). First, we present a system for summarizing videos by automatically creating a panorama for videos, detecting and tracking moving objects in the videos. Our video summarization experiments on the UAV dataset demonstrate that we can achieve efficient data reduction without losing significant activities of interest. Second, a distributed 3D reconstruction algorithm will be presented. Most methods for 3D reconstruction methods are either centralized or operate incrementally. The poor scalability affects the quality of solution for large-scale structure from motion (SfM). Our algorithm uses alternating direction method of multipliers (ADMM) to formulate a distributed bundle adjustment (BA) algorithm. The results are comparable to an alternate state-of-the-art centralized bundle adjustment algorithm on synthetic and real 3D reconstruction problems. The runtime of our implementation scales linearly with the number of observed points.


  1. 1.
    Agarwal S, Snavely N, Seitz SM, Szeliski R (2010) Bundle adjustment in the large. In: Computer Vision–ECCV 2010, Springer, pp 29–42Google Scholar
  2. 2.
    Aravkin A, Styer M, Moratto Z, Nefian A, Broxton M (2012) Student’s t robust bundle adjustment algorithm. In: Image Processing (ICIP), 2012 19th IEEE International Conference on, IEEE, pp 1757–1760Google Scholar
  3. 3.
    Bhattacharyya A (1943) On a measure of divergence between two statistical populations defined by their probability distributions. Bull Calcutta Math Soc 35:99–109MathSciNetzbMATHGoogle Scholar
  4. 4.
    Botchen RP, Bachthaler S, Schick F, Chen M, Mori G, Weiskopf D, Ertl T (2008) Action-based multifield video visualization. IEEE Transactions on Visualization and Computer Graphics 14(4):885–899CrossRefGoogle Scholar
  5. 5.
    Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3(1):1–122CrossRefGoogle Scholar
  6. 6.
    Byröd M, Åström K (2010) Conjugate gradient bundle adjustment. In: Computer Vision–ECCV 2010, Springer, pp 114–127Google Scholar
  7. 7.
    Comaniciu D, Ramesh V, Meer P (2000) Real-time tracking of non-rigid objects using mean shift. In: Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, IEEE, vol 2, pp 142–149Google Scholar
  8. 8.
    Eriksson A, Bastian J, Chin TJ, Isaksson M (2015) A consensus-based framework for distributed bundle adjustment. In: Computer Vision and Pattern Recognition, 2015. CVPR 2015. IEEE Conference on, IEEEGoogle Scholar
  9. 9.
    Feng S, Lei Z, Yi D, Li SZ (2012) Online content-aware video condensation. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IEEE, pp 2082–2087Google Scholar
  10. 10.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6):381–395MathSciNetCrossRefGoogle Scholar
  11. 11.
    Fusiello A, Crosilla F (2015) Solving bundle block adjustment by generalized anisotropic procrustes analysis. ISPRS Journal of Photogrammetry and Remote Sensing 102:209–221CrossRefGoogle Scholar
  12. 12.
    Goldman D, Curless B, Salesin D, Seitz S (2006) Schematic storyboards for video editing and visualization. In: Proc. ACM SIGGRAPH, vol 25, pp 862–871CrossRefGoogle Scholar
  13. 13.
    Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge university presszbMATHGoogle Scholar
  14. 14.
    Heinly J, Schonberger JL, Dunn E, Frahm JM (2015) Reconstructing the world* in six days*(as captured by the yahoo 100 million image dataset). In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3287–3295Google Scholar
  15. 15.
    Indelman V, Roberts R, Beall C, Dellaert F (2012) Incremental light bundle adjustment. In: Proceedings of the British Machine Vision Conference (BMVC 2012), pp 3–7Google Scholar
  16. 16.
    Irani M, Anandan P (1999) About direct methods. In: International Workshop on Vision Algorithms, Springer, pp 267–277Google Scholar
  17. 17.
    Irani M, Anandan P, Hsu S (1995) Mosaic based representations of video sequences and their applications. In: Computer Vision, 1995. Proceedings., Fifth International Conference on, IEEE, pp 605–611Google Scholar
  18. 18.
    Konolige K, Garage W (2010) Sparse sparse bundle adjustment. In: BMVC, Citeseer, pp 1–11Google Scholar
  19. 19.
    Kopf J, Cohen MF, Szeliski R (2014) First-person hyper-lapse videos. ACM Transactions on Graphics (TOG) 33(4):78CrossRefGoogle Scholar
  20. 20.
    Lin CC, Pankanti S, Ashour G, Porat D, Smith JR (2015) Moving camera analytics: Emerging scenarios, challenges, and applications. IBM Journal of Research and Development 59(2/3): 5–1CrossRefGoogle Scholar
  21. 21.
    Lions PL, Mercier B (1979) Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis 16(6):964–979MathSciNetCrossRefGoogle Scholar
  22. 22.
    Lourakis MI, Argyros AA (2009) SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software (TOMS) 36(1):2MathSciNetCrossRefGoogle Scholar
  23. 23.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference on, Ieee, vol 2, pp 1150–1157Google Scholar
  24. 24.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2):91–110CrossRefGoogle Scholar
  25. 25.
    Mitra K, Chellappa R (2008) A scalable projective bundle adjustment algorithm using the l infinity norm. In: Computer Vision, Graphics & Image Processing, 2008. ICVGIP’08. Sixth Indian Conference on, IEEE, pp 79–86Google Scholar
  26. 26.
    Moré JJ (1978) The Levenberg-Marquardt algorithm: implementation and theory. In: Numerical analysis, Springer, pp 105–116Google Scholar
  27. 27.
    Nocedal J, Wright S (1999) Numerical optimization. Springer Series in Operations Research, SpringerCrossRefGoogle Scholar
  28. 28.
    Oh S, Hoogs A, Perera A, Cuntoor N, Chen CC, Lee JT, Mukherjee S, Aggarwal J, Lee H, Davis L, et al (2011) A large-scale benchmark dataset for event recognition in surveillance video. In: Computer vision and pattern recognition (CVPR), 2011 IEEE conference on, IEEE, pp 3153–3160Google Scholar
  29. 29.
    Pritch Y, Rav-Acha A, Peleg S (2008) Nonchronological video synopsis and indexing. IEEE Transactions on Pattern Analysis & Machine Intelligence (11):1971–1984CrossRefGoogle Scholar
  30. 30.
    Pritt MD (2014) Fast orthorectified mosaics of thousands of aerial photographs from small UAVs. In: Applied Imagery Pattern Recognition Workshop (AIPR), 2014 IEEE, IEEE, pp 1–8Google Scholar
  31. 31.
    Ramamurthy KN, Lin CC, Aravkin AY, Pankanti S, Viguier R (2017) Distributed bundle adjustment. In: ICCV Workshops, pp 2146–2154Google Scholar
  32. 32.
    Rockafellar RT, Wets RJB (2009) Variational analysis, vol 317. Springer Science & Business MediaGoogle Scholar
  33. 33.
    Strecha C, von Hansen W, Gool LV, Fua P, Thoennessen U (2008) On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, Ieee, pp 1–8Google Scholar
  34. 34.
    Tomasi C, Kanade T (1991) Detection and tracking of point features. Tech. rep., International Journal of Computer VisionGoogle Scholar
  35. 35.
    Triggs B, McLauchlan PF, Hartley RI, Fitzgibbon AW (2000) Bundle adjustment - a modern synthesis. In: Vision algorithms: theory and practice, Springer, pp 298–372Google Scholar
  36. 36.
    Trinh H, Li J, Miyazawa S, Moreno J, Pankanti S (2012) Efficient UAV video event summarization. In: Pattern Recognition (ICPR), 2012 21st International Conference on, IEEE, pp 2226–2229Google Scholar
  37. 37.
    Tuytelaars T, Van Gool L (2004) Matching widely separated views based on affine invariant regions. International journal of computer vision 59(1):61–85CrossRefGoogle Scholar
  38. 38.
    Wang Y, Yin W, Zeng J (2015) Global convergence of ADMM in nonconvex nonsmooth optimization. arXiv preprint arXiv:151106324Google Scholar
  39. 39.
    Wu C (2013) Towards linear-time incremental structure from motion. In: 3D Vision-3DV 2013, 2013 International Conference on, IEEE, pp 127–134Google Scholar
  40. 40.
    Wu C, Agarwal S, Curless B, Seitz SM (2011) Multicore bundle adjustment. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, pp 3057–3064Google Scholar
  41. 41.
    Xiao J, Chen J, Yeung DY, Quan L (2008) Learning two-view stereo matching. In: Computer Vision–ECCV 2008, Springer, pp 15–27Google Scholar
  42. 42.
    Yilmaz A, Javed O, Shah M (2006) Object tracking: A survey. ACM computing surveys (CSUR) 38(4):13CrossRefGoogle Scholar
  43. 43.
    Zamir AR, Dehghan A, Shah M (2012) GMCP-tracker: Global multi-object tracking using generalized minimum clique graphs. In: Computer Vision–ECCV 2012, Springer, pp 343–356Google Scholar
  44. 44.
    Zhang J, Boutin M, Aliaga DG (2006) Robust bundle adjustment for structure from motion. In: Image Processing, 2006 IEEE International Conference on, IEEE, pp 2185–2188Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Chung-Ching Lin
    • 1
  • Karthikeyan Natesan Ramamurthy
    • 1
  • Sharathchandra U. Pankanti
    • 1
  1. 1.IBM Research AIYorktown HeightsUSA

Personalised recommendations