Skip to main content
SpringerLink
Log in

DIVS: A Real-Time Video Stabilization Framework for Consumer Camera

  • Conference paper
  • First Online:
Advanced Computational and Communication Paradigms (ICACCP 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 535))

  • 217 Accesses

Abstract

Video stabilization is one of the most interesting areas of research that can be implemented in consumer camera technology. Various works have been proposed to overcome this problem. However, there are many practical difficulties in deploying them in real time. Many frames are taken as an input to generate a stabilized output, which leads to high computation of the image sensing hardware. A novel approach to solve these challenges is proposed in this paper using the Hybrid Deep Neural Network Model (HDNNM) that uses gyroscopic sensor data and the optical flow analysis. This network is based on the deep unsupervised learning techniques, which require less computation and low buffer memory. The workflow of the proposed model is based on the integration of the optical flow with the joint motion representation to maintain a correspondence between the frames and the poses of scene objects. The LSTM module in the network wraps the grid and stabilizes the video frames. The relative motion along with the multi-stage training strategies was adopted to make the model unsupervised. The HDNNM is validated using the ablative dataset collected from various scenes, and the results of the model surpass the existing model’s performance metrics. To the best of the authors’ knowledge, this is the first unsupervised approach to overcome the challenges in video stabilization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Guilluy W, Oudre L, Beghdadi A (2021) Video stabilization: overview, challenges and perspectives. Signal Process Image Commun 90:116015. https://doi.org/10.1016/j.image.2020.116015

    Article  Google Scholar 

  2. Greeta S, Raajan NR, Meenu M (2019) Video stabilization. In: 2019 innovations in power and advanced computing technologies (i-PACT), pp 1–4. https://doi.org/10.1109/iPACT44901.2019.8959991

  3. Lee YG (2022) Low memory access video stabilization for low-cost camera SoC. Sensors (Basel). 22(6):2341. https://doi.org/10.3390/s22062341. PMID: 35336512; PMCID: PMC8949514

  4. Survey on digital video stabilization: concepts, methods, and challenges. ACM Comput Surv (2023). https://dl.acm.org/doi/10.1145/3494525

  5. Yu J, Ramamoorthi R (2019) Robust video stabilization by optimization in CNN weight space. In: CVPR

    Google Scholar 

  6. Bross B, Wang Y-K, Ye Y, Liu S, Chen J, Sullivan GJ, Ohm J-R (2021) Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans Circuits Syst Video Technol 31:3736–3764

    Article  Google Scholar 

  7. Lee YG (2019) Fast global motion estimation on single instruction multiple data processors for real-time devices. Electron Imaging 58:113105

    Google Scholar 

  8. Guo H, Liu S, Zhu S, Shen HT, Zeng B (2018) View-consistent MeshFlow for stereo-scopic video stabilization. IEEE Trans Comput Imaging 4(4):573–584

    Article  Google Scholar 

  9. Raut S et al (2019) Real-time high-resolution video stabilization using high-frame-rate jitter sensing. ROBOMECH J (SpringerOpen). https://doi.org/10.1186/s40648-019-0144-z

  10. Nie Y, Su T, Zhang Z, Sun H, Li G (2018) Dynamic video stitching via shakiness removing. IEEE Trans Image Process 27(1):164–178

    Article  MathSciNet  MATH  Google Scholar 

  11. Sen-Zhe X, Jun H, Wang M, Tai-Jiang M, Shi-Min H (2018) Deep video stabilization using adversarial networks. Comput Graph Forum 37:267–276

    Article  Google Scholar 

  12. Wang M, Yang G-Y, Lin J-K, Zhang S-H, Shamir A, Lu S-P, Hu S-M (2019) Deep online video stabilization with multi-grid warping transformation learning. IEEE Trans Image Process

    Google Scholar 

  13. Wang Z, Zhang L, Huang H (2018) High-quality real-time video stabilization using trajectory smoothing and mesh-based warping. IEEE Access 6:25157–25166

    Article  Google Scholar 

  14. Raptis GE, Fidas C, Avouris N (2018) Effects of mixed-reality on players behaviour and immersion in a cultural tourism game: a cognitive processing perspective. Int J Human Comput Stud 114:69–79

    Article  Google Scholar 

  15. Kamranian Z, Sadeghian H, Nilchi ARN, Mehran M (2021) Fast, yet robust end-to-end camera pose estimation for robotic applications. Appl Intell 51(6):3581–3599

    Google Scholar 

  16. Walter H, Li R, Munafo J, Curry C, Peterson N, Stoffregen T (2019) APAL coupling study

    Google Scholar 

  17. Jia C, Evans BL (2012) Probabilistic 3-D motion estimation for rolling shutter video rectification from visual and inertial measurements. In: Proceedings of the IEEE international workshop on multimedia signal processing (MMSP), pp 203–208

    Google Scholar 

  18. Grundmann M, Kwatra V, Essa I (2011) Auto-directed video stabilization with robust L1 optimal camera paths. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 225–232

    Google Scholar 

  19. Aguilar WG, Angulo C (2016) Real-time model-based video stabilization for microaerial vehicles. Neural Process Lett 43(2):459–477

    Article  Google Scholar 

  20. Tareen SAK, Saleem Z (2018) A comparative analysis of SIFT SURF KAZE AKAZE ORB and BRISK. In: 2018 international conference on computing mathematics and engineering technologies (iCoMET), pp 1–10

    Google Scholar 

  21. Noble FK (2016) Comparison of opencv’s feature detectors and feature matchers. In: 2016 23rd international conference on mechatronics and machine vision in practice (M2VIP), pp 1–6

    Google Scholar 

  22. Casser V, Pirk S, Mahjourian R, Angelova A (2019) Depth prediction without the sensors: leveraging structure for unsupervised learning from monocular videos. AAAI 33:8001–8008

    Article  Google Scholar 

  23. Gordon A, Li H, Jonschkowski R, Angelova A (2019) Depth from videos in the wild: unsupervised monocular depth learning from unknown cameras. In: ICCV, pp 8977–8986

    Google Scholar 

  24. Wang M, Yang G-Y, Lin J-K, Zhang S-H, Shamir A, Lu S-P, Hu S-M (2018) Deep online video stabilization with multi-grid warping transformation learning. In: IEEE TIP

    Google Scholar 

  25. Guervós E, Ruiz JJ, Pérez P, Muñoz JA, Díaz C, García N (2019) Using 360 VR video to improve the learning experience in veterinary medicine university degree. In: Electronic imaging. Society for imaging science and technology, pp 217-1–217-7

    Google Scholar 

  26. Fang M, Li H, Si S (2018) A video stabilization algorithm based on affine SIFT. In: 2018 international conference on computing, mathematics and engineering technologies (iCoMET), Sukkur, pp 1–4

    Google Scholar 

  27. Minh T, Hong M (2018) Video stabilization using feature-based classification. In: 2018 IEEE international conference on consumer electronics-Asia (ICCE-Asia), Jeju, 2018, pp 206–212

    Google Scholar 

  28. Huang TS, Netravali AN (2002) Motion and structure from feature correspondences: a review. In: Advances in image processing and understanding: a Festschrift for Thomas S Huang, pp 331–347

    Google Scholar 

  29. Luchetti A, Tomasin P, Fornaser A, Tallarico P, Bosetti P, De Cecco M (2019) The human being at the center of smart factories thanks to augmented reality. In: 2019 IEEE 5th international forum on research and technology for society and industry (RTSI), pp 51–56

    Google Scholar 

  30. Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77:1321–1329

    Google Scholar 

  31. Butaslac IIII, Luchetti A, Parolin E, Fujimoto Y, Kanbara M, De Cecco M, Kato H (2020) The feasibility of augmented reality as a support tool for motor rehabilitation. In: International conference on augmented reality, virtual reality and computer graphics. Springer, pp 165–173

    Google Scholar 

  32. Litleskare S, Calogiuri G (2019) Camera stabilization in 360\(^{\circ }\) videos and its impact on cyber sickness, environmental perceptions, and psychophysiological responses to a simulated nature walk: a singleblinded randomized trial. Front Psychol 10:2436

    Article  Google Scholar 

  33. Zhang G, Qin X, Hua W, Wong T-T, Heng P-A, Bao H (2007) Robust metric reconstruction from challenging video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–8

    Google Scholar 

  34. Yan C, Xie H, Yang D, Yin J, Zhang Y, Dai Q (2018) Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Trans Intell Transp Syst 19(1):284–295

    Article  Google Scholar 

  35. Dong J, Liu H (2017) Video stabilization for strict real-time applications. IEEE Trans Circuits Syst Video Technol 27(4):716–724

    Article  MathSciNet  Google Scholar 

  36. Kejriwal L, Singh I (2016) A hybrid filtering approach of digital video stabilization for UAV using Kalman and low pass filter. Procedia Comput Sci 93:359–366

    Article  Google Scholar 

  37. Bian J, Li Z, Wang N, Zhan H, Shen C, Cheng M-M et al (2019) Unsupervised scale-consistent depth and egomotion learning from monocular video. NeurIPS, pp 35–45

    Google Scholar 

  38. Choi J, Kweon IS (2019) Deep iterative frame interpolation for full-frame video stabilization. In: SIGGRAPH Asia

    Google Scholar 

  39. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778

    Google Scholar 

  40. Liu S, Li M, Zhu S, Zeng B (2017) Codingflow: enable video coding for video stabilization. IEEE TIP 26(7):3291–3302

    MathSciNet  MATH  Google Scholar 

  41. Liu S, Tan P, Yuan L, Sun J, Zeng B (2016) Meshflow: minimum latency online video stabilization. In: ECCV, pp 800–815

    Google Scholar 

  42. Luo X, Huang J-B, Szeliski R, Matzen K, Kopf J (2020) Consistent video depth estimation. ACM TOG 39(4)

    Google Scholar 

  43. Ranjan A, Jampani V, Balles L, Kim K, Sun D, Wulff J et al (2019) Competitive collaboration: joint unsupervised learning of depth camera motion optical flow and motion segmentation. In: CVPR 2019-June, no. 1, pp 12232–12241

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, VIT-AP University, Inavolu, India

    Varanasi L. V. S. K. B. Kasyap,  Sumathi & Abhijit Adhikari

  2. Department of Engineering Mathematics, Koneru Lakshmaiah Education Foundation, Vaddeswaram, India

    V. S. Bhagavan

Authors
  1. Varanasi L. V. S. K. B. Kasyap
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sumathi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abhijit Adhikari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. V. S. Bhagavan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Varanasi L. V. S. K. B. Kasyap .

Editor information

Editors and Affiliations

  1. Department of Computer Applications, Sikkim Manipal Institute of Technology, Rangpo, Sikkim, India

    Samarjeet Borah

  2. Department of Electrical Engineering, IIT Delhi, Delhi, India

    Tapan K. Gandhi

  3. Department of Computer Science, University of Milan, Milano, Milano, Italy

    Vincenzo Piuri

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kasyap, V.L.V.S.K.B., Sumathi, Adhikari, A., Bhagavan, V.S. (2023). DIVS: A Real-Time Video Stabilization Framework for Consumer Camera. In: Borah, S., Gandhi, T.K., Piuri, V. (eds) Advanced Computational and Communication Paradigms . ICACCP 2023. Lecture Notes in Networks and Systems, vol 535. Springer, Singapore. https://doi.org/10.1007/978-981-99-4284-8_28

Download citation

Publish with us

Policies and ethics

Search

Navigation