Skip to main content
SpringerLink
Log in

PanoSyn: immersive video synopsis for spherical surveillance video

  • Published:
Sādhanā Aims and scope Submit manuscript

Abstract

Finding an exciting event in a lengthy spherical (360\(^{\circ }\)) surveillance video having an unlimited Field of View (FoV) is challenging and time-consuming. Hence, this paper proposes a novel spherical video synopsis framework to condense the 360\(^{\circ }\) surveillance video. It also incorporates an action recognition module, making monitoring important activities easy, followed by less critical ones. The framework provides flexibility for generating synopsis from spherical videos based on the viewer’s preferences. The preferences include customized FoV visualization of the generated spherical synopsis video and FoV-based personalized synopsis video generation. In the first case, the viewer experience is customized by viewing the interested FoV based on the head movement over the generated spherical synopsis video. In the second case, the viewer experience is personalized by generating a synopsis video only for the viewer’s interested FoV dynamically. Overall, the proposed framework creates a condensed immersive video using various optimization techniques by reducing collisions between objects, preserving all events with interaction, chronological ordering, and showing the viewer only the specified number of objects per frame in the synopsis video recognizing important actions. Exhaustive experimental analysis of the used optimization algorithm is performed for the proposed video synopsis framework applicable for unlimited FoV. The analysis includes a video synopsis of varying lengths and predefined lengths.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24

Similar content being viewed by others

References

  1. Drisya S S, Ansuman M and Priyadharshini S 2022 360-degree image classification and viewport prediction using deep neural networks. In: Advances in Distributed Computing and Machine Learning. Springer, pp.483–492

  2. Priyadharshini S and Ansuman M 2020 360\(^{\circ }\) user-generated videos: Current research and future trends.High Performance Vision Intelligence: Recent Advances. pp.117–135

    Google Scholar 

  3. Yu-Chuan S, Dinesh J, and Kristen G 2016 Pano2vid: Automatic cinematography for watching 360\(^{\circ }\) videos.In: Asian Conference on Computer Vision. Springer, pp. 154–171

  4. Yu-Chuan S and Kristen G 2017 Making 360 video watchable in 2d: Learning videography for click free viewing. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 1368–1376

  5. Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang and Min Sun 2017 Deep 360 pilot: Learning a deep agent for piloting through 360 sports videos. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 1396–1405

  6. Youngjae Y, Sangho L, Joonil N, Jaeyun K, and Gunhee K 2018 A deep ranking model for spatio-temporal highlight detection from a 360 video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

  7. Sangho L, Jinyoung S, Youngjae Y, and Gunhee K 2018 A memory network approach for story-based temporal summarization of 360 videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1410–1419

  8. Wei-Sheng L, Yujia H, Neel J, Christopher B, Ming-Hsuan Y, and Sing Bing Kang 2017 Semantic-driven generation of hyperlapse from 360 degree video. IEEE transactions on visualization and computer graphics, 24: 2610–2621

    Google Scholar 

  9. Yael P, Alex R, and Shmuel P 2008 Nonchronological video synopsis and indexing. IEEE transactions on pattern analysis and machine intelligence, 30: 1971–1984

    Article  Google Scholar 

  10. Arif A, Samarjit K, Debi Prosad D, Renuka P, Lee S, Choi H and I Kim 2017 Video synopsis generation using spatio-temporal groups. In: 2017 IEEE International Conference on Signal and Image Processing Applications (ICSIPA). IEEE, pp. 512–517

  11. Giovanna P, Dario A, Filippo S, and Sebastiano B 2019 A new framework for studying tubes rearrangement strategies in surveillance video synopsis. In: 2019 IEEE international conference on image processing (ICIP). IEEE, pp. 664–668

  12. Sekh Arif A, Debi Prosad D, Samarjit K, Renuka P, Seung-Cheol L, Heeseung C, Gi Pyo N, and Ig-Jae K 2019 Query-based video synopsis for intelligent traffic monitoring applications. IEEE Transactions on Intelligent Transportation Systems,21: 3457–3468

    Google Scholar 

  13. Subhankar G, Suvendu R, Banshidhar M, and MNS S 2020 An improved surveillance video synopsis framework: a hsatlbo optimization approach. Multimedia Tools and Applications, 79: 4429–4461

  14. Shengbo C, Xianrui L, Yiyong H, Congcong Z, and Huaikou M 2020 Video synopsis based on attention mechanism and local transparent processing. IEEE Access, 8: 92603–92614

    Google Scholar 

  15. Namitha K, Athi N, and Geetha M 2021 Interactive visualization-based surveillance video synopsis. Applied Intelligence, pp. 1–22

  16. Subhankar G, Suvendu R, Himansu D and MNS S 2021 Gan based efficient foreground extraction and hgwosa based optimization for video synopsis generation. Digital Signal Processing, 111: 102988

  17. Jianqing Z, Shengcai L, and Stan Z L 2015 Multicamera joint video synopsis. IEEE Transactions on Circuits and Systems for Video Technology, 26: 1058–1069

    Google Scholar 

  18. Ansuman M, Pankaj K S, Banshidhar M, and Sudarshan P 2016 Mvs: A multi-view video synopsis framework. Signal Processing: Image Communication, 42:31–44

    Google Scholar 

  19. Shaoqing R, Kaiming H, Ross G, and Jian S 2015 Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91–99

    Google Scholar 

  20. Namitha K and Athi N 2020 Preserving interactions among moving objects in surveillance video synopsis. Multimedia Tools and Applications, 79: 32331–32360

    Article  Google Scholar 

  21. Nicolai W, Alex B, and Dietrich P 2017 Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP). IEEE, pp. 3645–3649

  22. Yu K and Yun F 2018 Human action recognition and prediction: A survey. arXiv preprintarXiv:1806.11230

  23. Hong-Bo Z, Yi-Xiang Z, Bineng Z, Qing L, Lijie Y, Ji-Xiang D and Duan-Sheng C 2019 A comprehensive survey of vision-based human action recognition methods. Sensors, 19: 1005

    Article  Google Scholar 

  24. Moez B, Franck M, Christian W, Christophe G, and Atilla B 2011 Sequential deep learning for human action recognition. In: International workshop on human behavior understanding. Springer, pp. 29–39

  25. Gao H, Zhuang L, Laurens Van D M, and Kilian Q W 2017 Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708

  26. Sepp H and Jürgen S 1997 Long short-term memory. Neural computation, 9: 1735–1780

    Article  Google Scholar 

  27. Christian S, Ivan L and Barbara C 2004 Recognizing human actions: a local svm approach. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.. volume 3, IEEE, pp. 32–36

  28. Fatma A H, Kashif H, Essam H H, Mai S M, and Walid Al-A 2021 Archimedes optimization algorithm: a new metaheuristic algorithm for solving optimization problems. Applied Intelligence, 51: 1531–1551

    Article  Google Scholar 

  29. Hazim N G and Károly J 2020 Dynamic differential annealed optimization: New metaheuristic optimization algorithm for engineering applications. Applied Soft Computing, 93: 106392

    Article  Google Scholar 

  30. Tong Y, Maosen X, Caiwen M, Chao S and Peng L 2014 Object based video synopsis. In: 2014 IEEE workshop on advanced research and technology in industry applications (WARTIA). IEEE, pp. 1138–1141

  31. Saleh S, Mohammad Sadegh H, and Akbar R 2017 A fuzzy multi-objective hybrid tlbo–pso approach to select the associated genes with breast cancer. Signal processing, 131: 58–65

    Article  Google Scholar 

  32. Yanhui C and Dengxu H 2021 A hybrid whale optimization with seagull algorithm for global optimization problems. Mathematical Problems in Engineering

  33. Reza A, Ramin H, Koorush Z, and Bahareh H 2012 A multi-objective artificial bee colony algorithm. Swarm and Evolutionary Computation, 2: 39–52

    Article  Google Scholar 

  34. Mona M M and Rasha S 2021 Object-based video synopsis approach using particle swarm optimization. Signal, Image and Video Processing, 15: 761–768

    Article  Google Scholar 

  35. Alex R, Yael P, and Shmuel P 2006 Making a long video short: Dynamic video synopsis. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). IEEE, volume 1, pp. 435–441

  36. Gaurav D and Vijay K 2019 Seagull optimization algorithm: Theory and its applications for large-scale industrial engineering problems. Knowledge-Based Systems, 165: 169–196

    Article  Google Scholar 

  37. R Venkata R 2016 Teaching-learning-based optimization algorithm. In: Teaching learning based optimization algorithm. Springer, pp. 9–39

  38. Seyedali M and Andrew L 2016 The whale optimization algorithm. Advances in engineering software, 95: 51–67

    Article  Google Scholar 

  39. George A A and Steven L F 2007 How many objects can you track?: Evidence for a resource-limited attentive tracking mechanism. Journal of vision, 7: 14–14

    Google Scholar 

  40. Patrick P, Michel G and Andrew B 2003 Poisson image editing. In: ACM SIGGRAPH 2003 Papers, pp. 313–318

  41. Yan Y, Elena A and Jill B Joint video exploration team (jvet) of itu-t sg 16 wp 3 and iso/iec jtc 1/sc 29/wg 11.

  42. Eric W W 2005 Spherical coordinates. https://mathworld.wolfram.com/

  43. Xinwei C, Ali T Z K and Walid S 2020 Deep learning for content-based personalized viewport prediction of 360-degree vr videos. IEEE Networking Letters, 2: 81–84

    Article  Google Scholar 

  44. Cort J W and Kenji M 2005 Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate research, 30: 79–82

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Puducherry, Puducherry, India

    S PRIYADHARSHINI & ANSUMAN MAHAPATRA

Authors
  1. S PRIYADHARSHINI
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. ANSUMAN MAHAPATRA
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S PRIYADHARSHINI.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

PRIYADHARSHINI, S., MAHAPATRA, A. PanoSyn: immersive video synopsis for spherical surveillance video. Sādhanā 47, 167 (2022). https://doi.org/10.1007/s12046-022-01937-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12046-022-01937-9

Keywords

Search

Navigation