Skip to main content
SpringerLink
Log in

Interactive visualization-based surveillance video synopsis

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Video synopsis is an effective technique for the efficient analysis of long videos in a short time. To generate a compact video, multiple tracks of moving objects, which we call as tubes are displayed simultaneously by rearranging them along the time axis. Contemporaneous video synopsis approaches focus on collision avoidance, or preservation of chronological order among tubes. However, generation of an adaptive personalized user-oriented synopsis video congruent to users’ preferences has yet not been thoroughly experimented. This paper propounds a framework for personalized visualization of synopsis video, integrating pertinent object attributes such as color, type, size, speed, travel path and direction towards generation of synopsis video for precise inference of user needs. The framework motivates users to interactively define queries for creation of the targeted synopsis. User queries are classified into visual-queries, temporal-queries, spatial-queries, and spatio-temporal queries concomitant with the visual and spatio-temporal attributes. Tubes relevant to a user-query are selected, and grouped according to original behavioral interactions followed by their rearrangement, to generate synopsis video with fewer false collisions. To evaluate the proffered technique, two evaluation metrics are proposed and extensive experiments of publicly available surveillance videos are conducted. The experimental results demonstrate the propriety and usability of the newer approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Chamasemani F F, Affendey L S, Mustapha N, Khalid F (2018) Video abstraction using density-based clustering algorithm. Vis Comput 34(10):1299–1314

    Article  Google Scholar 

  2. Nguyen H T, Jung S W, Won C S (2016) Order-preserving condensation of moving objects in surveillance videos. IEEE Trans Intell Transp Syst 17(9):2408–2418

    Article  Google Scholar 

  3. Murugan A S, Devi K S, Sivaranjani A, Srinivasan P (2018) A study on various methods used for video summarization and moving object detection for video surveillance applications. Multimed Tools Appl 77(18):23,273–23,290

    Article  Google Scholar 

  4. Elharrouss O, Almaadeed N, Al-Maadeed S, Bouridane A, Beghdadi A (2020) A combined multiple action recognition and summarization for surveillance video sequences. Appl Intell:1–23

  5. Rav-Acha A, Pritch Y, Peleg S (2006) Making a long video short: Dynamic video synopsis. In: IEEE Computer society conference on computer vision and pattern recognition (CVPR’06), vol 1, pp 435–441

  6. Pritch Y, Rav-Acha A, Gutman A, Peleg S (2007) Webcam synopsis: Peeking around the world. In: IEEE 11th International Conference on Computer Vision, pp 1–8

  7. Pritch Y, Rav-Acha A, Peleg S (2008) Nonchronological video synopsis and indexing. IEEE Trans Pattern Anal Mach Intell 30(11):1971–1984

    Article  Google Scholar 

  8. Namitha K, Narayanan A (2018) Video synopsis: State-of-the-art and research challenges. In: 2018 International conference on circuits and systems in digital enterprise technology (ICCSDET). IEEE, pp 1–10

  9. Li X, Wang Z, Lu X (2015) Surveillance video synopsis via scaling down objects. IEEE Trans Image Process 25(2):740– 755

    Article  MathSciNet  Google Scholar 

  10. He Y, Qu Z, Gao C, Sang N (2016) Fast online video synopsis based on potential collision graph. IEEE Signal Process Lett 24(1):22–26

    Article  Google Scholar 

  11. He Y, Gao C, Sang N, Qu Z, Han J (2017) Graph coloring based surveillance video synopsis. Neurocomputing 225:64–79

    Article  Google Scholar 

  12. Nie Y, Li Z, Zhang Z, Zhang Q, Ma T, Sun H (2019) Collision-free video synopsis incorporating object speed and size changes. IEEE Trans Image Process 29:1465–1478

    Article  MathSciNet  Google Scholar 

  13. Ra M, Kim W Y (2018) Parallelized tube rearrangement algorithm for online video synopsis. IEEE Signal Process Lett 25(8):1186–1190

    Article  Google Scholar 

  14. Ruan T, Wei S, Li J, Zhao Y (2019) Rearranging online tubes for streaming video synopsis: A dynamic graph coloring approach. IEEE Transactions on Image Processing

  15. Ghatak S, Rup S, Majhi B, Swamy M (2019) An improved surveillance video synopsis framework: a HSATLBO optimization approach. Multimed Tools Appl:1–33

  16. Moussa M M, Shoitan R (2020) Object-based video synopsis approach using particle swarm optimization. SIViP:1–8

  17. Zhu J, Liao S, Li S Z (2016) Multicamera joint video synopsis. IEEE Trans Circ Syst Video Technol 26(6):1058–1069

    Article  Google Scholar 

  18. Zhang Z, Nie Y, Sun H, Zhang Q, Lai Q, Li G, Xiao M (2019) Multi-view video synopsis via simultaneous object-shifting and view-switching optimization. IEEE Trans Image Process 29:971–985

    Article  MathSciNet  Google Scholar 

  19. Nie Y, Xiao C, Sun H, Li P (2012) Compact video synopsis via global spatiotemporal optimization. IEEE Trans Vis Comput Graph 19(10):1664–1676

    Article  Google Scholar 

  20. Pritch Y, Ratovitch S, Hendel A, Peleg S (2009) Clustered synopsis of surveillance video. In: Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp 195–200

  21. Zhu J, Feng S, Yi D, Liao S, Lei Z, Li S Z (2014) High-performance video condensation system. IEEE Trans Circ Syst Video Technol 25(7):1113–1124

    Google Scholar 

  22. Ahmed A, Kar S, Dogra D P, Patnaik R, Lee S, Choi H, Kim I (2017) Video synopsis generation using spatio-temporal groups. In: 2017 IEEE International conference on signal and image processing applications (ICSIPA). IEEE, pp 512–517

  23. Li X, Wang Z, Lu X (2018) Video synopsis in complex situations. IEEE Trans Image Process 27(8):3798–3812

    Article  MathSciNet  Google Scholar 

  24. Chou CL, Lin CH, Chiang TH, Chen HT, Lee SY (2015) Coherent event-based surveillance video synopsis using trajectory clustering. In: IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp 1–6

  25. Lin W, Zhang Y, Lu J, Zhou B, Wang J, Zhou Y (2015) Summarizing surveillance videos with local-patch-learning-based abnormality detection, blob sequence optimization, and type-based synopsis. Neurocomputing 155:84–98

    Article  Google Scholar 

  26. Wang W C, Chung P C, Huang C R, Huang WY (2017) Event based surveillance video synopsis using trajectory kinematics descriptors. In: 2017 Fifteenth IAPR international conference on machine vision applications (MVA). IEEE, pp 250–253

  27. Ahmed S A, Dogra D P, Kar S, Patnaik R, Lee S C, Choi H, Nam GP, Kim IJ (2019) Query-based video synopsis for intelligent traffic monitoring applications. IEEE Transactions on Intelligent Transportation Systems

  28. Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: CVPR, IEEE, pp 3457–3464

  29. Pérez-Hernández F, Tabik S, Lamas A, Olmos R, Fujita H, Herrera F (2020) Object detection binary classifiers methodology based on deep learning to identify small objects handled similarly: Application in video surveillance. Knowl-Based Syst 194:105,590

    Article  Google Scholar 

  30. Haritha H, Thangavel S K (2019) A modified deep learning architecture for vehicle detection in traffic monitoring system. Int J Comput Appl:1–10

  31. Yu X, Ye X, Gao Q (2020) Infrared handprint image restoration algorithm based on apoptotic mechanism, vol 8

  32. Liu X, Zhu X, Li M, Wang L, Zhu E, Liu T, Kloft M, Shen D, Yin J, Gao W (2019) Multiple kernel k k-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 42(5):1191–1204

    Google Scholar 

  33. Aarthi R, Amudha J, Boomika K, Varrier A (2016) Detection of moving objects in surveillance video by integrating bottom-up approach with knowledge base. Procedia Comput Sci 78:160– 164

    Article  Google Scholar 

  34. Subbiah U, Kumar D K, Thangavel S K, Parameswaran L (2020) An extensive study and comparison of the various approaches to object detection using deep learning. In: 2020 International conference on smart electronics and communication (ICOSEC). IEEE, pp 183–194

  35. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv

  36. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  37. Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. In: In: 2017 IEEE international conference on image processing (ICIP). IEEE, pp 3645–3649

  38. Tang Y, Zhang H, Xu B (2015) Metadata organization and retrieval with attribute tree for large-scale traffic surveillance videos. In: International Conference on Big Data Computing and Communications. Springer, pp 434–443

  39. Castañón G, Elgharib M, Saligrama V, Jodoin P M (2015) Retrieval in long-surveillance videos using user-described motion and object attributes. IEEE Trans Circ Syst Video Technol 26(12):2313–2327

    Article  Google Scholar 

  40. Momin BF, Mujawar TM (2015) Vehicle detection and attribute based search of vehicles in video surveillance system. In: 2015 International Conference on Circuits, Power and Computing Technologies [ICCPCT-2015]. IEEE, pp 1–4

  41. Thomas S S, Gupta S, Subramanian V K (2016) Perceptual synoptic view of pixel, object and semantic based attributes of video. J Vis Commun Image Represent 38:367–377

    Article  Google Scholar 

  42. Kamat V (1993) A survey of techniques for simulation of dynamic collision detection and response. Comput Graph 17(4):379– 385

    Article  Google Scholar 

  43. Kirkpatrick S, Gelatt C D, Vecchi M P (1983) Optimization by simulated annealing. Science 220(4598):671–680

    Article  MathSciNet  Google Scholar 

  44. Kolmogorov V, Zabih R (2002) What energy functions can be minimized via graph cuts? In: European conference on computer vision. Springer, pp 65–81

  45. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph (TOG) 22(3):313–318

    Article  Google Scholar 

  46. Baskurt K B, Samet R (2019) Video synopsis: a survey. Comput Vis Image Underst 181:26–38

    Article  Google Scholar 

  47. Guerrero-Gomez-Olmedo R, Lopez-Sastre R J, Maldonado-Bascon S, Fernandez-Caballero A (2013) Vehicle tracking by simultaneous detection and viewpoint estimation. In: IWINAC 2013, Part II, LNCS, vol 7931, pp 306–316

  48. Jodoin JP, Bilodeau GA, Saunier N (2014) Urban tracker: Multiple object tracking in urban mixed traffic. In: IEEE Winter Conference on Applications of Computer Vision. IEEE, pp 885–892

  49. Branch HOSD (2006) Imagery library for intelligent detection systems (i-lids). In: 2006 IET Conference on Crime and Security. IET, pp 445–448

  50. Fisher R, Santos-Victor J, Crowley J (2003) Ec funded caviar project/ist 2001 37540 Available at http://homepages.inf.ed.ac.uk/rbf/CAVIAR/ (accessed 21 March 2020)

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Amrita Vishwa Vidyapeetham, Amritapuri, India

    K. Namitha & M. Geetha

  2. Kimball Electronics (India), Technopark Campus, Thiruvananthapuram, India

    Athi Narayanan

Authors
  1. K. Namitha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Athi Narayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Geetha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. Namitha.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Namitha, K., Narayanan, A. & Geetha, M. Interactive visualization-based surveillance video synopsis. Appl Intell 52, 3954–3975 (2022). https://doi.org/10.1007/s10489-021-02636-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02636-4

Keywords

Search

Navigation