Skip to main content
SpringerLink
Log in

Retrospective analysis of time series for frame selection in surveillance video summarization

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In recent days, we have witnessed a dramatical growth of videos in various real-life scenarios. In this paper, we address the problem of surveillance video summarization. We present a new method of key-frame selection for this task: By virtue of retrospective analysis of time series, temporal cuts are first detected by sequentially measuring dissimilarities on a given video with threshold-based decision making; then, with the detected cuts, the video is segmented into a number of consecutive clips containing similar video contents; key frames are last selected by performing a typical clustering procedure in each resulted clip for final video summary. We have conducted extensive experiments on the benchmarking ViSOR dataset and the publicly available IVY LAB dataset. Excellent performances outperforming state-of-the-art competitors were demonstrated on key-frame selection for surveillance video summarization, which suggests good potentials of the proposed method in real-world applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Here, without going into details, the use of clustering for key-frame selection in a resulted clip is straightforward.

  2. http://www.openvisor.org/videocategories.asp.

  3. http://ivylab.kaist.ac.kr/demo/vs/dataset.htm.

References

  1. Chua, J.L., Chang, Y.C., Lim, W.K.: A simple vision-based fall detection technique for indoor video surveillance. SIViP 9(3), 623–633 (2015)

    Article  Google Scholar 

  2. Sharma, R.A., Gandhi, V., Chari, V., Jawahar, C.V.: Automatic analysis of broadcast football videos using contextual priors. SIViP (2016). doi:10.1007/s11760-016-0916-3

  3. Jiang, W., Cotton, C., Loui, A.C.: Automatic consumer video summarization by audio and visual analysis. In: International Conference of Multimedia and Expo (ICME), pp. 1–6. IEEE (2011)

  4. Liu, H., Meng, W., Liu, Z.: Key frame extraction of online video based on optimized frame difference. In: 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), pp. 1238–1242. IEEE (2012)

  5. Essa, A., Sidike, P., Asari, V.: A modular approach for key-frame selection in wide area surveillance video analysis. In: 2015 National Aerospace and Electronics Conference (NAECON), pp. 41–44. IEEE (2015)

  6. Kuanar, S.K., Panda, R., Chowdhury, A.S.: Video key frame extraction through dynamic Delaunay clustering with a structural constraint. J. Vis. Commun. Image Represent. 24(7), 1212–1227 (2013)

    Article  Google Scholar 

  7. Zhang, Q., Yu, S., Zhou, D.S., Wei, X.P.: An efficient method of key-frame extraction based on a cluster algorithm. J. Hum. Kinet. 39(1), 5–14 (2013)

    Article  Google Scholar 

  8. Mahmoud, K.M., Ghanem, N.M., Ismail, M.A.: Unsupervised video summarization via dynamic modeling-based hierarchical clustering. In: 12th International Conference on Machine Learning and Applications (ICMLA), vol. 2, pp. 303–308. IEEE (2013)

  9. Ejaz, N., Mehmood, I., Baik, S.W.: Efficient visual attention based framework for extracting key frames from videos. Signal Process. Image Commun. 28(1), 34–44 (2013)

    Article  Google Scholar 

  10. Potapov, D., Douze, M., Harchaoui, Z., Schmid, C.: Category-specific video summarization. In: European Conference on Computer Vision (ECCV), pp. 540–555. Springer International Publishing (2014)

  11. Evangelio, R.H., Senst, T., Keller, I., Sikora, T.: Video indexing and summarization as a tool for privacy protection. In: 18th International Conference on Digital Signal Processing (DSP), pp. 1–6. IEEE (2013)

  12. Elhamifar, E., Sapiro, G., Vidal, R.: See all by looking at a few: sparse modeling for finding representative objects. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1600–1607. IEEE (2012)

  13. Tu, Z., Sun, D., Luo, B.: Video summarization by robust low-rank subspace segmentation. In: Proceedings of The Eighth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA), pp. 929–937. Springer, Berlin (2013)

  14. Zhang, X., Sun, F., Liu, G., Ma, Y.: Fast low-rank subspace segmentation. IEEE Trans Knowl. Data Eng. 26(5), 1293–1297 (2014)

    Article  Google Scholar 

  15. Lu, G., Zhou, Y., Li, X., Yan, P.: Unsupervised, efficient and scalable key-frame selection for automatic summarization of surveillance videos. Multimedia Tools and Applications. (2016). doi:10.1007/s11042-016-3263-z

  16. Liu, S., Yamada, M., Collier, N., Sugiyama, M.: Change-point detection in time-series data by relative density-ratio estimation. Neural Netw. 43, 72–83 (2013)

    Article  MATH  Google Scholar 

  17. Das, J., Roy, H.: Human face detection in color images using HSV color histogram and WLD. In: 2014 International Conference on Computational Intelligence and Communication Networks (CICN), pp. 198–202. IEEE (2014)

  18. Lu, G., Kudo, M., Toyama, J.: Hierarchical foreground detection in dynamic background. In International Conference on Computer Analysis of Images and Patterns, pp. 413–420. Springer, Berlin (2011)

  19. De Avila, S.E.F., Lopes, A.P.B., da Luz, A., de Albuquerque Arajo, A.: VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recognit. Lett. 32(1), 56–68 (2011)

    Article  Google Scholar 

  20. Barbic, J., Safonova, A., Pan, J.Y., Faloutsos, C., Hodgins, J.K., Pollard, N.S.: Segmenting motion capture data into distinct behaviors. In: Proceedings of Graphics Interface, pp. 185–194. Canadian Human–Computer Communications Society (2004)

  21. Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. J. Mach. Learn. Res. 13(1), 723–773 (2012)

    MathSciNet  MATH  Google Scholar 

  22. Hofmann, T., Schölkopf, B., Smola, A.J.: Kernel methods in machine learning. Ann. Stat. 36(3), 1171–1220 (2008)

  23. Smola, A., Gretton, A., Song, L., Schölkopf, B.: A Hilbert space embedding for distributions. In: International Conference on Algorithmic Learning Theory, pp. 13–31. Springer, Berlin (2007)

  24. Gretton, A., Borgwardt, K., Rasch, M.J., Schölkopf, B., Smola, A.J.: A kernel method for the two-sample problem. In: Proceedings of Advances in neural information processing systems (NIPS), pp. 513–520. MIT Press, Cambridge (2006)

  25. Vezzani, R., Cucchiara, R.: Video surveillance online repository (visor): an integrated framework. Multimed. Tools Appl. 50(2), 359–380 (2010)

    Article  Google Scholar 

  26. Sohn, H., De Neve, W., Ro, Y.M.: Privacy protection in video surveillance systems: analysis of subband-adaptive scrambling in JPEG XR. IEEE Trans. Circuits Syst. Video Technol. 21(2), 170–177 (2011)

    Article  Google Scholar 

  27. Zhang, Y., Kwong, S., Jiang, G., Wang, X., Yu, M.: Statistical early termination model for fast mode decision and reference frame selection in multiview video coding. IEEE Trans. Broadcast. 58(1), 10–23 (2012)

    Article  Google Scholar 

Download references

Acknowledgments

The work is supported by National Natural Science Foundation of China (61403232, 61327003), Natural Science Foundation of Shandong Province, China (ZR2014FQ025), and the Open Projects Program of National Laboratory of Pattern Recognition (NLPR) of China (20147346).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shandong University, Jinan, China

    Zhen Gao, Guoliang Lu & Peng Yan

  2. Key Laboratory of High-efficiency and Clean Mechanical Manufacture, Ministry of Education, Jinan, China

    Zhen Gao, Guoliang Lu & Peng Yan

  3. National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China

    Liang Wang

Authors
  1. Zhen Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guoliang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guoliang Lu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Z., Lu, G., Yan, P. et al. Retrospective analysis of time series for frame selection in surveillance video summarization. SIViP 11, 581–588 (2017). https://doi.org/10.1007/s11760-016-0997-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-016-0997-z

Keywords

Search

Navigation