Abstract
Several computer vision applications such as e-learning, video editing, video compression, video-on-demand and surveillance etc. are popular in recent days. Most of the applications need videos to be retrieved and processed regularly. First and foremost step towards video retrieval and management is keyframe extraction. The perfect identification of shot transition boundaries is trivial in extracting keyframes. In present article, a framework for shot transition detection and keyframe extraction have been proposed. The proposed method is efficient, simple and does not require supervision which makes it attractive. The proposed method establishes the shot transition boundaries by estimating feature similarity (FSIM) between gradient magnitudes of consecutive frames. Then the frame with the highest mean and standard deviation is chosen as keyframe to that shot. In any situation if one feature fails to establish shot transition boundary another feature may succeed in establishment of shot transition boundary at proper frame locations of video. The proposed algorithm is tested on four different datasets, among them one is developed by us, two are well known standard datasets to evaluate keyframe extraction algorithm and the other one is standard surveillance video dataset. All the datasets are publicly available. Performance evaluation of the method is done in terms of Figure of merit, Detection percentage, Accuracy and Missing factor. The experimental results prove that the proposed method outperforms other state-of-art methods.
Similar content being viewed by others
References
Ayadi T, Ellouze M, Hamdani TM, Alimi AM (2013 Jun 1) Movie scenes detection with MIGSOM based on shots semi-supervised clustering. Neural Comput Appl 22(7–8):1387–1396
Birinci M, Kiranyaz S (2014 Mar 1) A perceptual scheme for fully automatic video shot boundary detection. Signal Process: Image Commun 29(3):410–423
Bommisetty RM, Prakash O, Khare A (2019) Keyframe extraction using Pearson correlation coefficient and color moments. Multimedia Systems:1-33. https://doi.org/10.1007/s00530-019-00642-8
Chen J, Ren J, Jiang J (2011) Modelling of content-aware indicators for effective determination of shot boundaries in compressed MPEG videos. Multimed Tools Appl 54(2):219–239
Dutta D, Saha SK, Chanda B (2016) A shot detection technique using linear regression of shot transition pattern. Multimed Tools Appl 75(1):93–113
Fei M, Jiang W, Mao W, Song Z (2016) New fusional framework combining sparse selection and clustering for key frame extraction. IET Comput Vis 10(4):280–288
Ferreira L, da Silva Cruz LA, Assuncao P (2016) Towards key-frame extraction methods for 3D video: a review. EURASIP J Image Video Process 2016(1):28
Gao G, Ma H (2014) To accelerate shot boundary detection by reducing detection region and scope. Multimed Tools Appl 71(3):1749–1770
Hannane R, Elboushaki A, Afdel K, Naghabhushan P, Javed M (2016 Jun 1) An efficient method for video shot boundary detection and keyframe extraction using SIFT-point distribution histogram. Int J Multimed Inform Retriev 5(2):89–104
https://computervisiononline.com/dataset/ (2020) Accessed on 18th August 2020.
https://www.sites.google.com/site/vsummsite/download. Accessed 15 Jan 2021
Hu W, Jin Y, Wen Y, Wang Z, Sun L (2017) Towards wi-fi ap-assisted content prefetching for on-demand tv series: A learning-based approach. IEEE Trans Circuits Syst Video Technol 28(7):1665–1676
Huang CR, Lee HP, Chen CS (2014) Shot change detection via local keypoint matching. IEEE Trans Multimed 10(6):1097–1108
Ioannidis A, Chasanis V, Likas A (2016 Mar 1) Weighted multi-view key-frame extraction. Pattern Recognition Lett 72:52–61
Jadhav MP, Jadhav DS (2015 Jan 1) Video Summarization Using Higher Order Color Moments (VSUHCM). Procedia Comput Sci 45:275–281
Kovesi P (1999) Image features from phase congruency. Videre: J Comp Vis Res, 1(3):1–26
Kumar K, Shrimankar DD, Singh N (2018 Mar 1) Eratosthenes sieve based key-frame extraction technique for event summarization in videos. Multimed Tools Appl 77(6):7383–7404
Lee (2018) VirtualDub home page. http://www.virtualdub.org/index.html. Accessed 27 Sept 2018
Lee H, Yu J, Im Y, Gil JM, Park D (2011) A unified scheme of shot boundary detection and anchor shot detection in news video story parsing. Multimed Tools Appl 51(3):1127–1145
Li Z, Liu G (2008) A novel scene change detection algorithm based on the 3D wavelet transform. In: Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on 2008 Oct 12. IEEE, pp 1536–1539
Liu H, Hao H (2014) Key frame extraction based on improved hierarchical clustering algorithm. In: Fuzzy Systems and Knowledge Discovery (FSKD), 2014 11th International Conference on 2014 Aug 19. IEEE, pp 793–797
Liu H, Meng W, Liu Z (2012) Key frame extraction of online video based on optimized frame difference. In: Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th International Conference on 2012 May 29. IEEE, pp 1238–1242
Liu H, Pan L, Meng W (2012) Key frame extraction from online video based on improved frame difference optimization. In: Communication Technology (ICCT), 2012 IEEE 14th International Conference on 2012 Nov 9. IEEE, pp 940–944
Liu XM, Hao AM, Zhao D (2013 Jan 1) Optimization-based key frame extraction for motion capture animation. Visual Comput 29(1):85–95
Lu ZM, Shi Y (2013 Dec 1) Fast video shot boundary detection based on SVD and pattern matching. IEEE Trans Image Process 22(12):5136–5145
Lu G, Zhou Y, Li X, Yan P (2017 Mar 1) Unsupervised, efficient and scalable key-frame selection for automatic summarization of surveillance videos. Multimed Tools Appl 76(5):6309–6331
Mohanta PP, Saha SK, Chanda B (2012 Feb) A model-based shot boundary detection technique using frame transition parameters. IEEE Trans Multimed 14(1):223–233
Mounika (2020) https://sites.google.com/site/mounikabrv3/research-profile, Accessed on 19th August2020
Mundur P, Rao Y, Yesha Y (2006 Apr 1) Keyframe-based video summarization using Delaunay clustering. Int J Digital Lib 6(2):219–232
Poornima K, Kanchana R (2012) A method to align images using image segmentation. Int J Soft Comput Eng 2(1):294–298
Shaker IF, Abd-Elrahman A, Abdel-Gawad AK, Sherief MA (2011 Apr 12) Building extraction from high resolution space images in high density residential areas in the Great Cairo region. Remote Sens 3(4):781–791
Sheena CV, Narayanan NK (2015 Jan 1) Key-frame extraction by analysis of histograms of video frames using statistical methods. Procedia Comput Sci 70:36–40
Shi Y, Yang H, Gong M, Liu X, Xia Y (2017) A fast and robust key frame extraction method for video copyright protection. J Electric Comput Eng 2017(3):1–7
Thakre KS, Rajurkar AM, Manthalkar RR (2016 Jan 1) Video Partitioning and Secured Keyframe Extraction of MPEG Video. Procedia Comput Sci 78:790–798
Warhade KK, Merchant SN, Desai UB (2011 Nov 1) Shot boundary detection in the presence of fire flicker and explosion using stationary wavelet transform. Signal Image Video Process 5(4):507–515
Yu L, Cao J, Chen M, Cui X (2018 Sep 1) Key frame extraction scheme based on sliding window and features. Peer-to-Peer Netw Appl 11(5):1141–1152
Zhang L, Zhang L, Mou X, Zhang D (2011 Jan 31) FSIM: A feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Mounika Bommisetty, R., Khare, A., Siddiqui, T.J. et al. Fusion of gradient and feature similarity for Keyframe extraction. Multimed Tools Appl 80, 15429–15467 (2021). https://doi.org/10.1007/s11042-020-10390-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10390-x