Replay and key-events detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine

  • Ali Javed
  • Aun Irtaza
  • Yasmeen Khaliq
  • Hafiz Malik
  • Muhammad Tariq MahmoodEmail author


Sports broadcasters generate enormous amount of video content viewed all over the world. To capture the user interests in the rebroadcasted content, the sports videos are summarized that need the manual inspection and analysis. However, the huge repository and long duration of videos make manual analysis and summarization a laborious and time-consuming job. To overcome this problem, efforts have been made for automatic video summarization. In this paper, a novel framework to summarize sports videos is presented. It has been observed that the replays within a sports video represent key-events and these events can be used for video summarization. It has been noted that replays are usually sandwiched between start and stop of gradual-transitions. A thresholding-based approach is used to identify gradual transition effect (i.e. fade-in, fade-out) in sports video. The Gaussian mixture model (GMM) is then applied to key-event candidates to extract silhouettes and generate motion history image (MHI) for each key-event. The MHIs are processed using Confined Elliptical Local Ternary Patterns (CE-LTPs) for feature extraction. Extreme learning machine (ELM) classifier is used to learn the underlying model for events. A trained ELM-based classifier is then used for key-event detection. The output of classifier is then used for key-event labeling, replay detection, and complete game summarization. Performance of the proposed framework is evaluated on a dataset consisting of 20 videos of four different sports. Experimental results indicate the effectiveness of the proposed framework in terms of replays and key-events detection from selected dataset.


Gradual transition Highlights Replay detection Events detection 



This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2016R1D1A1B03933860).


  1. 1.
    Panda R, Roy-Chowdhury AK (2017) Multi-view surveillance video summarization via joint embedding and sparse optimization. IEEE Trans Multimedia 19(9):2010–2021Google Scholar
  2. 2.
    Muhammad K, Ahmad J, Sajjad M, Baik SW (2016) Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems. Springer Plus 5(1):1495Google Scholar
  3. 3.
    Tran QD, Hwang D, Lee OJ, Jung JE (2017) Exploiting character networks for movie summarization. Multimed Tools Appl 76(8):10357–10369Google Scholar
  4. 4.
    Varini P, Serra G, Cucchiara R (2017) Personalized egocentric video summarization of cultural tour on user preferences input. IEEE Trans Multimed 19(12):2832Google Scholar
  5. 5.
    Javed A, Bajwa KB, Malik H, Irtaza A, Mahmood MT (2016) A hybrid approach for summarization of cricket videos. In: IEEE International conference on consumer electronics-Asia (ICCE-Asia). IEEE, pp 1–4Google Scholar
  6. 6.
    Javed A, Bajwa KB, Malik H, Irtaza A (2016) An efficient framework for automatic highlights generation from sports videos. IEEE Signal Process Lett 23(7):954–958Google Scholar
  7. 7.
    Li B, Pan H, Sezan I (2003) A general framework for sports video summarization with its application to soccer.. In: ICASSP, 2003: Proceedings of 30th IEEE international conference on acoustics, speech and signal processing; 2003 April 6; Hong Kong. IEEE, pp 169–172Google Scholar
  8. 8.
    Hung MH, Hsieh CH (2008) Event detection of broadcast baseball videos. IEEE Trans Circ Syst Vid Technol 18(12):1713–1726Google Scholar
  9. 9.
    Kolekar MH, Sengupta S (2015) Bayesian network-based customized highlight generation for broadcast soccer videos. IEEE Trans Broadcast 61(2):195–209Google Scholar
  10. 10.
    Chang P, Han M, Gong Y (2002) Extract highlights from baseball game video with hidden Markov models. In: ICIP, 2002: Proceedings of 9th IEEE international conference on image processing; 2002 Sep 22-25; Pittsburgh, USA. IEEE, pp 609–612Google Scholar
  11. 11.
    Jiang H, Lu Y, Xue J (2016) Automatic soccer video event detection based on a deep neural network combined CNN and RNN. In: Proc int conf in tools with artificial intelligence, San Jose, CA, USA, November 2016. IEEE, pp 490–494Google Scholar
  12. 12.
    Pan H, Van Beek P, Sezan MI (2001) Detection of slow-motion replay segments in sports video for highlights generation. In: ICASSP, 2001: Proceedings of 28th IEEE international conference on acoustics, speech and signal processing; 2001 May 7-11; Utah, USA. IEEE, pp 1649–1652Google Scholar
  13. 13.
    Pan H, Li B, Sezan MI (2002) Detection of slow-motion replay segments in sports video for highlights generation. In: ICASSP, 2001: Proceedings of 28th IEEE international conference on acoustics, speech and signal processing; 2001 May 7-11; Utah, USA. IEEE, pp 1649–1652Google Scholar
  14. 14.
    Tavassolipour M, Karimian M, Kasaei S (2014) Event detection and summarization in soccer videos using Bayesian network and copula. IEEE Trans Circ Syst Video Technol 24(2):291–304Google Scholar
  15. 15.
    Duan LY, Xu M, Tian Q, Xu CS et al (2004) Mean shift based video segment representation and applications to replay detection. In: ICASSP, 2004: Proceedings of 29th IEEE international conference on acoustics, speech and signal processing; 2004 May 17-21; Montreal, Canada. IEEE, pp 709–712Google Scholar
  16. 16.
    Soleymani M, Larson M, Pun T, Hanjalic A (2014) Corpus development for affective video indexing. IEEE Trans Multimed 16(4):1075–1089Google Scholar
  17. 17.
    Kapela R, McGuinness K, Connor NE (2017) Real-time field sports scene classification using colour and frequency space decompositions. J Real-Time Image Process 13(4):725–737Google Scholar
  18. 18.
    Zawbaa HM, El-Bendary N, Hassanien AE, Kim TH (2011) Machine learning-based soccer video summarization system. In: Multimedia, computer graphics and broadcasting. Springer, Berlin, pp 19–28Google Scholar
  19. 19.
    Wang L, Liu X, Lin S, Xu G, Shum HY (2004) Generic slow-motion replay detection in sports video. In: ICIP, 2004: Proceedings of 11th IEEE international conference on image processing; 2004 Oct 24-27; Singapore. IEEE, pp 1585–1588Google Scholar
  20. 20.
    Xu W, Yi Y (2011) A robust replay detection algorithm for soccer video. IEEE Signal Process Lett 18 (9):509–512Google Scholar
  21. 21.
    Zhao F, Dong Y, Wei Z, Wang H (2012) Matching logos for slow motion replay detection in broadcast sports video. In: ICASSP, 2012: Proceedings of 37th IEEE international conference on acoustics, speech and signal processing; 2012 Mar 25; Kyoto, Japan. IEEE, pp 1409–1412Google Scholar
  22. 22.
    Eldib MY, Zaid BSA, Zawbaa HM, El-Zahar M, El-Saban M (2009) Soccer video summarization using enhanced logo detection. In: ICIP, 2009: proceedings of 16th IEEE international conference on image processing; 2009 Nov 7-10; Cairo, Egypt. IEEE, pp 4345–4348Google Scholar
  23. 23.
    Su PC, Lan CH, Wu CS, Zeng ZX, Chen WY (2013) Transition effect detection for extracting highlights in baseball videos. EURASIP J Image Vid Process 2013(1):1–6Google Scholar
  24. 24.
    Wang J, Chng E, Xu C (2005) Soccer replay detection using scene transition structure analysis. In: ICASSP, 2005: Proceedings of 30th IEEE international conference on acoustics, speech, and signal processing; 2005 Mar 19-23; Philadelphia, PA, USA. IEEE, pp 433–436Google Scholar
  25. 25.
    Zhao Z, Jiang S, Huang Q, Zhu G (2006) Highlight summarization in sports video based on replay detection. In: ICME, 2006: Proceedings of international conference on multimedia and expo; 2006 Jul 9-12; Toronto, Canada. IEEE, pp 1613–1616Google Scholar
  26. 26.
    Chen CM, Chen LH (2015) A novel method for slow motion replay detection in broadcast basketball video. Multimed Tools Appl 74(21):9573–9593Google Scholar
  27. 27.
    Chen CM, Chen LH (2014) Novel framework for sports video analysis: a basketball case study. In: ICIP, 2014: Proceedings of international conference on image processing; 2014 Oct 27-30; Paris, France. IEEE, pp 961–965Google Scholar
  28. 28.
    Nguyen N, Yoshitaka A (2012) Shot type and replay detection for soccer video parsing. In: ISM, 2012: Proceedings of IEEE international symposium on multimedia; 2012 Dec 10-12; California, USA. IEEE, pp 344–347Google Scholar
  29. 29.
    Dang Z, Du J, Huang Q, Jiang S (2007) Replay detection based on semi-automatic logo template sequence extraction in sports video. In: ICIG 2007: Proceedings of 4th international conference on image and graphics; 2007 Aug 22; Sichuan, China. IEEE, pp 839–844Google Scholar
  30. 30.
    Li W, Chen S, Wang H (2009) A rule-based sports video event detection method. In: CISE, 2009: Proceedings of 21st IEEE international conference on computational intelligence and software engineering; 2009 Dec 11-13; Wuhan, China: IEEE, pp 1–4Google Scholar
  31. 31.
    Chen M, Wei X, Yang Q, Li Q, Wang G, Yang MH (2017) Spatiotemporal GMM for background subtraction with superpixel hierarchy. IEEE Trans Pattern Anal Mach Intell 40(6):1518Google Scholar
  32. 32.
    Bilen H, Fernando B, Gavves E, Vedaldi A (2017) Action recognition with dynamic image networks. IEEE Transactions on Pattern Analysis and Machine IntelligenceGoogle Scholar
  33. 33.
    Murala S, Maheshwari R, Balasubramanian R (2012) Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Trans Image Process 21(5):2874–2886MathSciNetzbMATHGoogle Scholar
  34. 34.
    Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybern) 42(2):513–529Google Scholar
  35. 35.
    Javed A, Malik H, Bajwa K, Irtaza A, Mahmood MT Data from: replay detection framework for automatic highlights generation from sports videos. Dryad Digital Repository.
  36. 36.
    Wilson S, Mohan CK, Murthy KS (2014) Event-based sports videos classification using HMM framework. In: Computer vision in sports. Springer, Cham, pp 229–244Google Scholar
  37. 37.
    Midhu K, Padmanabhan NA (2018) Highlight generation of cricket match using deep learning. In: Computational vision and bio inspired computing. Springer, Cham, pp 925–936Google Scholar
  38. 38.
    Wang Z, Yu J, He Y (2017) Soccer video event annotation by synchronization of attackdefense clips and match reports with coarse-grained time information. IEEE Trans Circ Syst Vid Technol 27(5):1104–1117Google Scholar
  39. 39.
    Godi M, Rota P, Setti F (2017) Indirect match highlights detection with deep convolutional neural networks. In: Proc int conf on image analysis and processing, Catania, Italy, September 2017, pp 87–96Google Scholar
  40. 40.
    Wang D, Xin J (2018) Emergent spatio-temporal multimodal learning using a developmental network. Appl Intell 1–18Google Scholar
  41. 41.
    Song X, Zhang W, Weng J (2015) Types, locations, and scales from cluttered natural video and actions. IEEE Trans Auton Ment Dev 7(4):273Google Scholar
  42. 42.
    Wang D, Wang J, Liu L (2017) Developmental network: an internal emergent object feature learning. Neural Process Lett 1–25Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Software Engineering DepartmentUniversity of Engineering and TechnologyTaxilaPakistan
  2. 2.Computer Science DepartmentUniversity of Engineering and TechnologyTaxilaPakistan
  3. 3.Computer Science DepartmentCOMSATS University Islamabad, Wah CampusWah CanttPakistan
  4. 4.College of Engineering and Computer ScienceUniversity of Michgan-DearbornDearbornUSA
  5. 5.School of Computer Science and EngineeringKorea University of Technology and EducationCheonanSouth Korea

Personalised recommendations