Skip to main content
SpringerLink
Log in

Replay and key-events detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Sports broadcasters generate enormous amount of video content viewed all over the world. To capture the user interests in the rebroadcasted content, the sports videos are summarized that need the manual inspection and analysis. However, the huge repository and long duration of videos make manual analysis and summarization a laborious and time-consuming job. To overcome this problem, efforts have been made for automatic video summarization. In this paper, a novel framework to summarize sports videos is presented. It has been observed that the replays within a sports video represent key-events and these events can be used for video summarization. It has been noted that replays are usually sandwiched between start and stop of gradual-transitions. A thresholding-based approach is used to identify gradual transition effect (i.e. fade-in, fade-out) in sports video. The Gaussian mixture model (GMM) is then applied to key-event candidates to extract silhouettes and generate motion history image (MHI) for each key-event. The MHIs are processed using Confined Elliptical Local Ternary Patterns (CE-LTPs) for feature extraction. Extreme learning machine (ELM) classifier is used to learn the underlying model for events. A trained ELM-based classifier is then used for key-event detection. The output of classifier is then used for key-event labeling, replay detection, and complete game summarization. Performance of the proposed framework is evaluated on a dataset consisting of 20 videos of four different sports. Experimental results indicate the effectiveness of the proposed framework in terms of replays and key-events detection from selected dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Panda R, Roy-Chowdhury AK (2017) Multi-view surveillance video summarization via joint embedding and sparse optimization. IEEE Trans Multimedia 19(9):2010–2021

  2. Muhammad K, Ahmad J, Sajjad M, Baik SW (2016) Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems. Springer Plus 5(1):1495

    Article  Google Scholar 

  3. Tran QD, Hwang D, Lee OJ, Jung JE (2017) Exploiting character networks for movie summarization. Multimed Tools Appl 76(8):10357–10369

    Article  Google Scholar 

  4. Varini P, Serra G, Cucchiara R (2017) Personalized egocentric video summarization of cultural tour on user preferences input. IEEE Trans Multimed 19(12):2832

    Article  Google Scholar 

  5. Javed A, Bajwa KB, Malik H, Irtaza A, Mahmood MT (2016) A hybrid approach for summarization of cricket videos. In: IEEE International conference on consumer electronics-Asia (ICCE-Asia). IEEE, pp 1–4

  6. Javed A, Bajwa KB, Malik H, Irtaza A (2016) An efficient framework for automatic highlights generation from sports videos. IEEE Signal Process Lett 23(7):954–958

    Article  Google Scholar 

  7. Li B, Pan H, Sezan I (2003) A general framework for sports video summarization with its application to soccer.. In: ICASSP, 2003: Proceedings of 30th IEEE international conference on acoustics, speech and signal processing; 2003 April 6; Hong Kong. IEEE, pp 169–172

  8. Hung MH, Hsieh CH (2008) Event detection of broadcast baseball videos. IEEE Trans Circ Syst Vid Technol 18(12):1713–1726

    Article  Google Scholar 

  9. Kolekar MH, Sengupta S (2015) Bayesian network-based customized highlight generation for broadcast soccer videos. IEEE Trans Broadcast 61(2):195–209

    Article  Google Scholar 

  10. Chang P, Han M, Gong Y (2002) Extract highlights from baseball game video with hidden Markov models. In: ICIP, 2002: Proceedings of 9th IEEE international conference on image processing; 2002 Sep 22-25; Pittsburgh, USA. IEEE, pp 609–612

  11. Jiang H, Lu Y, Xue J (2016) Automatic soccer video event detection based on a deep neural network combined CNN and RNN. In: Proc int conf in tools with artificial intelligence, San Jose, CA, USA, November 2016. IEEE, pp 490–494

  12. Pan H, Van Beek P, Sezan MI (2001) Detection of slow-motion replay segments in sports video for highlights generation. In: ICASSP, 2001: Proceedings of 28th IEEE international conference on acoustics, speech and signal processing; 2001 May 7-11; Utah, USA. IEEE, pp 1649–1652

  13. Pan H, Li B, Sezan MI (2002) Detection of slow-motion replay segments in sports video for highlights generation. In: ICASSP, 2001: Proceedings of 28th IEEE international conference on acoustics, speech and signal processing; 2001 May 7-11; Utah, USA. IEEE, pp 1649–1652

  14. Tavassolipour M, Karimian M, Kasaei S (2014) Event detection and summarization in soccer videos using Bayesian network and copula. IEEE Trans Circ Syst Video Technol 24(2):291–304

    Article  Google Scholar 

  15. Duan LY, Xu M, Tian Q, Xu CS et al (2004) Mean shift based video segment representation and applications to replay detection. In: ICASSP, 2004: Proceedings of 29th IEEE international conference on acoustics, speech and signal processing; 2004 May 17-21; Montreal, Canada. IEEE, pp 709–712

  16. Soleymani M, Larson M, Pun T, Hanjalic A (2014) Corpus development for affective video indexing. IEEE Trans Multimed 16(4):1075–1089

    Article  Google Scholar 

  17. Kapela R, McGuinness K, Connor NE (2017) Real-time field sports scene classification using colour and frequency space decompositions. J Real-Time Image Process 13(4):725–737

    Article  Google Scholar 

  18. Zawbaa HM, El-Bendary N, Hassanien AE, Kim TH (2011) Machine learning-based soccer video summarization system. In: Multimedia, computer graphics and broadcasting. Springer, Berlin, pp 19–28

  19. Wang L, Liu X, Lin S, Xu G, Shum HY (2004) Generic slow-motion replay detection in sports video. In: ICIP, 2004: Proceedings of 11th IEEE international conference on image processing; 2004 Oct 24-27; Singapore. IEEE, pp 1585–1588

  20. Xu W, Yi Y (2011) A robust replay detection algorithm for soccer video. IEEE Signal Process Lett 18 (9):509–512

    Article  Google Scholar 

  21. Zhao F, Dong Y, Wei Z, Wang H (2012) Matching logos for slow motion replay detection in broadcast sports video. In: ICASSP, 2012: Proceedings of 37th IEEE international conference on acoustics, speech and signal processing; 2012 Mar 25; Kyoto, Japan. IEEE, pp 1409–1412

  22. Eldib MY, Zaid BSA, Zawbaa HM, El-Zahar M, El-Saban M (2009) Soccer video summarization using enhanced logo detection. In: ICIP, 2009: proceedings of 16th IEEE international conference on image processing; 2009 Nov 7-10; Cairo, Egypt. IEEE, pp 4345–4348

  23. Su PC, Lan CH, Wu CS, Zeng ZX, Chen WY (2013) Transition effect detection for extracting highlights in baseball videos. EURASIP J Image Vid Process 2013(1):1–6

    Article  Google Scholar 

  24. Wang J, Chng E, Xu C (2005) Soccer replay detection using scene transition structure analysis. In: ICASSP, 2005: Proceedings of 30th IEEE international conference on acoustics, speech, and signal processing; 2005 Mar 19-23; Philadelphia, PA, USA. IEEE, pp 433–436

  25. Zhao Z, Jiang S, Huang Q, Zhu G (2006) Highlight summarization in sports video based on replay detection. In: ICME, 2006: Proceedings of international conference on multimedia and expo; 2006 Jul 9-12; Toronto, Canada. IEEE, pp 1613–1616

  26. Chen CM, Chen LH (2015) A novel method for slow motion replay detection in broadcast basketball video. Multimed Tools Appl 74(21):9573–9593

    Article  Google Scholar 

  27. Chen CM, Chen LH (2014) Novel framework for sports video analysis: a basketball case study. In: ICIP, 2014: Proceedings of international conference on image processing; 2014 Oct 27-30; Paris, France. IEEE, pp 961–965

  28. Nguyen N, Yoshitaka A (2012) Shot type and replay detection for soccer video parsing. In: ISM, 2012: Proceedings of IEEE international symposium on multimedia; 2012 Dec 10-12; California, USA. IEEE, pp 344–347

  29. Dang Z, Du J, Huang Q, Jiang S (2007) Replay detection based on semi-automatic logo template sequence extraction in sports video. In: ICIG 2007: Proceedings of 4th international conference on image and graphics; 2007 Aug 22; Sichuan, China. IEEE, pp 839–844

  30. Li W, Chen S, Wang H (2009) A rule-based sports video event detection method. In: CISE, 2009: Proceedings of 21st IEEE international conference on computational intelligence and software engineering; 2009 Dec 11-13; Wuhan, China: IEEE, pp 1–4

  31. Chen M, Wei X, Yang Q, Li Q, Wang G, Yang MH (2017) Spatiotemporal GMM for background subtraction with superpixel hierarchy. IEEE Trans Pattern Anal Mach Intell 40(6):1518

    Article  Google Scholar 

  32. Bilen H, Fernando B, Gavves E, Vedaldi A (2017) Action recognition with dynamic image networks. IEEE Transactions on Pattern Analysis and Machine Intelligence

  33. Murala S, Maheshwari R, Balasubramanian R (2012) Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Trans Image Process 21(5):2874–2886

    Article  MathSciNet  MATH  Google Scholar 

  34. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybern) 42(2):513–529

    Article  Google Scholar 

  35. Javed A, Malik H, Bajwa K, Irtaza A, Mahmood MT Data from: replay detection framework for automatic highlights generation from sports videos. Dryad Digital Repository. https://doi.org/10.5061/dryad.5b880

  36. Wilson S, Mohan CK, Murthy KS (2014) Event-based sports videos classification using HMM framework. In: Computer vision in sports. Springer, Cham, pp 229–244

  37. Midhu K, Padmanabhan NA (2018) Highlight generation of cricket match using deep learning. In: Computational vision and bio inspired computing. Springer, Cham, pp 925–936

  38. Wang Z, Yu J, He Y (2017) Soccer video event annotation by synchronization of attackdefense clips and match reports with coarse-grained time information. IEEE Trans Circ Syst Vid Technol 27(5):1104–1117

    Article  Google Scholar 

  39. Godi M, Rota P, Setti F (2017) Indirect match highlights detection with deep convolutional neural networks. In: Proc int conf on image analysis and processing, Catania, Italy, September 2017, pp 87–96

  40. Wang D, Xin J (2018) Emergent spatio-temporal multimodal learning using a developmental network. Appl Intell 1–18

  41. Song X, Zhang W, Weng J (2015) Types, locations, and scales from cluttered natural video and actions. IEEE Trans Auton Ment Dev 7(4):273

    Article  Google Scholar 

  42. Wang D, Wang J, Liu L (2017) Developmental network: an internal emergent object feature learning. Neural Process Lett 1–25

Download references

Acknowledgments

This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2016R1D1A1B03933860).

Author information

Authors and Affiliations

  1. Software Engineering Department, University of Engineering and Technology, Taxila, 47050, Pakistan

    Ali Javed

  2. Computer Science Department, University of Engineering and Technology, Taxila, 47050, Pakistan

    Aun Irtaza

  3. Computer Science Department, COMSATS University Islamabad, Wah Campus, GT Road, Wah Cantt, 47040, Pakistan

    Yasmeen Khaliq

  4. College of Engineering and Computer Science, University of Michgan-Dearborn, 4901 Evergreen Rd, Dearborn, MI, 48128, USA

    Hafiz Malik

  5. School of Computer Science and Engineering, Korea University of Technology and Education, 1600, Chungjeol-ro, Byeongcheon-myeon, 31253, Cheonan, South Korea

    Muhammad Tariq Mahmood

Authors
  1. Ali Javed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aun Irtaza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yasmeen Khaliq
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hafiz Malik
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muhammad Tariq Mahmood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Tariq Mahmood.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Javed, A., Irtaza, A., Khaliq, Y. et al. Replay and key-events detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine. Appl Intell 49, 2899–2917 (2019). https://doi.org/10.1007/s10489-019-01410-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-019-01410-x

Keywords

Search

Navigation