Skip to main content
SpringerLink
Log in

Content-Aware Summarization of Broadcast Sports Videos: An Audio–Visual Feature Extraction Approach

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

A large number of videos available on the internet belong to the category of sports. Generally, a sports video has a long duration and consists of only a few exciting moments. Sports enthusiasts keep themselves updated on the current happenings, in less time, by means of a summarized version of the sports video known as highlights. For the past few years, sports video summarization is regaining attention among the research community. Automatic generation of highlights form a sports video is a challenging task as different sports games have different rules and situations. In this paper, we propose a method for automatically generating highlights from broadcast sports videos. The proposed method generates highlights by extracting audio and visual features from a sports video. Our method automatically learns the scorebox template from a broadcast sports video using SIFT features, and then locates and extracts the template from a video stream. The extracted template is further analyzed to find out all the possible text regions. Afterward, the information is extracted from all the text regions by means of deep neural network. Based on user preferences, the most relevant information is extracted and converted to a keyframe representation which helps to generate highlights. Extensive experiments were performed to evaluate the effectiveness of the proposed method. Results of the experiments reveal the effectiveness and superiority of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://www.mathworks.com/matlabcentral/fileexchange/65266-speech2text.

References

  1. Agyeman R, Muhammad R, Choi GS (2019) Soccer video summarization using deep learning. In: 2nd IEEE conference on multimedia information processing and retrieval, MIPR 2019, San Jose, CA, USA, March 28–30, 2019, pp 270–273

  2. Akiyama Y, Barrantes RG, Hynes T (2019) Video scene extraction tool for soccer goalkeeper performance data analysis. In: Joint proceedings of the ACM IUI 2019 workshops co-located with the 24th ACM conference on intelligent user interfaces (ACM IUI 2019), Los Angeles, USA, March 20, 2019

  3. Berkun R, Sonn E, Rudoy D (2011) Detection of score changes in sport videos using textual overlays. In: 7th international symposium on image and signal processing and analysis (ISPA), pp 301–306

  4. Bettadapura V, Pantofaru C, Essa IA (2016) Leveraging contextual cues for generating basketball highlights. In: Proceedings of the 2016 ACM conference on multimedia conference, MM 2016, Amsterdam, The Netherlands, October 15–19, 2016, pp 908–917

  5. Cai J, Tang X (2018) RGB video based tennis action recognition using a deep weighted long short-term memory. CoRR. abs/1808.00845

  6. Chakraborty S, Tickoo O, Iyer R (2015) Adaptive keyframe selection for video summarization. In: 2015 IEEE winter conference on applications of computer vision, WACV 2015, Waikoloa, HI, USA, January 5–9, 2015, pp 702–709

  7. Chen C, Chen L (2014) Novel framework for sports video analysis: a basketball case study. In: 2014 IEEE international conference on image processing, ICIP 2014, Paris, France, October 27–30, 2014, pp 961–965

  8. Chen D, Hsiao M, Lee S (2006) Automatic closed caption detection and filtering in MPEG videos for video structuring. J Inf Sci Eng 22(5):1145–1162

    Google Scholar 

  9. Chen H, Tsai SS, Schroth G, Chen DM, Grzeszczuk R, Girod B (2011) Robust text detection in natural images with edge-enhanced maximally stable extremal regions. In: 18th IEEE international conference on image processing, ICIP 2011, Brussels, Belgium, September 11–14, 2011, pp 2609–2612

  10. Decroos T, Dzyuba V, Haaren JV, Davis J (2017) Predicting soccer highlights from spatio-temporal match event streams. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, February 4–9, 2017, San Francisco, California, USA, pp 1302–1308

  11. Deng G, Liu L, Zuo J (2019) Scoring framework of soccer matches using possession trajectory data. In: Proceedings of the ACM turing celebration conference—China, ACM TUR-C 2019, Chengdu, China, May 17–19, 2019, pp 59:1–59:2

  12. Ghosh A, Jawahar CV (2017) Smarttennistv: automatic indexing of tennis videos. In: Computer vision, pattern recognition, image processing, and graphics—6th national conference, NCVPRIPG 2017, Mandi, India, December 16–19, 2017, revised selected papers, pp 24–33

  13. Ghosh A, Jawahar CV (2018) SmartTennisTV: automatic indexing of tennis videos. CoRR. abs/1801.01430

  14. Gilbert AC, Zhang Y, Lee K, Zhang Y, Lee H (2017) Towards understanding the invertibility of convolutional neural networks. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI 2017, Melbourne, Australia, August 19–25, 2017, pp 1703–1710

  15. Godi M, Rota P, Setti F (2017) Indirect match highlights detection with deep convolutional neural networks. In: New trends in image analysis and processing—ICIAP 2017—ICIAP international workshops, WBICV, SSPandBE, 3AS, RGBD, NIVAR, IWBAAS, and MADiMa 2017, Catania, Italy, September 11–15, 2017, revised selected papers, pp 87–96

  16. Gong Y, Sin LT, Chuan CH, Zhang H, Sakauchi M (1995) Automatic parsing of TV soccer programs. In: Proceedings of the IEEE international conference on multimedia computing and systems, ICMCS 1995, Washington DC, USA, May 15–18, 1995, pp 167–174

  17. Gonzalez Á, Bergasa LM, Torres JJY, Bronte S (2012) Text location in complex images. In: Proceedings of the 21st international conference on pattern recognition, ICPR 2012, Tsukuba, Japan, November 11–15, 2012, pp 617–620

  18. Guo J, Gurrin C, Lao S, Foley C, Smeaton AF (2011) Localization and recognition of the scoreboard in sports video based on SIFT point matching. In: Advances in multimedia modeling—17th international multimedia modeling conference, MMM 2011, Taipei, Taiwan, January 5–7, 2011, proceedings, Part II, pp 337–347

  19. He C, Shao J, Zhang J, Zhou X (2019) Clustering-based multiple instance learning with multi-view feature. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2019.113027

    Article  Google Scholar 

  20. Hu R, Zhu X, Zhu Y, Gan J (2019) Robust svm with adaptive graph learning. World Wide Web. https://doi.org/10.1007/s11280-019-00766-x

    Article  Google Scholar 

  21. Huang C, Shih H, Chao C (2006) Semantic analysis of soccer video using dynamic bayesian network. IEEE Trans Multimed 8(4):749–760

    Article  Google Scholar 

  22. Hung M, Hsieh C (2008) Event detection of broadcast baseball videos. IEEE Trans Circuits Syst Video Technol 18(12):1713–1726

    Article  Google Scholar 

  23. Javed A, Irtaza A, Malik H, Mahmood MT, Adnan SM (2019) Multimodal framework based on audio–visual features for summarisation of cricket videos. IET Image Process 13(4):615–622

    Article  Google Scholar 

  24. Jiang H, Lu Y, Xue J (2016) Automatic soccer video event detection based on a deep neural network combined CNN and RNN. In: 28th IEEE international conference on tools with artificial intelligence, ICTAI 2016, San Jose, CA, USA, November 6–8, 2016, pp 490–494

  25. Kim W, Park J, Kim C (2008) Scorebox extraction from mobile sports videos using support vector machines. In: Proceedings of the SPIE 7073, applications of digital image processing XXXI

  26. Kim Y, Kim M (2019) ‘A wisdom of crowds’: social media mining for soccer match analysis. IEEE Access 7:52634–52639

    Article  Google Scholar 

  27. Kosmadakis I, Petrellis N, Birbas MK, Vardakas M (2018) Employing Savitzky–Golay smoothing in a low cost ehealth platform. In: 41st international conference on telecommunications and signal processing, TSP 2018, Athens, Greece, July 4–6, 2018, pp 1–5

  28. Li Y, Lu H (2012) Scene text detection via stroke width. In: Proceedings of the 21st international conference on pattern recognition, ICPR 2012, Tsukuba, Japan, November 11–15, 2012, pp 681–684

  29. Liang C, Chu W, Kuo J, Wu J, Cheng W (2005) Baseball event detection using game-specific feature sets and rules. In: International symposium on circuits and systems (ISCAS 2005), 23–26 May 2005. Kobe, Japan, pp 3829–3832

  30. Liao S, Wang Y, Xin Y (2015) Research on scoreboard detection and localization in basketball video. Int J Multimed Ubiquitous Eng 10(11):57–68

    Article  Google Scholar 

  31. Merler M, Joshi D, Nguyen Q, Hammer S, Kent J, Smith JR, Feris RS (2017) Automatic curation of golf highlights using multimodal excitement features. In: 2017 IEEE conference on computer vision and pattern recognition workshops, CVPR workshops 2017, Honolulu, HI, USA, July 21–26, 2017, pp 57–65

  32. Miao G, Zhu G, Jiang S, Huang Q, Xu C, Gao W (2007) The demo: a real-time score detection and recognition approach in broadcast basketball sports video. In: Proceedings of the 2007 IEEE international conference on multimedia and expo, ICME 2007, July 2–5, 2007, Beijing, China, p 1

  33. Mochizuki T, Tadenuma M, Yagi N (2005) Baseball video indexing using patternization of scenes and hidden Markov model. In: Proceedings of the 2005 international conference on image processing, ICIP 2005, Genoa, Italy, September 11–14, 2005, pp 1212–1215

  34. Narasimhan H, Satheesh S, Sriram D (2010) Automatic summarization of cricket video events using genetic algorithm. In: Genetic and evolutionary computation conference, GECCO 2010, proceedings, Portland, Oregon, USA, July 7–11, 2010, companion material, pp 2051–2054

  35. Neumann L, Matas J (2012) Real-time scene text localization and recognition. In: 2012 IEEE conference on computer vision and pattern recognition, Providence, RI, USA, June 16–21, 2012, pp 3538–3545

  36. Raventos A, Quijada R, Torres L, Tarres F, Carasusán E, Giribet D (2014) The importance of audio descriptors in automatic soccer highlights generation. In: IEEE 11th international multi-conference on systems, signals and devices, SSD 2014, Castelldefels-Barcelona, Spain, February 11–14, 2014, pp 1–6

  37. Rekik G, Khacharem A, Belkhir Y, Bali N, Jarraya M (2019) The instructional benefits of dynamic visualizations in the acquisition of basketball tactical actions. J Comput Assist Learn 35(1):74–81

    Article  Google Scholar 

  38. Roy S, Shivakumara P, Pal U, Lu T, Tan CL (2016) New tampered features for scene and caption text classification in video frame. In: 15th international conference on frontiers in handwriting recognition, ICFHR 2016, Shenzhen, China, October 23–26, 2016, pp 36–41

  39. Sankar KP, Pandey S, Jawahar CV (2006) Text driven temporal segmentation of cricket videos. In: Computer vision, graphics and image processing, 5th Indian conference, ICVGIP 2006, Madurai, India, December 13–16, 2006, proceedings, pp 433–444

  40. Santiago CB, Sousa A, Estriga ML, Reis LP, Lames M (2010) Survey on team tracking techniques applied to sports. In: Autonomous and intelligent systems—first international conference, AIS 2010, Povoa de Varzim, Portugal, June 21–23, 2010. Proceedings, pp 1–6

  41. Setti F, Conigliaro D, Rota P, Bassetti C, Conci N, Sebe N, Cristani M (2017) The s-hock dataset: a new benchmark for spectator crowd analysis. Comput Vis Image Underst 159:47–58

    Article  Google Scholar 

  42. Shih H (2018) A survey of content-aware video analysis for sports. IEEE Trans Circuits Syst Video Technol 28(5):1212–1231

    Article  Google Scholar 

  43. Shih H, Huang C (2006) A robust superimposed caption box content understanding for sports videos.In: Eigth IEEE international symposium on multimedia (ISM 2006), 11–13 December 2006, San Diego, CA, USA, pp 867–872

  44. Shukla P, Sadana H, Bansal A, Verma D, Elmadjian CEL, Raman B, Turk M (2018) Automatic cricket highlight generation using event-driven and excitement-based features. In: 2018 IEEE conference on computer vision and pattern recognition workshops, CVPR workshops 2018, Salt Lake City, UT, USA, June 18–22, 2018, pp 1800–1808

  45. Smith R, Antonova D, Lee D (2009) Adapting the Tesseract open source OCR engine for multilingual OCR. In: Proceedings of the international workshop on multilingual OCR, MOCR@ICDAR 2009, Barcelona, Spain, July 25, 2009, p 1

  46. Tang H, Kwatra V, Sargin ME, Gargi U (2011) Detecting highlights in sports videos: cricket as a test case. In: Proceedings of the 2011 IEEE international conference on multimedia and expo, ICME 2011, 11–15 July, 2011, Barcelona, Catalonia, Spain, pp 1–6

  47. Tran D, Bourdev LD, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, December 7–13, 2015, pp 4489–4497

  48. Uehira K, Tanaka G, Suzuki K, Komiya K, Ikeda H (2012) Content indexing for specific scenes in baseball videos utilizing two-dimensional matching of intensity patterns. In: IEEE international conference on consumer electronics, ICCE 2012, Las Vegas, NV, USA, January 13–16, 2012, pp 122–123

  49. Yoon Y, Hwang H, Choi Y, Joo M, Oh H, Park I, Lee K, Hwang J (2019) Analyzing basketball movements and pass relationships using realtime object tracking techniques based on deep learning. IEEE Access 7:56564–56576

    Article  Google Scholar 

  50. Yu J, Lei A, Hu Y (2019) Soccer video event detection based on deep learning. In: MultiMedia modeling—25th international conference, MMM 2019, Thessaloniki, Greece, January 8–11, 2019, proceedings, Part II, pp 377–389

  51. Zhang D, Chang S (2002) Event detection in baseball video using superimposed caption recognition. In: Proceedings of the 10th ACM international conference on multimedia 2002, Juan les Pins, France, December 1–6, 2002, pp 315–318

  52. Zhang F, Jiang Y (2019) Basketball action data processing method based on mode symmetric algorithm. Symmetry 11(4):560

    Article  Google Scholar 

  53. Zhu X, Gan J, Lu G, Li J, Zhang S (2019) Spectral clustering via half-quadratic optimization. World Wide Web. https://doi.org/10.1007/s11280-019-00731-8

  54. Zhu X, Zhang S, He W, Hu R, Lei C, Zhu P (2019) One-step multi-view spectral clustering. IEEE Trans Knowl Data Eng 31(10):2022–2034

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Nature Science Foundation of China (Grant Nos. 61672133 and 61832001) and Sichuan Science and Technology Program (Grant No. 2019YFG0535).

Author information

Authors and Affiliations

  1. Center for Future Media, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

    Abdullah Aman Khan, Jie Shao, Waqar Ali & Saifullah Tumrani

  2. Sichuan Artificial Intelligence Research Institute, Yibin, 644000, China

    Jie Shao

Authors
  1. Abdullah Aman Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Waqar Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saifullah Tumrani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Shao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, A.A., Shao, J., Ali, W. et al. Content-Aware Summarization of Broadcast Sports Videos: An Audio–Visual Feature Extraction Approach. Neural Process Lett 52, 1945–1968 (2020). https://doi.org/10.1007/s11063-020-10200-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10200-3

Keywords

Search

Navigation