FPGA-Based Vision Processing System for Automatic Online Player Tracking in Indoor Sports
- 162 Downloads
In recent years, there has been an increasing growth of using vision-based systems for tracking the players in team sports to evaluate and enhance their performance. Player tracking using vision systems is a very challenging task due to the nature of sports games, which includes severe and frequent interactions (e.g. occlusions) between the players. Additionally, these vision systems have high computational demands since they require processing of a huge amount of video data based on the utilization of multiple cameras with high resolution and high frame rate. As a result, most of the existing systems based on general-purpose computers are not able to perform online real-time player tracking, but track the players offline using pre-recorded video files, limiting, e.g., direct feedback on the player performance during the game. In this paper, we present a reconfigurable system to track the players in indoor sports automatically and without user interaction. The proposed system performs real-time processing of the incoming video streams from the cameras, achieving online player tracking. The teams are identified, and the players’ positions are detected based on the colors of their jerseys. FPGA technology is used to handle the compute-intensive vision processing tasks by implementing the video acquisition, video preprocessing, player segmentation, and team identification & player detection modules in hardware, realizing an online real-time system. While the pixel processing is performed in the FPGA, the less compute-intensive player tracking is performed on a general purpose computer. The maximum achieved frame rate for the FPGA implementation is 96.7 fps using a mature Xilinx Virtex-4 FPGA, and can be increased to 136.4 fps using a Xilinx Virtex-7 device. The Player tracking requires an average time of 2.5 ms per frame in the host-PC. As a result, the proposed reconfigurable system supports a maximum frame rate of 78.9 fps using two cameras with a resolution of 1392 × 1040 pixels each. Our results show that the achieved average precision and recall for player detection are up to 84.02% and 96.6%, respectively. Including player tracking, the achieved average precision and recall are up to 94.85% and 94.72%, respectively. Using the proposed FPGA implementation, a speedup by a factor of 15.2 is achieved compared to an OpenCV-based software implementation on a PC equipped with a 2.93 GHz Intel i7-870 CPU.
KeywordsFPGA Hardware acceleration Reconfigurable vision system Multiple Object Tracking Player tracking Indoor sports
This work was funded as part of the Cluster of Excellence Cognitive Interaction Technology ‘CITEC’ (EXC 277), Bielefeld University, and the European Fond for Regional Development (Europaeischer Fond fuer regionale Entwicklung (EFRE no. 0400079)), and the German Academic Exchange Service (DAAD). The authors are responsible for the contents of this publication.
- 1.APIDIS Basketball dataset. http://sites.uclouvain.be/ispgroup/index.php/Softwares/APIDIS.
- 2.Acuna, D. (2017). “Towards Real-Time Detection and Tracking of Basketball Players using Deep Neural Networks”. In: 31st Conference on Neural Information Processing Systems (NIPS 2017).Google Scholar
- 3.Automated Imaging Association (AIA). (2016). GigE Vision - True Plug and Play Connectivity. http://www.visiononline.org.
- 4.Alahi, A., Boursier, Y., Jacques, L., Vandergheynst, P. (2009). Sport players detection and tracking with a mixed network of planar and omnidirectional cameras. In 3rd ACM/IEEE international conference on distributed smart cameras, ICDSC 2009 (pp. 1–8), https://doi.org/10.1109/ICDSC.2009.52893406.
- 5.Albuquerque, E.S., Ferreira, A.P.A., Silva, G.M., Carlos, R.L.M., Albuquerque, D.S., Barros, E.N.S. (2016). An FPGA-based accelerator for multiple real-time template matching. In 29th symposium on integrated circuits and systems design (SBCCI) (pp. 1–4).Google Scholar
- 7.Benton, S. (2008). Background subtraction, MATLAB models. EETimes.Google Scholar
- 8.Bernardin, K., & Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: the CLEAR MOT metrics. Eurasip Journal on Image and Video Processing. https://doi.org/10.1155/2008/246309.
- 9.Biresaw, T.A., Nawaz, T., Ferryman, J., Dell, A.I. (2016). ViTBAT: Video tracking and behavior annotation tool. In 2016 13th IEEE international conference on advanced video and signal based surveillance, AVSS 2016 (Vol. 1, pp. 295–301). https://doi.org/10.1109/AVSS.2016.7738055.
- 10.Butt, A.A., & Collins, R.T. (2013). Multi-target tracking by lagrangian relaxation to min-cost network flow. In Proceedings of the IEEE Computer Society conference on computer vision and pattern recognition (pp. 1846–1853), https://doi.org/10.1109/CVPR.2013.241.
- 11.Cheung, S.C., & Kamath, C. (2004). Robust techniques for background subtraction in urban traffic video. Proceedings of Video Communications and Image Processing, SPIE Electronic Imaging, 881–892. https://doi.org/10.1117/12.526886.
- 14.Godil, A., Bostelman, R., Shneier, M., Shackleford, W. (2014). Performance metrics for evaluating object and human detection and tracking systems (pp. 1–13) https://doi.org/10.6028/NIST.IR.7972.
- 16.Ibraheem, O.W., Irwansyah, A., Hagemeyer, J., Porrmann, M., Rueckert, U. (2015). A resource-efficient multi-camera GigE vision IP Core for embedded vision processing platforms. In 2015 international conference on ReConFigurable computing and FPGAs (ReConFig) (pp. 1–6), https://doi.org/10.1109/ReConFig.2015.7393282.
- 17.Ibraheem, O.W., Irwansyah, A., Hagemeyer, J., Porrmann, M., Rueckert, U. (2017). Reconfigurable vision processing system for player tracking in indoor sports. In 2017 Conference on design and architectures for signal and image processing (DASIP) (pp. 1–6), https://doi.org/10.1109/DASIP.2017.8122114.
- 18.Irwansyah, A., Ibraheem, O.W., Hagemeyer, J., Porrmann, M., Rueckert, U. (2015). FGPA-based Circular Hough Transform with Grpah Clustering. In 2015 International Conference on ReConFigurable Computing and FPGAs (ReConFig) (pp. 1–8), https://doi.org/10.1109/ReConFig.2015.7393313.
- 20.Jacobsen, M., Sampangi, S., Freund, Y., Kastner, R. (2014). Improving FPGA accelerated tracking with multiple online trained classifiers. In 24th International Conference on Field Programmable Logic and Applications, FPL 2014, https://doi.org/10.1109/FPL.2014.6927505.
- 22.Lam, E. (2005). Combining gray world and retinex theory for automatic white balance in digital photography. In Proceedings of the ninth international symposium on consumer electronics, (ISCE 2005) (pp. 1–6).Google Scholar
- 23.Li, Y., Huang, C., Nevatia, R. (2009). Learning to associate: hybridboosted multi-target tracker for crowded scene. In 2009 IEEE Computer Society conference on computer vision and pattern recognition workshops, CVPR workshops 2009 (pp. 2953–2960), https://doi.org/10.1109/CVPRW.2009.5206735.
- 24.Li, C., Yee, L.Y., Maruyama, H., Yamaguchi, Y. (2017). FPGA-based volleyball player tracker. In ACM SIGARCH Computer Architecture News (Vol. 44, pp. 80–86), https://doi.org/10.1145/3039902.3039917.
- 26.Lu, W.l. (2011). Learning to track and identify players from broadcast sports videos. Ph.D. thesis, The University Of British Columbia.Google Scholar
- 28.Mazzeo, P.L., Giove, L., Moramarco, G.M., Spagnolo, P., Leo, M. (2011). HSV and RGB color histograms comparing for objects tracking among non overlapping FOVs, using CBTF. In 8th IEEE international conference on advanced video and signal based surveillance, AVSS (pp. 498–503), https://doi.org/10.1109/AVSS.2011.6027383.
- 30.Monier, E. (2011). Vision based tracking in team sports. Ph.D. thesis, Paderborn University, Germany.Google Scholar
- 31.Monier, E., Wilhelm, P., Rückert, U. (2009). A computer vision based tracking system for indoor team sports. In The fourth international conference on intelligent computing and information systems (pp. 1–5).Google Scholar
- 33.Okuma, K., Taleghani, A., Freitas, N., Little, J.J., Lowe, D.G. (2004). A boosted particle filter: multitarget detection and tracking. In European conference on computer vision (pp. 28–39).Google Scholar
- 35.Santiago, C., Gomes, L., Sousa, A., Reis, L., Estriga, M. (2012). Tracking players in indoor sports using a vision system inspired in fuzzy and parallel processing. Cutting Edge Research in New Technologies. https://doi.org/10.5772/2431.
- 38.STATS Website. www.stats.com.