Advertisement

Journal of Signal Processing Systems

, Volume 91, Issue 7, pp 703–729 | Cite as

FPGA-Based Vision Processing System for Automatic Online Player Tracking in Indoor Sports

  • Omar W. IbraheemEmail author
  • Arif Irwansyah
  • Jens Hagemeyer
  • Mario Porrmann
  • Ulrich Rueckert
Article
  • 269 Downloads

Abstract

In recent years, there has been an increasing growth of using vision-based systems for tracking the players in team sports to evaluate and enhance their performance. Player tracking using vision systems is a very challenging task due to the nature of sports games, which includes severe and frequent interactions (e.g. occlusions) between the players. Additionally, these vision systems have high computational demands since they require processing of a huge amount of video data based on the utilization of multiple cameras with high resolution and high frame rate. As a result, most of the existing systems based on general-purpose computers are not able to perform online real-time player tracking, but track the players offline using pre-recorded video files, limiting, e.g., direct feedback on the player performance during the game. In this paper, we present a reconfigurable system to track the players in indoor sports automatically and without user interaction. The proposed system performs real-time processing of the incoming video streams from the cameras, achieving online player tracking. The teams are identified, and the players’ positions are detected based on the colors of their jerseys. FPGA technology is used to handle the compute-intensive vision processing tasks by implementing the video acquisition, video preprocessing, player segmentation, and team identification & player detection modules in hardware, realizing an online real-time system. While the pixel processing is performed in the FPGA, the less compute-intensive player tracking is performed on a general purpose computer. The maximum achieved frame rate for the FPGA implementation is 96.7 fps using a mature Xilinx Virtex-4 FPGA, and can be increased to 136.4 fps using a Xilinx Virtex-7 device. The Player tracking requires an average time of 2.5 ms per frame in the host-PC. As a result, the proposed reconfigurable system supports a maximum frame rate of 78.9 fps using two cameras with a resolution of 1392 × 1040 pixels each. Our results show that the achieved average precision and recall for player detection are up to 84.02% and 96.6%, respectively. Including player tracking, the achieved average precision and recall are up to 94.85% and 94.72%, respectively. Using the proposed FPGA implementation, a speedup by a factor of 15.2 is achieved compared to an OpenCV-based software implementation on a PC equipped with a 2.93 GHz Intel i7-870 CPU.

Keywords

FPGA Hardware acceleration Reconfigurable vision system Multiple Object Tracking Player tracking Indoor sports 

Notes

Acknowledgements

This work was funded as part of the Cluster of Excellence Cognitive Interaction Technology ‘CITEC’ (EXC 277), Bielefeld University, and the European Fond for Regional Development (Europaeischer Fond fuer regionale Entwicklung (EFRE no. 0400079)), and the German Academic Exchange Service (DAAD). The authors are responsible for the contents of this publication.

References

  1. 1.
  2. 2.
    Acuna, D. (2017). “Towards Real-Time Detection and Tracking of Basketball Players using Deep Neural Networks”. In: 31st Conference on Neural Information Processing Systems (NIPS 2017).Google Scholar
  3. 3.
    Automated Imaging Association (AIA). (2016). GigE Vision - True Plug and Play Connectivity. http://www.visiononline.org.
  4. 4.
    Alahi, A., Boursier, Y., Jacques, L., Vandergheynst, P. (2009). Sport players detection and tracking with a mixed network of planar and omnidirectional cameras. In 3rd ACM/IEEE international conference on distributed smart cameras, ICDSC 2009 (pp. 1–8),  https://doi.org/10.1109/ICDSC.2009.52893406.
  5. 5.
    Albuquerque, E.S., Ferreira, A.P.A., Silva, G.M., Carlos, R.L.M., Albuquerque, D.S., Barros, E.N.S. (2016). An FPGA-based accelerator for multiple real-time template matching. In 29th symposium on integrated circuits and systems design (SBCCI) (pp. 1–4).Google Scholar
  6. 6.
    Bailey, D.G. (2011). Design for embedded image processing on FPGAs. New York: Wiley.CrossRefGoogle Scholar
  7. 7.
    Benton, S. (2008). Background subtraction, MATLAB models. EETimes.Google Scholar
  8. 8.
    Bernardin, K., & Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: the CLEAR MOT metrics. Eurasip Journal on Image and Video Processing.  https://doi.org/10.1155/2008/246309.
  9. 9.
    Biresaw, T.A., Nawaz, T., Ferryman, J., Dell, A.I. (2016). ViTBAT: Video tracking and behavior annotation tool. In 2016 13th IEEE international conference on advanced video and signal based surveillance, AVSS 2016 (Vol. 1, pp. 295–301).  https://doi.org/10.1109/AVSS.2016.7738055.
  10. 10.
    Butt, A.A., & Collins, R.T. (2013). Multi-target tracking by lagrangian relaxation to min-cost network flow. In Proceedings of the IEEE Computer Society conference on computer vision and pattern recognition (pp. 1846–1853),  https://doi.org/10.1109/CVPR.2013.241.
  11. 11.
    Cheung, S.C., & Kamath, C. (2004). Robust techniques for background subtraction in urban traffic video. Proceedings of Video Communications and Image Processing, SPIE Electronic Imaging, 881–892.  https://doi.org/10.1117/12.526886.
  12. 12.
    de Pádua, P.H., Pádua, F.L, de A. Pereira, M., Sousa, M.T., de Oliveira, M.B., Wanner, E.F. (2017). A vision-based system to support tactical and physical analyses in futsal. Machine Vision and Applications, 28(5–6), 475–496.  https://doi.org/10.1007/s00138-017-0849-z.CrossRefGoogle Scholar
  13. 13.
    Foley, J.D., van Dam, A., Feiner, S. K., Hughes, J.F. (1996). Computer graphics: principles and practice. Boston: Addison-Wesley.zbMATHGoogle Scholar
  14. 14.
    Godil, A., Bostelman, R., Shneier, M., Shackleford, W. (2014). Performance metrics for evaluating object and human detection and tracking systems (pp. 1–13)  https://doi.org/10.6028/NIST.IR.7972.
  15. 15.
    Hu, M.C., Chang, M.H., Wu, J.L., Chi, L. (2011). Robust camera calibration and player tracking in broadcast basketball video. IEEE Transactions on Multimedia, 13(2), 266–279.  https://doi.org/10.1109/TMM.2010.2100373.CrossRefGoogle Scholar
  16. 16.
    Ibraheem, O.W., Irwansyah, A., Hagemeyer, J., Porrmann, M., Rueckert, U. (2015). A resource-efficient multi-camera GigE vision IP Core for embedded vision processing platforms. In 2015 international conference on ReConFigurable computing and FPGAs (ReConFig) (pp. 1–6),  https://doi.org/10.1109/ReConFig.2015.7393282.
  17. 17.
    Ibraheem, O.W., Irwansyah, A., Hagemeyer, J., Porrmann, M., Rueckert, U. (2017). Reconfigurable vision processing system for player tracking in indoor sports. In 2017 Conference on design and architectures for signal and image processing (DASIP) (pp. 1–6),  https://doi.org/10.1109/DASIP.2017.8122114.
  18. 18.
    Irwansyah, A., Ibraheem, O.W., Hagemeyer, J., Porrmann, M., Rueckert, U. (2015). FGPA-based Circular Hough Transform with Grpah Clustering. In 2015 International Conference on ReConFigurable Computing and FPGAs (ReConFig) (pp. 1–8),  https://doi.org/10.1109/ReConFig.2015.7393313.
  19. 19.
    Irwansyah, A., Ibraheem, O.W., Hagemeyer, J., Porrmann, M., Rueckert, U. (2017). FPGA-based multi-robot tracking. Journal of Parallel and Distributed Computing, 107, 146–161.  https://doi.org/10.1016/j.jpdc.2017.03.008.CrossRefGoogle Scholar
  20. 20.
    Jacobsen, M., Sampangi, S., Freund, Y., Kastner, R. (2014). Improving FPGA accelerated tracking with multiple online trained classifiers. In 24th International Conference on Field Programmable Logic and Applications, FPL 2014,  https://doi.org/10.1109/FPL.2014.6927505.
  21. 21.
    Land, E.H., & McCann, J.J. (1971). Lightness and Retinex theory. Journal of the Optical Society of America, 61(1), 1–11.CrossRefGoogle Scholar
  22. 22.
    Lam, E. (2005). Combining gray world and retinex theory for automatic white balance in digital photography. In Proceedings of the ninth international symposium on consumer electronics, (ISCE 2005) (pp. 1–6).Google Scholar
  23. 23.
    Li, Y., Huang, C., Nevatia, R. (2009). Learning to associate: hybridboosted multi-target tracker for crowded scene. In 2009 IEEE Computer Society conference on computer vision and pattern recognition workshops, CVPR workshops 2009 (pp. 2953–2960),  https://doi.org/10.1109/CVPRW.2009.5206735.
  24. 24.
    Li, C., Yee, L.Y., Maruyama, H., Yamaguchi, Y. (2017). FPGA-based volleyball player tracker. In ACM SIGARCH Computer Architecture News (Vol. 44, pp. 80–86),  https://doi.org/10.1145/3039902.3039917.
  25. 25.
    Liu, J., Cao, X., Li, Y., Zhang, B. (2018). Online multi-object tracking using hierarchical constraints for complex scenarios. IEEE Transactions on Intelligent Transportation Systems, 19(1), 151–161.  https://doi.org/10.1109/TITS.2017.2750058.CrossRefGoogle Scholar
  26. 26.
    Lu, W.l. (2011). Learning to track and identify players from broadcast sports videos. Ph.D. thesis, The University Of British Columbia.Google Scholar
  27. 27.
    Lu, W.L., Ting, J.A., Little, J.J., Murphy, K.P. (2013). Learning to track and identify players from broadcast sports videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1704–1716.  https://doi.org/10.1109/TPAMI.2012.242.CrossRefGoogle Scholar
  28. 28.
    Mazzeo, P.L., Giove, L., Moramarco, G.M., Spagnolo, P., Leo, M. (2011). HSV and RGB color histograms comparing for objects tracking among non overlapping FOVs, using CBTF. In 8th IEEE international conference on advanced video and signal based surveillance, AVSS (pp. 498–503),  https://doi.org/10.1109/AVSS.2011.6027383.
  29. 29.
    McFarlane, N.J.B., & Schofield, C.P. (1995). Segmentation and tracking of piglets in images. Machine Vision and Applications, 8(3), 187–193.  https://doi.org/10.1007/BF01215814.CrossRefGoogle Scholar
  30. 30.
    Monier, E. (2011). Vision based tracking in team sports. Ph.D. thesis, Paderborn University, Germany.Google Scholar
  31. 31.
    Monier, E., Wilhelm, P., Rückert, U. (2009). A computer vision based tracking system for indoor team sports. In The fourth international conference on intelligent computing and information systems (pp. 1–5).Google Scholar
  32. 32.
    Munkres, J. (1957). Algorithms for the Assignment and Transportation Problems. Journal of the Society for Industrial and Applied Mathematics, 5(1), 32–38.  https://doi.org/10.1137/0105003. http://epubs.siam.org/doi/10.1137/0105003.MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Okuma, K., Taleghani, A., Freitas, N., Little, J.J., Lowe, D.G. (2004). A boosted particle filter: multitarget detection and tracking. In European conference on computer vision (pp. 28–39).Google Scholar
  34. 34.
    Porrmann, M., Hagemeyer, J., Romoth, J., Strugholtz, M., Pohl, C. (2010). RAPTOR-A scalable platform for rapid prototyping and FPGA-based cluster computing. Advances in Parallel Computing, 19, 592–599.  https://doi.org/10.3233/978-1-60750-530-3-592.Google Scholar
  35. 35.
    Santiago, C., Gomes, L., Sousa, A., Reis, L., Estriga, M. (2012). Tracking players in indoor sports using a vision system inspired in fuzzy and parallel processing. Cutting Edge Research in New Technologies.  https://doi.org/10.5772/2431.
  36. 36.
    Santiago, C.B., Sousa, A., Reis, L.P. (2013). Vision system for tracking handball players using fuzzy color processing. Machine Vision and Applications, 24(5), 1055–1074.  https://doi.org/10.1007/s00138-012-0471-z.CrossRefGoogle Scholar
  37. 37.
    Schaeffer, S.E. (2007). Graph clustering. Computer Science Review, 1, 27–64.  https://doi.org/10.1016/j.cosrev.2007.05.001.CrossRefzbMATHGoogle Scholar
  38. 38.
    STATS Website. www.stats.com.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Omar W. Ibraheem
    • 1
    Email author
  • Arif Irwansyah
    • 1
    • 2
  • Jens Hagemeyer
    • 1
  • Mario Porrmann
    • 1
  • Ulrich Rueckert
    • 1
  1. 1.Cognitronics and Sensor Systems Group, CITECBielefeld UniversityBielefeldGermany
  2. 2.Electronic Engineering Polytechnic Institute of SurabayaSurabayaIndonesia

Personalised recommendations