A Hardware/Software Co-design Approach for Real-Time Binocular Stereo Vision Based on ZYNQ (Short Paper)

  • Yukun Pan
  • Minghua ZhuEmail author
  • Jufeng Luo
  • Yunzhou Qiu
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 268)


Based on the ZYNQ platform, this paper proposes a hardware/software co-design approach, and implements a binocular stereo vision system with high real-time performance and good human-computer interaction, which can be used to assist advanced driver assistance systems to improve driving safety. Combining the application characteristics of binocular stereo vision, the approach firstly modularizes the system’s functions to perform hardware/software partitioning, accelerates the data processing on FPGA, and performs the data control on ARM cores; then uses the ARM instruction set to configure the registers within FPGA to design relevant interfaces to complete the data interaction between hardware and software; finally, combines the implementation of specific algorithms and logical control to complete the binocular stereo vision system. The test results show that the frame rate with an image resolution of 640 * 480 can reach 121.43 frames per second when the FPGA frequency is 100M, and the frame rate is also high for large resolution images. At the same time, the system can achieve real-time display and human-computer interaction with the control of the graphical user interface.


Binocular stereo vision system Hardware/software co-design Data interaction Processing system Programmable logic 



This work is supported by Shanghai Science and Technology Commission Project (17511106902), Shanghai Youth Science and Technology Phosphorus Project (No. 18QB1403900) and Shanghai Science and Technology Major Funding Project (No. 15DZ1100400).


  1. 1.
    Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, New York (2010)zbMATHGoogle Scholar
  2. 2.
    Woodfill, J.I., Gordon, G., Buck, R.: Tyzx DeepSea high speed stereo vision system. In: Conference on Computer Vision and Pattern Recognition Workshop, CVPRW 2004, p. 41. IEEE (2004)Google Scholar
  3. 3.
    Janai, J., Güney, F., Behl, A., Geiger, A.: Computer vision for autonomous vehicles: problems, datasets and state-of-the-art (2017).
  4. 4.
    Bimbraw, K.: Autonomous cars: past, present and future a review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology. In: International Conference on Informatics in Control, Automation and Robotics, pp. 191–198. IEEE (2015)Google Scholar
  5. 5.
    Song, W., Yang, Y., Fu, M., et al.: Lane detection and classification for forward collision warning system based on stereo vision. IEEE Sens. J. PP(99), 1 (2018)Google Scholar
  6. 6.
    Jin, S., Cho, J., Xuan, D.P., et al.: FPGA design and implementation of a real-time stereo vision system. IEEE Trans. Circ. Syst. Video Technol. 20(1), 15–26 (2010)CrossRefGoogle Scholar
  7. 7.
    Pérez-Patricio, M., Aguilar-González, A.: FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. 1–17 (2015)Google Scholar
  8. 8.
    Perri, S., Frustaci, F., Spagnolo, F., et al.: Design of real-time FPGA-based embedded system for stereo vision. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp.1–5. IEEE (2018)Google Scholar
  9. 9.
    De Michell, G., Gupta, R.K.: Hardware/software co-design. Proc. IEEE 85(3), 349–365 (1997)CrossRefGoogle Scholar
  10. 10.
    Wolf, W.H.: Hardware-software co-design of embedded systems. Proc. IEEE 82(7), 967–989 (1994)CrossRefGoogle Scholar
  11. 11.
    Adams, J.K., Thomas, D.E.: The design of mixed hardware/software systems. In: Design Automation Conference Proceedings, pp. 515–520. IEEE (1996)Google Scholar
  12. 12.
    Edwards, S., Lavagno, L., Lee, E.A., et al.: Design of embedded systems: formal models, validation, and synthesis. Proc. IEEE 85(3), 366–390 (1997)CrossRefGoogle Scholar
  13. 13.
    Lu, J., Jiang, D., Ma, M.: Embedded system software and hardware cooperative design practical guide based on Xilinx ZYNQ. China Machine Press, Beijing (2013, Chinese)Google Scholar
  14. 14.
    Yan, Y., et al.: Camera calibration in binocular stereo vision of moving robot. In: The Sixth World Congress on Intelligent Control and Automation, WCICA 2006, pp. 9257–9261. IEEE (2006)Google Scholar
  15. 15.
    Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 666–673. IEEE (2002)Google Scholar
  16. 16.
    Bouguet, J.: Camera calibration toolbox for matlab (2004).
  17. 17.
    Fetić, A., Jurić, D., Osmanković, D.: The procedure of a camera calibration using camera calibration toolbox for MATLAB. In: 2012 Proceedings of the International Convention Mipro, pp. 1752–1757. IEEE (2012)Google Scholar
  18. 18.
    Tang, Y.P., Pang, C.J., Zhou, Z.S., et al.: Binocular omni-directional vision sensor and epipolar rectification in its omni-directional images. J. Zhejiang Univ. Technol. 1, 20 (2011)Google Scholar
  19. 19.
    Bouguet, J.: The calibration toolbox for matlab, example 5: stereo rectification algorithm. Code and instructions only.
  20. 20.
    Mei, X., Sun, X., Zhou, M., et al.: On building an accurate stereo matching system on graphics hardware. In: IEEE International Conference on Computer Vision Workshops, pp. 467–474. IEEE Computer Society (2011)Google Scholar
  21. 21.
    Zynq-7000 All Programmable SoC Overview DS190 (v1.2). Xilinx (2012)Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2019

Authors and Affiliations

  • Yukun Pan
    • 1
  • Minghua Zhu
    • 1
    Email author
  • Jufeng Luo
    • 2
  • Yunzhou Qiu
    • 2
  1. 1.Hardware/Software Co-Design Technology and Application Engineering Research CenterEast China Normal UniversityShanghaiChina
  2. 2.Shanghai Internet of Things CO., LTDShanghaiChina

Personalised recommendations