Abstract
Based on the ZYNQ platform, this paper proposes a hardware/software co-design approach, and implements a binocular stereo vision system with high real-time performance and good human-computer interaction, which can be used to assist advanced driver assistance systems to improve driving safety. Combining the application characteristics of binocular stereo vision, the approach firstly modularizes the system’s functions to perform hardware/software partitioning, accelerates the data processing on FPGA, and performs the data control on ARM cores; then uses the ARM instruction set to configure the registers within FPGA to design relevant interfaces to complete the data interaction between hardware and software; finally, combines the implementation of specific algorithms and logical control to complete the binocular stereo vision system. The test results show that the frame rate with an image resolution of 640 * 480 can reach 121.43 frames per second when the FPGA frequency is 100M, and the frame rate is also high for large resolution images. At the same time, the system can achieve real-time display and human-computer interaction with the control of the graphical user interface.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, New York (2010)
Woodfill, J.I., Gordon, G., Buck, R.: Tyzx DeepSea high speed stereo vision system. In: Conference on Computer Vision and Pattern Recognition Workshop, CVPRW 2004, p. 41. IEEE (2004)
Janai, J., Güney, F., Behl, A., Geiger, A.: Computer vision for autonomous vehicles: problems, datasets and state-of-the-art (2017). https://arxiv.org/abs/1704.05519
Bimbraw, K.: Autonomous cars: past, present and future a review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology. In: International Conference on Informatics in Control, Automation and Robotics, pp. 191–198. IEEE (2015)
Song, W., Yang, Y., Fu, M., et al.: Lane detection and classification for forward collision warning system based on stereo vision. IEEE Sens. J. PP(99), 1 (2018)
Jin, S., Cho, J., Xuan, D.P., et al.: FPGA design and implementation of a real-time stereo vision system. IEEE Trans. Circ. Syst. Video Technol. 20(1), 15–26 (2010)
Pérez-Patricio, M., Aguilar-González, A.: FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. 1–17 (2015)
Perri, S., Frustaci, F., Spagnolo, F., et al.: Design of real-time FPGA-based embedded system for stereo vision. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp.1–5. IEEE (2018)
De Michell, G., Gupta, R.K.: Hardware/software co-design. Proc. IEEE 85(3), 349–365 (1997)
Wolf, W.H.: Hardware-software co-design of embedded systems. Proc. IEEE 82(7), 967–989 (1994)
Adams, J.K., Thomas, D.E.: The design of mixed hardware/software systems. In: Design Automation Conference Proceedings, pp. 515–520. IEEE (1996)
Edwards, S., Lavagno, L., Lee, E.A., et al.: Design of embedded systems: formal models, validation, and synthesis. Proc. IEEE 85(3), 366–390 (1997)
Lu, J., Jiang, D., Ma, M.: Embedded system software and hardware cooperative design practical guide based on Xilinx ZYNQ. China Machine Press, Beijing (2013, Chinese)
Yan, Y., et al.: Camera calibration in binocular stereo vision of moving robot. In: The Sixth World Congress on Intelligent Control and Automation, WCICA 2006, pp. 9257–9261. IEEE (2006)
Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 666–673. IEEE (2002)
Bouguet, J.: Camera calibration toolbox for matlab (2004). http://www.vision.caltech.edu/bouguetj/calibdoc/
Fetić, A., Jurić, D., Osmanković, D.: The procedure of a camera calibration using camera calibration toolbox for MATLAB. In: 2012 Proceedings of the International Convention Mipro, pp. 1752–1757. IEEE (2012)
Tang, Y.P., Pang, C.J., Zhou, Z.S., et al.: Binocular omni-directional vision sensor and epipolar rectification in its omni-directional images. J. Zhejiang Univ. Technol. 1, 20 (2011)
Bouguet, J.: The calibration toolbox for matlab, example 5: stereo rectification algorithm. Code and instructions only. http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html
Mei, X., Sun, X., Zhou, M., et al.: On building an accurate stereo matching system on graphics hardware. In: IEEE International Conference on Computer Vision Workshops, pp. 467–474. IEEE Computer Society (2011)
Zynq-7000 All Programmable SoC Overview DS190 (v1.2). Xilinx (2012)
Acknowledgments
This work is supported by Shanghai Science and Technology Commission Project (17511106902), Shanghai Youth Science and Technology Phosphorus Project (No. 18QB1403900) and Shanghai Science and Technology Major Funding Project (No. 15DZ1100400).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Pan, Y., Zhu, M., Luo, J., Qiu, Y. (2019). A Hardware/Software Co-design Approach for Real-Time Binocular Stereo Vision Based on ZYNQ (Short Paper). In: Gao, H., Wang, X., Yin, Y., Iqbal, M. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2018. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 268. Springer, Cham. https://doi.org/10.1007/978-3-030-12981-1_50
Download citation
DOI: https://doi.org/10.1007/978-3-030-12981-1_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-12980-4
Online ISBN: 978-3-030-12981-1
eBook Packages: Computer ScienceComputer Science (R0)