Skip to main content
SpringerLink
Log in

Real-time low-power binocular stereo vision based on FPGA

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Binocular stereo vision is a commonly applied computer vision technique with a wide range of applications in 3D scene perception. However, binocular stereo matching algorithms are computationally intensive and complicated. In addition, some traditional platforms are unable to meet the real-time and energy efficient dual requirements. In this paper, we proposed a hardware/software co-design FPGA (Field Programmable Gate Array) approach to overcome these limitations. Based on the characteristics of binocular stereo vision, we modularize the system functions to achieve the hardware/software partitioning. This accelerates the data processing on the FPGA, while simultaneously performing data control on the ARM (Advanced RISC Machine) cores. The parallelism of the FPGA allows for a full-pipeline design that is synchronized with an identical system clock for the simultaneous running of multiple stereo processing components, thus improving the processing speed. Furthermore, to minimize hardware costs, the collected images and data are compressed prior to matching, while the precision is subsequently enhanced during post-processing. The proposed system was evaluated on the PYNQ-Z2 development board, with experimental results revealing its high real-time performance and low power consumption for a 100M clock frequency. Compared with existing designs, the simple yet flexible system demonstrated a higher image processing speed and less hardware resource overhead (thus lower power consumption). The average error rate of the BM matching algorithm was also improved, particularly with the limited PYNQ-Z2 hardware resource. The proposed system has been opened on GitHub.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Adams, J.K., Thomas, D.E.: The design of mixed hardware/software systems. In: Proceedings of the 33rd annual Design Automation Conference (DAC '96). Association for Computing Machinery, New York, NY, USA, pp 515–520. (1996) https://doi.org/10.1145/240518.240616

  2. Ambrosch, K., Humenberger, M., Kubinger, W., Steininger, A.: SAD-Based stereo matching using FPGAs. In: Kisačanin, B., Bhattacharyya, S.S., Chai, S. (eds.) Embedded Computer Vision. Advances in Pattern Recognition. Springer, London, pp. 121-138. https://doi.org/10.1007/978-1-84800-304-0_6

  3. Finnerty, A., Ratigner, H.: Reduce Power and Cost by Converting from Floating Point to Fixed Point, WP491 (v1.0). Xilinx (2017). https://www.xilinx.com/support/documentation/white_papers/wp491-floating-to-fixed-point.pdf

  4. Angoletta, M.E.: Digital signal processor fundamentals and system design. In : CAS - CERN Accelerator School: Digital Signal Processing, pp. 167–229 (2008)

  5. Bouguet, J.Y.: Fifth calibration example - calibrating a stereo system, stereo image rectification and 3d stereo triangulation. http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html

  6. Bouguet, J.Y.: Camera calibration toolbox for matlab (2015). http://www.vision.caltech.edu/bouguetj/calib_doc/

  7. Crockett, L.H., Elliot, R.A., Enderwitz, M.A., Stewart, R.W.: The Zynq Book: Embedded Processing with the Arm Cortex-A9 on the Xilinx Zynq-7000 All Programmable Soc. Strathclyde Academic Media, Glasgow (2014)

    Google Scholar 

  8. DeMicheli, G., Sami, M.: Hardware/software Co-design, . Springer Science & Business Media, Berlin 310 (2013)

    Google Scholar 

  9. Einspruch, N.: Application Specific Integrated Circuit (ASIC) Technology, vol. 23. Academic Press, Cambridge (2012)

    Google Scholar 

  10. Fetić, A., Jurić, D., Osmanković, D.: The procedure of a camera calibration using Camera Calibration Toolbox for MATLAB, In: Proceedings of the 35th International Convention MIPRO, pp. 1752-1757 (2012)

  11. Geiger, A., Roser, M., Urtasun, R.: Efficient Large-Scale Stereo Matching. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) Computer Vision – ACCV 2010. Lecture Notes in Computer Science, vol. 6492. Springer, Berlin, Heidelberg. (2011). https://doi.org/10.1007/978-3-642-19315-6_3

  12. Guo, Y., Yao, Y., Di, X.: Research on Structural Parameter Optimization of Binocular Vision Measuring System for Parallel Mechanism. In: 2006 International Conference on Mechatronics and Automation, pp. 1131-1135 (2006) https://doi.org/10.1109/ICMA.2006.257784

  13. Jin, S., Cho, J., Dai Pham, X., Lee, K.M., Park, S.K., Kim, M., Jeon, J.W.: Fpga design and implementation of a real-time stereo vision system. IEEE Trans. Circuits Syst. Video Technol. 20(1), 15–26 (2009)

    Google Scholar 

  14. Li, J., Wu, J., You, Y., Jeon, G.: Parallel binocular stereo-vision-based gpu accelerated pedestrian detection and distance computation. J. Real-Time Image Process. 17(3), 447–457 (2020)

    Article  Google Scholar 

  15. Lin, C.Y., Chiu, Y.P., Lin, C.Y., Tsai, C.R.: Development of a binocular vision-based catcher robot system using dsp platform. J. Chin. Inst. Eng. 37(2), 210–223 (2014)

    Article  Google Scholar 

  16. Madisetti, V., Madisetti, V.: VLSI Digital Signal Processors. Butterworth-Heinemann, Oxford (1995)

    MATH  Google Scholar 

  17. Mahammed, M.A., Melhum, A.I., Kochery, F.A.: Object distance measurement by stereo vision. Int. J. Sci. Appl. Inform. Technol. (IJSAIT) 2(2), 05–08 (2013)

    Google Scholar 

  18. Nan, Y.: Binocular vision system based on pynq (2017). http://www.digilent.com.cn/project/details/140.html

    Google Scholar 

  19. Pan, Y., Zhu, M., Luo, J., Qiu, Y.: A Hardware/Software Co-design Approach for Real-Time Binocular Stereo Vision Based on ZYNQ (Short Paper). In: Gao, H., Wang, X., Yin, Y., Iqbal, M. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2018. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol. 268. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12981-1_50

  20. Pérez-Patricio, M., Aguilar-González, A.: Fpga implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching. J. Real-Time Image Process. 16(2), 271–287 (2019)

    Article  Google Scholar 

  21. Perri, S., Frustaci, F., Spagnolo, F., Corsonello, P.: Design of Real-Time FPGA-based Embedded System for Stereo Vision. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5 (2018).https://doi.org/10.1109/ISCAS.2018.8351886

  22. Rahnama, O., Cavallari, T., Golodetz, S., Tonioni, A., Joy, T., Di Stefano, L., Walker, S., Torr, P.H.: Real-time highly accurate dense depth on a power budget using an fpga-cpu hybrid soc. IEEE Trans. Circ. Syst. II 66(5), 773–777 (2019)

    Google Scholar 

  23. Rodriguez-Andina, J.J., Moure, M.J., Valdes, M.D.: Features, design tools, and application domains of fpgas. IEEE Trans. Ind. Electron. 54(4), 1810–1823 (2007)

    Article  Google Scholar 

  24. Salmen, J., Schlipsing, M., Edelbrunner, J., Hegemann, S., Lüke, S.: Real-time stereo vision: making more out of dynamic programming. In: International Conference on Computer Analysis of Images and Patterns, pp. 1096–1103. Springer (2009)

  25. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)

    Article  Google Scholar 

  26. Schauwecker, K.: Real-Time Stereo Vision on FPGAs with SceneScan (2018). CoRR abs/1809.07977

  27. Tang, Y., Pang, C., Zhou, Z., Chen, Y.: Binocular omni-directional vision sensor and epipolar rectification in its omni-directional images. J. Zhejiang Univ. Technol. 1, 20 (2011)

    Google Scholar 

  28. Werner, M., Stabernack, B., Riechert, C.: Hardware implementation of a full hd real-time disparity estimation algorithm. IEEE Trans. Consumer Electron. 60(1), 66–73 (2014)

    Article  Google Scholar 

  29. Yan, Y., Zhu, Q., Lin, Z., Chen, Q.: Camera Calibration in Binocular Stereo Vision of Moving Robot. In: 2006 6th World Congress on Intelligent Control and Automation, pp. 9257–9261. https://doi.org/10.1109/WCICA.2006.1713792

  30. Zhang, K., Lu, J., Lafruit, G., Lauwereins, R., Gool, L.V.: Real-time accurate stereo with bitwise fast voting on CUDA. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 794-800 (2009). https://doi.org/10.1109/ICCVW.2009.5457623

  31. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

The work presented in this paper was originally designed for a student contest in the Xilinx International Summer School. We express sincere appreciation to Xilinx for partially sponsoring this project. We thank TopEdit (www.topeditsci.com) for its linguistic assistance during the preparation of this manuscript. We thank Yongzheng Chen for completing the experiments and analysis in Sect. 5.3.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Northeastern University, Shenyang, China

    Gang Wu, Jinglei Yang & Hao Yang

  2. Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Shenyang, China

    Gang Wu

Authors
  1. Gang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinglei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work is supported by the National Nature Science Foundation of China (Grant No.61872072 and No.61672144).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, G., Yang, J. & Yang, H. Real-time low-power binocular stereo vision based on FPGA. J Real-Time Image Proc 19, 29–39 (2022). https://doi.org/10.1007/s11554-021-01158-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01158-z

Keywords

Search

Navigation