Skip to main content

Advertisement

SpringerLink
Log in

An optimized stereo vision implementation for embedded systems: application to RGB and infra-red images

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

The aim of this paper is to demonstrate the applicability and the effectiveness of a computationally demanding stereo-matching algorithm in different low-cost and low-complexity embedded devices, by focusing on the analysis of timing and image quality performances. Various optimizations have been implemented to allow its deployment on specific hardware architectures while decreasing memory and processing time requirements: (1) reduction of color channel information and resolution for input images; (2) low-level software optimizations such as parallel computation, replacement of function calls or loop unrolling; (3) reduction of redundant data structures and internal data representation. The feasibility of a stereo vision system on a low-cost platform is evaluated by using standard datasets and images taken from infra-red cameras. Analysis of the resulting disparity map accuracy with respect to a full-size dataset is performed as well as the testing of suboptimal solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The digits in the sequences can be seen as pixel values of a given channel in the input image pairs or, more in general, as elements to align. For the sake of simplicity, by stating \(gap = egap\), effects of extended gap have not been taken into account.

  2. For near infra-red images a grayscale color encoding is assumed.

  3. The total amount \(M\) has not been rounded to the upper multiple of 32 since other small data structures not considered in Eq. (6) can be used to keep the alignment and save memory. Moreover, the unrounded version \(M\) is useful to have a more accurate comparison of memory requirements in case of use of different datasets.

  4. According to the grayscale-space representation, lets assume that value 0 is black and the maximum value, i.e. 255 at 8 bits per pixel, is white. Therefore, higher values in the disparity map correspond to lighter areas in the image, i.e. objects closer to cameras.

References

  1. Ambrosch K., Kubinger W., Accurate hardware-based stereo vision. Comput. Vis. Image Underst. 114(11):1303–1316 (2010). doi:10.1016/j.cviu.2010.07.008

  2. Bacon D., Rabbah R., Shukla S.: FPGA Programming for the Masses. In: ACM Queue, Vol. 11, pp. 40, (2013). doi:10.1145/2436696.2443836

  3. Baker H.H., Binford T.O.: Depth from Edge and Intensity Based Stereo. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, Vol. 2, Morgan Kaufmann Publishers Inc., San Francisco, IJCAI’81, pp 631–636 (1981)

  4. Banahan M., Brady D., Doran M.: The C Book: featuring the ANSI C Standard. Addison-Wesley (1991)

  5. Bertozzi M., Broggi A., Caraffi C., Rose M.D., Felisa M., Vezzoni G., Pedestrian detection by means of far-infrared stereo vision. Computer Vision and Image Understanding 106(23):194–204 (2007). doi:10.1016/j.cviu.2006.07.016

  6. Bobick, A.F., Intille, S.S.: Large occlusion stereo. Int. J. Comput. Vis. 33(3), 181–200 (1999). doi:10.1023/A:1008150329890

    Article  Google Scholar 

  7. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. Pattern Anal. Mach Intell IEEE Trans. 23(11), 1222–1239 (2001). doi:10.1109/34.969114

    Article  Google Scholar 

  8. Buttazzo, G.C.: Hard real-time computing systems: predictable scheduling algorithms and applications (real-time systems series). Springer-Verlag TELOS, Santa Clara (2004)

    MATH  Google Scholar 

  9. Chandrapala T.: Real time Stereo Vision based on biologically motivated algorithms using GPU. 17th ERU Research Symposium: University of Moratuwa, Sri Lanka (2011)

  10. Chen D., Varodayan D., Flierl M., Girod B.: Distributed stereo image coding with improved disparity and noise estimation. In: International Conference on Acoustics, Speech and Signal Processing (2008)

  11. Cox, I.J., Hingorani, S.L., Maggs, B.M., Rao, S.B.: A maximum likelihood stereo algorithm. Comput. Vis. Image Underst. 63(3), 542–567 (1996)

    Article  Google Scholar 

  12. D-Link DCS-942L Datasheet (2011). http://www.dlink.com/. Accessed 11 Sept 2013

  13. Deng Y., Lin X.: A Fast Line Segment Based Dense Stereo Algorithm Using Tree Dynamic Programming. In: Leonardis A., Bischof H., Pinz A. (eds) Computer Vision ECCV 2006, Lecture Notes in Computer Science, vol. 3953, Springer, Berlin Heidelberg, pp 201–212 (2006). doi:10.1007/11744078_16

  14. Devy M., Boizard J.L., Galeano D.B., Lindado H.C., Manzano M.I., Irki Z., Naoulou A., Lacroix P., Fillatreau P., Fourniols J.Y., Parra C.: Stereovision Algorithm to be Executed at 100Hz on a FPGA-based Architecture. In: Advances in Theory and Applications of Stereo Vision, Dr. Asim Bhatti (2011). doi:10.5772/14037

  15. Dieny R., Thevenon J., Martinez-del-Rincon J., Nebel J.C.: Bioinformatics inspired algorithm for stereo correspondence. In: VISAPP, Vilamoura (2011)

  16. Dos Santos-Paulino, A., Nebel, J.C., Florez-Revuelta, F.: Evolutionary algorithm for dense pixel matching in presence of distortions, pp. 23–25. EvoStar, Granada (2014)

    Google Scholar 

  17. Dubbelman G., Van der Mark W., Van Den Heuvel J.C., Groen F.C.A.: Obstacle detection during day and night conditions using stereo vision. In: Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on, pp 109–116 (2007). doi:10.1109/IROS.2007.4399055

  18. Faggioli D., Checconi F., Trimarchi M., Scordino C.: An EDF scheduling class for the Linux kernel. 11th Real-Time Linux Workshop (RTLWS), Dresden, September 2009

  19. FLIR Thermal imaging guidebook for industrial applications (2011)

  20. Forstmann S., Kanou Y., Ohya J., Thuering S., Schmitt A.: Real-time stereo by using dynamic programming. In: Computer Vision and Pattern Recognition Workshop, 2004. CVPRW ’04. Conference on. pp 29–29 (2004). doi 10.1109/CVPR.2004.154

  21. Forsyth D.A., Ponce J.: Computer vision: a modern approach. Prentice Hall Professional Technical Reference (2002)

  22. Fowers J., Brown G., Cooke P., Stitt G.: A performance and energy comparison of FPGAs, GPUs, and multicores for sliding-window applications. In: Proceedings of the ACM/SIGDA international symposium on Field Programmable Gate Arrays (FPGA ’12). ACM, New York, 47–56 (2012)

  23. Free Software Foundation Inc.: GCC, the GNU Compiler Collection (2013). http://gcc.gnu.org/. Accessed 23 Apr 2013

  24. Gai P., Bini E., Lipari G., Natale M.D., Abeni L.: Architecture for a Portable Open Source Real Time Kernel Environment. In: Proceedings of the Second Real-Time Linux Workshop and Hand’s on Real-Time Linux Tutorial (2000)

  25. Gaisler J.: A Portable and Fault-Tolerant Microprocessor Based on the SPARC V8 Architecture. In: Proceedings of the 2002 International Conference on Dependable Systems and Networks (DSN ’02). IEEE Computer Society, Washington, DC, 409–415 (2002)

  26. Geiger, D., Ladendorf, B., Yuille, A.: Occlusions and binocular stereo. Int. J. Comput. Vis. 14(3), 211–226 (1995). doi:10.1007/BF01679683

    Article  Google Scholar 

  27. Goss C.F.: Machine Code Optimization - Improving Executable Object Code. CoRR abs/1308.4815 (2013)

  28. Guan S., Klette R., Woo Y. W.: Belief propagation for stereo analysis of night-vision sequences. In: Proceedings of the 3rd Pacific Rim Symposium on Advances in Image and Video Technology pp. 932–943 (2008). doi:10.1007/978-3-540-92957-4_81

  29. Gudis E., Van der Wal G., Kuthirummal S., Chai S.: Multi-Resolution Real-Time Dense Stereo Vision Processing in FPGA. In: Field-Programmable Custom Computing Machines (FCCM), 2012 IEEE 20th Annual International Symposium on. pp 29–32 (2012). doi:10.1109/FCCM.2012.15

  30. Held M., Karp R.M.: A dynamic programming approach to sequencing problems. In: Proceedings of the 1961 16th ACM National Meeting (ACM ’61), New York, NY, USA, pp 71.201-71.204 (1961). doi:10.1145/800029.808532

  31. Hengstler S., Aghajan H.: A smart camera mote architecture for distributed intelligent surveillance. In: ACM SenSys Workshop on Distributed Smart Cameras (2006)

  32. Higgins, D., Thompson, J., Gibson, T., Thompson, J.D., Higgins, D.G., Gibson, T.J.: Improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucl. Acids Res. 22, 4673–4680 (1994)

    Article  Google Scholar 

  33. Hirschmuller H., Scharstein D.: Evaluation of Cost Functions for Stereo Matching. In: Computer Vision and Pattern Recognition, 2007. CVPR ’07. IEEE Conference on. pp 1–8 (2007). doi:10.1109/CVPR.2007.383248

  34. Hosseini A., Bacara D., Lienkamp M.: A system design for automotive augmented reality using stereo night vision. In: Intelligent Vehicles Symposium Proceedings, 2014 IEEE, pp. 127–133. doi:10.1109/IVS.2014.6856484.

  35. Humenberger M., Zinner C., Weber M., Kubinger W., Vincze M., A fast stereo matching algorithm suitable for embedded real-time systems. Comput Vis Image Underst. 114(11):1180–1202 (2010). doi: 10.1016/j.cviu.2010.03.012

  36. Intel: Intelcore™quad processors specifications (2007). http://www.intel.com

  37. The IPERMOB project: A Pervasive and Heterogeneous Infrastructure to control Urban Mobility in Real-Time (2009). http://www.ipermob.org

  38. Jin, S., Cho, J., Pham, X.D., Lee, K.M., Park, S.K., Kim, M., Jeon, J.W.: FPGA Design and Implementation of a Real-Time Stereo Vision System. Circuits Syst Video Technol. IEEE Trans. 20(1), 15–26 (2010). doi:10.1109/TCSVT.2009.2026831

    Article  Google Scholar 

  39. Lassmann, T., Sonnhammer, E.L.L.: Kalign - an accurate and fast multiple sequence alignment algorithm. BMC Bioinformatics 6, 298 (2005)

    Article  Google Scholar 

  40. Lempitsky V., Rother C., Blake A.: LogCut - Efficient Graph Cut Optimization for Markov Random Fields. In: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. pp 1–8 (2007). doi:10.1109/ICCV.2007.4408907

  41. Levitin, A.V.: Introduction to the Design and Analysis of Algorithms. Addison-Wesley Longman Publishing Co., Inc, Boston, MA, USA (2002)

    Google Scholar 

  42. Lewine D.: POSIX programmers guide. O’Reilly Media, Inc (1991).

  43. MacLean, W.J., Sabihuddin, S., Islam, J.: Leveraging cost matrix structure for hardware implementation of stereo disparity computation using dynamic programming. Comput. Vis. Image Understanding 114(11), 1126–1138 (2010). doi:10.1016/j.cviu.2010.03.011

    Article  Google Scholar 

  44. Maggiani L., Salvadori C., Petracca M., Pagano P., Saletti, R.: Reconfigurable FPGA architecture for computer vision applications in smart camera networks. In: Distributed Smart Cameras (ICDSC), 2013 Seventh International Conference on, pp. 1–6. doi:10.1109/ICDSC.2013.6778212

  45. Martinez del Rincon J., Thevenon J., Dieny R., Nebel J.C.: Dense pixel matching between unrectified and distorted images using dynamic programming. In: International Conference on Computer Vision Theory and Applications, pp 24–26, Rome (2012).

  46. Mouser Electronics. 579–32MX795F512L80VF 32-bit Microcontroller. http://www.mouser.com. Accessed 11 Sept 2013

  47. Microchip MPLABX IDE (2011). http://microchip.com/. Accessed 11 Sept 2013

  48. Nalpantidis, L., Sirakoulis, G.C., Gasteratos, A.: Review of stereo vision algorithms: from software to hardware. Int. J. Optomech. 2(4), 435–462 (2008)

    Article  Google Scholar 

  49. Nalpantidis L., Kalomiros J., Gasteratos A.: Robust 3D vision for robots using dynamic programming. Imaging Systems and Techniques (IST), 2011 IEEE International Conference on, pp. 89–93, (2011). doi:10.1109/IST.2011.5962204.

  50. Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. J. Mol. Biol. 48, 443–453 (1970)

    Article  Google Scholar 

  51. Notredame, C., Higgins, D., Heringa, J.: T-coffee: A novel method for multiple sequence alignments. J. Mol. Biol. 302, 205–217 (2000)

    Article  Google Scholar 

  52. Ohta Y., Kanade T.: Stereo by intra- and inter-scanline search using dynamic programming. Pattern Anal.Mach. Intell. IEEE Trans. PAMI 7(2):139–154 (1985). doi:10.1109/TPAMI.1985.4767639.

  53. Owens K., Matthies L.: Passive Night Vision Sensor Comparison for Unmanned Ground Vehicle Stereo Vision Navigation. In: Proceedings of the IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications (1999). doi:10.1109/ROBOT.2000.844049.

  54. Panda, P.R., Catthoor, F., Dutt, N.D., Danckaert, K., Brockmeyer, E., Kulkarni, C., Vandercappelle, A., Kjeldsberg, P.G.: Data and memory optimization techniques for embedded systems. ACM Trans. Des. Autom. Electron. Syst. 6(2), 149–206 (2001). doi:10.1145/375977.375978

    Article  Google Scholar 

  55. Pandaboard.org. PandaBoard (2010). http://pandaboard.org/

  56. Pandey, J.G., Purushottam, S., Karmakar, A., Shekhar, C.: Platform-based extensible hardware-software video streaming module for a smart camera system. Int. J. Model. Optim. 2(4), 482–487 (2012)

    Article  Google Scholar 

  57. Quenot, G.M.: Image matching using dynamic programming: application to stereovision and image interpolation, p. 2657. Image, Communication (1996)

  58. Raspberry Pi Foundation. Raspberry Pi Board (2011). http://www.raspberrypi.org/. Accessed 2 May 2013

  59. Raspberry Pi Foundation. Raspbian OS (2012). http://www.raspbian.org/. Accessed 2 May 2013

  60. Salehian B., Raie A.A., Fotouhi A.M., Norouzi M.: Efficient interscanline consistency enforcing method for dynamic programming-based dense stereo matching algorithms, J. Electr. Imaging 22(4), (2013). doi: 10.1117/1.JEI.22.4.043028

  61. Salmen J., Schlipsing M., Edelbrunner J., Hegemann S., Lüuke S.: Real-time stereo vision: making more out of dynamic programming. In: Jiang X., Petkov N. (eds.) Computer Analysis of Images and Patterns, Lecture Notes in Computer Science, vol. 5702, pp 1096–1103. Springer, Berlin Heidelberg (2009). doi:10.1007/978-3-642-03767-2_133

  62. Salvadori C., Petracca M., Bocchino S., Pelliccia R., Pagano P.: A low-cost vehicle counter for next-generations ITS. In: Journal of Real Time Image Processing (2014). doi:10.1007/s11554-014-0411-4

  63. Sarkar V.: Optimized Unrolling of Nested Loops, vol 29, Springer, pp 545–581 (2001)

  64. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)

    Article  MATH  Google Scholar 

  65. Scuola Superiore Sant’Anna and Evidence s.r.l.: SeedEye board (2011). http://www.evidence.eu.com. Accessed 17 Feb 2013

  66. Singh A.K., Kumar S.A.: Microcontroller and Embedded System. New Age International (P) Limited (2008).

  67. Summit, S.: C programming FAQs: frequently asked questions. Addison Wesley Longman Publishing Inc., Redwood City (1995)

    Google Scholar 

  68. Torr P.H.S., Criminisi A., Dense stereo using pivoted dynamic programming. Image Vis.Comput. 22(10), 795–806 (2004). doi:10.1016/j.imavis.2004.02.012

  69. Tseng Y.C., Chang N., Chang T.S.: Low memory cost block-based belief propagation for stereo correspondence. In: Multimedia and Expo, 2007 IEEE International Conference on. pp 1415–1418 (2007). doi:10.1109/ICME.2007.4284925

  70. Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. Int J Comput Vis47(1–3), 275–285 (2002). doi:10.1023/A:1014562312225

    Article  MATH  Google Scholar 

  71. Veksler, O.: Stereo correspondence by dynamic programming on a tree. IEEE Comput Soc Conf Comput Vis Pattern Recognit2, 384–390 (2005). doi:10.1109/CVPR.2005.334

    Google Scholar 

  72. Vigliar M., Fratello M., Puglia L., Raiconi G.: SASC: A hardware string alignment coprocessor for stereo correspondence. In: Electronics Design, Systems and Applications (ICEDSA), 2012 IEEE International Conference, Kuala Lumpur, Malaysia, pp. 56–62, 5–6 (2012).

  73. Wang L., Liao M., Gong M., Yang R., Nister D.: High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In: 3D Data Processing, Visualization, and Transmission, Third International Symposium on. pp 798–805 (2006). doi:10.1109/3DPVT.2006.75

  74. Wei, Y., Tsuhan, C., Franchetti, F., Hoe, J.: High performance stereo vision designed for massively data parallel platforms. IEEE Trans Circuits Sys Video Technol 20(11), 1509–1519 (2010). doi:10.1109/TCSVT.2010.2077771

    Article  Google Scholar 

  75. Xia, L., Fujimura, K.: Pedestrian detection using stereo night vision. IEEE Trans. Veh. Tech. 53(6), 1657–1665 (2004). doi:10.1109/TVT.2004.834876

    Article  Google Scholar 

  76. Zinner C., Humenberger M., Ambrosch K., Kubinger W.: An Optimized software-based implementation of a census-based stereo matching algorithm. In: Bebis G., Boyle R., Parvin B., Koracin D., Remagnino P., Porikli F., Peters J., Klosowski J., Arns L., Chun Y., Rhyne T., Monroe L. (eds.) Advances in Visual Computing, Lecture Notes in Computer Science, vol. 5358, pp 216–227. Springer, Berlin Heidelberg (2008). doi:10.1007/978-3-540-89639-5_21

Download references

Author information

Authors and Affiliations

  1. TeCIP Institute, Scuola Superiore Sant’Anna, 56124, Pisa, Italy

    Simone Madeo, Riccardo Pelliccia & Claudio Salvadori

  2. The Institute of Electronics, Communications and Information Technology (ECIT), Queen’s University of Belfast, BT3 9DT, Belfast , United Kingdom

    Jesus Martinez del Rincon

  3. Digital Imaging Research Centre, Kingston University, KT1 2EE, Kingston upon Thames, United Kingdom

    Jean-Christophe Nebel

Authors
  1. Simone Madeo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Riccardo Pelliccia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claudio Salvadori
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jesus Martinez del Rincon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jean-Christophe Nebel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simone Madeo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Madeo, S., Pelliccia, R., Salvadori, C. et al. An optimized stereo vision implementation for embedded systems: application to RGB and infra-red images. J Real-Time Image Proc 12, 725–746 (2016). https://doi.org/10.1007/s11554-014-0461-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-014-0461-7

Keywords

Search

Navigation