Journal of Real-Time Image Processing

, Volume 12, Issue 4, pp 725–746 | Cite as

An optimized stereo vision implementation for embedded systems: application to RGB and infra-red images

  • Simone Madeo
  • Riccardo Pelliccia
  • Claudio Salvadori
  • Jesus Martinez del Rincon
  • Jean-Christophe Nebel
Special Issue Paper

Abstract

The aim of this paper is to demonstrate the applicability and the effectiveness of a computationally demanding stereo-matching algorithm in different low-cost and low-complexity embedded devices, by focusing on the analysis of timing and image quality performances. Various optimizations have been implemented to allow its deployment on specific hardware architectures while decreasing memory and processing time requirements: (1) reduction of color channel information and resolution for input images; (2) low-level software optimizations such as parallel computation, replacement of function calls or loop unrolling; (3) reduction of redundant data structures and internal data representation. The feasibility of a stereo vision system on a low-cost platform is evaluated by using standard datasets and images taken from infra-red cameras. Analysis of the resulting disparity map accuracy with respect to a full-size dataset is performed as well as the testing of suboptimal solutions.

Keywords

Stereo vision Embedded optimization Embedded systems Smart camera Near infra-red 

References

  1. 1.
    Ambrosch K., Kubinger W., Accurate hardware-based stereo vision. Comput. Vis. Image Underst. 114(11):1303–1316 (2010). doi:10.1016/j.cviu.2010.07.008
  2. 2.
    Bacon D., Rabbah R., Shukla S.: FPGA Programming for the Masses. In: ACM Queue, Vol. 11, pp. 40, (2013). doi:10.1145/2436696.2443836
  3. 3.
    Baker H.H., Binford T.O.: Depth from Edge and Intensity Based Stereo. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, Vol. 2, Morgan Kaufmann Publishers Inc., San Francisco, IJCAI’81, pp 631–636 (1981)Google Scholar
  4. 4.
    Banahan M., Brady D., Doran M.: The C Book: featuring the ANSI C Standard. Addison-Wesley (1991)Google Scholar
  5. 5.
    Bertozzi M., Broggi A., Caraffi C., Rose M.D., Felisa M., Vezzoni G., Pedestrian detection by means of far-infrared stereo vision. Computer Vision and Image Understanding 106(23):194–204 (2007). doi:10.1016/j.cviu.2006.07.016
  6. 6.
    Bobick, A.F., Intille, S.S.: Large occlusion stereo. Int. J. Comput. Vis. 33(3), 181–200 (1999). doi:10.1023/A:1008150329890 CrossRefGoogle Scholar
  7. 7.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. Pattern Anal. Mach Intell IEEE Trans. 23(11), 1222–1239 (2001). doi:10.1109/34.969114 CrossRefGoogle Scholar
  8. 8.
    Buttazzo, G.C.: Hard real-time computing systems: predictable scheduling algorithms and applications (real-time systems series). Springer-Verlag TELOS, Santa Clara (2004)MATHGoogle Scholar
  9. 9.
    Chandrapala T.: Real time Stereo Vision based on biologically motivated algorithms using GPU. 17th ERU Research Symposium: University of Moratuwa, Sri Lanka (2011)Google Scholar
  10. 10.
    Chen D., Varodayan D., Flierl M., Girod B.: Distributed stereo image coding with improved disparity and noise estimation. In: International Conference on Acoustics, Speech and Signal Processing (2008)Google Scholar
  11. 11.
    Cox, I.J., Hingorani, S.L., Maggs, B.M., Rao, S.B.: A maximum likelihood stereo algorithm. Comput. Vis. Image Underst. 63(3), 542–567 (1996)CrossRefGoogle Scholar
  12. 12.
    D-Link DCS-942L Datasheet (2011). http://www.dlink.com/. Accessed 11 Sept 2013
  13. 13.
    Deng Y., Lin X.: A Fast Line Segment Based Dense Stereo Algorithm Using Tree Dynamic Programming. In: Leonardis A., Bischof H., Pinz A. (eds) Computer Vision ECCV 2006, Lecture Notes in Computer Science, vol. 3953, Springer, Berlin Heidelberg, pp 201–212 (2006). doi:10.1007/11744078_16
  14. 14.
    Devy M., Boizard J.L., Galeano D.B., Lindado H.C., Manzano M.I., Irki Z., Naoulou A., Lacroix P., Fillatreau P., Fourniols J.Y., Parra C.: Stereovision Algorithm to be Executed at 100Hz on a FPGA-based Architecture. In: Advances in Theory and Applications of Stereo Vision, Dr. Asim Bhatti (2011). doi:10.5772/14037
  15. 15.
    Dieny R., Thevenon J., Martinez-del-Rincon J., Nebel J.C.: Bioinformatics inspired algorithm for stereo correspondence. In: VISAPP, Vilamoura (2011)Google Scholar
  16. 16.
    Dos Santos-Paulino, A., Nebel, J.C., Florez-Revuelta, F.: Evolutionary algorithm for dense pixel matching in presence of distortions, pp. 23–25. EvoStar, Granada (2014)Google Scholar
  17. 17.
    Dubbelman G., Van der Mark W., Van Den Heuvel J.C., Groen F.C.A.: Obstacle detection during day and night conditions using stereo vision. In: Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on, pp 109–116 (2007). doi:10.1109/IROS.2007.4399055
  18. 18.
    Faggioli D., Checconi F., Trimarchi M., Scordino C.: An EDF scheduling class for the Linux kernel. 11th Real-Time Linux Workshop (RTLWS), Dresden, September 2009Google Scholar
  19. 19.
    FLIR Thermal imaging guidebook for industrial applications (2011)Google Scholar
  20. 20.
    Forstmann S., Kanou Y., Ohya J., Thuering S., Schmitt A.: Real-time stereo by using dynamic programming. In: Computer Vision and Pattern Recognition Workshop, 2004. CVPRW ’04. Conference on. pp 29–29 (2004). doi 10.1109/CVPR.2004.154
  21. 21.
    Forsyth D.A., Ponce J.: Computer vision: a modern approach. Prentice Hall Professional Technical Reference (2002)Google Scholar
  22. 22.
    Fowers J., Brown G., Cooke P., Stitt G.: A performance and energy comparison of FPGAs, GPUs, and multicores for sliding-window applications. In: Proceedings of the ACM/SIGDA international symposium on Field Programmable Gate Arrays (FPGA ’12). ACM, New York, 47–56 (2012)Google Scholar
  23. 23.
    Free Software Foundation Inc.: GCC, the GNU Compiler Collection (2013). http://gcc.gnu.org/. Accessed 23 Apr 2013
  24. 24.
    Gai P., Bini E., Lipari G., Natale M.D., Abeni L.: Architecture for a Portable Open Source Real Time Kernel Environment. In: Proceedings of the Second Real-Time Linux Workshop and Hand’s on Real-Time Linux Tutorial (2000)Google Scholar
  25. 25.
    Gaisler J.: A Portable and Fault-Tolerant Microprocessor Based on the SPARC V8 Architecture. In: Proceedings of the 2002 International Conference on Dependable Systems and Networks (DSN ’02). IEEE Computer Society, Washington, DC, 409–415 (2002)Google Scholar
  26. 26.
    Geiger, D., Ladendorf, B., Yuille, A.: Occlusions and binocular stereo. Int. J. Comput. Vis. 14(3), 211–226 (1995). doi:10.1007/BF01679683 CrossRefGoogle Scholar
  27. 27.
    Goss C.F.: Machine Code Optimization - Improving Executable Object Code. CoRR abs/1308.4815 (2013)Google Scholar
  28. 28.
    Guan S., Klette R., Woo Y. W.: Belief propagation for stereo analysis of night-vision sequences. In: Proceedings of the 3rd Pacific Rim Symposium on Advances in Image and Video Technology pp. 932–943 (2008). doi:10.1007/978-3-540-92957-4_81
  29. 29.
    Gudis E., Van der Wal G., Kuthirummal S., Chai S.: Multi-Resolution Real-Time Dense Stereo Vision Processing in FPGA. In: Field-Programmable Custom Computing Machines (FCCM), 2012 IEEE 20th Annual International Symposium on. pp 29–32 (2012). doi:10.1109/FCCM.2012.15
  30. 30.
    Held M., Karp R.M.: A dynamic programming approach to sequencing problems. In: Proceedings of the 1961 16th ACM National Meeting (ACM ’61), New York, NY, USA, pp 71.201-71.204 (1961). doi:10.1145/800029.808532
  31. 31.
    Hengstler S., Aghajan H.: A smart camera mote architecture for distributed intelligent surveillance. In: ACM SenSys Workshop on Distributed Smart Cameras (2006)Google Scholar
  32. 32.
    Higgins, D., Thompson, J., Gibson, T., Thompson, J.D., Higgins, D.G., Gibson, T.J.: Improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucl. Acids Res. 22, 4673–4680 (1994)CrossRefGoogle Scholar
  33. 33.
    Hirschmuller H., Scharstein D.: Evaluation of Cost Functions for Stereo Matching. In: Computer Vision and Pattern Recognition, 2007. CVPR ’07. IEEE Conference on. pp 1–8 (2007). doi:10.1109/CVPR.2007.383248
  34. 34.
    Hosseini A., Bacara D., Lienkamp M.: A system design for automotive augmented reality using stereo night vision. In: Intelligent Vehicles Symposium Proceedings, 2014 IEEE, pp. 127–133. doi:10.1109/IVS.2014.6856484.
  35. 35.
    Humenberger M., Zinner C., Weber M., Kubinger W., Vincze M., A fast stereo matching algorithm suitable for embedded real-time systems. Comput Vis Image Underst. 114(11):1180–1202 (2010). doi: 10.1016/j.cviu.2010.03.012
  36. 36.
    Intel: Intelcore™quad processors specifications (2007). http://www.intel.com
  37. 37.
    The IPERMOB project: A Pervasive and Heterogeneous Infrastructure to control Urban Mobility in Real-Time (2009). http://www.ipermob.org
  38. 38.
    Jin, S., Cho, J., Pham, X.D., Lee, K.M., Park, S.K., Kim, M., Jeon, J.W.: FPGA Design and Implementation of a Real-Time Stereo Vision System. Circuits Syst Video Technol. IEEE Trans. 20(1), 15–26 (2010). doi:10.1109/TCSVT.2009.2026831 CrossRefGoogle Scholar
  39. 39.
    Lassmann, T., Sonnhammer, E.L.L.: Kalign - an accurate and fast multiple sequence alignment algorithm. BMC Bioinformatics 6, 298 (2005)CrossRefGoogle Scholar
  40. 40.
    Lempitsky V., Rother C., Blake A.: LogCut - Efficient Graph Cut Optimization for Markov Random Fields. In: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. pp 1–8 (2007). doi:10.1109/ICCV.2007.4408907
  41. 41.
    Levitin, A.V.: Introduction to the Design and Analysis of Algorithms. Addison-Wesley Longman Publishing Co., Inc, Boston, MA, USA (2002)Google Scholar
  42. 42.
    Lewine D.: POSIX programmers guide. O’Reilly Media, Inc (1991).Google Scholar
  43. 43.
    MacLean, W.J., Sabihuddin, S., Islam, J.: Leveraging cost matrix structure for hardware implementation of stereo disparity computation using dynamic programming. Comput. Vis. Image Understanding 114(11), 1126–1138 (2010). doi:10.1016/j.cviu.2010.03.011 CrossRefGoogle Scholar
  44. 44.
    Maggiani L., Salvadori C., Petracca M., Pagano P., Saletti, R.: Reconfigurable FPGA architecture for computer vision applications in smart camera networks. In: Distributed Smart Cameras (ICDSC), 2013 Seventh International Conference on, pp. 1–6. doi:10.1109/ICDSC.2013.6778212
  45. 45.
    Martinez del Rincon J., Thevenon J., Dieny R., Nebel J.C.: Dense pixel matching between unrectified and distorted images using dynamic programming. In: International Conference on Computer Vision Theory and Applications, pp 24–26, Rome (2012).Google Scholar
  46. 46.
    Mouser Electronics. 579–32MX795F512L80VF 32-bit Microcontroller. http://www.mouser.com. Accessed 11 Sept 2013
  47. 47.
    Microchip MPLABX IDE (2011). http://microchip.com/. Accessed 11 Sept 2013
  48. 48.
    Nalpantidis, L., Sirakoulis, G.C., Gasteratos, A.: Review of stereo vision algorithms: from software to hardware. Int. J. Optomech. 2(4), 435–462 (2008)CrossRefGoogle Scholar
  49. 49.
    Nalpantidis L., Kalomiros J., Gasteratos A.: Robust 3D vision for robots using dynamic programming. Imaging Systems and Techniques (IST), 2011 IEEE International Conference on, pp. 89–93, (2011). doi:10.1109/IST.2011.5962204.
  50. 50.
    Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. J. Mol. Biol. 48, 443–453 (1970)CrossRefGoogle Scholar
  51. 51.
    Notredame, C., Higgins, D., Heringa, J.: T-coffee: A novel method for multiple sequence alignments. J. Mol. Biol. 302, 205–217 (2000)CrossRefGoogle Scholar
  52. 52.
    Ohta Y., Kanade T.: Stereo by intra- and inter-scanline search using dynamic programming. Pattern Anal.Mach. Intell. IEEE Trans. PAMI 7(2):139–154 (1985). doi:10.1109/TPAMI.1985.4767639.
  53. 53.
    Owens K., Matthies L.: Passive Night Vision Sensor Comparison for Unmanned Ground Vehicle Stereo Vision Navigation. In: Proceedings of the IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications (1999). doi:10.1109/ROBOT.2000.844049.
  54. 54.
    Panda, P.R., Catthoor, F., Dutt, N.D., Danckaert, K., Brockmeyer, E., Kulkarni, C., Vandercappelle, A., Kjeldsberg, P.G.: Data and memory optimization techniques for embedded systems. ACM Trans. Des. Autom. Electron. Syst. 6(2), 149–206 (2001). doi:10.1145/375977.375978 CrossRefGoogle Scholar
  55. 55.
    Pandaboard.org. PandaBoard (2010). http://pandaboard.org/
  56. 56.
    Pandey, J.G., Purushottam, S., Karmakar, A., Shekhar, C.: Platform-based extensible hardware-software video streaming module for a smart camera system. Int. J. Model. Optim. 2(4), 482–487 (2012)CrossRefGoogle Scholar
  57. 57.
    Quenot, G.M.: Image matching using dynamic programming: application to stereovision and image interpolation, p. 2657. Image, Communication (1996)Google Scholar
  58. 58.
    Raspberry Pi Foundation. Raspberry Pi Board (2011). http://www.raspberrypi.org/. Accessed 2 May 2013
  59. 59.
    Raspberry Pi Foundation. Raspbian OS (2012). http://www.raspbian.org/. Accessed 2 May 2013
  60. 60.
    Salehian B., Raie A.A., Fotouhi A.M., Norouzi M.: Efficient interscanline consistency enforcing method for dynamic programming-based dense stereo matching algorithms, J. Electr. Imaging 22(4), (2013). doi: 10.1117/1.JEI.22.4.043028
  61. 61.
    Salmen J., Schlipsing M., Edelbrunner J., Hegemann S., Lüuke S.: Real-time stereo vision: making more out of dynamic programming. In: Jiang X., Petkov N. (eds.) Computer Analysis of Images and Patterns, Lecture Notes in Computer Science, vol. 5702, pp 1096–1103. Springer, Berlin Heidelberg (2009). doi:10.1007/978-3-642-03767-2_133
  62. 62.
    Salvadori C., Petracca M., Bocchino S., Pelliccia R., Pagano P.: A low-cost vehicle counter for next-generations ITS. In: Journal of Real Time Image Processing (2014). doi:10.1007/s11554-014-0411-4
  63. 63.
    Sarkar V.: Optimized Unrolling of Nested Loops, vol 29, Springer, pp 545–581 (2001)Google Scholar
  64. 64.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)CrossRefMATHGoogle Scholar
  65. 65.
    Scuola Superiore Sant’Anna and Evidence s.r.l.: SeedEye board (2011). http://www.evidence.eu.com. Accessed 17 Feb 2013
  66. 66.
    Singh A.K., Kumar S.A.: Microcontroller and Embedded System. New Age International (P) Limited (2008).Google Scholar
  67. 67.
    Summit, S.: C programming FAQs: frequently asked questions. Addison Wesley Longman Publishing Inc., Redwood City (1995)Google Scholar
  68. 68.
    Torr P.H.S., Criminisi A., Dense stereo using pivoted dynamic programming. Image Vis.Comput. 22(10), 795–806 (2004). doi:10.1016/j.imavis.2004.02.012
  69. 69.
    Tseng Y.C., Chang N., Chang T.S.: Low memory cost block-based belief propagation for stereo correspondence. In: Multimedia and Expo, 2007 IEEE International Conference on. pp 1415–1418 (2007). doi:10.1109/ICME.2007.4284925
  70. 70.
    Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. Int J Comput Vis47(1–3), 275–285 (2002). doi:10.1023/A:1014562312225 CrossRefMATHGoogle Scholar
  71. 71.
    Veksler, O.: Stereo correspondence by dynamic programming on a tree. IEEE Comput Soc Conf Comput Vis Pattern Recognit2, 384–390 (2005). doi:10.1109/CVPR.2005.334 Google Scholar
  72. 72.
    Vigliar M., Fratello M., Puglia L., Raiconi G.: SASC: A hardware string alignment coprocessor for stereo correspondence. In: Electronics Design, Systems and Applications (ICEDSA), 2012 IEEE International Conference, Kuala Lumpur, Malaysia, pp. 56–62, 5–6 (2012).Google Scholar
  73. 73.
    Wang L., Liao M., Gong M., Yang R., Nister D.: High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In: 3D Data Processing, Visualization, and Transmission, Third International Symposium on. pp 798–805 (2006). doi:10.1109/3DPVT.2006.75
  74. 74.
    Wei, Y., Tsuhan, C., Franchetti, F., Hoe, J.: High performance stereo vision designed for massively data parallel platforms. IEEE Trans Circuits Sys Video Technol 20(11), 1509–1519 (2010). doi:10.1109/TCSVT.2010.2077771 CrossRefGoogle Scholar
  75. 75.
    Xia, L., Fujimura, K.: Pedestrian detection using stereo night vision. IEEE Trans. Veh. Tech. 53(6), 1657–1665 (2004). doi:10.1109/TVT.2004.834876 CrossRefGoogle Scholar
  76. 76.
    Zinner C., Humenberger M., Ambrosch K., Kubinger W.: An Optimized software-based implementation of a census-based stereo matching algorithm. In: Bebis G., Boyle R., Parvin B., Koracin D., Remagnino P., Porikli F., Peters J., Klosowski J., Arns L., Chun Y., Rhyne T., Monroe L. (eds.) Advances in Visual Computing, Lecture Notes in Computer Science, vol. 5358, pp 216–227. Springer, Berlin Heidelberg (2008). doi:10.1007/978-3-540-89639-5_21

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Simone Madeo
    • 1
  • Riccardo Pelliccia
    • 1
  • Claudio Salvadori
    • 1
  • Jesus Martinez del Rincon
    • 2
  • Jean-Christophe Nebel
    • 3
  1. 1.TeCIP InstitutePisaItaly
  2. 2.The Institute of Electronics, Communications and Information Technology (ECIT)Queen’s University of BelfastBelfast United Kingdom
  3. 3.Digital Imaging Research CentreKingston UniversityKingston upon ThamesUnited Kingdom

Personalised recommendations