Advertisement

Journal of Real-Time Image Processing

, Volume 16, Issue 5, pp 1643–1660 | Cite as

An evaluation of real-time RGB-D visual odometry algorithms on mobile devices

  • Vincent AngladonEmail author
  • Simone Gasparini
  • Vincent Charvillat
  • Tomislav Pribanić
  • Tomislav Petković
  • Matea Ðonlić
  • Benjamin Ahsan
  • Frédéric Bruel
Original Research Paper

Abstract

We present an evaluation and a comparison of different visual odometry algorithms selected to be tested on a mobile device equipped with a RGB-D camera. The most promising algorithms from the literature are tested on different mobile devices, some equipped with the Structure Sensor. We evaluate the accuracy of each selected algorithm as well as the memory and CPU consumption, and we show that even without specific optimization, some of them can provide a reliable measure of the device motion in real time.

Keywords

RGB-D Camera Visual odometry Benchmark Mobile device 

Notes

Acknowledgements

This research was supported by the CIFRE grant ANRT contract #2014/0014 funding the collaboration between the French company Telequid and INPT-IRIT laboratory, and in part by the Croatian Science Foundation’s funding of the project IP-11-2013-3717. The authors also acknowledge the French-Croatian Program “Cogito”, Hubert Curien partnership, funding the project “3D reconstruction using smartphone” (project code 33059RF).

References

  1. 1.
    Andreasson, H., Stoyanov, T.: Real time registration of RGB-D data using local visual features and 3D-NDT registration. In: Proceedings of International Conference on Robotics and Automation (ICRA) Workshop on Semantic Perception, Mapping and Exploration (SPME). IEEE Computer Society (2012)Google Scholar
  2. 2.
    ARM. Arm compiler toolchain assembler reference v5.0—neon instructions. http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0489c/CJAJIIGG.html (2016)
  3. 3.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) Proceedings of the 2006 European Conference on Computer Vision (ECCV 2006), volume 3951 of Lecture Notes in Computer Science, pp. 404–417. Springer, Berlin (2006)CrossRefGoogle Scholar
  4. 4.
    Ben, A.: Principles of Concurrent and Distributed Programming, 2nd edn. Addison-Wesley, London (2006)Google Scholar
  5. 5.
    Besl, P.J., McKay, H.D.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)CrossRefGoogle Scholar
  6. 6.
    Borenstein, J., Everett, H.R., Feng, L.: Navigating Mobile Robots: Systems and Techniques. A. K. Peters Ltd, Natick, MA (1996)zbMATHGoogle Scholar
  7. 7.
    Brunetto, N., Salti, S., Fioraio, N., Cavallari, T., Di Stefano, L.: Fusion of inertial and visual measurements for RGB-D SLAM on mobile devices. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops. IEEE Computer Society (2015)Google Scholar
  8. 8.
    Calonder, M., Lepetit, V., Strecha, C., Fua, P.: BRIEF: Binary robust independent elementary features. In: Proceedings of the European Conference on Computer Vision (ECCV), volume 6314 LNCS, pp. 778–792. Springer, Berlin (2010)CrossRefGoogle Scholar
  9. 9.
    Chen, Y., Medioni, G.: Object modeling by registration of multiple range images. In: Proceedings of the 1991 IEEE International Conference on Robotics and Automation (ICRA), vol. 3, pp. 2724–2729 (1991)Google Scholar
  10. 10.
    Driesen, K., Hölzle, U.: The direct cost of virtual function calls in C++. SIGPLAN Not. 31(10), 306–323 (1996)CrossRefGoogle Scholar
  11. 11.
    Dryanovski, I., Valenti, R.G., Xiao, J.: Fast visual odometry and mapping from RGB-D data. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 2305–2310 (2013)Google Scholar
  12. 12.
    Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., Burgard, W.: An evaluation of the RGB-D SLAM system. In: Proceedings of the 2012 IEEE International Conference Robotics and Automation (ICRA), vol. 1, pp. 1691–1696. IEEE Computer Society (2012)Google Scholar
  13. 13.
    Fang, Z., Zhang, Y.: Experimental evaluation of RGB-D visual odometry methods. Int. J. Adv. Robot. Syst. 12(3), 1–16 (2015)CrossRefGoogle Scholar
  14. 14.
    Geiger, A.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), CVPR ’12, pp. 3354–3361. IEEE Computer Society, Washington, DC (2012)Google Scholar
  15. 15.
    Geiger, A., Ziegler, J., Stiller, C.: StereoScan: Dense 3D reconstruction in real-time. In: Proceedings of the Intelligent Vehicles Symposium (IV 2011). IEEE Computer Society, Washington (2011)Google Scholar
  16. 16.
    Google. ATAP Project Tango—Google. http://www.google.com/atap/projecttango/ (2014)
  17. 17.
    Gunther, N.J.: White paper: UNIX Load Average—Part 1: How It Works. White paper http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf, TeamQuest Corporation (2010)
  18. 18.
    Handa, A., Whelan, T., Mcdonald, J., Davison, A.J.: A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA 2014), pp. 1524–1531. IEEE Computer Society (2014)Google Scholar
  19. 19.
    Horn, B.K.P.: Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. A 4(4), 629–642 (1987)CrossRefGoogle Scholar
  20. 20.
    Huang, A.S., Bachrach, A., Henry, P., Krainin, M., Maturana, D., Fox, D., Roy, N.: Visual odometry and mapping for autonomous flight using an RGB-D camera. In: Proceedings of the International Symposium of Robotics Research (ISRR 2011), pp. 1–16 (2011)Google Scholar
  21. 21.
    Hudelist, M.A., Cobârzan, C., Schoeffmann, K.: OpenCV performance measurements on mobile devices. In: Proceedings of International Conference on Multimedia Retrieval—ICMR ’14, pp. 479–482. ACM Press, New York, NY (2014)Google Scholar
  22. 22.
    Jones, G.: Accurate and computationally-inexpensive recovery of ego-motion using optical flow and range flow with extended temporal support. In: Proceedings of the British machine vision conference 2013, pp. 75.1–75.11. British Machine Vision Association (2013)Google Scholar
  23. 23.
    Kerl, C., Sturm, J., Cremers, D.: Robust odometry estimation for RGB-D cameras. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2013), pp. 3748–3754. IEEE Computer Society (2013)Google Scholar
  24. 24.
    Khoshelham, K., Elberink, S.O.: Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors (Basel, Switzerland) 12(2), 1437–54 (2012)CrossRefGoogle Scholar
  25. 25.
    Klingensmith, M., Dryanovski, I., Srinivasa, S., Xiao, J.: Chisel: Real time large scale 3d reconstruction onboard a mobile device. In: Proceedings of the 2015 Robotics Science and Systems Conference (2015)Google Scholar
  26. 26.
    Kolev, K., Tanskanen, P., Speciale, P., Pollefeys, M.: Turning mobile phones into 3d scanners. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3946–3953. IEEE Computer Society (2014)Google Scholar
  27. 27.
    Leutenegger, S., Chli, M., Siegwart, R.Y.: BRISK: binary robust invariant scalable keypoints. In: Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV2011), pp. 2548–2555. IEEE Computer Society (2011)Google Scholar
  28. 28.
    Limare, N.: Integer and floating-point arithmetic speed vs precision. http://nicolas.limare.net/pro/notes/2014/12/12_arit_speed/#index3h2 (2014)
  29. 29.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  30. 30.
    Mantis Vision. MV4D depth sensor for mobile devices. http://www.mv4d.com/mobile.php (2014)
  31. 31.
    Morell-Gimenez, V., Saval-Calvo, M., Azorin-Lopez, J., Garcia-Rodriguez, J., Cazorla, M., Orts-Escolano, S., Fuster-Guillo, A.: A comparative study of registration methods for RGB-D video of static scenes. Sensors (Switzerland) 14(5), 8547–8576 (2014)CrossRefGoogle Scholar
  32. 32.
    Newcombe, R.A., Molyneaux, D., Kim, D., Davison, A.J., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2011), pp. 127–136. IEEE Computer Society (2011)Google Scholar
  33. 33.
    Nguyen, C.V., Izadi, S., Lovell, D.: Modeling kinect sensor noise for improved 3d reconstruction and tracking. In: Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, 3DIMPVT ’12, pp. 524–530. IEEE Computer Society, Washington, DC (2012)Google Scholar
  34. 34.
    Nister, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2004), vol. 1, pp. I–652–I–659. IEEE Computer Society (2004)Google Scholar
  35. 35.
    Occipital. ST depth. STTracker instance with the kSTTrackerTypeKey set to STTrackerDepthBased (2013)Google Scholar
  36. 36.
    Occipital. ST hybrid. STTracker instance with the kSTTrackerTypeKey set to STTrackerDepthAndColorBased (2013)Google Scholar
  37. 37.
    Occipital Inc. The structure sensor. http://structure.io (2014)
  38. 38.
    Ondrúška, P., Kohli, P., Izadi, S.: Mobilefusion: real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE Trans. Vis. Comput. Graph. 21(11), 1251–1258 (2015)CrossRefGoogle Scholar
  39. 39.
    Pomerleau, F., Colas, F., Siegwart, R.: A review of point cloud registration algorithms for mobile robotics. Found. Trends Robot. 4(1–104), 1–104 (2015)Google Scholar
  40. 40.
    Prisacariu, V., Kähler, O., Murray, D., Reid, I.: Simultaneous 3d tracking and reconstruction on a mobile phone. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2013), pp. 89–98. IEEE Computer Society (2013)Google Scholar
  41. 41.
    Prisacariu, V.A., Kähler, Murray: Real-time 3d tracking and reconstruction on mobile phones. IEEE Trans. Vis. Comput. Graph. 21(5), 557–570x (2012)CrossRefGoogle Scholar
  42. 42.
    Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)Google Scholar
  43. 43.
    Rabaud, V., Dimashova, M.: OpenCV ICP and RGBD visual odometry. Class RgbdICPOdometry and function RGBDICPOdometryImpl() in the file https://github.com/Itseez/opencv_contrib/tree/master/modules/rgbd/src/odometry.cpp (2012)
  44. 44.
    Rabaud, V., Dimashova, M.: OpenCV ICP-based visual odometry. Class ICPOdometry and function RGBDICPOdometryImpl() in the file https://github.com/Itseez/opencv_contrib/tree/master/modules/rgbd/src/odometry.cpp (2012)
  45. 45.
    Rabaud, V., Dimashova, M.: OpenCV RGBD-based visual odometry. Class RgbdOdometry and function RGBDICPOdometryImpl() in the file https://github.com/Itseez/opencv_contrib/tree/master/modules/rgbd/src/odometry.cpp (2012)
  46. 46.
    Rabaud, V., Dimashova, M.: OpenCV RGBD module. https://github.com/Itseez/opencv_contrib/tree/master/modules/rgbd (2012)
  47. 47.
    Raynal, M.: Concurrent Programming: Algorithms, Principles, and Foundations. Springer, Berlin (2012)zbMATHGoogle Scholar
  48. 48.
    Rosten, E., Porter, R., Drummond, T.: Faster and better: a machine learning approach to corner detection. IEEE Trans. Pattern Anal. Mach. Intell. 32(1), 105–119 (2010)CrossRefGoogle Scholar
  49. 49.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: An efficient alternative to SIFT or SURF. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2564–2571. IEEE Computer Society (2011)Google Scholar
  50. 50.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling, pp. 145–152. IEEE (2001)Google Scholar
  51. 51.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: Proceedings of the 2011 IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China. IEEE Computer Society, May 9–13 (2011)Google Scholar
  52. 52.
    Scaramuzza, D., Fraundorfer, F.: Visual odometry [tutorial]. IEEE Robot. Autom. Mag. 18(4), 80–92 (2011)CrossRefGoogle Scholar
  53. 53.
    Schöps, T., Engel, J., Cremers, D.: Semi-dense visual odometry for AR on a smartphone. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2014). IEEE Computer Society (2014)Google Scholar
  54. 54.
    Schöps, T., Sattler, T., Häne, C., Pollefeys, M.: 3D modeling on the go: interactive 3D reconstruction of large-scale scenes on mobile devices. In: Proceedings of the International Conference on 3D Vision, Piscataway, NJ. IEEE Computer Society (2015)Google Scholar
  55. 55.
    Steinbrücker, F., Sturm, J., Cremers, D.: Real-time visual odometry from dense RGB-D images. In: Proceedings of the IEEE International Conference on Computer Vision, number 3, pp. 719–722. IEEE Computer Society (2011)Google Scholar
  56. 56.
    Stückler, J., Behnke, S.: Multi-resolution surfel maps for efficient dense 3D modeling and tracking. J. Vis. Commun. Image Represent. 25(1), 137–147 (2014)CrossRefGoogle Scholar
  57. 57.
    Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS 2012), pp. 573–580. IEEE Computer Society (2012)Google Scholar
  58. 58.
    Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M.: Live metric 3d reconstruction on mobile phones. In: Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV), pp. 65–72. IEEE Computer Society (2013)Google Scholar
  59. 59.
    Tomasi, C., Manduchi, R.: Bilateral Filtering for Gray and Color Images. In: Proceedings of the 1998 IEEE International Conference on Computer Vision (ICCV), pp. 839–846. IEEE Computer Society (1998)Google Scholar
  60. 60.
    Whelan, T., Johannsson, H., Kaess, M., Leonard, J., McDonald, J.: Robust real-time visual odometry for dense RGB-D mapping. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2013), Karlsruhe, Germany. IEEE Computer Society (2013)Google Scholar
  61. 61.
    Yang, D.: C++ and Object-Oriented Numeric Computing for Scientists and Engineers. Springer, New York (2001)CrossRefGoogle Scholar
  62. 62.
    Yousif, K., Bab-Hadiashar, A., Hoseinnezhad, R.: An overview to visual odometry and visual slam: applications to mobile robotics. Intell. Ind. Syst. 1(4), 289–311 (2015)CrossRefGoogle Scholar
  63. 63.
    Zhang, J., Kaess, M., Singh, S.: Real-time depth enhanced monocular odometry. In: Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4973–4980. IEEE Computer Society (2014)Google Scholar
  64. 64.
    Zhang, J., Kaess, M., Singh, S.: A real-time method for depth enhanced visual odometry. Auton. Robots 41(1), 31–43 (2015)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.INPT–IRITUniversité de ToulouseToulouseFrance
  2. 2.TelequidToulouseFrance
  3. 3.Faculty of Electrical Engineering and ComputingUniversity of ZagrebZagrebCroatia

Personalised recommendations