Journal of Real-Time Image Processing

, Volume 14, Issue 3, pp 667–683 | Cite as

Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels

  • Claudia Cruz-Martinez
  • José Martínez-Carranza
  • Walterio Mayol-Cuevas
Special Issue Paper


In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.


Superpixel-based segmentation SLIC (simple linear iterative clustering) Visual SLAM (simultaneous localization and mapping) 3D mapping GPU architecture 



The first author is supported by the Mexican National Council for Science and Technology (CONACyT) studentship number 624142. This work has been partially funded by the Royal Society through the Newton Advanced Fellowship with reference NA140454.


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Alcantarilla, P., Stent, S., Ros, G., Arroyo, R., Gherardi, R.: Street-view change detection with deconvolutional networks. In: Robotics: Science and Systems (RSS), Michigan, USA (2016)Google Scholar
  3. 3.
    Choi, K.S.., Oh, K.W..: Fast simple linear iterative clustering by early candidate cluster elimination. In: Iberian Conference on Pattern Recognition and Image Analysis, Springer, pp. 579–586 (2015)Google Scholar
  4. 4.
    Concha, A., Civera, J.: Using superpixels in monocular SLAM. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 365–372 (2014)Google Scholar
  5. 5.
    Concha, A., Civera, J.: DPPTAM: Dense piecewise planar tracking and mapping from a monocular sequence. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 5686–5693 (2015)Google Scholar
  6. 6.
    Concha, A., Hussain, W., Montano, L., Civera, J.: Incorporating scene priors to dense monocular mapping. Auton. Robots 39(3), 279–292 (2015)CrossRefGoogle Scholar
  7. 7.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  8. 8.
    Cruz Martinez, C., Martinez Carranza, J., Mayol-Cuevas, W., Arias Estrada, M.O.: Enhancing 3d mapping via real-time superpixel-based segmentation. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Adjunct Proceedings, IEEE, pp. 90–95 (2016)Google Scholar
  9. 9.
    Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13(2), 99–110 (2006)CrossRefGoogle Scholar
  10. 10.
    Eade, E., Drummond, T.: Edge landmarks in monocular SLAM. In: Proceedings of British Machine Vision Conference, Citeseer (2006)Google Scholar
  11. 11.
    Engel, J., Sturm, J., Cremers, D.: Semi-dense visual odometry for a monocular camera. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1456 (2013)Google Scholar
  12. 12.
    Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Computer Vision–ECCV 2014, pp. 834–849. Springer (2014)Google Scholar
  13. 13.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)CrossRefGoogle Scholar
  14. 14.
    Forster, C., Pizzoli, M., Scaramuzza, D.: SVO: fast semi-direct monocular visual odometry. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 15–22 (2014)Google Scholar
  15. 15.
    Hartley, R., Andrew, Z.: Multiple View Geometry in Computer Vision. Cambridge University Press, Espaa (2003)zbMATHGoogle Scholar
  16. 16.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: using kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 31(5), 647–663 (2012)CrossRefGoogle Scholar
  17. 17.
    Hornung, A., Wurm, K.M., Bennewitz, M., Stachniss, C., Burgard, W.: OctoMap: an efficient probabilistic 3D mapping framework based on octrees. Auton. Robots 34(3), 189–206 (2013)CrossRefGoogle Scholar
  18. 18.
    Jiang, L., Lu, H., Koch, A., Zell, A.: Superpixel segmentation based gradient maps on RGB-D dataset. In: 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 1359–1364 (2015)Google Scholar
  19. 19.
    Karlsson, N., Di Bernardo, E., Ostrowski, J., Goncalves, L., Pirjanian, P., Munich, M.E.: The vSLAM algorithm for robust localization and mapping. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, 2005. ICRA 2005, IEEE, pp. 24–29 (2005)Google Scholar
  20. 20.
    Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, 2007. ISMAR 2007. IEEE, pp. 225–234 (2007)Google Scholar
  21. 21.
    Klein, G., Murray, D.: Improving the agility of keyframe-based SLAM. In: European Conference on Computer Vision, Springer, pp. 802–815 (2008)Google Scholar
  22. 22.
    Li, M., Mourikis, A.I.: High-precision, consistent EKF-based visual-inertial odometry. Int. J. Robot. Res. 32(6), 690–711 (2013)CrossRefGoogle Scholar
  23. 23.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Eighth IEEE International Conference on Computer Vision, 2001. ICCV 2001. Proceedings. IEEE, vol. 2, pp. 416–423 (2001)Google Scholar
  24. 24.
    Müller, A.C., Behnke, S.: Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 6232–6237 (2014)Google Scholar
  25. 25.
    Mur-Artal, R., Tardós, J.D.: Probabilistic semi-dense mapping from highly accurate feature-based monocular slam. In: Proceedings of Robotics: Science and Systems, Rome, Italy 1 (2015)Google Scholar
  26. 26.
    Mur-Artal, R., Montiel, J., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)CrossRefGoogle Scholar
  27. 27.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor Segmentation and Support Inference from RGBD Images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol 7576. Springer, Berlin, Heidelberg (2012)Google Scholar
  28. 28.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, pp. 127–136 (2011)Google Scholar
  29. 29.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J., DTAM: dense tracking and mapping in real-time. In: 2011 IEEE International Conference on Computer Vision (ICCV), IEEE, pp. 2320–2327 (2011)Google Scholar
  30. 30.
    Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 2609–2616 (2014)Google Scholar
  31. 31.
    Ren, C.Y., Reid, I.: gSLIC: a real-time implementation of SLIC superpixel segmentation. University of Oxford, Department of Engineering, technical report (2011)Google Scholar
  32. 32.
    Ren, C.Y., Prisacariu, V.A.: Reid ID gSLICr: SLIC superpixels at over 250 Hz. arXiv preprint arXiv:150904232 (2015)
  33. 33.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: 2011 International conference on computer vision, IEEE, pp. 2564–2571 (2011)Google Scholar
  34. 34.
    Rusu, R.B., Cousins, S.: 3d is here: point cloud library (PCL). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 1–4 (2011)Google Scholar
  35. 35.
    Salas-Moreno, R.F., Newcombe, R., Strasdat, H., Kelly, P.H., Davison, A.J., et al.: Slam++: simultaneous localisation and mapping at the level of objects. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 1352–1359 (2013)Google Scholar
  36. 36.
    Schneider, P., Eberly, D.H.: Geometric Tools for Computer Graphics. Morgan Kaufmann, San Francisco (2002)Google Scholar
  37. 37.
    Schöps, T., Engel, J., Cremers, D. Semi-dense visual odometry for AR on a smartphone. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, pp. 145–150 (2014)Google Scholar
  38. 38.
    Steinbrücker, F., Sturm, J., Cremers, D.: Volumetric 3d mapping in real-time on a cpu. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 2021–2028 (2014)Google Scholar
  39. 39.
    Stühmer, J., Gumhold, S., Cremers, D.: Real-time dense geometry from a handheld camera. In: Joint Pattern Recognition Symposium, Springer, pp. 11–20 (2010)Google Scholar
  40. 40.
    Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: Proceedings of the International Conference on Intelligent Robot Systems (IROS) (2012)Google Scholar
  41. 41.
    Weikersdorfer, D., Gossow, D., Beetz, M.: Depth-adaptive superpixels. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp. 2087–2090 (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.National Institute of Astrophysics, Optics and ElectronicsPueblaMexico
  2. 2.University of BristolBristolUK

Personalised recommendations