Object Recognition and Tracking for Indoor Robots Using an RGB-D Sensor

  • Lixing Jiang
  • Artur Koch
  • Andreas Zell
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 302)


In this paper, we extend and generalize our previously published approach on RGB-D based fruit recognition to be able to recognize different kinds of objects in front of our mobile system. We therefore first extend our segmentation to use depth filtering and clustering with a watershed algorithm on the depth data to detect the target to be recognized. We forward the processed data to extract RGB-D descriptors that are used to recoup complementary object information for the classification and recognition task. After having detected the object once, we apply a simple tracking method to reduce the object search space and the computational load through frequent detection queries. The proposed method is evaluated using the random forest (RF) classifier. Experimental results highlight the effectiveness as well as real-time suitability of the proposed extensions for our mobile system based on real RGB-D data.


RGB-D Mobile systems Segmentation Tracking Classification Recognition 


  1. 1.
    L. Jiang, A. Koch, S. A. Scherer, and A. Zell, "Multi-class fruit classification using RGB-D data for indoor robots," in IEEE Int. Conf. Robotics and Biomimetics (ROBIO), (Shenzhen), 2013.Google Scholar
  2. 2.
    M. Bastan, H. Cam, U. Gudukbay, and O. Ulusoy, "Bilvideo-7: An MPEG-7- compatible video indexing and retrieval system," IEEE Multimedia, vol. 17, no. 3, pp. 62–73, 2010.CrossRefGoogle Scholar
  3. 3.
    B. S. Manjunath, J.-R. Ohm, V. V. Vasudevan, and A. Yamada, "Color and texture descriptors," IEEE Trans. Circuits and Systems for Video Technology (CSVT), vol. 11, pp. 703–715, 2002.Google Scholar
  4. 4.
    G. R. Bradski, “Real time face and object tracking as a component of a perceptual user interface,” in Proc. of the Fourth IEEE Workshop on Applications of Computer Vision (WACV’98), pp. 214–219, Oct. 1998.Google Scholar
  5. 5.
    Y. Khan, A. Masselli, and A. Zell, "Visual terrain classification by flying robots," in IEEE Int. Conf. Robotics and Automation (ICRA), (Saint Paul, MN), pp. 498–503, May 2012.Google Scholar
  6. 6.
    K. Lai, L. Bo, X. Ren, and D. Fox, "A large-scale hierarchical multi-view RGB-D object dataset," in IEEE Int. Conf. Robotics and Automation (ICRA), (Shanghai, China), pp. 1817–1824, 2011.Google Scholar
  7. 7.
    C. Gu, J. Lim, P. Arbelaez, and J. Malik, "Recognition using regions," in IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR), (Miami, FL), pp. 1030–1037, 2009.Google Scholar
  8. 8.
    D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Computer Vision, vol. 60, pp. 91–110, 2004.Google Scholar
  9. 9.
    H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "Speeded-up robust features (SURF)," Comput. Vis. Image Underst., vol. 110, no. 3, pp. 346–359, 2008.Google Scholar
  10. 10.
    A. Frome, D. Huber, R. Kolluri, T. Bülow, and J. Malik, "Recognizing objects in range data using regional point descriptors," in IEEE Pro. European Conf. Computer Vision (ECCV), pp. 224–237, May 2004.Google Scholar
  11. 11.
    A. E. Johnson and M. Hebert, "Using spin images for efficient object recognition in cluttered 3D scenes," IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), pp. 433–449, 1999.Google Scholar
  12. 12.
    A. Karpathy, S. Miller, and L. Fei-Fei, "Object discovery in 3D scenes via shape analysis," in IEEE Int. Conf. Robotics and Automation (ICRA), (Karlsruhe, Germany), pp. 290–294, May 2013.Google Scholar
  13. 13.
    L. Bo, X. Ren, and D. Fox, "Depth kernel descriptors for object recognition," in IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), (California), pp. 821–826, 2011.Google Scholar
  14. 14.
    J. Fischer, R. Bormann, G. Arbeiter, and A. Verl, "A feature descriptor for texture-less object representation using 2D and 3D cues from RGB-D data," in IEEE Int. Conf. Robotics and Automation (ICRA), (Karlsruhe, Germany), pp. 2104–2109, May 2013.Google Scholar
  15. 15.
    R. Socher, B. Huval, B. Bhat, C. D. Manning, and A. Y. Ng, "Convolutional-recursive deep learning for 3D object classification," in Advances in Neural Information Processing Systems (NIPS), 2012.Google Scholar
  16. 16.
    M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, "The WEKA data mining software: an update," SIGKDD Explor. Newsl., vol. 11, pp. 10–18, 2009.Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of TuebingenTuebingenGermany

Personalised recommendations