Object detection based on color and shape features for service robot in semi-structured indoor environment

  • Haojie Li
  • Qijie ZhaoEmail author
  • Xianfa Li
  • Xudong Zhang
Regular Paper


Intelligent service robot is a challenging area of research that is rapidly expanding in our daily life. To meet robot’s object detection requirements in a messy surrounding, this paper provides a visual object detection method based on color and shape features. Firstly, a color hierarchical model and a multi-size filter are built to obtain initial object regions from scene image. Then a straight line-corner-arc strategy is presented to detect shape features. After comparing color and shape features with known-objects’ features stored in database, the detection scope is narrowed. So speeded up robust features algorithm is used to quickly match object features. The proposed method is tested by mobile robot in a semi-structured indoor environment. Finally, combined with the above steps, the total detection accuracy achieves 88.5% that confirms the feasibility of the proposed method.


Service robot Object detection Shape detection Scene segmentation Speeded up robust features (SURF) 



This work was supported by the National Natural Science Foundation of China under Grant no. 61101177.


  1. Banerjee, J., Banerji, S., Ray, R., Shome, S.N.: Fuzzy based object shape recognition using translation, rotation and scale invariant parameters: an automatic approach. Springer, New Delhi (2016)CrossRefGoogle Scholar
  2. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (SURF). Comput Vis Image Underst 110(3), 346–359 (2008)CrossRefGoogle Scholar
  3. Budiharto W, Gunawan AAS (2016) Development of coffee maker service robot using speech and face recognition systems using POMDP. In: First International Workshop on Pattern Recognition 2016, p. 1001110. International Society for Optics and PhotonicsGoogle Scholar
  4. Busta M, Neumann L, Matas J (2017) Deep TextSpotter: an end-to-end trainable scene text localization and recognition framework. In: IEEE International Conference on Computer Vision 2017, pp. 2223–2231Google Scholar
  5. Cheng MM, Liu Y, Hou Q, Bian J, Torr P, Hu SM, Tu Z (2016) HFS Hierarchical feature selection for efficient image segmentation.In: European Conference on Computer Vision 2016, Springer, Cham, pp. 867–882CrossRefGoogle Scholar
  6. Chung, H.-Y., Hou, C.-C., Chen, Y.-S.: Indoor intelligent mobile robot localization using fuzzy compensation and Kalman filter to fuse the data of gyroscope and magnetometer. IEEE Trans Industr Electron 62(10), 6436–6447 (2015)CrossRefGoogle Scholar
  7. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on 2005, pp. 886–893Google Scholar
  8. Ekvall, S., Kragic, D., Jensfelt, P.: Object detection and mapping for service robot tasks. Robotica 25(2), 175–187 (2007)CrossRefGoogle Scholar
  9. Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE Trans Pattern Anal Mach Intell 28(4), 594–611 (2006)CrossRefGoogle Scholar
  10. Ge L, Ju R, Ren T, Wu G (2015) Interactive RGB-D image segmentation using hierarchical graph cut and geodesic distance. In: Pacific Rim Conference on Multimedia 2015, Springer, Cham, pp. 114–124Google Scholar
  11. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference 1988, vol. 50, p. 10.5244. Manchester, UKGoogle Scholar
  12. Hernandez-Lopez, J.-J., Quintanilla-Olvera, A.-L., López-Ramírez, J.-L., Rangel-Butanda, F.-J., Ibarra-Manzano, M.-A., Almanza-Ojeda, D.-L.: Detecting objects using color and depth segmentation with Kinect sensor. Procedia Technol 3, 196–204 (2012)CrossRefGoogle Scholar
  13. Hossain, M.A., Kurnia, R., Nakamura, A., Kuno, Y.: Interactive object recognition through hypothesis generation and confirmation. IEICE TRANS Inf Syst 89(7), 2197–2206 (2006)CrossRefGoogle Scholar
  14. Jiang, L., Koch, A., Zell, A.: Object recognition and tracking for indoor robots using an RGB-D sensor, pp. 859–871. Springer, Cham (2016a)Google Scholar
  15. Jiang, L., Ye, Y., Xu, G.: An efficient curve detection algorithm. Optik Int J Light Electron Optics 127(1), 232–238 (2016b)CrossRefGoogle Scholar
  16. Jing, D., Xia, Y., Qifeng, Y.: Fast line segment detection based on edge connecting. Acta Optica Sinica 33(3), 0315003 (2013)CrossRefGoogle Scholar
  17. Karpathy A, Fei-Fei L (2015) Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015, pp. 3128–3137Google Scholar
  18. Kimoto M, Iio T, Shiomi M et al (2015) Improvement of object reference recognition through human robot alignment[C]//IEEE International Symposium on Robot and Human Interactive Communication.Google Scholar
  19. Kumar V, Pandey S, Pal A, Sharma S (2016) Edge detection based shape identification arXiv:1604.02030.
  20. Landsiedel, C., Rieser, V., Walter, M., Wollherr, D.: A review of spatial reasoning and interaction for real-world robotics. Adv Robot 31(5), 222–242 (2017)CrossRefGoogle Scholar
  21. Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. J Mach Learn Res 17(1), 1334–1373 (2016)MathSciNetzbMATHGoogle Scholar
  22. Liberda A, Lilja A, Langborn B, Lindström J, Kahl F, Larsson M (2016) Image segmentation and convolutional neural networks as tools for indoor scene understanding. Final report. Bachelor’s Thesis at Signals and systems - SSYX02-16-31Google Scholar
  23. Maeyama, S., Takahashi, Y., Watanabe, K.: A solution to SLAM problems by simultaneous estimation of kinematic parameters including sensor mounting offset with an augmented UKF. Adv Robot 29(17), 1137–1149 (2015)CrossRefGoogle Scholar
  24. Mansur A, Sakata K, Rukhsana T, Kobayashi Y, Kuno Y (2008) Human robot interaction through simple expressions for object recognition. In: Robot and Human Interactive Communication, 2008. RO-MAN 2008. The 17th IEEE International Symposium on 2008, pp. 647–652Google Scholar
  25. Matas J, Murino V, Rosenhahn B, Leal-Taixé L (2015) Holistic scene understanding (Dagstuhl Seminar 15081). In: Dagstuhl Reports 2015, vol. 2. Schloss Dagstuhl-Leibniz-Zentrum fuer InformatikGoogle Scholar
  26. Mekhtiche, M., Bencherif, M., Algabri, M., Alsulaiman, M., Hedjar, R., Faisal, M., AlMutib, K.: Real time object detection tracking over a mobile platform. Indian J Sci Technol 8(22), 1–7 (2016)Google Scholar
  27. Muthugnanambika M, Padmavathi S (2017) Feature detection for color images using SURF. In: International Conference on Advanced Computing and Communication Systems 2017, pp. 1–4Google Scholar
  28. Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vision 42(3), 145–175 (2001)CrossRefGoogle Scholar
  29. Oliva, A., Torralba, A.: The role of context in object recognition. Trends Cognit Sci 11(12), 520 (2007)CrossRefGoogle Scholar
  30. Pan, H.P., Li-Hua, H.U., Liu, Y.: A target recognition algorithm based on color clustering and seed filling. J Mech Elect Eng 2011, 7 (2011)Google Scholar
  31. Patraucean, V., Gurdjos, P., Gioi, R.G.V.: Joint A contrario ellipse and line detection. IEEE Trans Pattern Anal Mach Intell 39(4), 788–802 (2017)CrossRefGoogle Scholar
  32. Randelli, G., Bonanni, T.M., Iocchi, L., Nardi, D.: Knowledge acquisition through human–robot multimodal interaction. Intel Serv Robot 6(1), 19–31 (2013)CrossRefGoogle Scholar
  33. Rangel, J.C., Cazorla, M., García-Varea, I., Martínez-Gómez, J., Fromont, É., Sebban, M.: Scene classification based on semantic labeling. Adv Robot 30(11–12), 758–769 (2016)CrossRefGoogle Scholar
  34. Richtsfeld, A., Mörwald, T., Prankl, J., Zillich, M., Vincze, M.: Learning of perceptual grouping for object segmentation on RGB-D data. J Vis Commun Image Represent 25(1), 64–73 (2014)CrossRefGoogle Scholar
  35. Shen, L.L., Zhen, J.I.: Gabor wavelet selection and SVM classification for object recognition. Acta Autom Sin 35(4), 350–355 (2009)MathSciNetzbMATHGoogle Scholar
  36. Shi J (2002) Good features to track. In: Computer Vision and Pattern Recognition, 1994. Proceedings CVPR 1994, IEEE Computer Society Conference on 2002, pp. 593–600Google Scholar
  37. Shotton, J., Blake, A., Cipolla, R.: Multiscale categorical object recognition using contour fragments. IEEE Trans Pattern Anal Mach Intell 30(7), 1270–1281 (2008)CrossRefGoogle Scholar
  38. Stark M, Goesele M, Schiele B (2009) A shape-based object class model for knowledge transfer. In: Computer Vision, 2009 IEEE 12th International Conference on 2009, pp. 373–380Google Scholar
  39. Teo, C.L., Fermüller, C., Aloimonos, Y.: A Gestaltist approach to contour-based object recognition: combining bottom-up and top-down cues. Int J Robot Res 34(4–5), 627–652 (2015)CrossRefGoogle Scholar
  40. Turk, M.A., Morgenthaler, D.G., Gremban, K.D., Marra, M.: VITS-A vision system for autonomous land vehicle navigation. IEEE Trans Pattern Anal Mach Intell 10(3), 342–361 (1988)CrossRefGoogle Scholar
  41. Vincze M, Bajones M, Suchi M, Wolf D, Weiss A, Fischinger D, da la Puente P (2016) Learning and detecting objects with a mobile robot to assist older adults in their homes. In: European Conference on Computer Vision 2016, Springer, Cham, pp. 316–330Google Scholar
  42. Yang, C., Feinen, C., Tiebe, O., Shirahama, K., Grzegorzek, M.: Shape-based object matching using interesting points and high-order graphs. Pattern Recogn Lett 83(P3), 251–260 (2016)CrossRefGoogle Scholar
  43. Zaki, H.F.M., Shafait, F., Mian, A.: Learning a deeply supervised multi-modal RGB-D embedding for semantic scene and object category recognition[J]. Robot Auton Syst 92, 41–52 (2017)CrossRefGoogle Scholar
  44. Zhang Xd, Zhao QJ, Meng QX, Tu DW, Yi JG (2017) A new scene segmentation method based on color information for mobile robot in indoor environment. In: Wearable Sensors and Robots. Springer, Cham, pp. 353–363Google Scholar
  45. Zhang L, Jie-Xin PU, Fan QH (2008) Objects recognition based on genetic algorithm and BP neural network. Computer Engineering and DesignGoogle Scholar
  46. Zhao J, Liu H, Feng Y, Yuan S, Cai W (2015) BE-SIFT: A more brief and efficient sift image matching algorithm for computer vision. In: Computer and information technology; ubiquitous computing and communications; dependable, autonomic and secure computing; pervasive intelligence and computing (CIT/IUCC/DASC/PICOM), 2015. IEEE International Conference on 2015, pp. 568–574Google Scholar
  47. Zhao, Q., Li, X., Lu, J., Yi, J.: Monocular vision-based parameter estimation for mobile robotic painting. IEEE Trans Instrum Meas 68(10), 3589–3599 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.School of Mechatronic Engineering and AutomationShanghai UniversityShanghaiChina

Personalised recommendations