Enabling Imagination: Generative Adversarial Network-Based Object Finding in Robotic Tasks

  • Huimin Che
  • Ben Hu
  • Bo Ding
  • Huaimin Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10639)


The skill to find objects in a real world situation is important for mobile robots. Existing works of robotic vision-based object finding is based on the traditional training and classification paradigm, which means that a robot can only detect objects with the fixed and pre-trained classification labels. It is of great challenge for robots to find an untrained object, even if a complex description of the object has been given. In this paper, we proposed a vision-based object detection approach for robotic finding names Generative Search. It is inspired by the object detection model that when an unfamiliar object needs to be found through a complex description, human would “imagine” the object in his or her brain and then find the object which is mostly like the imagined object profile. By adopting a Generative Adversarial Network (GAN), our approach enables the robot to generate the object virtually according to the given description. Then, we use pre-trained deep neural networks to match the generated image with images in the robotic vision. At the implementation level, we adopt the cloud robotic architecture to promote the algorithm efficiency. The experiments on both open datasets and real robotic scenarios have proved the significant promotion of object finding accuracy when a robot searching an unfamiliar object with a complex description.


Robotic object finding GAN Image matching 



This work is partially supported by the National Natural Science Foundation of China (nos. 91118008 and 61202117), the special program for the applied basic research of the National University of Defense Technology (no. ZDYYJCYJ20140601), and the Jiangsu Future Networks Innovation Institute Prospective Research Project on Future Networks (no. BY2013095-2-08).


  1. 1.
    Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems, pp. 2553–2561 (2013)Google Scholar
  2. 2.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  3. 3.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)Google Scholar
  4. 4.
    Aydemir, A., Sjöö, K., Jensfelt, P.: Object search on a mobile robot using relational spatial information. In: Proceedings of International Conference on Intelligent Autonomous Systems, pp. 111–120 (2010)Google Scholar
  5. 5.
    Saigol, Z., Ridder, B., Wang, M., Dearden, R., Fox, M., Hawes, N., Lane, D.M., Long, D.: Efficient search for known objects in unknown environments using autonomous indoor robots. In: IROS Workshop on Task Planning for Intelligent Robots in Service and Manufacturing (2015)Google Scholar
  6. 6.
    Garvey, T.D.: Perceptual strategies for purposive vision (1976)Google Scholar
  7. 7.
    Torralba, A., Murphy, K.P., Freeman, W.T., Rubin, M.A., et al.: Context-based vision system for place and object recognition. In: ICCV, vol. 3, pp. 273–280 (2003)Google Scholar
  8. 8.
    Li, Y., Wang, H., Ding, B., Shi, P., Liu, X.: Toward QoS-aware cloud robotic applications: a hybrid architecture and its implementation. In: 2016 International IEEE Conferences Ubiquitous Intelligence & Computing, pp. 33–40. IEEE (2016)Google Scholar
  9. 9.
    López, D.G., Sjo, K., Paul, C., Jensfelt, P.: Hybrid laser and vision based object search and localization. In: IEEE International Conference on Robotics and Automation, ICRA 2008, pp. 2636–2643. IEEE (2008)Google Scholar
  10. 10.
    Kim, H.S., Jain, R., Volz, R.: Object recognition using multiple views. In: Proceedings 1985 IEEE International Conference on Robotics and Automation, vol. 2, pp. 28–33. IEEE (1985)Google Scholar
  11. 11.
    Zhang, H., Xu, T., Li, H., Zhang, S., Huang, X., Wang, X., Metaxas, D.: Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks. arXiv preprint arXiv:1612.03242 (2016)
  12. 12.
    Santana, E., Hotz, G.: Learning a driving simulator. arXiv preprint arXiv:1608.01230 (2016)
  13. 13.
    Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems, pp. 4565–4573 (2016)Google Scholar
  14. 14.
    Lin, K., Yang, H.F., Hsiao, J.H., Chen, C.S.: Deep learning of binary hash codes for fast image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 27–35 (2015)Google Scholar
  15. 15.
    Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013)CrossRefGoogle Scholar
  16. 16.
    Hu, B., Wang, H., Zhang, P., Ding, B., Che, H.: Cloudroid: a cloud framework for transparent and QoS-aware robotic computation outsourcing. arXiv preprint arXiv:1705.05691 (2017)
  17. 17.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)Google Scholar
  18. 18.
    Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, pp. 722–729. IEEE (2008)Google Scholar
  19. 19.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.National Key Lab of Parallel and Distributed Processing, College of ComputerNational University of Defense TechnologyChangshaChina

Personalised recommendations