Multi-stage Reinforcement Learning for Object Detection

  • Jonas König
  • Simon Malberg
  • Martin Martens
  • Sebastian NiehausEmail author
  • Artus Krohn-Grimberghe
  • Arunselvan Ramaswamy
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 943)


We present a reinforcement learning approach for detecting objects within an image. Our approach performs a step-wise deformation of a bounding box with the goal of tightly framing the object. It uses a hierarchical tree-like representation of predefined region candidates, which the agent can zoom in on. This reduces the number of region candidates that must be evaluated so that the agent can afford to compute new feature maps before each step to enhance detection quality. We compare an approach that is based purely on zoom actions with one that is extended by a second refinement stage to fine-tune the bounding box after each zoom step. We also improve the fitting ability by allowing for different aspect ratios of the bounding box. Finally, we propose different reward functions to lead to a better guidance of the agent while following its search trajectories. Experiments indicate that each of these extensions leads to more correct detections. The best performing approach comprises a zoom stage and a refinement stage, uses aspect-ratio modifying actions and is trained using a combination of three different reward metrics.


Deep reinforcement leaning Q-learning Object detection 

Supplementary material


  1. 1.
    Itti, L., Rees, G., Tsotsos, J.K.: Neurobiology of attention (2005)CrossRefGoogle Scholar
  2. 2.
    Mathe, S., Pirinen, A., Sminchisescu, C.: Reinforcement learning for visual object detection. IEEE (2016)Google Scholar
  3. 3.
    Bueno, M.B., Nieto, X.G., Marqués, F., Torres, J.: Hierarchical object detection with deep reinforcement learning. arXiv:1611.03718v2 (2016)
  4. 4.
    Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. Int. J. Comput. Vis. 104, 154–171 (2013)CrossRefGoogle Scholar
  5. 5.
    Zitnick, C.L., Piotr, D.: Edge boxes: locating object proposals from edges. In: European Conference on Computer Vision (2014)Google Scholar
  6. 6.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  7. 7.
    Caicedo, J.C., Lazebnik, S.: Active object localization with deep reinforcement learning. IEEE (2015)Google Scholar
  8. 8.
    Maicas, G., Carneiro, G., Bradley, A.P., Nascimento, J.C., Reid, I.: Deep reinforcement learning for active breast lesion detection from DCE-MRI. In: 2007 International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham (2017)Google Scholar
  9. 9.
    Watkins, C.J.C.H.: Learning from Delayed Rewards (1989)Google Scholar
  10. 10.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6 (2015)
  11. 11.
    Mataric, M.J.: Reward functions for accelerated learning. Mach. Learn. Proc. 1994, 181–189 (1994)Google Scholar
  12. 12.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-2010), pp. 807–814 (2010)Google Scholar
  13. 13.
    Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. In: Journal of Machine Learning Research, pp. 1929–1958, 2014Google Scholar
  14. 14.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  15. 15.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 303–338 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Jonas König
    • 1
  • Simon Malberg
    • 1
  • Martin Martens
    • 1
  • Sebastian Niehaus
    • 2
    Email author
  • Artus Krohn-Grimberghe
    • 3
  • Arunselvan Ramaswamy
    • 1
  1. 1.Paderborn UniversityPaderbornGermany
  2. 2.AICURA Medical GmbHBerlinGermany
  3. 3.Lytiq GmbHPaderbornGermany

Personalised recommendations