Skip to main content
SpringerLink
Log in

Image Registration Algorithm for Deep Learning-Based Stereo Visual Control of Mobile Robots

  • Chapter
  • First Online:
Deep Learning for Unmanned Systems

Part of the book series: Studies in Computational Intelligence ((SCI,volume 984))

  • 1629 Accesses

Abstract

Since the emergence of deep learning as a dominant technique for numerous tasks in the computer vision domain, the robotics community has strived to utilize its potential. Deep learning represents a framework capable of learning the most complex models necessary to carry out various robotic tasks. We propose to integrate deep learning and one of the fundamental robotic algorithms—visual servoing. Fully convolutional neural networks are used for semantic segmentation, which represents the process of labeling every pixel within the image. The obtained information from labeled (categorical) images can be crucial for mobile robot control in dynamic environments. To adequately utilize semantic segmentation for mobile robot control, the segmented images acquired at the desired and the current pose need to be registered (aligned). Since the accuracy of visual servoing depends on the accuracy of the image registration process, we propose to increase the accuracy of mobile robot positioning by analyzing three different optimization algorithms devoted to the registration of categorical images. The standard gradient descent algorithm is compared to the OnePlusOneEvolutionary algorithm, and simulated annealing. Moreover, different cost functions such as Mattes mutual information, global accuracy, and mean intersection over union are also investigated. All the algorithms are tested on our own wheeled mobile robot RAICO (Robot with Artificial Intelligence based COgnition) developed within the Laboratory for robotics and artificial intelligence. The results indicate that the algorithm with a larger exploration to exploitation ratio provides better results. Moreover, the cost function with the steepest convex domain is more advantageous.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

RAICO:

Robot with artificial intelligence based cognition

AI:

Artificial intelligence

ML:

Machine learning

DNNs:

Deep neural networks

IBVS:

Image-based visual servoing

PBVS:

Position-based visual servoing

DVS:

Direct visual servoing

CNN:

Convolutional neural network

VGG16:

Visual geometry group

DOF:

Degree of freedom

FCN:

Fully convolutional network

SGD:

Stochastic gradient descent

ReLU:

Rectified linear unit

mIoU:

Mean intersection over union

MI:

Mutual information

GA:

Global accuracy

SA:

Simulated annealing

GD:

Gradient descent

EV:

OnePlusOneEvolutionary

References

  1. Sünderhauf N et al (2018) The limits and potentials of deep learning for robotics. Int J Rob Res 37(4–5):405–420. https://doi.org/10.1177/0278364918770733

    Article  Google Scholar 

  2. Mitić M, Vuković N, Petrović M, Miljković Z (2018) Chaotic metaheuristic algorithms for learning and reproduction of robot motion trajectories. Neural Comput Appl 30(4):1065–1083. https://doi.org/10.1007/s00521-016-2717-6

    Article  Google Scholar 

  3. Vuković N, Mitić M, Miljković Z (2015) Trajectory learning and reproduction for differential drive mobile robots based on GMM/HMM and dynamic time warping using learning from demonstration framework. Eng Appl Artif Intell 45:388–404. https://doi.org/10.1016/j.engappai.2015.07.002

    Article  Google Scholar 

  4. Miljković Z, Vuković N, Mitić M, Babić B (2013) New hybrid vision-based control approach for automated guided vehicles. Int J Adv Manuf Technol 66(1–4):231–249. https://doi.org/10.1007/s00170-012-4321-y

    Article  Google Scholar 

  5. Petrović M, Miljković Z, Jokić A (2019) A novel methodology for optimal single mobile robot scheduling using whale optimization algorithm. Appl Soft Comput 81:105520. https://doi.org/10.1016/j.asoc.2019.105520

  6. Petrović M, Miljković Z, Babić B, Vuković N, Čović N (2012) Towards a conceptual design of intelligent material transport using artificial intelligence. Strojarstvo 54(3):205–219

    Google Scholar 

  7. Petrović M, Miljković Z, Babić B (2013) Integration of process planning, scheduling, and mobile robot navigation based on TRIZ and multi-agent methodology. FME Trans 41(2):120–129

    Google Scholar 

  8. Pierson HA, Gashler MS (2017) Deep learning in robotics: a review of recent research. Adv Rob 31(16):821–835. https://doi.org/10.1080/01691864.2017.1365009

    Article  Google Scholar 

  9. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Rob Res 37(5):421–436. https://doi.org/10.1177/0278364917710318

    Article  Google Scholar 

  10. Sadeghi F, Levine S (2016) CAD2RL: real single-image flight without a single real image. arXiv Prepr. arXiv:1611.04201

  11. Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In: IEEE International conference on intelligent robots and systems (IROS), pp 23–30. https://doi.org/10.1109/IROS.2017.8202133

  12. Chaumette F, Hutchinson S (2006) Visual servo control. I. Basic approaches. IEEE Rob Autom Mag 13(4):82–90. https://doi.org/10.1109/MRA.2006.250573

    Article  Google Scholar 

  13. Jokić A, Petrović M, Miljković Z (2018) Methods for visual servoing of robotic systems: a state of the art survey. Tehnika 73(6):801–816. https://doi.org/10.5937/tehnika1806801j

    Article  Google Scholar 

  14. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations (ICLR), pp 1–14

    Google Scholar 

  15. Sadeghi F, Toshev A, Jang E, Levine S (2018) Sim2Real viewpoint invariant visual servoing by recurrent control. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 4691–4699. https://doi.org/10.1109/CVPR.2018.00493

  16. Petrović M, Mystkowski A, Jokić A, Đokić L, Miljković Z (2020) Deep learning-based algorithm for mobile robot control in textureless environment. In: 2020 IEEE International conference mechatronic systems and materials (MSM), pp 1–4. https://doi.org/10.1109/msm49833.2020.9201666

  17. Bateux Q, Marchand E, Leitner J, Chaumette F (2018) Training deep neural networks for visual servoing. In: 2018 IEEE International conference on robotics and automation (ICRA), pp 1–8, [Online]. Available: https://hal.inria.fr/hal-01716679/document

  18. Krizhevsky A, Sutskever I, Hinton GE (2020) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1097–1105. Accessed: 18 Aug 2020 [Online]. Available: http://code.google.com/p/cuda-convnet/

  19. Russakovsky O et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  20. Saxena A, Pandya H, Kumar G, Gaud A, Krishna KM (2017) Exploring convolutional networks for end-to-end visual servoing, [Online]. Available: http://arxiv.org/abs/1706.03220

  21. Dosovitskiy A et al. (2015) FlowNet: learning optical flow with convolutional networks. In: IEEE international conference on computer vision, pp 2758–2766

    Google Scholar 

  22. Zhang F, Leitner J, Milford M, Upcroft B, Corke P (2015) Towards vision-based deep reinforcement learning for robotic motion control. arXiv:1511.03791

  23. Zhang F, Leitner J, Ge Z, Milford M, Corke P (2019) Adversarial discriminative sim-to-real transfer of visuo-motor policies. Int J Rob Res 38(10–11):1229–1245

    Article  Google Scholar 

  24. Mitić M, Miljković Z (2014) Neural network learning from demonstration and epipolar geometry for visual control of a nonholonomic mobile robot. Soft Comput 18(5):1011–1025. https://doi.org/10.1007/s00500-013-1121-8

    Article  Google Scholar 

  25. Miljković Z, Mitić M, Lazarević M, Babić B (2013) Neural network reinforcement learning for visual control of robot manipulators. Expert Syst Appl 40(5):1721–1736. https://doi.org/10.1016/j.eswa.2012.09.010

    Article  Google Scholar 

  26. Mithun P, Mehta SA, Shah SV, Bhatnagar G (2020) Student mixture model based visual servoing. arXiv:2006.11347v1

  27. Crombez N, Caron G, Mouaddib EM (2015) Photometric Gaussian mixtures based visual servoing. In: 2015 IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 5486–5491. https://doi.org/10.1109/IROS.2015.7354154

  28. Marchand E (2019) Subspace-based direct visual servoing. IEEE Rob Autom Lett 4(3):2699–2706

    Article  Google Scholar 

  29. Kallem V, Swensen JP, Hager GD, Cowan NJ (2007) Kernel-based visual servoing. In: IEEE/RSJ International conference on intelligent robots and systems, pp 1975–1980

    Google Scholar 

  30. Collewet C, Marchand E (2011) Photometric visual servoing. IEEE Trans Rob 27(4):828–834. https://doi.org/10.1109/TRO.2011.2112593

    Article  Google Scholar 

  31. Bateux Q, Marchand E (2016) Particle filter-based direct visual servoing. In: IEEE International conference on intelligent robots and systems (IROS), pp 4180–4186. https://doi.org/10.1109/IROS.2016.7759615

  32. Dame A, Marchand E (2011) Mutual information-based visual servoing. IEEE Trans Rob 27(5):958–969. https://doi.org/10.1109/TRO.2011.2147090

    Article  Google Scholar 

  33. Bertrand D, Marchand E (2012) Visual servoing using the sum of conditional variance. In: Intelligent robots and systems (IROS), pp 1689–1694

    Google Scholar 

  34. Teulière C, Marchand E (2014) A dense and direct approach to visual servoing using depth maps. IEEE Trans Rob 30(5):1242–1249

    Article  Google Scholar 

  35. Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Garcia-Rodriguez J (2017) A review on deep learning techniques applied to semantic segmentation, [Online]. Available: http://arxiv.org/abs/1704.06857

  36. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

  37. Mitić M, Vuković N, Petrović M, Miljković Z (2015) Chaotic fruit fly optimization algorithm. Knowl Based Syst 89:446–458. https://doi.org/10.1016/j.knosys.2015.08.010

  38. Miljković Z, Petrović M (2017) Application of modified multi-objective particle swarm optimisation algorithm for flexible process planning problem. Int J Comput Integr Manuf 30(2–3):271–291. https://doi.org/10.1080/0951192X.2016.1145804

    Article  Google Scholar 

  39. Petrović M, Mitić M, Vuković N, Miljković Z (2016) Chaotic particle swarm optimization algorithm for flexible process planning. Int J Adv Manuf Technol 85(9–12):2535–2555. https://doi.org/10.1007/s00170-015-7991-4

    Article  Google Scholar 

  40. Petrović M, Vuković N, Mitić M, Miljković Z (2016) Integration of process planning and scheduling using chaotic particle swarm optimization algorithm. Expert Syst Appl 64:569–588. https://doi.org/10.1016/j.eswa.2016.08.019

    Article  Google Scholar 

  41. Styner M, Brechbuhler C, Szekely G, Gerig G (2000) Parametric estimate of intensity inhomogeneities applied to MRI. IEEE Trans Med Imaging 19(3):153–165. https://doi.org/10.1109/42.845174

    Article  Google Scholar 

Download references

Acknowledgements

This work has been financially supported by the Ministry of Education, Science and Technological Development of the Serbian Government, through the project “Integrated research in macro, micro, and nano mechanical engineering—Deep learning of intelligent manufacturing systems in production engineering”, under the contract number 451-03-9/2021-14/200105, and by the Science Fund of the Republic of Serbia, Grant No. 6523109, AI - MISSION4.0, 2020–2022.

Author information

Authors and Affiliations

  1. Faculty of Mechanical Engineering, University of Belgrade, Belgrade, Serbia

    Zoran Miljković, Aleksandar Jokić & Milica Petrović

Authors
  1. Zoran Miljković
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aleksandar Jokić
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Milica Petrović
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aleksandar Jokić .

Editor information

Editors and Affiliations

  1. College of Computer and Information Sciences, Prince Sultan University, Riyadh, Saudi Arabia

    Anis Koubaa

  2. College of Computer and Information Sciences, Prince Sultan University, Riyadh, Saudi Arabia

    Ahmad Taher Azar

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Miljković, Z., Jokić, A., Petrović, M. (2021). Image Registration Algorithm for Deep Learning-Based Stereo Visual Control of Mobile Robots. In: Koubaa, A., Azar, A.T. (eds) Deep Learning for Unmanned Systems. Studies in Computational Intelligence, vol 984. Springer, Cham. https://doi.org/10.1007/978-3-030-77939-9_13

Download citation

Publish with us

Policies and ethics

Search

Navigation