Robot Learning for Persistent Autonomy

  • Petar KormushevEmail author
  • Seyed Reza Ahmadzadeh
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 42)


Autonomous robots are not very good at being autonomous. They work well in structured environments, but fail quickly in the real world facing uncertainty and dynamically changing conditions. In this chapter, we describe robot learning approaches that help to elevate robot autonomy to the next level, the so-called ‘persistent autonomy’. For a robot to be ‘persistently autonomous’ means to be able to perform missions over extended time periods (e.g. days or months) in dynamic, uncertain environments without need for human assistance. In particular, persistent autonomy is extremely important for robots in difficult-to-reach environments such as underwater, rescue, and space robotics. There are many facets of persistent autonomy, such as: coping with uncertainty, reacting to changing conditions, disturbance rejection, fault tolerance, energy efficiency and so on. This chapter presents a collection of robot learning approaches that address many of these facets. Experiments with robot manipulators and autonomous underwater vehicles demonstrate the usefulness of these learning approaches in real world scenarios.


Fuzzy System Autonomous Underwater Vehicle Autonomous Robot Iterative Learn Control Imitation Learning 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We would like to thank Professor David Lane from the Ocean Systems Laboratory, Heriot-Watt University, UK, for introducing us to the topic of persistent autonomy.

We are grateful to Arnau Carrera, Narcís Palomeras, and Marc Carreras from the Computer Vision and Robotics Group (VICOROB), University of Girona, Spain, for making it possible to conduct real-world experiments with the Girona 500 AUV.

This work was supported by the European project PANDORA: Persistent Autonomy through learNing, aDaptation, Observation and ReplAnning, contract FP7-ICT-288273 (PANDORA 2012).


  1. Abidi MA, Eason RO, Gonzalez RC (1991) Autonomous robotic inspection and manipulation using multisensor feedback. Computer 24(4):17–31CrossRefGoogle Scholar
  2. Ahmadzadeh SR, Kormushev P, Caldwell DG (2013a) Autonomous robotic valve turning: a hierarchical learning approach. In: 2013 IEEE international conference on robotics and automation (ICRA). IEEE, pp 4614–4619Google Scholar
  3. Ahmadzadeh SR, Leonetti M, Kormushev P (2013b) Online direct policy search for thruster failure recovery in autonomous underwater vehicles. In: 6th international workshop on evolutionary and reinforcement learning for autonomous robot system (ERLARS 2013), Taormina, ItalyGoogle Scholar
  4. Ahmadzadeh SR, Jamisola RS, Kormushev P, Caldwell DG (2014a) Learning reactive robot behavior for autonomous valve turning. In: Proceedings of the IEEE international conference on humanoid robots (Humanoids 2014), Madrid, SpainGoogle Scholar
  5. Ahmadzadeh SR, Leonetti M, Carrera A, Carreras M, Kormushev P, Caldwell DG (2014b) Online discovery of AUV control policies to overcome thruster failure. In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6522–6528Google Scholar
  6. Ajoudani A, Lee J, Rocchi A, Ferrati M, Mingo E, Settimi A, Caldwell DG, Bicchi A, Tsagarakis N (2014) A manipulation framework for compliant humanoid COMAN: application to a valve turning task. In: 2014 IEEE-RAS international conference on humanoid robots (Humanoids 2014). IEEE, pp 664–670Google Scholar
  7. Alessandri A, Caccia M, Veruggio G (1998) A model-based approach to fault diagnosis in unmanned underwater vehicles. In: OCEANS’98 conference proceedings, vol 2. IEEE, pp 825–829Google Scholar
  8. Alunni N, Phillips-Grafftin C, Suay HB, Lofaro D, Berenson D, Chernova S, Lindeman RW, Oh P (2013) Toward a user-guided manipulation framework for high-dof robots with limited communication. In: 2013 IEEE international conference on technologies for practical robot applications (TePRA). IEEE, pp 1–6Google Scholar
  9. Anisi DA, Persson E, Heyer C (2011) Real-world demonstration of sensor-based robotic automation in oil & gas facilities. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 235–240Google Scholar
  10. Anisi DA, Skourup C, Petrochemicals A (2012) A step-wise approach to oil and gas robotics. In: IFAC workshop on automatic control in offshore oil and gas production, Trondheim, Norway, vol 31Google Scholar
  11. Antonelli G (2003) A survey of fault detection/tolerance strategies for AUVs and ROVs. In: Caccavale F, Villani L (eds) Fault diagnosis and fault tolerance for mechatronic systems: recent advances. Springer, Berlin, pp 109–127CrossRefGoogle Scholar
  12. Antonelli G (2006) Underwater robots: motion and force control of vehicle-manipulator systems. Springer tracts in advanced robotics. Springer, New YorkGoogle Scholar
  13. Bristow D, Tharayil M, Alleyne AG et al (2006) A survey of iterative learning control. IEEE Control Syst 26(3):96–114CrossRefGoogle Scholar
  14. Caccia M, Bono R, Bruzzone G, Bruzzone G, Spirandelli E, Veruggio G (2001) Experiences on actuator fault detection, diagnosis and accomodation for ROVs. In: International symposiyum of unmanned untethered sub-mersible technologyGoogle Scholar
  15. Carrera A, Ahmadzadeh S, Ajoudani A, Kormushev P, Carreras M, Caldwell D (2012) Towards autonomous robotic valve turning. Cybern Inf Technol 12(3):17–26Google Scholar
  16. Cheng ASF, Leonard NE (1999) Fin failure compensation for an unmanned underwater vehicle. In: Proceedings of the 11th international symposium on unmanned untethered submersible technologyGoogle Scholar
  17. Das SN, Das SK (2004) Determination of coupled sway, roll, and yaw motions of a floating body in regular waves. Int J Math Math Sci 41:2181–2197CrossRefGoogle Scholar
  18. Hamilton K, Lane D, Taylor N, Brown K (2001) Fault diagnosis on autonomous robotic vehicles with recovery: an integrated heterogeneous-knowledge approach. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA, 2001, vol 4. IEEE, pp 3232–3237Google Scholar
  19. Ijspeert AJ, Nakanishi J, Hoffmann H, Pastor P, Schaal S (2013) Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput 25(2):328–373CrossRefMathSciNetzbMATHGoogle Scholar
  20. Jamisola RS, Oetomo DN, Ang MH, Khatib O, Lim TM, Lim SY (2005) Compliant motion using a mobile manipulator: an operational space formulation approach to aircraft canopy polishing. Adv Robot 19(5):613–634CrossRefGoogle Scholar
  21. Karras GC, Bechlioulis CP, Leonetti M, Palomeras N, Kormushev P, Kyriakopoulos KJ, Caldwell DG (2013) On-line identification of autonomous underwater vehicles through global derivative-free optimization. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS)Google Scholar
  22. Khatib O (1987) A unified approach for motion and force control of robot manipulators: the operational space formulation. IEEE J Robot Autom 3(1):43–53CrossRefGoogle Scholar
  23. Kirkpatrick S, Gelatt CD Jr, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680CrossRefMathSciNetzbMATHGoogle Scholar
  24. Konidaris G, Osentoski S, Thomas PS (2011) Value function approximation in reinforcement learning using the fourier basis. In: AAAIGoogle Scholar
  25. Kormushev P, Caldwell DG (2013a) Improving the energy efficiency of autonomous underwater vehicles by learning to model disturbances. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), Tokyo, JapanGoogle Scholar
  26. Kormushev P, Caldwell DG (2013b) Towards improved AUV control through learning of periodic signals. In: Proceedings of the MTS/IEEE international conference on OCEANS 2013, San DiegoGoogle Scholar
  27. Kormushev P, Calinon S, Caldwell DG (2011) Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. Adv Robot 25(5):581–603CrossRefGoogle Scholar
  28. Lane DM, Maurelli F, Kormushev P, Carreras M, Fox M, Kyriakopoulos K (2012) Persistent autonomy: the challenges of the PANDORA project. In: Proceedings of the IFAC MCMCGoogle Scholar
  29. Leonetti M, Kormushev P, Sagratella S (2012) Combining local and global direct derivative-free optimization for reinforcement learning. Cybern Inf Technol 12(3):53–65Google Scholar
  30. Leonetti M, Ahmadzadeh SR, Kormushev P (2013) On-line learning to recover from thruster failures on autonomous underwater vehicles. In: OCEANS 2013. IEEEGoogle Scholar
  31. Moore KL (2012) Iterative learning control for deterministic systems. Springer Science & Business Media, LondonGoogle Scholar
  32. Orsag M, Korpela C, Bogdan S, Oh P (2014) Valve turning using a dual-arm aerial manipulator. In: 2014 international conference on unmanned aircraft systems (ICUAS). IEEE, pp 836–841Google Scholar
  33. PANDORA (2012) Persistent autonomy through learning, adaptation, observation and re-planning., PANDORA European Project
  34. Perrault D, Nahon M (1998) Fault-tolerant control of an autonomous underwater vehicle. In: OCEANS’98 conference proceedings, vol 2. IEEE, pp 820–824Google Scholar
  35. Podder T, Antonelli G, Sarkar N (2000) Fault tolerant control of an autonomous underwater vehicle under thruster redundancy: simulations and experiments. In: Proceedings of the IEEE international conference on robotics and automation, ICRA’00, 2000, vol 2. IEEE, pp 1251–1256Google Scholar
  36. Podder TK, Sarkar N (2001) Fault-tolerant control of an autonomous underwater vehicle under thruster redundancy. Robot Auton Syst 34(1):39–52CrossRefGoogle Scholar
  37. Raibert MH, Craig JJ (1981) Hybrid position/force control of manipulators. J Dyn Syst, Measur, Control 103(2):126–133CrossRefGoogle Scholar
  38. Ribas D, Palomeras N, Ridao P, Carreras M, Mallios A (2012) Girona 500 AUV: from survey to intervention. IEEE/ASME Trans Mechatron 17(1):46–53CrossRefGoogle Scholar
  39. Schaal S, Ijspeert A, Billard A (2003) Computational approaches to motor learning by imitation. Philoso Trans R Soc Lond Ser B: Biol Sci 358(1431):537–547CrossRefGoogle Scholar
  40. Seto ML (2011) An agent to optimally re-distribute control in an underactuated AUV. Int J Intell Def Support Syst 4(1):3–19MathSciNetGoogle Scholar
  41. Storn R, Price K (1997) Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359CrossRefMathSciNetzbMATHGoogle Scholar
  42. Wang L (1999) A course on fuzzy systems. Prentice-Hall press, Upper Saddle RiverGoogle Scholar
  43. Yoshikawa T, Zheng XZ (1993) Coordinated dynamic hybrid position/force control for multiple robot manipulators handling one constrained object. Int J Robot Res 12(3):219–230CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Dyson School of Design EngineeringImperial College LondonLondonUK
  2. 2.iCub FacilityIstituto Italiano di TecnologiaGenoaItaly

Personalised recommendations