Toward Effective Soft Robot Control via Reinforcement Learning

  • Haochong Zhang
  • Rongyun Cao
  • Shlomo Zilberstein
  • Feng Wu
  • Xiaoping ChenEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10462)


A soft robot is a kind of robot that is constructed with soft, deformable and elastic materials. Control of soft robots presents complex modeling and planning challenges. We introduce a new approach to accomplish that, making two key contributions: designing an abstract representation of the state of soft robots, and developing a reinforcement learning method to derive effective control policies. The reinforcement learning process can be trained quickly by ignoring the specific materials and structural properties of the soft robot. We apply the approach to the Honeycomb PneuNets Soft Robot and demonstrate the effectiveness of the training method and its ability to produce good control policies under different conditions.


Soft robot control Reinforcement learning PneuNets 



Feng Wu was supported in part by National Natural Science Foundation of China (No. 61603368), the Youth Innovation Promotion Association of CAS (No. 2015373), and Natural Science Foundation of Anhui Province (No. 1608085QF134).


  1. 1.
    Bai, A., Wu, F., Chen, X.: Bayesian mixture modelling and inference based thompson sampling in monte-carlo tree search. In: Proceedings of NIPS, pp. 1646–1654 (2013)Google Scholar
  2. 2.
    Bai, A., Wu, F., Chen, X.: Online planning for large markov decision processes with hierarchical decomposition. ACM Trans. Intell. Syst. Technol. 6(4) (2015). Article No. 45Google Scholar
  3. 3.
    Baird, L., et al.: Residual algorithms: Reinforcement learning with function approximation. In: Proceedings of ICML, pp. 30–37 (1995)Google Scholar
  4. 4.
    Chen, Y., Wu, F., Shuai, W., Wang, N., Chen, R., Chen, X.: Kejia robot - an attractive shopping mall guider. In: Proceedings of ICSR, pp. 145–154 (2015)Google Scholar
  5. 5.
    Chen, Y., Wu, F., Wang, N., Tang, K., Cheng, M., Chen, X.: KeJia-LC: a low-cost mobile robot platform - champion of demo challenge on benchmarking service robots at RoboCup 2015. In: Proceedings of RoboCup, vol. 9513, pp. 60–71 (2015)Google Scholar
  6. 6.
    Cheng, B., Sun, H., Chen, X.: Evolving honeycomb pneumatic finger in bullet physics engine. Robot Intell. Tech. App. 3, 411–423 (2015)Google Scholar
  7. 7.
    Cheng, M., Chen, X., Tang, K., Wu, F., Kupcsik, A., Iocchi, L., Chen, Y., Hsu, D.: Synthetical benchmarking of service robots: a first effort on domestic mobile platforms. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup 2015. LNCS (LNAI), vol. 9513, pp. 377–388. Springer, Cham (2015). doi: 10.1007/978-3-319-29339-4_32 CrossRefGoogle Scholar
  8. 8.
    Duriez, C.: Control of elastic soft robots based on real-time finite element method. In: Proceedings of ICRA, pp. 3982–3987 (2013)Google Scholar
  9. 9.
    Giorelli, M., Renda, F., Ferri, G., Laschi, C.: A feed-forward neural network learning the inverse kinetics of a soft cable-driven manipulator moving in three-dimensional space. In: Proceedings of IROS, pp. 5033–5039 (2013)Google Scholar
  10. 10.
    Hannan, M.W., Walker, I.D.: Kinematics and the implementation of an elephant’s trunk manipulator and other continuum style robots. J. Robotic Syst. 20(2), 45–63 (2003)CrossRefzbMATHGoogle Scholar
  11. 11.
    Inoue, T., Hirai, S.: Modeling of soft fingertip for object manipulation using tactile sensing. In: Proceedings of IROS, pp. 2654–2659 (2003)Google Scholar
  12. 12.
    Kober, J., Peters, J.: Reinforcement learning in robotics: a survey. In: Reinforcement Learning, pp. 579–610 (2012)Google Scholar
  13. 13.
    Laschi, C., Cianchetti, M., Mazzolai, B., Margheri, L., Follador, M., Dario, P.: Soft robot arm inspired by the octopus. Adv. Robot. 26(7), 709–727 (2012)CrossRefGoogle Scholar
  14. 14.
    Laschi, C., Mazzolai, B., Cianchetti, M.: Soft robotics: technologies and systems pushing the boundaries of robot abilities. Sci. Robot. (2016)Google Scholar
  15. 15.
    Lu, D., Zhou, Y., Wu, F., Zhang, Z., Chen, X.: Integrating answer set programming with semantic dictionaries for robot task planning. In: Proceedings of IJCAI (2017)Google Scholar
  16. 16.
    Luo, M., Pan, Y., Skorina, E.H., Tao, W., Chen, F., Ozel, S., Onal, C.D.: Slithering towards autonomy: a self-contained soft robotic snake platform with integrated curvature sensing. Bioinspir. Biomim. 10(5), 055001 (2015)CrossRefGoogle Scholar
  17. 17.
    Mazzolai, B., Margheri, L., Cianchetti, M., Dario, P., Laschi, C.: Soft-robotic arm inspired by the octopus: II. From artificial requirements to innovative technological solutions. Bioinspir. Biomim. 7(2), 025005 (2012)CrossRefGoogle Scholar
  18. 18.
    McMahan, W., Chitrakaran, V., Csencsits, M., Dawson, D., Walker, I.D., Jones, B.A., Pritts, M., Dienno, D., Grissom, M., Rahn, C.D.: Field trials and testing of the OctArm continuum manipulator. In: Proceedings of ICRA, pp. 2336–2341 (2006)Google Scholar
  19. 19.
    Pfeifer, R.: soft robotics - the next generation of intelligent machines. Invited talk on IJCAI (2013)Google Scholar
  20. 20.
    Ramchurn, S.D., Huynh, T.D., Wu, F., Ikuno, Y., Flann, J., Moreau, L., Fischer, J.E., Jiang, W., Rodden, T., Simpson, E., Reece, S., Roberts, S., Jennings, N.R.: A disaster response system based on human-agent collectives. J. Artif. Intell. Res. 57, 661–708 (2016)zbMATHGoogle Scholar
  21. 21.
    Renda, F., Giorelli, M., Calisti, M., Cianchetti, M., Laschi, C.: Dynamic model of a multibending soft robot arm driven by cables. IEEE Trans. Robot. 30(5), 1109–1122 (2014)CrossRefGoogle Scholar
  22. 22.
    Rus, D., Tolley, M.T.: Design, fabrication and control of soft robots. Nature 521(7553), 467–475 (2015)CrossRefGoogle Scholar
  23. 23.
    Shibata, M., Hirai, S.: Soft object manipulation by simultaneous control of motion and deformation. In: Proceedings ICRA, pp. 2460–2465 (2006)Google Scholar
  24. 24.
    Sun, H., Chen, X.-P.: Towards honeycomb pneunets robots. In: Kim, J.-H., Matson, E.T., Myung, H., Xu, P., Karray, F. (eds.) Robot Intelligence Technology and Applications 2. AISC, vol. 274, pp. 331–340. Springer, Cham (2014). doi: 10.1007/978-3-319-05582-4_28 CrossRefGoogle Scholar
  25. 25.
    Tolley, M.T., Shepherd, R.F., Mosadegh, B., Galloway, K.C., Wehner, M., Karpelson, M., Wood, R.J., Whitesides, G.M.: A resilient, untethered soft robot. Soft Robot. 1(3), 213–223 (2014)CrossRefGoogle Scholar
  26. 26.
    Webster, R.J., Jones, B.: Design and kinematic modeling of constant curvature continuum robots: a review. Int. J. Robot. Res. 29(13), 1661–1683 (2010)CrossRefGoogle Scholar
  27. 27.
    Wu, F., Chen, X.: Solving large-scale and sparse-reward DEC-POMDPs with correlation-MDPs. In: Proceedings of RoboCup, pp. 208–219 (2007)Google Scholar
  28. 28.
    Wu, F., Ramchurn, S., Chen, X.: Coordinating human-UAV teams in disaster response. In: Proceedings of IJCAI, pp. 524–530 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Haochong Zhang
    • 1
  • Rongyun Cao
    • 1
  • Shlomo Zilberstein
    • 2
  • Feng Wu
    • 1
  • Xiaoping Chen
    • 1
    Email author
  1. 1.University of Science and Technology of ChinaHefeiChina
  2. 2.University of Massachusetts AmherstAmherstUSA

Personalised recommendations