Skip to main content
SpringerLink
Log in

Dynamical Perceptual-Motor Primitives for Better Deep Reinforcement Learning Agents

  • Conference paper
  • First Online:
Advances in Practical Applications of Agents, Multi-Agent Systems, and Social Good. The PAAMS Collection (PAAMS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12946))

Abstract

Recent innovations in Deep Reinforcement Learning (DRL) and Artificial Intelligence (AI) techniques have allowed for the development of artificial agents that can outperform human counterparts. But when it comes to multiagent task contexts, the behavioral patterning of AI agents is just as important as their performance. Indeed, successful multi-agent interaction requires that co-actors behave reciprocally, anticipate each other’s behaviors, and readily perceive each other’s behavioral intentions. Thus, developing AI agents that can produce behaviors compatible with human co-actors is of vital importance. Of particular relevance here, research exploring the dynamics of human behavior has demonstrated that many human behaviors and actions can be modeled using a small set of dynamical perceptual-motor primitives (DPMPs) and, moreover, that these primitives can also capture the complex behavior of humans in multiagent scenarios. Motived by this understanding, the current paper proposes methodologies which use DPMPs to augment the training and action dynamics of DRL agents to ensure that the agents inherit the essential pattering of human behavior while still allowing for optimal exploration of the task solution space during training. The feasibility of these methodologies is demonstrated by creating hybrid DPMP-DRL agents for a multiagent herding task. Overall, this approach leads to faster training of DRL agents while also exhibiting behavior characteristics of expert human actors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Berner, C., et al.: Dota 2 with large scale deep reinforcement learning. arXiv arXiv:1912.06680 (2019)

  2. Vinyals, O., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019)

    Article  Google Scholar 

  3. Pohlen, T., et al.: Observe and look further: achieving consistent performance on Atari. arXiv arXiv:1805.11593 (2018)

  4. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  5. Shek, J.: Takeaways from OpenAI Five [AI/ML, Dota Summary] (2019). https://senrigan.io/blog/takeaways-from-openai-5/

  6. Carroll, M., et al.: On the utility of learning about humans for human-AI coordination. In: Advances in Neural Information Processing Systems, NeurIPS 2019, vol. 32 (2019)

    Google Scholar 

  7. Nalepka, P., et al.: Interaction flexibility in artificial agents teaming with humans. In: CogSci 2021 (2021). https://escholarship.org/uc/item/9ks6n70q

  8. Bain, M., Sammut, C.: A framework for behavioural cloning. In: Machine Intelligence 15. Intelligent Agents, pp. 103–129. Oxford University, GBR (1999). St. Catherine’s College, Oxford, July 1995

    Google Scholar 

  9. Ho, J., Ermon, S.: Generative adversarial imitation learning. arXiv arXiv:1606.03476 (2016)

  10. Maclin, R., et al.: Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In: AAAI (2005)

    Google Scholar 

  11. Amodei, D., et al.: Concrete problems in AI safety. arXiv arXiv:1606.06565 (2016)

  12. Osa, T., et al.: An algorithmic perspective on imitation learning. Found. Trends Robot. 7, 1–179 (2018)

    Article  Google Scholar 

  13. Patil, G., et al.: Hopf bifurcations in complex multiagent activity: the signature of discrete to rhythmic behavioral transitions. Brain Sci. 10, 536 (2020)

    Article  Google Scholar 

  14. Nalepka, P., et al.: Human social motor solutions for human–machine interaction in dynamical task contexts. Proc. Natl. Acad. Sci. U. S. A. 116, 1437–1446 (2019)

    Article  Google Scholar 

  15. Richardson, M.J., et al.: Modeling embedded interpersonal and multiagent coordination. In: Proceedings of the 1st International Conference on Complex Information Systems, pp. 155–164. SCITEPRESS - Science and Technology Publications (2016)

    Google Scholar 

  16. Warren, W.H.: The dynamics of perception and action. Psychol. Rev. 113, 358–389 (2006)

    Article  Google Scholar 

  17. Kelso, J.A.S.: Dynamic Patterns: The Self-Organization of Brain and Behavior. MIT Press, Cambridge (1997)

    Google Scholar 

  18. Schmidt, R.C., Richardson, M.J.: Dynamics of interpersonal coordination. In: Fuchs, A., Jirsa, V.K. (eds.) Coordination: Neural, Behavioral and Social Dynamics, pp. 281–308. Springer , Heidelberg (2008). https://doi.org/10.1007/978-3-540-74479-5_14

    Chapter  Google Scholar 

  19. Nalepka, P., et al.: Herd those sheep: emergent multiagent coordination and behavioral-mode switching. Psychol. Sci. 28, 630–650 (2017)

    Article  Google Scholar 

  20. Sternad, D., et al.: Bouncing a ball: tuning into dynamic stability. J. Exp. Psychol. Hum. Percept. Perform. 27, 1163–1184 (2001)

    Article  Google Scholar 

  21. Fajen, B.R., et al.: A dynamical model of visually-guided steering, obstacle avoidance, and route selection. Int. J. Comput. Vis. 54, 13–34 (2003). https://doi.org/10.1023/A:1023701300169

    Article  MATH  Google Scholar 

  22. Lamb, M., et al.: To pass or not to pass: modeling the movement and affordance dynamics of a pick and place task. Front. Psychol. 8, 1061 (2017)

    Article  Google Scholar 

  23. Ijspeert, A.J., et al.: Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25, 328–373 (2013). https://doi.org/10.1162/NECO_a_00393

    Article  MathSciNet  MATH  Google Scholar 

  24. Hogan, N., Sternad, D.: On rhythmic and discrete movements: reflections, definitions and implications for motor control. Exp. Brain Res. 181(1), 13–30 (2007). https://doi.org/10.1007/s00221-007-0899-y

    Article  Google Scholar 

  25. Kay, B.A., et al.: Space-time behavior of single and bimanual rhythmical movements: data and limit cycle model. J. Exp. Psychol. Hum. Percept. Perform. 13, 178–192 (1987)

    Article  Google Scholar 

  26. Vesper, C., et al.: Joint action: mental representations, shared information and general mechanisms for coordinating with others. Front. Psychol. 07, 2039 (2017)

    Article  Google Scholar 

  27. Repp, B.H., Keller, P.E.: Adaptation to tempo changes in sensorimotor synchronization: effects of intention, attention, and awareness. Q. J. Exp. Psychol. Sect. A Hum. Exp. Psychol. 57, 499–521 (2004)

    Article  Google Scholar 

  28. Lagarde, J.: Challenges for the understanding of the dynamics of social coordination. Front. Neurorobot. 7, 18 (2013)

    Article  Google Scholar 

  29. Richardson, M.J., et al.: Challenging the egocentric view of coordinated perceiving, acting, and knowing. In: Mind Context, pp. 307–333 (2010)

    Google Scholar 

  30. Schmidt, R.C., O’Brien, B.: Evaluating the dynamics of unintended interpersonal coordination. Ecol. Psychol. 9, 189–206 (1997)

    Article  Google Scholar 

  31. Lamb, M., et al.: A hierarchical behavioral dynamic approach for naturally adaptive human-agent pick-and-place interactions. Complexity, 2019 , 16 (2019). John Wiley & Sons, Inc., USA. https://doi.org/10.1155/2019/5964632

  32. Yokoyama, K., Yamamoto, Y.: Three people can synchronize as coupled oscillators during sports activities. PLoS Comput. Biol. 7, e1002181 (2011)

    Article  MathSciNet  Google Scholar 

  33. Zhang, M., et al.: Critical diversity: divided or United States of social coordination. PLoS ONE 13, e0193843 (2018)

    Article  Google Scholar 

  34. Schaal, S., Peters, J., Nakanishi, J., Ijspeert, A.: Learning movement primitives. In: Dario, P., Chatila, R. (eds.) Robotics Research. The Eleventh International Symposium. STAR, vol. 15, pp. 561–572. Springer, Heidelberg (2005). https://doi.org/10.1007/11008941_60

    Chapter  Google Scholar 

  35. Schaal, S., et al.: Nonlinear dynamical systems as movement primitives. In: International Conference on Humanoid Robots, Cambridge, MA, vol. 38, pp. 117–124 (2001)

    Google Scholar 

  36. Ijspeert, A.J., et al.: Movement imitation with nonlinear dynamical systems in humanoid robots. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 2, pp. 1–6 (2002)

    Google Scholar 

  37. Mukovskiy, A., et al.: Modeling of coordinated human body motion by learning of structured dynamic representations. In: Laumond, J.-P., Mansard, N., Lasserre, J.-B. (eds.) Geometric and Numerical Foundations of Movements. STAR, vol. 117, pp. 237–267. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51547-2_11

    Chapter  Google Scholar 

  38. Nalepka, P., et al.: “Human-like” emergent behavior in an evolved agent for a cooperative shepherding task. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017), Vancouver, Canada (2017)

    Google Scholar 

  39. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge (2017)

    MATH  Google Scholar 

  40. Arulkumaran, K., et al.: Deep reinforcement learning: a brief survey. IEEE Sig. Process. Mag. 34, 26–38 (2017)

    Article  Google Scholar 

  41. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Machine Learning (2016)

    Google Scholar 

  42. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings. International Conference on Learning Representations, ICLR (2016)

    Google Scholar 

  43. Tampuu, A., et al.: Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12, e0172395 (2017)

    Article  Google Scholar 

  44. Hester, T., et al.: Learning from demonstrations for real world reinforcement learning. arXiv arXiv:1704.03732 (2017)

  45. Hussein, A., et al.: Imitation learning: a survey of learning methods. ACM Comput. Surv. 50, 1–35 (2017)

    Article  Google Scholar 

  46. Nalepka, P., Kallen, R.W., Chemero, A., Saltzman, E., Richardson, M.J.: Practical applications of multiagent shepherding for human-machine interaction. In: Demazeau, Y., Matson, E., Corchado, J.M., De la Prieta, F. (eds.) PAAMS 2019. LNCS (LNAI), vol. 11523, pp. 168–179. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24209-1_14

    Chapter  Google Scholar 

  47. Auletta, F., et al.: Herding stochastic autonomous agents via local control rules and online global target selection strategies. arXiv arXiv:2010.00386 (2020)

  48. Auletta, F., et al.: Human-inspired strategies to solve complexjoint tasks in multi agent systems (2021)

    Google Scholar 

  49. Rigoli, L.M., et al.: Employing models of human social motor behavior for artificial agent trainers. In: An, B., et al. (eds.) Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020), p. 9. International Foundation for Autonomous Agents and Multiagent Systems, Auckland (2020)

    Google Scholar 

  50. Juliani, A., et al.: Unity: a general platform for intelligent agents. arXiv (2018)

    Google Scholar 

Download references

Acknowledgments

This work was supported by ARC Future Fellowship (Richardson, FT180 100447), Macquarie University Research Fellowship (Nalepka), and Human Performance Research Network (HPRnet, partnership grant ID9024).

Author information

Authors and Affiliations

  1. Department of Psychology, Macquarie University, Sydney, NSW, 2019, Australia

    Gaurav Patil, Patrick Nalepka, Lillian Rigoli, Rachel W. Kallen & Michael J. Richardson

  2. Centre for Elite Performance, Expertise and Training, Macquarie University, Sydney, NSW, 2019, Australia

    Gaurav Patil, Patrick Nalepka, Rachel W. Kallen & Michael J. Richardson

Authors
  1. Gaurav Patil
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Nalepka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lillian Rigoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rachel W. Kallen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael J. Richardson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gaurav Patil .

Editor information

Editors and Affiliations

  1. Umeå University, Umeå, Sweden

    Frank Dignum

  2. Universidad de Salamanca, Salamanca, Salamanca, Spain

    Juan Manuel Corchado

  3. University of Salamanca, Salamanca, Spain

    Fernando De La Prieta

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Patil, G., Nalepka, P., Rigoli, L., Kallen, R.W., Richardson, M.J. (2021). Dynamical Perceptual-Motor Primitives for Better Deep Reinforcement Learning Agents. In: Dignum, F., Corchado, J.M., De La Prieta, F. (eds) Advances in Practical Applications of Agents, Multi-Agent Systems, and Social Good. The PAAMS Collection. PAAMS 2021. Lecture Notes in Computer Science(), vol 12946. Springer, Cham. https://doi.org/10.1007/978-3-030-85739-4_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85739-4_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85738-7

  • Online ISBN: 978-3-030-85739-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation