Skip to main content
SpringerLink
Log in

Teaching Reinforcement Learning Agents with Adaptive Instructional Systems

  • Conference paper
  • First Online:
Adaptive Instructional Systems. Design and Evaluation (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12792))

Included in the following conference series:

  • 1512 Accesses

Abstract

Traditionally, adaptive instructional systems (AISs) are built to instruct human students. However, they are not the only students that might benefit from an AIS. The field of reinforcement learning (RL), a subfield of machine learning, studies the instruction of synthetic students called agents, by means of various algorithms. In this paper, we advocate the use of an AIS as a conceptual framework to design and teach RL agents. We form our argument by deconstructing what it means to build and use an AIS for a human student, and discuss how the various concepts and relationships may apply to RL agents. We illustrate our findings by means of examples from the reinforcement learning literature and show a domain implementation of an AIS for RL agents.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sottilare, R., Brawner, K.: Exploring standardization opportunities by examining interaction between common adaptive instructional system components. In: Proceedings of the First Adaptive Instructional Systems (AIS) Standards Workshop, Orlando, Florida (2018)

    Google Scholar 

  2. Berner, C., et al.: Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019)

  3. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    Google Scholar 

  4. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  5. Brockman, G., et al.: OpenAI Gym (2016)

    Google Scholar 

  6. Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40 (2017)

    Google Scholar 

  7. Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10 (2009)

    Google Scholar 

  8. van Oijen, J., Roessingh, J.J., Poppinga, G., García, V.: Learning analytics of playing space fortress with reinforcement learning. In: International Conference on Human-Computer Interaction. pp. 363–378. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_29

  9. Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)

    Google Scholar 

  10. Brown, T.B., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

  11. M Spronck, P.H., Ponsen, M.J.V., Sprinkhuizen-Kuyper, I.G., Postma, E.O.: Adaptive game AI with dynamic scripting. Mach. Learn. 63, 217–248 (2006)

    Google Scholar 

  12. Kulkarni, T.D., Narasimhan, K., Saeedi, A., Tenenbaum, J.: Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Adv. Neural. Inf. Process. Syst. 29, 3675–3683 (2016)

    Google Scholar 

  13. Branch, R.M.: Instructional Design: The ADDIE Approach. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-09506-6

  14. Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017)

    Google Scholar 

  15. Bakker, B.: Reinforcement learning with long short-term memory. In: NIPS, pp. 1475–1482 (2001)

    Google Scholar 

  16. Rae, J.W., Potapenko, A., Jayakumar, S.M., Lillicrap, T.P.: Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507 (2019)

  17. Hedegaard, M.: The zone of proximal development as basis for instruction. In: Moll, L.C.E. (ed.) Vygotsky and Education: Instructional Implications and Applications of Sociohistorical Psychology, pp. 349–371. Cambridge University Press (1990)

    Google Scholar 

  18. Fleer, S.: Scaffolding for learning from reinforcement: Improving interaction learning (2020)

    Google Scholar 

  19. Laud, A.D.: Theory and application of reward shaping in reinforcement learning (2004)

    Google Scholar 

  20. Niehaus, J., Riedl, M.O.: Scenario adaptation: An approach to customizing computer-based training games and simulations. In: Proceedings of the AIED 2009 Workshop on intelligent Educational Games, pp. 89–98 (2009)

    Google Scholar 

  21. Elman, J.L.: Learning and development in neural networks: the importance of starting small. Cognition 48, 71–99 (1993)

    Article  Google Scholar 

  22. Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M.E., Stone, P.: Curriculum learning for reinforcement learning domains: a framework and survey. J. Mach. Learn. Res. 21, 1–50 (2020)

    MathSciNet  MATH  Google Scholar 

  23. Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. ACM Comput. Surv. (CSUR) 50, 1–35 (2017)

    Article  Google Scholar 

  24. Borsa, D., Piot, B., Munos, R., Pietquin, O.: Observational learning by reinforcement learning. arXiv preprint arXiv:1706.06617 (2017)

  25. Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

  26. Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S.: D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 (2020)

  27. Toubman, A.: Validating air combat behaviour models for adaptive training of teams. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 557–571. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_44

    Chapter  Google Scholar 

  28. Garcıa, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 16, 1437–1480 (2015)

    MathSciNet  MATH  Google Scholar 

  29. Mondesire, S.C., Wiegand, R.P.: A demonstration of stability-plasticity imbalance in multi-agent, decomposition-based learning. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 1070–1075. IEEE (2015)

    Google Scholar 

  30. Sottilare, R.: Understanding the AIS problem space. In: Proceedings of the 2nd Adaptive Instructional Systems (AIS) Standards Workshop (2019)

    Google Scholar 

  31. Aubret, A., Matignon, L., Hassas, S.: A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.06976 (2019)

  32. Gupta, A., Eysenbach, B., Finn, C., Levine, S.: Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv:1806.04640 (2018)

  33. Karli, M., Efe, M.Ö., Sever, H.: Air combat learning from F-16 flight information. In: 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–6. IEEE (2017)

    Google Scholar 

  34. Toubman, A.: Calculated moves: Generating air combat behaviour. Ph.D. dissertation (2020)

    Google Scholar 

  35. Zhang, X., Liu, G., Yang, C., Wu, J.: Research on air confrontation maneuver decision-making method based on reinforcement learning. Electronics 7, 279 (2018)

    Article  Google Scholar 

  36. Doyle, M.J., Portrey, A.M.: Rapid adaptive realistic behavior modeling is viable for use in training. In: Proceedings of the 23rd Conference on Behavior Representation in Modeling and Simulation (BRIMS), pp. 73–80 (2014)

    Google Scholar 

  37. Freeman, J., Watz, E., Bennett, W.: Adaptive agents for adaptive tactical training: the state of the art and emerging requirements. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 493–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_39

    Chapter  Google Scholar 

  38. van Oijen, J., Toubman, A., Poppinga, G.: Effective behaviour modelling for computer generated forces. In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC). I/ITSEC (2019)

    Google Scholar 

  39. Warwick, W., Rodgers, S.: Wrong in the right way: balancing realism against other constraints in simulation-based training. In: Sottilare, R., Schwarz, J. (eds.) Adaptive Instructional Systems. HCII 2019. LNCS, vol 11597, pp. 379–388. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_30

  40. Ludwig, J., Presnell, B.: Developing an adaptive opponent for tactical training. In: Sottilare, R., Schwarz, J. (eds.) Adaptive Instructional Systems. HCII 2019. LNCS, vol 11597. pp. 532–541. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_42

  41. Luotsinen, L.J., Løvlid, R.A.: Data-driven behavior modeling for computer generated forces. In: NATO Modelling and Simulation Group Symposium M&S Support to Operational Tasks Including War Gaming, Logistics, Cyber Defence (MSG-133), pp. 1–13 (2015)

    Google Scholar 

  42. Sottilare, R.: Exploring methods to promote interoperability in adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 227–238. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_19

    Chapter  Google Scholar 

  43. Brawner, K.: Bridging conceptual models and architectural interchange for adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 34–44. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_3

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Royal Netherlands Aerospace Centre, Amsterdam, The Netherlands

    Joost van Oijen, Armon Toubman & Olivier Claessen

Authors
  1. Joost van Oijen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Armon Toubman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olivier Claessen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joost van Oijen .

Editor information

Editors and Affiliations

  1. Soar Technology, Inc., Orlando, FL, USA

    Robert A. Sottilare

  2. Fraunhofer FKIE, Wachtberg, Germany

    Jessica Schwarz

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

van Oijen, J., Toubman, A., Claessen, O. (2021). Teaching Reinforcement Learning Agents with Adaptive Instructional Systems. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Design and Evaluation. HCII 2021. Lecture Notes in Computer Science(), vol 12792. Springer, Cham. https://doi.org/10.1007/978-3-030-77857-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77857-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77856-9

  • Online ISBN: 978-3-030-77857-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation