Abstract
Specifying reward functions without causing side effects is still a challenge to be solved in Reinforcement Learning. Attainable Utility Preservation (AUP) seems promising to preserve the ability to optimize for a correct reward function in order to minimize negative side-effects. Current approaches however assume the existence of a no-op action in the environment’s action space, which limits AUP to solve tasks where doing nothing for a single time-step is a valuable option. Depending on the environment, this cannot always be guaranteed. We introduce four different baselines that do not build on such actions and therefore extend the concept of AUP to a broader class of environments. We evaluate all introduced variants on different AI safety gridworlds and show that this approach generalizes AUP to a broader range of tasks, with only little performance losses.
S. Eresheim and F. Kovac—Both authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alamdari, P.A., Klassen, T.Q., Icarte, R.T., McIlraith, S.A.: Be considerate: objectives, side effects, and deciding how to act (2021). http://arxiv.org/abs/2106.02617
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety (2016). http://arxiv.org/abs/1606.06565
Armstrong, S., Levinstein, B.: Low impact artificial intelligences (2017). https://doi.org/10.48550/arXiv.1705.10720, http://arxiv.org/abs/1705.10720
Bellemare, M.G., et al.: Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588(7836), 77–82 (2020)
Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
Fawzi, A., et al.: Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610(7930), 47–53 (2022)
Gleave, A., Toyer, S.: A primer on maximum causal entropy inverse reinforcement learning (2022). https://doi.org/10.48550/arXiv.2203.11409, http://arxiv.org/abs/2203.11409
Krakovna, V., Orseau, L., Martic, M., Legg, S.: Penalizing side effects using stepwise relative reachability. In: Espinoza, H., et al. (eds.) Proceedings of the Workshop on Artificial Intelligence Safety 2019. CEUR Workshop Proceedings, vol. 2419. CEUR (2019). http://ceur-ws.org/Vol-2419/#paper1
Krakovna, V., Orseau, L., Ngo, R., Martic, M., Legg, S.: Avoiding side effects by considering future tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 19064–19074. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/file/dc1913d422398c25c5f0b81cab94cc87-Paper.pdf
Leech, G., Kubicki, K., Cooper, J., McGrath, T.: Preventing side-effects in gridworlds (2018). https://www.gleech.org/grids
Leike, J., et al.: AI safety gridworlds (2017). https://doi.org/10.48550/arXiv.711.09883, http://arxiv.org/abs/1711.09883
Berner, C., et al.: Dota 2 with large scale deep reinforcement learning (2019). https://doi.org/10.48550/arXiv.1912.06680, http://arxiv.org/abs/1912.06680
Saunders, W., Sastry, G., Stuhlmüller, A., Evans, O.: Trial without error: towards safe reinforcement learning via human intervention. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, pp. 2067–2069. International Foundation for Autonomous Agents and Multiagent Systems (2018)
Schrittwieser, J., et al.: Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588(7839), 604–609 (2020). https://doi.org/10.1038/s41586-020-03051-4, https://www.nature.com/articles/s41586-020-03051-4
Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., Dragan, A.: Preferences implicit in the state of the world (2022). https://openreview.net/forum?id=rkevMnRqYQ
Silver, D., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016). https://doi.org/10.1038/nature16961, http://www.nature.com/articles/nature16961
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series, 2nd edn. The MIT Press (2018). http://incompleteideas.net/book/the-book.html
Turner, A.M., Hadfield-Menell, D., Tadepalli, P.: Conservative agency via attainable utility preservation. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES 2020, pp. 385–391. Association for Computing Machinery (2020). https://doi.org/10.1145/3375627.3375851
Turner, A.M., Ratzlaff, N., Tadepalli, P.: Avoiding side effects in complex environments. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H.T. (eds.) Advances in Neural Information Processing Systems. NeurIPS 2020, vol. 33, pp. 21406–21415. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/hash/f50a6c02a3fc5a3a5d4d9391f05f3efc-Abstract.html
Turner, A.M., Smith, L.R., Shah, R., Critch, A., Tadepalli, P.: Optimal policies tend to seek power (2021). https://openreview.net/forum?id=l7-DBWawSZH
Vinyals, O., et al.: Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature 575(7782), 350–354 (2019)
Acknowledgements
This research was funded in whole, or in part, by the Austrian Science Fund (FWF) [P 33656-N]. For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Eresheim, S., Kovac, F., Adrowitzer, A. (2023). Standing Still Is Not an Option: Alternative Baselines for Attainable Utility Preservation. In: Holzinger, A., Kieseberg, P., Cabitza, F., Campagner, A., Tjoa, A.M., Weippl, E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2023. Lecture Notes in Computer Science, vol 14065. Springer, Cham. https://doi.org/10.1007/978-3-031-40837-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-40837-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40836-6
Online ISBN: 978-3-031-40837-3
eBook Packages: Computer ScienceComputer Science (R0)
