Using model uncertainty for robust optimization in approximate inference control
- 154 Downloads
Abstract
Recently, the optimization-by-inference approach has been proposed as a new means for solving high-dimensional optimization problems quickly. Approximate Inference COntrol (AICO) is one of the most successful and promising methods that implement the optimization-by-inference approach. AICO is able to solve stochastic optimal control problems and has already been successfully used in many applications. However, it is known that the iterative inference of AICO sometimes fails to converge to the optimal solution. To make the optimization more robust, in this paper, we propose to take model uncertainty into account. In AICO, the cost function to be minimized is accurate around a particular state of a given stochastic system, but the accuracy is uncertain in regions far from that state. Because using such an uncertain function is harmful to the convergence, we modify AICO, so that it does not use the function in uncertain regions. Our method is easy to implement and does not add much computational time to the original AICO. Experiments using two different scenarios show that our method substantially improves AICO in terms of the rate at which the algorithm produces convergent results.
Keywords
Approximate inference control Stochastic optimal control Optimization-by-inference approach Model uncertaintyNotes
Acknowledgements
We would like to thank the reviewers for their valuable comments. This study was partially supported by the Ministry of Education, Culture, Sports, Science and Technology in Japan, Grant-in-Aid for Scientific Research (C) 15K00341.
References
- 1.Attias H (2003) Planning by probabilistic inference. In: Proceedings of the ninth international workshop on artificial intelligence and statisticsGoogle Scholar
- 2.Verma D, Rao RP (2005) Goal-based imitation as probabilistic inference over graphical models. In: Advances in neural information processing systems, pp 1393–1400Google Scholar
- 3.Toussaint M, Storkey A (2006) Probabilistic inference for solving discrete and continuous state Markov decision processes. In: Proceedings of the 23rd international conference on machine learning, pp 945–952Google Scholar
- 4.Toussaint M (2009) Robot trajectory optimization using approximate inference. In: Proceedings of the 26th international conference on machine learning, pp 1049–1056Google Scholar
- 5.Kappen HJ, Gómez V, Opper M (2012) Optimal control as a graphical model inference problem. Mach Learn 87(2):159–182MathSciNetCrossRefMATHGoogle Scholar
- 6.Kumar A, Zilberstein S, Toussaint M (2015) Probabilistic inference techniques for scalable multiagent decision making. J Artif Intell Res 53(1):223–270MathSciNetMATHGoogle Scholar
- 7.Minka TP (2001), Expectation propagation for approximate Bayesian inference. In: Proceedings of the 17th conference on uncertainty in artificial intelligence, pp 362–369Google Scholar
- 8.Rawlik K, Toussaint M, Vijayakumar S (2010), An approximate inference approach to temporal optimization in optimal control. In: Advances in neural information processing systems, pp 2011–2019Google Scholar
- 9.Jetchev N, Toussaint M (2013) Fast motion planning from experience: trajectory prediction for speeding up movement generation. Auton Robots 34(1–2):111–127CrossRefGoogle Scholar
- 10.Ivan V, Zarubin D, Toussaint M, Komura T, Vijayakumar S (2013) Topology-based representations for motion planning and generalization in dynamic environments with interactions. Int J Robot Res 32(9–10):1151–1163CrossRefGoogle Scholar
- 11.Zarubin D, Pokorny FT, Song D, Toussaint M, Kragic D (2013) Topological synergies for grasp transfer. In: Hand synergies-how to tame the complexity of grapsing, workshop, IEEE international conference on robotics and automationGoogle Scholar
- 12.Kadoya T, Itoh H, Fukumoto H, Wakuya H, Furukawa T (2014) Movement imitation in a humanoid robot with approximate inference control. In: Proceedings of the 19th international symposium on artificial life and robotics, pp 260–263Google Scholar
- 13.Watter M, Springenberg J, Boedecker J, Riedmiller M (2015) Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in neural information processing systems, pp 2728–2736Google Scholar
- 14.Toussaint M (2009) Pros and cons of truncated Gaussian EP in the context of approximate inference control. NIPS workshop on probabilistic approaches for robotics and controlGoogle Scholar
- 15.Rüeckert E, Mindt M, Peters J, Neumann G (2014) Robust policy updates for stochastic optimal control. In: Proceedings of the 14th IEEE-RAS international conference on humanoid robots (humanoids), pp 388–393Google Scholar
- 16.Zarubin D, Ivan V, Toussaint M, Komura T, Vijayakumar S (2012) Hierarchical motion planning in topological representations. In: International conference on robotics science and systemsGoogle Scholar
- 17.Rawlik K, Toussaint M, Vijayakumar S (2012) On stochastic optimal control and reinforcement learning by approximate inference. In: International conference on robotics science and systemsGoogle Scholar