Abstract
An adaptive max-plus eigenvector method is proposed for approximating the solution of continuous time nonlinear optimal control problems. At each step of the method, given a set of quadratic basis functions, a standard max-plus eigenvector method is applied to yield an approximation of the value function of interest. Using this approximation, an approximate level set of the back substitution error defined by the Hamiltonian is tessellated according to where each basis function is active in approximating the value function. The polytopes obtained, and their vertices, are sorted according to this back substitution error, allowing “worst-case” basis functions to be identified. The locations of these basis functions are subsequently evolved to yield new basis functions that reduce this worst-case. Basis functions that are inactive in the value function approximation are pruned, and the aforementioned steps repeated. Underlying algebraic properties associated with max-plus linearity, dynamic programming, and semiconvex duality are provided as a foundation for the development, and the utility of the proposed method is illustrated by example.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
McEneaney, W.: Max-Plus Methods for Nonlinear Control and Estimation. Systems & Control: Foundations & Application. Birkhauser, Basel (2006)
McEneaney, W.: A curse-of-dimensionality-free numerical method for solution of certain HJB PDEs. SIAM J. Control Optim. 46(4), 1239–1276 (2007)
Akian, M., Gaubert, S., Lakhoua, A.: The max-plus finite element method for solving deterministic optimal control problems: basic properties and convergence analysis. SIAM J. Control Optim. 47(2), 817–848 (2008)
Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)
McEneaney, W.: A new fundamental solution for differential Riccati equations arising in control. Automatica 44, 920–936 (2008)
Dower, P., McEneaney, W., Zhang, H.: Max-plus fundamental solution semigroups for optimal control problems. In: Proceedings of SIAM Conference on Control Theory and Its Applications (Paris), 2015, pp. 368–375 (2015)
Qu, Z.: A max-plus based randomized algorithm for solving a class of HJB PDEs. In: Proceedings of 53rd IEEE Conference on Decision and Control (Los Angeles, CA) (2014)
Dower, P.: An approximation arising in max-plus based optimal stopping. In: Proceedings of Australian Control Conference (Sydney), pp. 271–276 (2012)
Grune, L., Dower, P.: Hamiltonian based a posteriori error estimation for Hamilton-Jacobi-Bellman equations, Technical report. Universitat Bayreuth. https://epub.uni-bayreuth.de/id/eprint/3578 (2018)
Rockafellar, R.: Conjugate Duality and Optimization. SIAM Regional Conference Series in Applied Mathematics, vol. 16. SIAM, Philadelphia (1974)
Baccelli, F., Cohen, G., Olsder, G., Quadrat, J.-P.: Synchronization and Linearity. Wiley, New York (1992)
Kolokoltsov, V., Maslov, V.: Idempotent Analysis and Applications. Kluwer Publishing House, Dordrecht (1997)
Litvinov, G., Maslov, V., Shpiz, G.: Idempotent functional analysis: an algebraic approach. Math. Notes 69(5), 696–729 (2001)
Cohen, G., Gaubert, S., Quadrat, J.-P.: Duality and separation theorems in idempotent semimodules. Linear Algebra Appl. 379, 395–422 (2004)
Fleming, W., McEneaney, W.: A max-plus-based algorithm for a Hamilton-Jacobi-Bellman equation of nonlinear filtering. SIAM J. Control Optim. 38(3), 683–710 (2000)
McEneaney, W., Dower, P.: The principle of least action and fundamental solutions of mass-spring and n-body two-point boundary value problems. SIAM J. Control Optim. 53(5), 2898–2933 (2015)
Zhang, H., Dower, P.: Max-plus fundamental solution semigroups for a class of difference Riccati equations. Automatica 52, 103–110 (2015)
Dower, P., McEneaney, W.: A max-plus dual space fundamental solution for a class of operator differential Riccati equations. SIAM J. Control Optim. 53(2), 969–1002 (2015)
Dower, P., Zhang, H.: A max-plus primal space fundamental solution for a class of differential Riccati equations. Math. Control Signals Syst. 29(3), 1–33 (2017) [Online]. http://dx.doi.org/10.1007/s00498-017-0200-2
Dower, P., McEneaney, W.: Solving two-point boundary value problems for a wave equation via the principle of stationary action and optimal control. SIAM J. Control Optim. 55(4), 2151–2205 (2017)
Dower, P.: Basis adaptation for a max-plus eigenvector method arising in optimal control. In: Proceedings of 23rd International Symposium on Mathematical Theory of Networks and Systems (Hong Kong), pp. 350–355 (2018)
Avis, D., Fukuda, K.: A pivoting algorithm for convex hulls and vertex enumeration of arrangements and polyhedra. Discret. Comput. Geom. 8, 295–313 (1992)
Bremner, D., Fukuda, K., Marzetta, A.: Primal-dual methods for vertex and facet enumeration. Discret. Comput. Geom. 20, 333–357 (1998)
de Berg, M., Cheong, O., van Kreveld, M., Overmars, M.: Computational Geometry: Algorithms and Applications, 3rd ed. Springer, Berlin (2008)
Rockafellar, R., Wets, R.: Variational Analysis. Springer, Berlin (1997)
Acknowledgements
This research was partially supported by AFOSR/AOARD grant FA2386-16-1-4066.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
In the statement of Theorem 3, the semiconvex transform D φ is as specified by (12), and its candidate inverse is as per (13). For convenience, formally define
in which \({{\mathcal {R}}_-^{{\mathsf {{K}}}}}\) is as per (14).
Lemma 16
With −K ∈ Σ M ∪{M} fixed, D φ and \({\mathsf {{D}}}_\varphi ^{\sharp }\) of (12) and (54) satisfy the following properties:
-
1)
\({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^\sharp ) {{ \ = {\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ) = {{{{\mathcal {R}}_-^{{\mathsf {{K}}}}}}} \subset {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}}}\);
-
2)
\({\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^{\sharp }) {{ \ = {\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ) = {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}} \subset {{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}}}\);
-
3)
\({\mathsf {{D}}}_\varphi \, {\mathsf {{D}}}_\varphi ^{\sharp } = {\mathsf {{I}}}\) on \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^{\sharp }) = {{\mathcal {R}}_-^{{\mathsf {{K}}}}}\);
-
4)
\({\mathsf {{D}}}_\varphi ^{\sharp }\, {\mathsf {{D}}}_\varphi = {\mathsf {{I}}}\) on \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ) = {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\);
-
5)
\({{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}} = {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\);
-
6)
If −K > M then \({{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\setminus {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}} \ne \emptyset \) and \({{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\setminus {{\mathcal {R}}_-^{{\mathsf {{K}}}}} \ne \emptyset \).
The following observations are useful in establishing Lemma 16.
-
1.
Given invertible M ∈ Σ,
$$\displaystyle \begin{aligned} & \langle x,\, z \rangle + {\,{\frac{1}{2}}}\, \langle x,\, {\mathsf{{M}}}\, x\rangle = {\,{\frac{1}{2}}}\, \langle x - \zeta(z),\, {\mathsf{{M}}} \, (x - \zeta(z)) \rangle - {\,{\frac{1}{2}}}\, \langle \zeta(z),\, {\mathsf{{M}}}\, \zeta(z) \rangle {} \end{aligned} $$(55)for all \(x,z\in {\mathbb {R}}^n\), where .
-
2.
Given O 1, O 2 ∈ Σ satisfying O 1 < O 2,
$$\displaystyle \begin{aligned} {{{\mathcal{S}}_+^{{{{{\,\mathsf{{O}}}_1}}}}}} \subset {{{\mathcal{S}}_+^{{{{{\,\mathsf{{O}}}_2}}}}}}\,, \qquad {{{\mathcal{S}}_-^{{{{{\,\mathsf{{O}}}_1}}}}}} \subset {{{\mathcal{S}}_-^{{{{{\,\mathsf{{O}}}_2}}}}}}\,. {} \end{aligned} $$(56)
Proof of Lemma 16
Assertion 1) By definitions (14) and (54),
Fix an arbitrary \(a\in {{\mathcal {R}}_-^{{\mathsf {{K}}}}}\). Applying definition (14), there exists a \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) such that D φ ψ = a. Define \(\psi _+,\psi _\pm :{\mathbb {R}}^n{\rightarrow }{{{{\overline {{\mathbb {R}}}}}}}\) by
for all \(x\in {\mathbb {R}}^n\), and note that
for all \(x\in {\mathbb {R}}^n\). By inspection of (57), ψ + is convex and lower closed on \({\mathbb {R}}^n\), as \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\). Hence, ψ ± is also convex and lower closed on \({\mathbb {R}}^n\), as − (K + M) ≥ 0 by definition of −K ∈ Σ M ∪{M}. Hence, the convex conjugate \(\psi _\pm ^*:{\mathbb {R}}^n{\rightarrow }{{{{\overline {{\mathbb {R}}}}}}}\) of ψ ± is also convex and lower closed [25, Theorem 5, p. 16], with
for all \(z\in {\mathbb {R}}^n\). Define \(\widetilde a:{\mathbb {R}}^n{\rightarrow }{{{{\overline {{\mathbb {R}}}}}}}\) by
for all \(\zeta \in {\mathbb {R}}^n\). Note that \(\widetilde a\) is upper closed, as \(\psi _\pm ^*\) and \({\, {\frac {1}{2}}}\, \langle \cdot ,\, {\mathsf {{M}}}\, \cdot \rangle \) are lower closed. Furthermore, \(\zeta \mapsto \tilde a(\zeta ) - {\, {\frac {1}{2}}}\, \langle \zeta , \, -{\mathsf {{M}}}\, \zeta \rangle \) is concave, as \(\psi _\pm ^*\) is convex. Hence, \(\widetilde a\in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\). Recalling that \(\psi _\pm ^*\) is the convex conjugate of ψ ±, (58), (60) imply that
for all \(\zeta \in {\mathbb {R}}^n\). That is, \(a = \widetilde a\in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\), and as \(a\in {{\mathcal {R}}_-^{{\mathsf {{K}}}}}\) is arbitrary, \({{\mathcal {R}}_-^{{\mathsf {{K}}}}} \subset {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\).
2) By definitions (12) and (54), \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ) = {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) and \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^\sharp ) = {{\mathcal {R}}_-^{{\mathsf {{K}}}}}\). Fix an arbitrary \(a\in {\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^\sharp ) = {{\mathcal {R}}_-^{{\mathsf {{K}}}}}\). Following the proof of Assertion 1), there exists a \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) such that D φ ψ = a, defining a convex and lower closed \(\psi _\pm :{\mathbb {R}}^n\rightarrow {{{{\overline {{\mathbb {R}}}}}}}\) as per (57), (58). Note further that \(a = \widetilde a\in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\), where \(\widetilde a\) is defined as per (60). As \(\psi _\pm = \psi _\pm ^{**}\), see [25, Theorem 5, p. 16], (54), (58) and (60) imply that
where the third equality uses (55) and the change of variable . Hence, \({\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^\sharp ) \subset {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\), as \(\psi = {\mathsf {{D}}}_\varphi ^\sharp \, a\in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\), and \(a\in {\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^\sharp )\) is arbitrary.
Alternatively, fix an arbitrary \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\), and construct ψ ± directly using (57), (58). Subsequently define \(a = \widetilde a\in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\) via (60), and note that \(\psi = {\mathsf {{D}}}_\varphi ^\sharp \, a\) as per (62). That is, \({{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\subset {\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^\sharp )\), as \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) is arbitrary. Recalling the earlier conclusion \({\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^\sharp ) \subset {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) yields \({{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}={\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^\sharp )\). Finally, as −K ∈ Σ M ∪{M}, i.e. K ≤−M, (56) implies that \({{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\subset {{{\mathcal {S}}_+^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\) as required.
3), 4) Applying Assertions 1) and 2), the compositions \({\mathsf {{D}}}_\varphi \, {\mathsf {{D}}}_\varphi ^\sharp :{{\mathcal {R}}_-^{{\mathsf {{K}}}}}{\rightarrow }{{\mathcal {R}}_-^{{\mathsf {{K}}}}}\) and \({\mathsf {{D}}}_\varphi ^\sharp \, {\mathsf {{D}}}_\varphi :{{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}{\rightarrow }{{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) are well-defined. Applying (61) and (62) yields \(a = {\mathsf {{D}}}_\varphi \, \mathsf {D}_\varphi ^\sharp \, a\) and \(\psi = \mathsf {D}_\varphi ^\sharp \, \mathsf {D}_\varphi \, \psi \) for any \(a\in {{\mathcal {R}}_-^{\mathsf {K}}} = {\mathrm{dom}\,}(\mathsf {D}_\varphi ^\sharp )\), \(\psi \in {{\mathcal {S}}_+^{\,\mathsf {K}}} = {\mathrm{dom}\,}(\mathsf {D}_\varphi )\), i.e.,
5) Fix any \(a\in {{{\mathcal {S}}_-^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\). Define \(a_\mp :{\mathbb {R}}^n{\rightarrow }{{{{\overline {{\mathbb {R}}}}}}}\) by
for all \(\zeta \in {\mathbb {R}}^n\). Observe that a ∓ is convex and lower closed by definition of a, so that \(a_\mp = a_\mp ^{**}\), see [10, Theorem 5, p. 16]. Hence,
where the final equality follows by application of (55). Define \(\psi _\mp :{\mathbb {R}}^n{\rightarrow }{{{{\overline {{\mathbb {R}}}}}}}\) by
for all \(\zeta \in {\mathbb {R}}^n\). As \(a_\mp ^*\) is also convex and lower closed [10], \(\psi _\mp \in {{{\mathcal {S}}_+^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\). Substituting (64) in (63), and recalling (12) with selected in the definition of dom (D φ),
for all \(y\in {\mathbb {R}}^n\). Hence, \(a\in {{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\) by inspection of (14). As \(a\in {{{\mathcal {S}}_-^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\) is arbitrary, it follows immediately that \({{{\mathcal {S}}_-^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\subset {{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\). However, applying Assertions 1) and 2) for K = −M yields that \({{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\subset {{{\mathcal {S}}_-^{{\,\mathsf {{-{\mathsf {{M}}}}}}}}}\), so that the claimed property holds.
6) Let \({\mathsf {{D}}}_\varphi ^{\mathsf {{M}}}\) and \({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}\sharp }\) denote the operators D φ and \({\mathsf {{D}}}_\varphi ^\sharp \) of (12) and (54) with respective domains given by \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^{\mathsf {{M}}}) = {{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\) and \({\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}\sharp }) = {{\mathcal {R}}_-^{{\mathsf {{-M}}}}}\). Fix any −K ∈ Σ M, i.e. so that −K > M. Observe that \({{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\setminus {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\ne \emptyset \), as it contains \({\, {\frac {1}{2}}}\, \langle \cdot ,\, {\mathsf {{N}}} \, \cdot \rangle \) for any N ∈ Σ satisfying M < N < −K. Fix any \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\setminus {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\). By Assertions 1) through 5) applied to \({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}}\) and \({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}\sharp }\), there exists an \(a\in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}} = {{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\supset {{{\mathcal {R}}}_-^{{{\mathsf {{K}}}}}}\) such that \({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}}\, \psi = a\) and \({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}\sharp }\, a = \psi \). Suppose that \(a\in {{{\mathcal {R}}}_-^{{{\mathsf {{K}}}}}} = {\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^\sharp )\subset {\mathrm{dom}\,}({\mathsf {{D}}}_\varphi ^{{\mathsf {{M}}}\sharp })\). By (12) and Assertion 4), applied to D φ and \({\mathsf {{D}}}_\varphi ^\sharp \),
Hence, \(\psi \in {\mathrm{ran}\,}({\mathsf {{D}}}_\varphi ^\sharp ) = {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\), by assertion 2), which contradicts \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\setminus {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\). That is, \(a\in {{\mathcal {R}}_-^{{\mathsf {{-M}}}}}\) and \(a\not \in {{{\mathcal {R}}}_-^{{{\mathsf {{K}}}}}}\), so that \(a\in {{{\mathcal {R}}}_-^{{-{\mathsf {{M}}}}}}\setminus {{{\mathcal {R}}}_-^{{{\mathsf {{K}}}}}}\ne \emptyset \). \(\square \)
Corollary 17
Given −K ∈ Σ M ∪{M}, and φ as per (10), the semiconvex transform \({\mathsf {{D}}}_\varphi :{{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}{\rightarrow }{{\mathcal {R}}_-^{{\mathsf {{K}}}}}\) of (12) has a well-defined inverse \({\mathsf {{D}}}_\varphi ^{-1}:{{\mathcal {R}}_-^{{\mathsf {{K}}}}} {\rightarrow } {{\mathcal {S}}_+^{{\,\mathsf {{K}}}}}\) given by
where \({\mathsf {{D}}}_\varphi ^\sharp \) is as per (54).
Proof of Corollary 17 and Theorem 3
Immediate by inspection of Lemma 16. \(\square \)
Lemma 18
The asserted forms (16), (17) of the semiconvex transform and its inverse (12), (13) hold.
Proof
Given any \(\psi \in {{\mathcal {S}}_+^{{\,\mathsf {{-M}}}}}\), Theorem 3 implies that \({\mathsf {{D}}}_\varphi \, \psi \in {{\mathcal {S}}_-^{{\,\mathsf {{-M}}}}}\). Define \(a_\mp :{\mathbb {R}}^n\rightarrow {{{{\overline {{\mathbb {R}}}}}}}\) by
for all \(z\in {\mathbb {R}}^n\), and observe that a ∓ is convex and lower closed, i.e. a ∓ = cl− co a ∓, see [10]. Motivated by the form of the argument of the convex hull operation in the right-hand side of (16), define \(\widetilde a_\mp :{\mathbb {R}}^n\rightarrow {{{{\overline {{\mathbb {R}}}}}}}\) by for all \(z\in {\mathbb {R}}^n\). Note by definition (8) of \(\delta _{z_i}^-\) that
for all \(z\in {\mathbb {R}}^n\). As \(\{z_i\}_{i\in {\mathbb {N}}}\) is dense in \({\mathbb {R}}^n\), inspection of (65) and (66) yields that \(a_\mp = {\text{cl}}^-\, {\text{co}}\ \widetilde a_\mp \). Recalling the definition of \(\widetilde a_\mp \) subsequently yields (16).
The remaining assertion (17) is an immediate consequence of (13), as \(\{z_i\}_{i\in {\mathbb {N}}}\) is dense in \({\mathbb {R}}^n\). \(\square \)
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Dower, P.M. (2018). An Adaptive Max-Plus Eigenvector Method for Continuous Time Optimal Control Problems. In: Falcone, M., Ferretti, R., Grüne, L., McEneaney, W. (eds) Numerical Methods for Optimal Control Problems. Springer INdAM Series, vol 29. Springer, Cham. https://doi.org/10.1007/978-3-030-01959-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-01959-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01958-7
Online ISBN: 978-3-030-01959-4
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)