Abstract
Recent results in the literature have provided connections between the so-called turnpike property, near optimality of closed-loop solutions, and strict dissipativity. Motivated by applications in economics, optimal control problems with discounted stage cost are of great interest. In contrast to non-discounted optimal control problems, it is more likely that several asymptotically stable optimal equilibria coexist. Due to the discounting and transition cost from a local to the global equilibrium, it may be more favourable staying in a local equilibrium than moving to the global—cheaper—equilibrium. In the literature, strict dissipativity was shown to provide criteria for global asymptotic stability of optimal equilibria and turnpike behaviour. In this paper, we propose a local notion of discounted strict dissipativity and a local turnpike property, both depending on the discount factor. Using these concepts, we investigate the local behaviour of (near-)optimal trajectories and develop conditions on the discount factor to ensure convergence to a local asymptotically stable optimal equilibrium.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
In recent years, dissipativity as introduced into systems theory by Willems [20, 21] has turned out to be a highly useful concept in order to understand the qualitative behaviour of optimally controlled systems. While related ideas were already present in early works by Willems [19] in a linear quadratic setting, the approach has been revived and extended to fully nonlinear problems motivated by the observation of the importance of dissipativity concepts in model predictive control [1, 3, 14, 15] and for the characterization of the turnpike property [8]. The turnpike property expresses the fact that optimal (and possible also near-optimal) trajectories stay within a vicinity of an optimal equilibrium for most of the time. It can be seen as a way to generalize asymptotic stability properties of optimal equilibria to finite- and infinite-horizon optimal control problems. While the references just discussed addressed non-discounted control problems, the results from [4, 5, 10, 12] show that central results from this theory can be carried over to discounted optimal control problems and complement detectability-based approaches such as [16, 17] for analysing global asymptotic stability of equilibria of discounted optimally controlled systems.
A crucial difference between discounted and non-discounted optimal control problems is that in discounted problems it is much more likely that several asymptotically stable optimal equilibria coexist. Indeed, assuming complete controllability, in non-discounted optimal control two optimal equilibria can only coexist for arbitrary long (or infinite) horizons if they yield exactly the same optimal cost. Otherwise, for sufficiently long time it will always be beneficial to steer the system from the more expensive equilibrium to the cheaper one. In contrast to this, in discounted optimal control, due to the discounting it may not be possible to compensate for the transition cost from one equilibrium to the other with the lower cost of staying in the cheaper equilibrium. Therefore, in the discounted case locally asymptotically stable equilibria with different costs may coexist even for infinite horizon problems. In mathematical economy, where discounted optimal control problems are an important modelling tool, this is a well known fact at least since the pioneering work of Skiba [18] and Dechert and Nishimura [2], and since then it was observed in many other papers, see, e.g., [13] and the references therein.
It is the goal of this paper to show that a local version of the strict dissipativity property for discounted optimal control problems can be used for obtaining local convergence results to optimal equilibria. More precisely, we show that in the presence of local strict dissipativity and appropriate growth conditions on the optimal value functions there exist two thresholds for the discount factor \(\beta \in (0,1)\), denoted by \(\beta _1\) and \(\beta _2\), with the following properties: Whenever \(\beta \ge \beta _1\), any optimal trajectory that stays near a locally optimal equilibrium converges to this equilibrium. Whenever \(\beta \le \beta _2\), any optimal trajectory that starts near this equilibrium will stay near the equilibrium. Together, this yields an interval \([\beta _1,\beta _2]\), which—provided that \(\beta _1\le \beta _2\) holds—contains the discount factors for which convergence of optimal trajectories to the locally optimal equilibrium holds locally. We formalize this convergence behaviour using the formalism from turnpike theory (see, e.g., [11]), because this provides a convenient way to express these properties in a mathematically precise way also for near-optimal trajectories and to link our results to the recent literature on the relation between dissipativity and turnpike properties. We carry out our analysis in discrete time because this simplifies some of our arguments, yet we think that conceptually similar results can also be achieved for continuous time problems.
The remainder of this paper is organised as follows. In Sect. 2 we introduce the precise problem formulation and notation. Section 3 summarises the known results for globally strictly dissipative discounted problems. In Sect. 4 we show how this result can be reformulated in case that only local strict dissipativity holds, provided the trajectories under consideration satisfy an invariance condition. In Sect. 5 we then show that this invariance condition is “automatically” satisfied under suitable conditions. Section 6 then contains the main result by bringing together the two results from Sects. 4 and 5. In Sect. 7 we illustrate our results by several examples and the final Sect. 8 provides a brief concluding discussion.
2 Setting and Preliminaries
2.1 System Class and Notation
We consider discrete time nonlinear systems of the form
for a map \(f: X\times U\rightarrow X\), where X and U are normed spaces. We impose the constraints \((x,u)\in {\mathbb {Y}}\subset X\times U\) on the state x and the input u and define \({\mathbb {X}}:=\{x\in X \mid \exists u\in U: (x,u)\in {\mathbb {Y}}\}\) and \({\mathbb {U}}:=\{u\in U\mid \exists x\in X: (x,u)\in {\mathbb {Y}}\}\). A control sequence \(u\in {\mathbb {U}}^N\) is called admissible for \(x_0\in {\mathbb {X}}\) if \((x(k),u(k))\in {\mathbb {Y}}\) for \(k=0,\dots ,N-1\) and \(x(N)\in {\mathbb {X}}\). In this case, the corresponding trajectory x(k) is also called admissible. The set of admissible control sequences is denoted by \({\mathbb {U}}^N(x_0)\). Likewise, we define \({\mathbb {U}}^\infty (x_0)\) as the set of all control sequences \(u\in {\mathbb {U}}^\infty \) with \((x(k),u(k))\in {\mathbb {Y}}\) for all \(k\in {\mathbb {N}}_0\). Furthermore, we assume that \({\mathbb {X}}\) is controlled invariant, i.e. that \({\mathbb {U}}^\infty (x_0)\ne \) for all \(x_0\in {\mathbb {X}}\). The trajectories of (1) are denoted by \(x_u(k,x_0)\) or simply by x(k) if there is no ambiguity about \(x_0\) and u.
We will make use of comparison-functions defined by
Moreover, with \({\mathcal {B}}_\varepsilon (x_0)\) we denote the open ball with radius \(\varepsilon >0\) around \(x_0\).
In this paper we consider infinite horizon discounted optimal control problems, i.e. problems of the type
Herein, the number \(\beta \in (0,1)\) is called the discount factor.
For such problems it was shown in [5] that if the optimal control problem is strictly dissipative at an optimal equilibrium \(x^\beta \), then for sufficiently large \(\beta \in (0,1)\) all optimal trajectories converge to a neighbourhood of \(x^\beta \). This neighbourhood shrinks down to \(x^\beta \) when \(\beta \rightarrow 1\), cf. [5, Theorem 4.4]. Under slightly stronger conditions on the problem data one can even show that the optimal trajectories converge to the optimal equilibrium \(x^\beta \) itself and not only to a neighbourhood, cf. [5, Section 6]. We will show in Theorem 3.4, below, that this result can be rewritten in the language of turnpike theory, in which convergence is weakened to the property that the trajectories stay in a neighbourhood of the optimal equilibrium for a (quantifiable) amount of time, but not necessarily forever. While only the optimal trajectories satisfy convergence to the optimal equilibrium, we will show that also near-optimal trajectories satisfy the turnpike property.Footnote 1
While this global turnpike result follows from a relatively straightforward modification of the arguments in [5], the main question that we want to address in this paper is more difficult: assume that strict dissipativity does not hold globally but only in a neighbourhood of a locally optimal equilibrium \(x_l^\beta \). Can we still expect to see a turnpike property of trajectories starting close to \(x_l^\beta \)?
For the derivation of our technical results, we make frequent use of the dynamic programming principle
where
denotes the optimal value function of (2). If \(u^*\in {\mathbb {U}}^\infty (x_0)\) is an optimal control sequence for an initial value \(x_0\in {\mathbb {X}}\), i.e. if \(J_\infty (x_0, u^*)=V_\infty (x_0)\) holds, then the identity
holds. Proofs for these statements can be found, e.g., in [9, Section 4.2].
We denote optimal trajectories by \(x^*(k,x_0)\) and we say that a set \({\mathbb {X}}_{inv}\subset {\mathbb {X}}\) is forward invariant for the optimally controlled system, if for each \(x_0\in {\mathbb {X}}_{inv}\) it follows that \(x^*(k,x_0)\in {\mathbb {X}}_{inv}\) for all \(k\ge 0\) and all optimal trajectories starting in \(x_0\).
3 The Global Discounted Turnpike Property
In this section we first consider the optimal control problem (2) assuming global strict dissipativity. We show that under similar technical assumptions and with a similar proof technique as in [5] we can obtain a global turnpike result for near-optimal trajectories. To this end, we first introduce discounted strict dissipativity and afterwards we use it to conclude the turnpike property.
3.1 Global Discounted Strict Dissipativity
We denote an equilibrium of system (1) in the discounted case by \((x^\beta ,u^\beta )\) since the equilibria are dependent on the discount factor \(\beta \in (0,1)\).
Definition 3.1
Given a discount factor \(\beta \in (0,1)\), we say that the system (1) is discounted strictly dissipative at an equilibrium \((x^\beta ,u^\beta )\) with supply rate \(s:{\mathbb {Y}}\rightarrow {\mathbb {R}}\) if there exists a storage function \(\lambda :{\mathbb {X}}\rightarrow {\mathbb {R}}\) bounded from below with \(\lambda (x^\beta )=0\) and a class \({\mathcal {K}}_\infty \)-function \(\alpha \) such that the inequality
holds for all \((x,u)\in {\mathbb {Y}}\) with \(f(x,u)\in {\mathbb {X}}\).
The following lemma is Proposition 3.2 from [12]. Since its proof is short and simple, we provide it here for convenience of the readers. It shows that we can replace the stage cost \(\ell \) by a modified—usually called rotated—stage cost \({\tilde{\ell }}\) that is positive definite without changing the optimal trajectories.
Lemma 3.2
Consider the discounted optimal control problem (2) with discount factor \(\beta \in (0,1)\) and assume the system (1) is discounted strictly dissipative at an equilibrium \((x^\beta ,u^\beta )\) with supply rate \(s(x,u)=\ell (x,u)-\ell (x^\beta ,u^\beta )\) and bounded storage function \(\lambda \). Then the optimal trajectories of (2) coincide with those of the problem
with rotated stage cost
which is positive definite in \(x^\beta \) at \((x^\beta ,u^\beta )\), i.e. it satisfies the inequality \({\tilde{\ell }}(x,u) \ge \alpha (\Vert x-x^\beta \Vert )\) with \(\alpha \in {\mathcal {K}}_\infty \) from (3) for all \((x,u)\in {\mathbb {Y}}\).
Proof
We rearrange
and a straightforward calculation shows that
Since \(\lambda \) is bounded and \(\beta \in (0,1)\), the last limit exists and is equal to 0. Hence, the objectives differ only by expressions which are independent of U, from which the identity of the optimal trajectories immediately follows. The positive definiteness of \({\tilde{\ell }}\) follows from its definition, using strict dissipativity and the fact that \(\lambda (x^\beta )=0\) implies \({\tilde{\ell }}(x^\beta ,u^\beta )=0\). \(\square \)
Remark 3.3
The requirement that \(\ell (x^\beta ,u^\beta )=0\) is the reason for imposing
\(\lambda (x^\beta )=0\) as a condition in Definition 3.1. Readers familiar with dissipativity for undiscounted problems will know that in the undiscounted case \(\lambda (x^\beta )=0\) can be assumed without loss of generality, since if \(\lambda \) is a storage function then \(\lambda + c\) is a storage function for all \(c\in {\mathbb {R}}\). In the discounted case, this invariance with respect to addition of constants no longer holds.
3.2 The Global Turnpike Property
In the non-discounted setting it is known that strict dissipativity (together with suitable regularity assumptions on the problem data) implies that optimal as well as near-optimal trajectories exhibit the turnpike property. In the discounted setting, it was observed already in [11] that for merely near-optimal trajectories the turnpike property can only be guaranteed on a finite discrete interval \(\{0,\ldots ,M\}\). In this case M depends on the deviation from optimality (denoted by \(\delta \) in the following theorem) and tends to infinity as this distance tends to 0. In fact, M and \(\delta \) depend on each other, i.e. the smaller we choose \(\delta \), the larger M becomes and, conversely, the larger we want M to be, the smaller we need to choose \(\delta \). In the following theorem we take this second point of view, i.e. we fix M and choose \(\delta >0\) accordingly. As the theorem shows, under the assumption of global discounted dissipativity we obtain precisely the turnpike property in the form which is originally given in [11, Definition 4.2]. The corresponding proof is a variation of the statement and proof of Theorem 3.1 and Corollary 4.3 in [5]. While in [5] only optimal trajectories were considered, here we extend the statement to near-optimal trajectories. We provide the proof in an appropriate way such that we can refer to it later in the proof of Theorem 4.3.
Theorem 3.4
Consider the infinite horizon optimal control problem (2) with discount factor \(\beta \in (0,1)\) and assume the system (1) is discounted strictly dissipative at an equilibrium \((x^\beta ,u^\beta )\). Assume that the optimal value function \({{\widetilde{V}}}_\infty \) of the modified problem satisfies \(\widetilde{V}_\infty (x) \le \alpha _V(\Vert x-x^\beta \Vert )\) and
for all \(x\in {\mathbb {X}}\), a function \(\alpha _V\in {\mathcal {K}}_\infty \), and a constant \(C\ge 1\) satisfying
Then the optimal control problem has the following turnpike property (cf. [11, Definition 4.2]):
For each \(\varepsilon >0\) and each bounded set \({\mathbb {X}}_b\subset {\mathbb {X}}\) there exist a constant \(P>0\) such that for each \(M\in {\mathbb {N}}\) there is a \(\delta >0\), such that for all \(x_0\in {\mathbb {X}}_b\) and \(u\in {\mathbb {U}}^\infty (x_0)\) with \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \), the set \({\mathcal {Q}}(x_0,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}\) has at most P elements.
Proof
It follows from the proof of Lemma 3.2 that the inequality \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \) implies \({{\widetilde{J}}}_\infty (x_0,u)\le {{\widetilde{V}}}_\infty (x_0)+\delta \). Together with the dynamic programming principle for \(\widetilde{V}_\infty \) this yields
This implies \({{\widetilde{J}}}_\infty (x_u(1,x_0),u(\cdot +1))\le {{\widetilde{V}}}_\infty (x_u(1,x_0))+\delta /\beta \), and proceeding inductively we obtain
for all \(k\in {\mathbb {N}}\). This implies
where \(\kappa = (1-\beta )-1/C < 0\) because of (8).
Moreover, strict discounted dissipativity implies \(\widetilde{V}_\infty (x) \ge {{\tilde{\ell }}}(x,u) \ge \alpha (\Vert x-x^\beta \Vert )\). Together with the upper bound \({{\widetilde{V}}}_\infty (x) \le \alpha _V(\Vert x-x^\beta \Vert )\) from the assumption this yields that for fixed \(M\in {\mathbb {N}}\) and \(k\in \{0,\ldots ,M\}\) the function \(\widetilde{V}_\infty \) is a practical Lyapunov function. Using [7, Theorem 2.4] restricted to \(\{0,\ldots ,M\}\) and the fact that \({\mathbb {X}}_b\) is bounded we can conclude that there is a sequence \(\eta _k\rightarrow 0\) (depending on \({\mathbb {X}}_b\)) and a function \(\gamma \in {\mathcal {K}}_\infty \) with
for all \(k\in \{0,\ldots ,M\}\). This implies the desired claim by choosing \(P\in {\mathbb {N}}\) (depending on \(\varepsilon \) and \(\eta _k\), hence on \({\mathbb {X}}_b\)) such that \(\eta _k < \varepsilon /2\) for all \(k\ge P\) and \(\delta >0\) (depending on \(\beta \), \(\varepsilon \) and M) such that \(\gamma (\delta /\beta ^M)<\varepsilon /2\). \(\square \)
For an illustration of the described turnpike property we refer to Fig. 1. We note again that in the formulation of the discounted turnpike property the level \(\delta \) which measures the deviation from optimality of the trajectory \(x_u(\cdot ,x_0)\) depends on M. For guaranteeing the turnpike property on \(\{0,\ldots ,M\}\), \(\delta \rightarrow 0\) may be required if \(M\rightarrow \infty \), cf. also Remark 3.5 (iv).
The following remark discusses aspects of the assumptions of Theorem 3.4. For the turnpike property to hold, it is obviously necessary that the state of the system can be steered to \(x^\beta \), at least asymptotically. This is made precise in part (i) of the remark. Part (ii) shows that if the state can be steered to \(x^\beta \) fast enough, then a constant C satisfying (7) for all \(\beta \in (0,1)\) holds. Finally, part (iii) of the remark discusses how inequality (7) can be relaxed if such a C cannot be found.
Remark 3.5
-
(i)
A necessary condition for the turnpike property to hold is that for each \(\varepsilon >0\), each bounded subset \({\mathbb {X}}_b\subseteq {\mathbb {X}}\) and each \(x_0\in {\mathbb {X}}_b\) there exists a control sequence \(u\in {\mathbb {U}}^{P+1}(x_0)\) with \(x_u(k,x_0)\in {\mathcal {B}}_\varepsilon (x^\beta )\) for some \(k\le P+1\), where P is the constant from the turnpike property in Theorem 3.4. This is immediately clear, because if such a control does not exist, then the number of points \(x_u(k,x_0)\not \in {\mathcal {Q}}(x_0,u,\varepsilon ,M,\beta )\) is larger than P for all u.
-
(ii)
If a constant C satisfying (7) for all \(\beta \in (0,1)\) exists, then (8) will hold for all sufficiently large \(\beta \in (0,1)\). A sufficient condition for the existence of such a C is the following exponential stabilizability assumption of the cost at the equilibrium \((x^\beta ,u^\beta )\): there are constants \(\sigma ,\lambda >0\) such that for each \(x_0\in {\mathbb {X}}\) there is \(u\in {\mathbb {U}}^\infty (x_0)\) with
$$\begin{aligned} {\tilde{\ell }}(x_u(k,x_0),u(k)) \le \sigma e^{-\lambda k}\inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}). \end{aligned}$$(10)Then, since \({\tilde{\ell }}\ge 0\) we obtain
$$\begin{aligned} {{\widetilde{V}}}_\infty (x_0)&\le \sum _{k=0}^\infty \beta ^k{\tilde{\ell }}(x_u(k,x_0),u(k)) \le \sum _{k=0}^\infty {\tilde{\ell }}(x_u(k,x_0),u(k))\\&\le \sum _{k=0}^\infty \sigma e^{-\lambda k}\inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}) = \frac{\sigma }{1-e^{-\lambda }} \inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}), \end{aligned}$$implying (7) with \(C=\sigma /(1-e^{-\lambda })\). We note that (10) holds in particular if the system itself is exponentially stabilizable to \(x^\beta \) with exponentially bounded controls and \({\tilde{\ell }}\) is a polynomial.Footnote 2 Exponential stabilizability of the system, in turn, follows locally around \(x^\beta \) from stabilizability of its linearization in \(x^\beta \). If, in addition, the necessary condition from part (i) of this remark holds, then local exponential stabilizability implies exponential stabilizability for bounded \({\mathbb {X}}\). We refer to [5, Section 6] for a more detailed discussion on these conditions. Note that if \({\tilde{\ell }}\) is continuous, then \(\inf _{u\in {\mathbb {U}}}{\tilde{\ell }}(x,u) \le \alpha _\ell (\Vert x-x^\beta \Vert )\) holds for an appropriate \(\alpha _\ell \in {\mathcal {K}}_\infty \). In this case, (7) implies the assumed upper bound \(\alpha _V\) on \({{\widetilde{V}}}_\infty \).
-
(iii)
If a C meeting (8) and (7) for all \(x\in {\mathbb {X}}\) does not exist, then we may still be able to find a C satisfying (8) and (7) for all \(x\in {\mathbb {X}}\) with \(\vartheta \le \Vert x-x^\beta \Vert \le \Theta \), for parameters \(0\le \vartheta <\Theta \). In this case we can follow the reasoning in the proof of Corollary 4.3 from [5] to conclude that we still obtain a turnpike property for \(\varepsilon >\varepsilon _0\) and \({\mathbb {X}}_b={\mathcal {B}}_\Delta (x^\beta )\cap {\mathbb {X}}\), with \(\varepsilon _0\rightarrow 0\) as \(\vartheta \rightarrow 0\) and \(\Delta \rightarrow \infty \) as \(\Theta \rightarrow \infty \).
-
(iv)
Optimal trajectories, i.e., trajectories for which \(J_\infty (x_0,u)=V_\infty (x_0)\) holds, satisfy the assumptions of Theorem 3.4 for each \(\delta >0\). Hence, the assertion of the theorem holds for each \(\varepsilon >0\) and each \(M\in {\mathbb {N}}\), implying that \(x_u(k,x_0)\) converges to \(x^\beta \) as \(k\rightarrow \infty \) and thus \(x^\beta \) behaves similarly to an asymptotically stable equilibrium. Whenever \(\delta >0\), however, the trajectory will typically move away from \(x^\beta \) for large times. This is more similar to the classical turnpike phenomenon in optimal control than to asymptotic stability, which is why we prefer this term over asymptotic stability or practical asymptotic stability.
4 The Local Discounted Turnpike Property Assuming Invariance
In the previous section, we have shown that an equilibrium at which the system is globally strictly dissipative has the turnpike property. Now, we consider an equilibrium denoted by \((x^\beta _l,u^\beta _l)\) at which discounted strict dissipativity holds only locally, i.e., for all x in a neighbourhood \({\mathbb {X}}_{{\mathcal {N}}}\) of \(x_l^\beta \), in the following sense.
Definition 4.1
Given a discount factor \(\beta \in (0,1)\), we say that the system (1) is locally discounted strictly dissipative at an equilibrium \((x^\beta _l,u^\beta _l)\) with supply rate \(s:{\mathbb {Y}}\rightarrow {\mathbb {R}}\) if there exists a storage function \(\lambda :{\mathbb {X}}\rightarrow {\mathbb {R}}\) bounded from below with \(\lambda (x_l^\beta )=0\) and a class \({\mathcal {K}}_\infty \)-function \(\alpha _\beta \) such that the inequality
holds for all \((x,u)\in {\mathbb {X}}_{\mathcal {N}}\times {\mathbb {U}}\) with \(f(x,u)\in {\mathbb {X}}\).
Further, we say that system (1) is locally discounted strictly (x, u)-dissipative at the equilibrium \((x^\beta _l,u^\beta _l)\) with supply rate \(s:{\mathbb {X}}\times {\mathbb {U}}\rightarrow {\mathbb {R}}\) if the same holds with the inequality
As in the global case we define the rotated stage cost by
We note that this definition is local since we only require it for \(x\in {\mathbb {X}}_{\mathcal {N}}\). However, also in this local version we require the dissipation inequalities to hold for all \(u\in {\mathbb {U}}\) with \(f(x,u)\in {\mathbb {X}}\), i.e. also for control values that drive the state out of the neighbourhood \({\mathbb {X}}_{\mathcal {N}}\) of the equilibrium \(x_l^\beta \). This property will be important in the proof of Lemma 5.2, below.
Obviously, with this definition Lemma 3.2 remains valid. Moreover, for \(x\in {\mathbb {X}}_{{\mathcal {N}}}\) the function \({\tilde{\ell }}\) satisfies the same properties as in the globally dissipative case. This will enable us to derive a local turnpike property, provided the neighbourhood \({\mathbb {X}}_{{\mathcal {N}}}\) contains an invariant set \({\mathbb {X}}_{inv}\subset {\mathbb {X}}_{{\mathcal {N}}}\) for the optimally controlled system. The following lemma gives a consequence of this assumption for the modified optimal value function, which will be important for concluding the local turnpike property.
Lemma 4.2
Consider the optimal control problem (2) with given discount factor \(\beta \in (0,1)\) and assume that the system is locally strictly dissipative in \((x^\beta _l,u^\beta _l)\in {\mathbb {X}}_{\mathcal {N}}\subset {\mathbb {X}}\). Consider a subset \({\mathbb {X}}_{inv}\subset {\mathbb {X}}_{\mathcal {N}}\) such that all optimal solutions \(x^*(k,x_0)\) with \(x_0\in {\mathbb {X}}_{inv}\) satisfy \(x^*(k,x_0) \in {\mathbb {X}}_{inv}\) for all \(k\ge 0\).
Then the modified optimal value function \({{\widetilde{V}}}_\infty \) satisfies
for all \(x\in {\mathbb {X}}_{inv}\).
Proof
For all \(x\in {\mathbb {X}}_{\mathcal {N}}\) and \(u\in {\mathbb {U}}\) the modified cost satisfies \({{\tilde{\ell }}}(x,u) \ge \alpha _\beta (\Vert x-x_l^\beta \Vert )\ge 0\). This implies
which shows the claim. \(\square \)
The following now gives a local version of Theorem 3.4.
Theorem 4.3
Consider the infinite horizon optimal control problem (2) with discount factor \(\beta \in (0,1)\) and assume that the system is locally strictly dissipative at \((x^\beta _l,u^\beta _l)\in {\mathbb {X}}_{\mathcal {N}}\subset {\mathbb {X}}\). Consider a subset \({\mathbb {X}}_{inv}\subset {\mathbb {X}}_{\mathcal {N}}\) such that all optimal solutions \(x^*(k,x_0)\) with \(x_0\in {\mathbb {X}}_{inv}\) satisfy \(x^*(k,x_0) \in {\mathbb {X}}_{inv}\) for all \(k\ge 0\) and suppose that the assumptions of Theorem 3.4 hold for all \(x \in {\mathbb {X}}_{inv}\).
Then the optimal control problem has the following turnpike property on \({\mathbb {X}}_{inv}\):
For each \(\varepsilon >0\) and each bounded set \({\mathbb {X}}_b\subset {\mathbb {X}}_{inv}\) there exist a constant \(P>0\) such that for each \(M\in {\mathbb {N}}\) there is a \(\delta >0\), such that for all \(u\in {\mathbb {U}}^\infty (x_0)\) with \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \) and \(x_u(k,x_0)\in {\mathbb {X}}_{inv}\) for all \(k\in \{0,\ldots ,M\}\), the set \({\mathcal {Q}}(x,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}\) has at most P elements.
Proof
The proof proceeds completely identical to the proof of Theorem 3.4, using the fact that all inqualities used in this proof remain valid as long as the considered solutions stay in \({\mathbb {X}}_{inv}\) which is guaranteed by the assumptions. We note that Lemma 4.2 is needed for establishing the lower bound on \({{\widetilde{V}}}_\infty \) required from a practical Lyapunov function. \(\square \)
Remark 4.4
Instead of assuming the existence of the invariant set \({\mathbb {X}}_{inv}\) we could also assume (14) for all \(x\in {\mathbb {X}}_{{\mathcal {N}}}\). Then by standard Lyapunov function arguments the largest sublevel set of \({{\widetilde{V}}}_\infty \) contained in \({\mathbb {X}}_{{\mathcal {N}}}\) is forward invariant for the optimal solutions and can then be used as set \({\mathbb {X}}_{inv}\). Using (9) we can even ensure that this sublevel set is also forward invariant for all solutions satisfying \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \) provided \(\delta >0\) is sufficiently small. Hence, for this choice of \({\mathbb {X}}_{inv}\) the assumption that \(x_u(k,x_0)\in {\mathbb {X}}_{inv}\) for all \(k\in \{0,\ldots ,M\}\) in Theorem 4.3 would be automatically satisfies if \(\delta \) is not too large.
5 Optimal Trajectories Stay near a Locally Dissipative Equilibrium
Theorem 4.3 shows that the local turnpike property holds if the optimal solutions stay in the neighbourhood of \(x_l^\beta \) in which the strict dissipativity property holds. In this section we show that this condition is “automatically” satisfied for appropriate discount factors. This will enable us to conclude a local turnpike property from local strict dissipativity. To this end, we aim to show that there exists a range of discount factors \(\beta \) for which it is more favourable to stay near the locally dissipative equilibrium than to move to other parts of the state space. The first lemma we need to this end shows a property of trajectories that move out of a neighbourhood of \(x_l^\beta \). In contrast to the previous result, now we need the stronger (x, u)-dissipativity.
Lemma 5.1
Consider a discounted optimal control problem (2) subject to system (1) with continuous f. Assume local strict (x, u)-dissipativity at an equilibrium \((x^\beta _l,u^\beta _l)\) according to Definition 4.1 and let \(\rho >0\) be such that \({\mathcal {B}}_\rho (x_l^\beta )\subset {\mathbb {X}}_{{\mathcal {N}}}\) holds for the neighbourhood \({\mathbb {X}}_{{\mathcal {N}}}\) from Definition 4.1. Then there exists \(\eta >0\) such that for each \(K\in {\mathbb {N}}\) and any trajectory \(x(\cdot )\) with \(x_0=x(0)\in {\mathcal {B}}_\eta (x_l^\beta )\) and \(x(K)\notin {\mathcal {B}}_\rho (x_l^\beta )\) there is a \(M\in \{0,\dots , K-1\}\) such that \(x(0),\ldots ,x(M)\in {\mathcal {B}}_\eta (x_l^\beta )\) and either
holds.
Proof
The continuity of f implies that there exists \(\varepsilon >0\) such that \(\Vert f(x,u)-x_l^\beta \Vert <\rho \) for all \((x,u)\in {\mathbb {Y}}\) with \(\Vert x-x_l^\beta \Vert <\varepsilon \) and \(\Vert u-u_l^\beta \Vert <\varepsilon \). We let \(K_{\min }\) be minimal with \(x(K_{\min })\notin {\mathcal {B}}_\rho (x_l^\beta )\), set \(\eta :=\min \{\varepsilon , \rho \}\), and claim that this implies the assertion for \(M=K_{\min }-1\).
We prove this claim by contradiction. To this end, we assume that for \(M=K_{\min }-1\) neither assertion (i) nor assertion (ii) holds. This implies on the one hand that \(\Vert x(M)-x_l^\beta \Vert <\eta \), since \(x(M)\in {\mathcal {B}}_\rho (x_l^\beta )\) by minimality of \(K_{\min }\) and (i) is not fulfilled. On the other hand, it implies \(\Vert u(M)-u_l^\beta \Vert <\eta \), because (ii) does not hold. Then, however, since \(\eta \le \varepsilon \), the continuity of f implies
This means that \(x(K_{\min })\in {\mathcal {B}}_\rho (x_l^\beta )\), which is a contradiction to the choice of \(K_{\min }\). Hence, either assertion (i) or assertion (ii) must hold for \(M=K_{\min }-1\). \(\square \)
The next lemma shows that the behaviour characterized in Lemma 5.1 induces a lower bound for the rotated discounted functional \({\widetilde{J}}_\infty \) from (4) along trajectories that start in a neighbourhood of \(x_l^\beta \) and leave this neighbourhood. To this end, we note that even if merely local strict dissipativity holds, the modified stage cost \({\tilde{\ell }}\) from (13) is well defined, since \(\lambda \) is defined for all \(x\in {\mathbb {X}}\). However, the inequality \({\tilde{\ell }}(x,u) \ge \alpha _\beta (\Vert x-x_l^\beta \Vert +\Vert u-u_l^\beta \Vert )\) and, more generally, positivity of \({\tilde{\ell }}\) are only guaranteed for \(x\in {\mathbb {X}}_{{\mathcal {N}}}\).
Lemma 5.2
Let the assumptions of Lemma 5.1 hold. In addition, assume that \(\lambda \) from Definition 4.1 is bounded and the stage cost \(\ell \) is bounded from below. Then, there exists \(\beta ^\star \in (0,1)\) with the following property: for any \(\beta \in (0,\beta ^\star )\) and any \(K\in {\mathbb {N}}\) there is \(\sigma (\beta ,K)>0\) such that for any trajectory \(x(\cdot )\) with \(x_0=x(0)\in {\mathcal {B}}_\eta (x_l^\beta )\) and \(x(P)\notin {\mathcal {B}}_\rho (x_l^\beta )\) for some \(P\in \{1,\ldots ,K\}\) the inequality
holds.
Proof
First observe that boundedness from below of \(\ell \) and boundedness of \(\lambda \) imply boundedness from below of \({\tilde{\ell }}\). Let \({\tilde{\ell }}_{\min } := \inf _{(x,u)\in {\mathbb {Y}}} {\tilde{\ell }}(x,u)\). Since \({\tilde{\ell }}(x^\beta ,u^\beta )=0\) it follows that \({\tilde{\ell }}_{\min }\le 0\). Moreover, local dissipativity implies that \({\tilde{\ell }}(x,u)\ge 0\) for all \(x\in {\mathbb {X}}_{{\mathcal {N}}}\) and all \(u\in {\mathbb {U}}\) with \(f(x,u)\in {\mathbb {X}}\). We note that it is important for the remainder of the proof that this inequality holds for all these \(u\in {\mathbb {U}}\) and not only when \(f(x,u)\in {\mathbb {X}}_{\mathcal {N}}\).
Since the trajectory under consideration satisfies the assumptions of Lemma 5.1 with \(K=P\), there exists \(M\in \{0,\dots ,P\}\) such that either assertion (i) or assertion (ii) of this lemma holds. Within the strict dissipativity, in case (i), we obtain that
and in case (ii) we obtain
Hence, we get the same inequality in both cases and we abbreviate \(\delta :=\alpha _\beta (\eta )>0\). In addition, Lemma 5.1 yields \(x(0),\ldots ,x(M)\in {\mathcal {B}}_\eta (x_l^\beta )\subset {\mathbb {X}}_{{\mathcal {N}}}\), which implies \({\tilde{\ell }}(x(k),u(k))\ge 0\) for all \(k=0,\ldots ,M-1\), and the lower bound on \({\tilde{\ell }}\) implies \({\tilde{\ell }}(x(k),u(k))\ge {\tilde{\ell }}_{\min }\) for all \(k\ge M+1\). Together this yields
We now claim that the assertion holds for \(\sigma = \frac{\beta ^K\delta }{2(1-\beta )} \le \frac{\beta ^M\delta }{2(1-\beta )}\). To this end, it is sufficient to show the existence of \(\beta ^\star \) with
for all \(\beta \in (0,\beta ^\star )\). This is equivalent to
since \({\tilde{\ell }}_{\min }-\delta <0\). This inequality holds for all \(\beta \in (0,\beta ^\star )\) if \(\beta ^\star =\delta /(2(\delta -{\tilde{\ell }}_{\min }))\). \(\square \)
Remark 5.3
The choice of the fraction \(\frac{1}{2}\) for \(\sigma \) in the proof of Lemma 5.2 is arbitrary. We can also use a more general fraction \(\frac{1}{k+1}\) with \(k\in {\mathbb {N}}\). Then, with the same calculation as above we get that \(\beta ^\star = \dfrac{k}{k+1}\dfrac{\delta }{\delta -{\tilde{\ell }}_{\min }}\).
Based on the estimate from Lemma 5.2 we can now conclude that near-optimal solutions starting near \(x_l^\beta \) stay in \({\mathbb {X}}_{\mathcal {N}}\) for a certain amount of time.
Lemma 5.4
Consider a discounted optimal control problem (2) subject to system (1) with f continuous and stage cost \(\ell \) bounded from below. Assume local strict (x, u)-dissipativity at an equilibrium \((x^\beta _l,u^\beta _l)\) according to Definition 4.1 with bounded storage function \(\lambda \). Assume furthermore that there is \(\gamma \in {\mathcal {K}}_\infty \) and \({\hat{\beta \in }}(0,1]\) such that \({{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )\) for all \(x\in {\mathbb {X}}_{{\mathcal {N}}}\) and all \(\beta \in (0,{\hat{\beta ]}}\). Then there exists \(\beta _2\in (0,1)\) with the following property: for any \(\beta \in (0,\beta _2)\) and any \(K\in {\mathbb {N}}\) there exists a neighbourhood \({\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )\) and a threshold value \(\theta (\beta ,K)>0\) such that all trajectories with \(x_0\in {\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )\) and \(J_\infty (x_0,u) < V_\infty (x_0) + \theta (\beta ,K)\) satisfy \(x(k)\in {\mathbb {X}}_{{\mathcal {N}}}\) for all \(k\in \{0,\ldots ,K\}\).
Proof
We choose \(\beta _2\) as the minimum of \(\beta ^\star \) from Lemma 5.2 and \({\hat{\beta }}\). We further use \(\sigma (\beta ,K)>0\) from Lemma 5.2 to set \(\varepsilon (\beta ,K):= \gamma ^{-1}(\sigma (\beta ,K)/2)\) and \(\theta (\beta ,K):= \sigma (\beta ,K)/2\). Now consider a trajectory meeting the assumptions and observe that since \(J_\infty \) and \({{\widetilde{J}}}_\infty \) differ only by a term that is independent of \(u(\cdot )\), the assumption \(J_\infty (x_0,u) \le V_\infty (x_0) + \theta (\beta ,K)\) together with the assumption on \(x_0\) implies
The definition of \(\theta \) and \(\varepsilon \) then implies
Since by Lemma 5.2 any trajectory leaving \({\mathbb {X}}_{{\mathcal {N}}}\) (and thus also \({\mathcal {B}}_\rho (x_l^\beta )\)) up to time K has a rotated value satisfying
the trajectory under consideration cannot leave \({\mathbb {X}}_{{\mathcal {N}}}\) for \(k\in \{0,\ldots ,K\}\). \(\square \)
We note that \({{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )\) can be ensured if we can locally steer the system to \(x^\beta \) fast enough and \({\tilde{\ell }}\) is continuous. For a more detailed discussion see Remark 3.5(ii).
6 The Local Discounted Turnpike Property Without Assuming Invariance
With the preparations from the previous sections, we are now able to formulate our main theorem on the existence of a local turnpike property.
Theorem 6.1
Consider a discounted optimal control problem (2) subject to system (1) with f continuous and stage cost \(\ell \) bounded from below. Assume local strict (x, u)-dissipativity at an equilibrium \((x^\beta _l,u^\beta _l)\) according to Definition 4.1 with bounded storage function \(\lambda \). Assume furthermore that there is \(\gamma \in {\mathcal {K}}_\infty \) and \({\hat{\beta \in }}(0,1]\) such that \({{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )\) for all \(x\in {\mathbb {X}}_{{\mathcal {N}}}\) and all \(\beta \in (0,{\hat{\beta }})\), and that there is an interval \([\beta _1,\beta ^\star ]\) of discount rates with \(\beta _1<{\hat{\beta }}\) and \(\beta ^\star \) from Lemma 5.2, such that for each \(\beta \in (\beta _1,\beta ^\star )\) the assumptions of Theorem 3.4 hold for all \(x \in {\mathbb {X}}_{{\mathcal {N}}}\).
Then there is \(\beta _2\in (0,1)\) such that for all \(\beta \in (\beta _1,\beta _2)\) there exists a neighbourhood \({\mathcal {N}}\) of \(x_l^\beta \) on which the system exhibits a local turnpike property in the following sense:
For each \(\varepsilon >0\) there exist a constant \(P>0\) such that for each \(M\in {\mathbb {N}}\) there is a \(\delta >0\), such that for all \(x_0\in {\mathcal {N}}\) and all \(u\in {\mathbb {U}}^\infty (x_0)\) with \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \), the set \({\mathcal {Q}}(x,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}\) has at most P elements.
Particularly, if \(J_\infty (x_0,u) = V_\infty (x_0)\), i.e., if the trajectory is optimal, then for each \(\varepsilon >0\) the set \({\mathcal {Q}}(x,u,\varepsilon ,\infty ,\beta ):=\bigcup _{M\in {\mathbb {N}}}{\mathcal {Q}}(x,u,\varepsilon ,M,\beta )\) has at most P elements, implying the convergence \(x_u(k,x_0)\rightarrow x^\beta \) as \(k\rightarrow \infty \).
Proof
The idea of the proof is to use \(\beta _2\) from Lemma 5.4 and, for each \(\beta \in (\beta _1,\beta _2)\), to construct a neighbourhood \({\mathcal {N}}\) of \(x_l^\beta \) and a \(\delta >0\) such that all trajectories starting in \(x_0\in {\mathcal {N}}\) and satisfying \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \) stay in \({\mathcal {N}}\) for all future times. Then the turnpike property follows from Theorem 4.3 applied with \({\mathbb {X}}_{inv}={\mathcal {N}}\).
To this end, we take \(\beta _2\) from Lemma 5.4, fix \(\beta \in (\beta _1,\beta _2)\), and consider the neighbourhood \({\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )\) and the threshold value \(\theta (\beta ,1)\) from Lemma 5.4 for \(K=1\). We choose \({\mathcal {N}}\) as the largest sublevel set of \({{\widetilde{V}}}_\infty \) that is contained in \({\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )\) and denote the level by \(\mu >0\), i.e., \({\mathcal {N}}=\{x\in {\mathbb {X}}_{{\mathcal {N}}}\,|\, {{\widetilde{V}}}_\infty (x) < {\mu }\}\). We abbreviate \(\kappa = (1-\beta )-1/C\), observing that \(\kappa <0\) because of (8) (cf. also the proof of Theorem 3.4), and set
Now let \(x_0\) and u be as in the assertion, i.e., satisfying \(J_\infty (x_0,u)\le V_\infty (x_0)+\delta \), and denote the corresponding trajectory by \(x(\cdot )\). Then, just as in the first part of the proof of Theorem 3.4, we obtain the estimate
for all \(k\in {\mathbb {N}}\). By definition of \(\delta \) this in particular implies
for all \(k=0,\ldots ,M\).
Now we prove by induction that \(x(k)\in {\mathcal {N}}\) for all \(k=0,\ldots ,M\). For \(k=0\) this follows from the choice of \(x_0\). For \(k\rightarrow k+1\), we make the induction assumption that \(x(k)\in {\mathcal {N}}\), i.e., \(\widetilde{V}_\infty (x(k))<{\mu }\). Then, because of (16) and \({\mathcal {N}}\subseteq {\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )\), Lemma 5.4 (applied with initial value \(x_0=x(k)\) and control \(u(\cdot +k)\)) implies that \(x(k+1)\in {\mathbb {X}}_{{\mathcal {N}}}\). Hence, all the (in)equalities leading to inequality (9) in the proof of Theorem 3.4 are valid and, together with the definition of \(\delta \), yield
Now if \({{\widetilde{V}}}_\infty (x(k)) \ge { \mu }/2\), then second term in the minimum defining \(\delta \) implies
implying \({{\widetilde{V}}}_\infty (x(k+1)) \le {{\widetilde{V}}}_\infty (x(k)) <{ \mu }\) and thus \(x(k+1)\in {\mathcal {N}}\).
If \({{\widetilde{V}}}_\infty (x(k)) < {\mu }/2\), then the third term in the minimum defining \(\delta \) implies
Here we have used that \({{\widetilde{V}}}_\infty (x(k))\ge 0\). This inequality follows since \(x(k)\in {\mathcal {N}}\): If the optimal trajectory starting in x(k) stays in \({\mathcal {N}}\), then it follows from strict dissipativity on \({\mathbb {X}}_{\mathcal {N}}\supset {\mathcal {N}}\) and if the optimal trajectory leaves \({\mathcal {N}}\) then it follows from Lemma 5.2. This implies \({{\widetilde{V}}}_\infty (x(k+1)) \le {{\widetilde{V}}}_\infty (x(k)) + {\mu }/2< {\mu }\), i.e., again \(x(k+1)\in {\mathcal {N}}\). This proves the induction step and hence \(x(k)\in {\mathcal {N}}\) for all \(k=0,\ldots ,M\).
Now the turnpike property follows from Theorem 4.3 applied with \({\mathbb {X}}_{inv}={\mathcal {N}}\).\(\square \)
Remark 6.2
We note that the interval \((\beta _1,\beta _2)\) may be empty. This is because
-
(i)
the condition (7) needed for proving the turnpike property for trajectories staying near \(x_l^\beta \) may require sufficiently large \(\beta \) to hold
-
(ii)
a trajectory starting near \(x_l^\beta \) will in general only stay near \(x_l^\beta \) for sufficiently small \(\beta \)
More precisely, the lower bound in (ii) as identified at the end of the proof of Lemma 5.2 depends on the cost \({\tilde{\ell }}\) outside a neighbourhood of \(x_l^\beta \) and the cost to leave this neighbourhood. The upper bound in (i), in turn, depends on the cost to reach the equilibrium \(x_l^\beta \) from a neighbourhood. If this cost is high and, in addition, the cost to leave the neighbourhood and the cost outside the neighbourhood are low, then the set of discount rates for which a local turnpike behaviour occurs may be empty.
Remark 6.3
The attentive reader may have noted that we apply Lemma 5.4 with \(K=1\) in this proof, rather than with \(K=M\), which might appear more natural given that we want to make a statement for \(\{0,\ldots ,M\}\). This is because the size of the neighbourhood \({\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )\) delivered by Lemma 5.4 depends on K. Hence, if we applied Lemma 5.4 with \(K=M\) in order to construct the neighbourhood \({\mathcal {N}}\), this neighbourhood may shrink down to \(\{x_l^\beta \}\) as M increases. In contrast to this, the fact that \({{\widetilde{V}}}_\infty \) is a (practical) Lyapunov function allows us to construct a neighbourhood \({\mathcal {N}}\) that does not depend on M.
7 Examples
We end our paper with a couple of examples illustrating our theoretical results. All numerical solutions were obtained using a dynamic programming algorithm as described in [6]. We start with two examples exhibiting a locally and a globally optimal equilibrium.
Example 7.1
Consider the dynamics \(f(x,u) = x+ u\) and the stage cost \(\ell (x,u)=x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2\).
As visualized in Fig. 2, the stage cost \(\ell \) has a local minimum in \(x = \frac{3-\sqrt{905}}{32}\), a maximum in \(x=0\) and a global minimum in \(x = \frac{3+\sqrt{905}}{32}\). Following [12, Section 4] we can calculate the storage function \(\lambda \) by using the optimality conditions for optimal equilibria. We remark that the procedure for computing global storage functions described in this reference also works for the local dissipativity in case of local convexity which is given in this example, cf. also the discussion after Example 7.2, below. Thus, by a straightforward calculation, we get the local equilibrium \((x^\beta _l,u^\beta _l)= (\frac{3-\sqrt{905}}{32},0)\) and the storage function \(\lambda \equiv 0\). Inserting this, we get the rotated stage cost \({\tilde{\ell }}(x,u) = x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2 - \ell (x_l^\beta , 0)\) and local discounted strict (x, u)-dissipativity of the system \(f(x,u) = x+ u\) at \(x_l^\beta \) for any \(\beta \in (0,1)\). Thus, the assumptions of Lemma 5.1 and Lemma 5.2 are fulfilled. Hence, following the proof of Lemma 5.2 we can estimate \(\beta _2\approx 0.67\) with \(\delta \approx 1\) and \({\tilde{\ell }}_{\min }\approx -0.42\). Further, since \(\left\Vert {\tilde{\ell }}(x,u)\right\Vert \) is bounded for x in a neighbourhood \({\mathcal {B}}_\varepsilon (x_0)\), \(\varepsilon >0\), Theorem 6.1 can be applied. For illustrating the theoretical results, we set \({\mathbb {U}}=[-0.75, 0.75]\).
On the left hand side of Fig. 3 we show the behaviour of the trajectory x and the control u for different discount factors \(\beta \). On the right hand side, we can observe the optimal feedback control values \(u_x\) and therefore the domain of attraction of the equilibria dependent on \(\beta \). After a maximum of three time instants, the trajectory reaches the global equilibrium for \(\beta \) large enough. In contrast, for \(\beta \le 0.67\) we can observe that it is more favourable to stay in a neighbourhood of the local equilibrium \(x_l^\beta \). We remark that is sufficient to depict \(\beta = 0.8\) as a representative for all \(\beta \in (0.67, 1)\) since the behaviour of the trajectory, the control and the stage cost does not change significantly. In this example \(\beta _1\) can be chosen arbitrarily close to 0, because due to the absence of u-dependent terms in \({\tilde{\ell }}\) it is always cheaper to approach \(x^\beta \) than to stay elsewhere in the neighbourhood of \(x^\beta \).
Figure 4, for fixed \(\beta = 0.7\) we consider different initial values \(x_0\). As we can see, the initial value determines to which equilibrium the trajectory converges. This underpins the theoretical results of Theorem 6.1 and especially of Lemma 5.2. We note that for a completely controllable system such a behaviour cannot occur in undiscounted problems.
Example 7.1 with \(x_0 = -0.8\)
Example 7.1 with \(\beta = 0.7\) (left) and \(\beta = 0.6\) (right) for different start values \(x_0\)
The following modified example illustrates the case that the interval \((\beta _1, \beta _2)\) is empty.
Example 7.2
Consider again the system \(f(x,u)=x+u\), but now with stage cost \(\ell (x,u)=x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2+\gamma |u|\) with \(\gamma \ne 0\). As the added term has no influence on the conditions of Theorem 6.1 we can again estimate \(\beta _2\approx 0.67\). Further, for \(\gamma =0\) we get the same stage cost as in Example 7.1 above. In contrast to Example 7.1, now for \(\gamma \) large enough we can observe that \((\beta _1, \beta _2)\) is empty. This fact is illustrated in Fig. 5 for \(\gamma = 10\). For the numerical results we use the same setting as in example 7.1.
In contrast, in the graph with \(\gamma = 10\) we can clearly observe that independent of the discount factor \(\beta \) we do not get convergence to the local equilibrium anymore. If we introduce state constraints that restrict the optimal solutions to a neighbourhood of the local equilibrium \(x_l^\beta \), then we only get convergence to \(x_l^\beta \) for \(\beta \approx 1\). More precisely, we have numerically determined the threshold of \(\beta _1\approx 0.999\). Without such state constraints, already for \(\beta \approx 0.95\) we observe convergence to the optimal equilibrium, which suggests that \(\beta _2\le 0.95\) and thus \((\beta _1,\beta _2)\) is empty.
In order to examine this behaviour in more detail we illustrate the behaviour of different values of \(\gamma \) for fixed discount factors \(\beta \) in Fig. 6. For \(\gamma >1\) and \(\beta \lesssim 0.95\) we can observe that the trajectories stay near by the start value and do not move away. In contrast, for \(\beta \approx 1\) the trajectories converge to the global equilibrium. Thus, we do not get convergence to the local equilibrium anymore.
Example 7.2 with \(\gamma = 10\) for different discount factor \(\beta \)
Example 7.2 with \(\beta = 0.7\) (left) and \(\beta = 0.95\) (right) for different \(\gamma \)
The two examples, above, have the particular feature that the dynamics is affine and the stage cost \(\ell \) is strictly convex in a neighbourhood of the optimal equilibria. In this case, similar arguments as used in the proof of Theorem 4.2 in [12] show that local strict dissipativity always holds. More precisely, we can restrict the proof of Theorem 4.2 in [12] to a bounded neighbourhood \({\mathbb {X}}_{{\mathcal {N}}}\subset {\mathbb {X}}\) of the local equilibrium \(x_l^\beta \), e.g., \({\mathcal {B}}_\varepsilon (x_l^\beta )\), \(\varepsilon >0\), instead of \({\mathbb {X}}\), and a local strict convex stage cost function \(\ell \). Following the proof, \(\mathrm {D}{\tilde{\ell }}(x^\beta _l,u^\beta _l)=0\) holds in the neighbourhood \({\mathbb {X}}_{{\mathcal {N}}}\), which by the local strict convexity of \({\tilde{\ell }}\) implies that \((x^\beta _l,u^\beta _l)\) is a strict local minimum. Together with the boundedness of \({\mathbb {X}}_{{\mathcal {N}}}\), this implies the existence of \(\alpha _\beta \in {\mathcal {K}}_\infty \) and thus local discounted strict dissipativity. We remark that the calculation of \(\lambda \) is the same as in the global case and yields a linear storage function. In the special case of Example 7.1, above, it yields the storage function \(\lambda \equiv 0\). In conclusion, local strict dissipativity always holds if the dynamics is affine and the stage cost \(\ell \) is strictly convex near the locally optimal equilibrium.
With this observation, our dissipativity based analysis provides a complementary approach to the stable manifold based analysis carried out, e.g., in [13]. Particularly, we can conclude that the model from this reference exhibits two equilibria at which the local turnpike property holds, which explains why the optimal trajectories are correctly reproduced by nonlinear model predictive control as shown in [10, Section 5.1].
Our final example demonstrates that strict convexity of \(\ell \) is not needed for obtaining strict dissipativity, thus showing that a dissipativity based analysis allows for strictly weaker assumptions than strict convexity of \(\ell \).
Example 7.3
Consider the 1d control system
with state constraints \({\mathbb {X}}=[-1,1]\), control constraints \({\mathbb {U}}=[-3,3]\), and stage cost
Obviously, the stage cost is strictly concave in x and strictly convex in u. Nevertheless, we can establish discounted strict (x, u)-dissipativity in \((x^*,u^*)=(0,0)\) (in this example even global) for \(\beta \ge 3/5\) with \(\lambda (x) = -x^2\). This follows from the fact that with \(a=2\beta /\sqrt{1+\beta }\) and \(b=\sqrt{1+\beta }\) we have
where the last inequality holds since the term in the large brackets is \(\ge 0\) for \(\beta \ge 3/5\).
Since the system is completely controllable in finite time, hence exponentially stabilizable, Theorem 3.4 in conjunction with Remark 3.5 (ii) implies that for sufficiently large \(\beta \) turnpike behaviour occurs at \(x^*=0\). This is confirmed for \(\beta =0.7\) in the left graph in Fig. 7. In contrast to this, the right graph in Fig. 7 shows that for \(\beta =0.6\) the turnpike behaviour for \(x^*=0\) does not occur. Rather, the optimal solution converges to the upper bound \(x=1\) of the state constraint set. In this example, the numerical computations indicate that \(\beta = 3/5=0.6\) is a relatively precise estimate of the threshold for the occurrence of the turnpike property at \(x^*=0\), although for \(\beta \) decreasing from 0.7 to 0.6 the set of initial values around \(x^*=0\) for which the turnpike behaviour can be seen shrinks down rapidly.
Optimal trajectories for Example 7.3 with \(\beta = 0.7\) and \(x_0=1\) (left) and with \(\beta = 0.59\) and \(x_0=0.004\) (right)
8 Conclusion
In this paper we have shown that a local strict dissipativity assumption in conjunction with an appropriate growth condition on the optimal value function can be used in order to conclude a local turnpike property at an optimal equilibrium. The turnpike property holds for discount factors from an interval \([\beta _1,\beta _2]\), where \(\beta _1\) is determined by local quantities while \(\beta _2\) is also determined by properties of the optimal control problem away from the local equilibrium. Hence, local and global properties together determine whether the interval is not empty. This is in accordance with other approaches for analysing local stability of equilibria in discounted optimal control such as those based on stable and unstable manifolds [13]. In contrast to other approaches, however, the dissipativity based approach is not limited to (locally) strictly convex problems, as our last example showed.
Notes
We note that the turnpike property can also be defined for finite horizon optimal control problems. Still, we restrict ourselves to the infinite horizon case, since it was shown in [11] that under mild conditions on the problem data the finite horizon turnpike property holds if and only if the infinite horizon turnpike property holds.
We could further relax this assumption to \({\tilde{\ell }}\) being bounded by \(C_1 P\) and \(C_2 P\) from below and above, respectively, for constants \(C_1>C_2>0\) and a polynomial P.
References
Angeli, D., Amrit, R., Rawlings, J.B.: On average performance and stability of economic model predictive control. IEEE Trans. Autom. Control 57(7), 1615–1626 (2012)
Dechert, W.D., Nishimura, K.: A complete characterization of optimal growth paths in an aggregated model with a non-concave production function. J. Econ. Theory 31, 332–354 (1983)
Diehl, M., Amrit, R., Rawlings, J.B.: A Lyapunov function for economic optimizing model predictive control. IEEE Trans. Autom. Control 56, 703–707 (2011)
Gaitsgory, V., Grüne, L., Thatcher, N.: Stabilization with discounted optimal control. Syst. Control Lett. 82, 91–98 (2015)
Gaitsgory, V., Grüne, L., Höger, M., Kellett, C.M., Weller, S.R.: Stabilization of strictly dissipative discrete time systems with discounted optimal control. Automatica 93, 311–320 (2018). https://doi.org/10.1016/j.automatica.2018.03.076
Grüne, L., Semmler, W.: Using dynamic programming with adaptive grid scheme for optimal control problems in economics. J. Econ. Dyn. Control 28, 2427–2456 (2004)
Grüne, L., Stieler, M.: Asymptotic stability and transient optimality of economic mpc without terminal conditions. J. Process Control 24(8), 1187–1196 (2014)
Grüne, L., Müller, M.A.: On the relation between strict dissipativity and the turnpike property. Syst. Control Lett. 90, 45–53 (2016)
Grüne, L., Pannek, J.: Nonlinear model predictive control : theory and algorithms. In: Isidori, A., van Schuppen, J.H. (eds.) Communications and Control Engineering, 2nd edn. Springer, Cham (2017)
Grüne, L., Semmler, W., Stieler, M.: Using nonlinear model predictive control for dynamic decision problems in economics. J. Econ. Dyn. Control 60, 112–133 (2015)
Grüne, L., Kellett, C., Weller, S.: On the relation between turnpike properties for finite and infinite horizon optimal control problems. J. Optim. Theory Appl. (2017). https://doi.org/10.1007/s10957-017-1103-6
Grüne, L., Müller, M.A., Kellett, C.M., Weller, S.R.: Strict dissipativity for discrete time discounted optimal control problems. Math. Control Relat. Fields (2020). https://doi.org/10.3934/mcrf.2020046
Haunschmied, J.L., Kort, P.M., Hartl, R.F., Feichtinger, G.: A DNS-curve in a two state capital accumulation model: a numerical analysis. J. Econ. Dyn. Control 27, 701–716 (2003)
Müller, M.A., Angeli, D., Allgöwer, F.: On necessity and robustness of dissipativity in economic model predictive control. IEEE Trans. Autom. Control 60(6), 1671–1676 (2015)
Müller, M.A., Grüne, L., Allgöwer, F.: On the role of dissipativity in economic model predictive control. In: Proceedings of the 5th IFAC Conference on Nonlinear Model Predictive Control, IFAC PapersOnLine, vol. 23, pp. 110–116 (2015)
Postoyan, R., Buşoniu, L., Nešić, D., Daafouz, J.: Stability of infinite-horizon optimal control with discounted cost. In: Proceedings of the 53rd IEEE Conference on Decision and Control, pp. 3903–3908 (2014)
Postoyan, R., Buşoniu, L., Nešić, D., Daafouz, J.: Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. IEEE Trans. Autom. Control 62(6), 2736–2749 (2017). https://doi.org/10.1109/TAC.2016.2616644
Skiba, A.K.: Optimal growth with a convex-concave production function. Econometrica 46, 527–540 (1978)
Willems, J.C.: Least squares stationary optimal control and the algebraic Riccati equation. IEEE Trans. Autom. Control 16, 621–634 (1971)
Willems, J.C.: Dissipative dynamical systems. I. General theory. Arch. Ration. Mech. Anal. 45, 321–351 (1972)
Willems, J.C.: Dissipative dynamical systems. II. Linear systems with quadratic supply rates. Arch. Ration. Mech. Anal. 45, 352–393 (1972)
Funding
Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The authors are supported by DFG Grant Gr 1569/13-2.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Grüne, L., Krügel, L. Local Turnpike Analysis Using Local Dissipativity for Discrete Time Discounted Optimal Control. Appl Math Optim 84 (Suppl 2), 1585–1606 (2021). https://doi.org/10.1007/s00245-021-09805-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00245-021-09805-4