Local Turnpike Analysis Using Local Dissipativity for Discrete Time Discounted Optimal Control

Abstract

Recent results in the literature have provided connections between the so-called turnpike property, near optimality of closed-loop solutions, and strict dissipativity. Motivated by applications in economics, optimal control problems with discounted stage cost are of great interest. In contrast to non-discounted optimal control problems, it is more likely that several asymptotically stable optimal equilibria coexist. Due to the discounting and transition cost from a local to the global equilibrium, it may be more favourable staying in a local equilibrium than moving to the global—cheaper—equilibrium. In the literature, strict dissipativity was shown to provide criteria for global asymptotic stability of optimal equilibria and turnpike behaviour. In this paper, we propose a local notion of discounted strict dissipativity and a local turnpike property, both depending on the discount factor. Using these concepts, we investigate the local behaviour of (near-)optimal trajectories and develop conditions on the discount factor to ensure convergence to a local asymptotically stable optimal equilibrium.

1 Introduction

In recent years, dissipativity as introduced into systems theory by Willems [20, 21] has turned out to be a highly useful concept in order to understand the qualitative behaviour of optimally controlled systems. While related ideas were already present in early works by Willems [19] in a linear quadratic setting, the approach has been revived and extended to fully nonlinear problems motivated by the observation of the importance of dissipativity concepts in model predictive control [1, 3, 14, 15] and for the characterization of the turnpike property [8]. The turnpike property expresses the fact that optimal (and possible also near-optimal) trajectories stay within a vicinity of an optimal equilibrium for most of the time. It can be seen as a way to generalize asymptotic stability properties of optimal equilibria to finite- and infinite-horizon optimal control problems. While the references just discussed addressed non-discounted control problems, the results from [4, 5, 10, 12] show that central results from this theory can be carried over to discounted optimal control problems and complement detectability-based approaches such as [16, 17] for analysing global asymptotic stability of equilibria of discounted optimally controlled systems.

A crucial difference between discounted and non-discounted optimal control problems is that in discounted problems it is much more likely that several asymptotically stable optimal equilibria coexist. Indeed, assuming complete controllability, in non-discounted optimal control two optimal equilibria can only coexist for arbitrary long (or infinite) horizons if they yield exactly the same optimal cost. Otherwise, for sufficiently long time it will always be beneficial to steer the system from the more expensive equilibrium to the cheaper one. In contrast to this, in discounted optimal control, due to the discounting it may not be possible to compensate for the transition cost from one equilibrium to the other with the lower cost of staying in the cheaper equilibrium. Therefore, in the discounted case locally asymptotically stable equilibria with different costs may coexist even for infinite horizon problems. In mathematical economy, where discounted optimal control problems are an important modelling tool, this is a well known fact at least since the pioneering work of Skiba [18] and Dechert and Nishimura [2], and since then it was observed in many other papers, see, e.g., [13] and the references therein.

It is the goal of this paper to show that a local version of the strict dissipativity property for discounted optimal control problems can be used for obtaining local convergence results to optimal equilibria. More precisely, we show that in the presence of local strict dissipativity and appropriate growth conditions on the optimal value functions there exist two thresholds for the discount factor $\beta \in (0,1)$, denoted by $\beta _1$ and $\beta _2$, with the following properties: Whenever $\beta \ge \beta _1$, any optimal trajectory that stays near a locally optimal equilibrium converges to this equilibrium. Whenever $\beta \le \beta _2$, any optimal trajectory that starts near this equilibrium will stay near the equilibrium. Together, this yields an interval $[\beta _1,\beta _2]$, which—provided that $\beta _1\le \beta _2$ holds—contains the discount factors for which convergence of optimal trajectories to the locally optimal equilibrium holds locally. We formalize this convergence behaviour using the formalism from turnpike theory (see, e.g., [11]), because this provides a convenient way to express these properties in a mathematically precise way also for near-optimal trajectories and to link our results to the recent literature on the relation between dissipativity and turnpike properties. We carry out our analysis in discrete time because this simplifies some of our arguments, yet we think that conceptually similar results can also be achieved for continuous time problems.

The remainder of this paper is organised as follows. In Sect. 2 we introduce the precise problem formulation and notation. Section 3 summarises the known results for globally strictly dissipative discounted problems. In Sect. 4 we show how this result can be reformulated in case that only local strict dissipativity holds, provided the trajectories under consideration satisfy an invariance condition. In Sect. 5 we then show that this invariance condition is “automatically” satisfied under suitable conditions. Section 6 then contains the main result by bringing together the two results from Sects. 4 and 5. In Sect. 7 we illustrate our results by several examples and the final Sect. 8 provides a brief concluding discussion.

2 Setting and Preliminaries

2.1 System Class and Notation

We consider discrete time nonlinear systems of the form

$$\begin{aligned} x(k+1)=f(x(k),u(k)),\quad x(0)=x_0 \end{aligned}$$

(1)

for a map $f: X\times U\rightarrow X$, where X and U are normed spaces. We impose the constraints $(x,u)\in {\mathbb {Y}}\subset X\times U$ on the state x and the input u and define ${\mathbb {X}}:=\{x\in X \mid \exists u\in U: (x,u)\in {\mathbb {Y}}\}$ and ${\mathbb {U}}:=\{u\in U\mid \exists x\in X: (x,u)\in {\mathbb {Y}}\}$. A control sequence $u\in {\mathbb {U}}^N$ is called admissible for $x_0\in {\mathbb {X}}$ if $(x(k),u(k))\in {\mathbb {Y}}$ for $k=0,\dots ,N-1$ and $x(N)\in {\mathbb {X}}$. In this case, the corresponding trajectory x(k) is also called admissible. The set of admissible control sequences is denoted by ${\mathbb {U}}^N(x_0)$. Likewise, we define ${\mathbb {U}}^\infty (x_0)$ as the set of all control sequences $u\in {\mathbb {U}}^\infty$ with $(x(k),u(k))\in {\mathbb {Y}}$ for all $k\in {\mathbb {N}}_0$. Furthermore, we assume that ${\mathbb {X}}$ is controlled invariant, i.e. that ${\mathbb {U}}^\infty (x_0)\ne$ for all $x_0\in {\mathbb {X}}$. The trajectories of (1) are denoted by $x_u(k,x_0)$ or simply by x(k) if there is no ambiguity about $x_0$ and u.

We will make use of comparison-functions defined by

$$\begin{aligned} {\mathcal {K}} :=\{\alpha :{\mathbb {R}}_0^+\rightarrow {\mathbb {R}}^+_0&| \alpha \text { is continuous and}\\&\text {strictly increasing with }\alpha (0)=0\}\\ {\mathcal {K}}_\infty :=\{\alpha :{\mathbb {R}}_0^+\rightarrow {\mathbb {R}}^+_0&|\alpha \in {\mathcal {K}}, \alpha \text { is unbounded}\}\\ {\mathcal {L}}:=\{\delta :{\mathbb {R}}_0^+\rightarrow {\mathbb {R}}^+_0&|\delta \text { is continuous and}\\&\text {strictly decreasing with} \lim _{t\rightarrow \infty }\delta (t)=0\}\\ \mathcal {KL}:=\{\beta :{\mathbb {R}}_0^+\times {\mathbb {R}}_0^+\rightarrow {\mathbb {R}}_0^+&|\beta \text { is continuous, } \beta (\cdot ,t)\in {\mathcal {K}}, \beta (r,\cdot )\in {\mathcal {L}}\}. \end{aligned}$$

Moreover, with ${\mathcal {B}}_\varepsilon (x_0)$ we denote the open ball with radius $\varepsilon >0$ around $x_0$.

In this paper we consider infinite horizon discounted optimal control problems, i.e. problems of the type

$$\begin{aligned} \min _{u\in {\mathbb {U}}\infty (x_0)} J_\infty (x_0,u) \quad \text {with } J_\infty (x_0,u)=\sum _{k=0}^\infty \beta ^k\ell (x(k,x_0),u(k)). \end{aligned}$$

(2)

Herein, the number $\beta \in (0,1)$ is called the discount factor.

For such problems it was shown in [5] that if the optimal control problem is strictly dissipative at an optimal equilibrium $x^\beta$, then for sufficiently large $\beta \in (0,1)$ all optimal trajectories converge to a neighbourhood of $x^\beta$. This neighbourhood shrinks down to $x^\beta$ when $\beta \rightarrow 1$, cf. [5, Theorem 4.4]. Under slightly stronger conditions on the problem data one can even show that the optimal trajectories converge to the optimal equilibrium $x^\beta$ itself and not only to a neighbourhood, cf. [5, Section 6]. We will show in Theorem 3.4, below, that this result can be rewritten in the language of turnpike theory, in which convergence is weakened to the property that the trajectories stay in a neighbourhood of the optimal equilibrium for a (quantifiable) amount of time, but not necessarily forever. While only the optimal trajectories satisfy convergence to the optimal equilibrium, we will show that also near-optimal trajectories satisfy the turnpike property.¹

While this global turnpike result follows from a relatively straightforward modification of the arguments in [5], the main question that we want to address in this paper is more difficult: assume that strict dissipativity does not hold globally but only in a neighbourhood of a locally optimal equilibrium $x_l^\beta$. Can we still expect to see a turnpike property of trajectories starting close to $x_l^\beta$?

For the derivation of our technical results, we make frequent use of the dynamic programming principle

$$\begin{aligned} V_\infty (x_0)= \inf _{u\in {\mathbb {U}}^1(x_0)}\{\ell (x,u)+\beta V_\infty (f(x_0,u))\}, \end{aligned}$$

where

$$\begin{aligned} V_\infty (x_0):=\min _{u\in {\mathbb {U}}^\infty (x_0)}J_\infty (x_0,u) \end{aligned}$$

denotes the optimal value function of (2). If $u^*\in {\mathbb {U}}^\infty (x_0)$ is an optimal control sequence for an initial value $x_0\in {\mathbb {X}}$, i.e. if $J_\infty (x_0, u^*)=V_\infty (x_0)$ holds, then the identity

$$\begin{aligned} V_\infty (x_0)=\ell (x_0,u^*(0))+\beta V_\infty (f(x_0,u^*(0))) \end{aligned}$$

holds. Proofs for these statements can be found, e.g., in [9, Section 4.2].

We denote optimal trajectories by $x^*(k,x_0)$ and we say that a set ${\mathbb {X}}_{inv}\subset {\mathbb {X}}$ is forward invariant for the optimally controlled system, if for each $x_0\in {\mathbb {X}}_{inv}$ it follows that $x^*(k,x_0)\in {\mathbb {X}}_{inv}$ for all $k\ge 0$ and all optimal trajectories starting in $x_0$.

3 The Global Discounted Turnpike Property

In this section we first consider the optimal control problem (2) assuming global strict dissipativity. We show that under similar technical assumptions and with a similar proof technique as in [5] we can obtain a global turnpike result for near-optimal trajectories. To this end, we first introduce discounted strict dissipativity and afterwards we use it to conclude the turnpike property.

3.1 Global Discounted Strict Dissipativity

We denote an equilibrium of system (1) in the discounted case by $(x^\beta ,u^\beta )$ since the equilibria are dependent on the discount factor $\beta \in (0,1)$.

Definition 3.1

Given a discount factor $\beta \in (0,1)$, we say that the system (1) is discounted strictly dissipative at an equilibrium $(x^\beta ,u^\beta )$ with supply rate $s:{\mathbb {Y}}\rightarrow {\mathbb {R}}$ if there exists a storage function $\lambda :{\mathbb {X}}\rightarrow {\mathbb {R}}$ bounded from below with $\lambda (x^\beta )=0$ and a class ${\mathcal {K}}_\infty$-function $\alpha$ such that the inequality

$$\begin{aligned} s(x,u)+\lambda (x)-\beta \lambda (f(x,u))\ge \alpha (\Vert x-x^\beta \Vert ) \end{aligned}$$

(3)

holds for all $(x,u)\in {\mathbb {Y}}$ with $f(x,u)\in {\mathbb {X}}$.

The following lemma is Proposition 3.2 from [12]. Since its proof is short and simple, we provide it here for convenience of the readers. It shows that we can replace the stage cost $\ell$ by a modified—usually called rotated—stage cost ${\tilde{\ell }}$ that is positive definite without changing the optimal trajectories.

Lemma 3.2

Consider the discounted optimal control problem (2) with discount factor $\beta \in (0,1)$ and assume the system (1) is discounted strictly dissipative at an equilibrium $(x^\beta ,u^\beta )$ with supply rate $s(x,u)=\ell (x,u)-\ell (x^\beta ,u^\beta )$ and bounded storage function $\lambda$. Then the optimal trajectories of (2) coincide with those of the problem

$$\begin{aligned} \min _{u\in {\mathbb {U}}^\infty (x_0)}{\widetilde{J}}_\infty (x_0,u) \quad \text {with } {\widetilde{J}}_\infty (x_0,u):=\sum _{k=0}^\infty \beta ^k{\tilde{\ell }}(x(k,x_0),u(k)) \end{aligned}$$

(4)

with rotated stage cost

$$\begin{aligned} {\tilde{\ell }}(x,u)=\ell (x,u)-\ell (x^\beta ,u^\beta )+\lambda (x)-\beta \lambda (f(x,u)) \end{aligned}$$

(5)

which is positive definite in $x^\beta$ at $(x^\beta ,u^\beta )$, i.e. it satisfies the inequality ${\tilde{\ell }}(x,u) \ge \alpha (\Vert x-x^\beta \Vert )$ with $\alpha \in {\mathcal {K}}_\infty$ from (3) for all $(x,u)\in {\mathbb {Y}}$.

Proof

We rearrange

$$\begin{aligned} {\widetilde{J}}_\infty (x_0,u)=&\sum _{k=0}^\infty \beta ^k{\tilde{\ell }}(x(k,x_0),u(k)) =\sum _{k=0}^\infty \beta ^k\big (\ell (x(k,x_0),u(k))-\\&\ell (x^\beta ,u^\beta )+\lambda (x(k,x_0))-\beta \lambda (x(k+1,x_0))\big ) \end{aligned}$$

and a straightforward calculation shows that

$$\begin{aligned} {\widetilde{J}}_\infty (x_0,u)=J_\infty (x_0,u)-\dfrac{\ell (x^\beta ,u^\beta )}{1-\beta }+\lambda (x_0)-\lim _{k\rightarrow \infty }\beta ^k\lambda (x_u(k)). \end{aligned}$$

(6)

Since $\lambda$ is bounded and $\beta \in (0,1)$, the last limit exists and is equal to 0. Hence, the objectives differ only by expressions which are independent of U, from which the identity of the optimal trajectories immediately follows. The positive definiteness of ${\tilde{\ell }}$ follows from its definition, using strict dissipativity and the fact that $\lambda (x^\beta )=0$ implies ${\tilde{\ell }}(x^\beta ,u^\beta )=0$. $\square$

Remark 3.3

The requirement that $\ell (x^\beta ,u^\beta )=0$ is the reason for imposing

$\lambda (x^\beta )=0$ as a condition in Definition 3.1. Readers familiar with dissipativity for undiscounted problems will know that in the undiscounted case $\lambda (x^\beta )=0$ can be assumed without loss of generality, since if $\lambda$ is a storage function then $\lambda + c$ is a storage function for all $c\in {\mathbb {R}}$. In the discounted case, this invariance with respect to addition of constants no longer holds.

3.2 The Global Turnpike Property

In the non-discounted setting it is known that strict dissipativity (together with suitable regularity assumptions on the problem data) implies that optimal as well as near-optimal trajectories exhibit the turnpike property. In the discounted setting, it was observed already in [11] that for merely near-optimal trajectories the turnpike property can only be guaranteed on a finite discrete interval $\{0,\ldots ,M\}$. In this case M depends on the deviation from optimality (denoted by $\delta$ in the following theorem) and tends to infinity as this distance tends to 0. In fact, M and $\delta$ depend on each other, i.e. the smaller we choose $\delta$, the larger M becomes and, conversely, the larger we want M to be, the smaller we need to choose $\delta$. In the following theorem we take this second point of view, i.e. we fix M and choose $\delta >0$ accordingly. As the theorem shows, under the assumption of global discounted dissipativity we obtain precisely the turnpike property in the form which is originally given in [11, Definition 4.2]. The corresponding proof is a variation of the statement and proof of Theorem 3.1 and Corollary 4.3 in [5]. While in [5] only optimal trajectories were considered, here we extend the statement to near-optimal trajectories. We provide the proof in an appropriate way such that we can refer to it later in the proof of Theorem 4.3.

Theorem 3.4

Consider the infinite horizon optimal control problem (2) with discount factor $\beta \in (0,1)$ and assume the system (1) is discounted strictly dissipative at an equilibrium $(x^\beta ,u^\beta )$. Assume that the optimal value function ${{\widetilde{V}}}_\infty$ of the modified problem satisfies $\widetilde{V}_\infty (x) \le \alpha _V(\Vert x-x^\beta \Vert )$ and

$$\begin{aligned} {{\widetilde{V}}}_\infty (x) \le C \inf _{u\in {\mathbb {U}}}{{\tilde{\ell }}}(x,u) \end{aligned}$$

(7)

for all $x\in {\mathbb {X}}$, a function $\alpha _V\in {\mathcal {K}}_\infty$, and a constant $C\ge 1$ satisfying

$$\begin{aligned} C < 1/(1-\beta ) . \end{aligned}$$

(8)

Then the optimal control problem has the following turnpike property (cf. [11, Definition 4.2]):

For each $\varepsilon >0$ and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$ there exist a constant $P>0$ such that for each $M\in {\mathbb {N}}$ there is a $\delta >0$, such that for all $x_0\in {\mathbb {X}}_b$ and $u\in {\mathbb {U}}^\infty (x_0)$ with $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$, the set ${\mathcal {Q}}(x_0,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}$ has at most P elements.

Proof

It follows from the proof of Lemma 3.2 that the inequality $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$ implies ${{\widetilde{J}}}_\infty (x_0,u)\le {{\widetilde{V}}}_\infty (x_0)+\delta$. Together with the dynamic programming principle for $\widetilde{V}_\infty$ this yields

$$\begin{aligned} \delta\ge & {} {{\widetilde{J}}}_\infty (x_0,u) - {{\widetilde{V}}}_\infty (x_0) \\= & {} {{\tilde{\ell }}}(x_0,u(0)) + \beta {{\widetilde{J}}}_\infty (x_u(1,x_0),u(\cdot +1)) - \inf _{u\in {\mathbb {U}}}\left\{ {{\tilde{\ell }}}(x_0,u) + \beta {{\widetilde{V}}}_\infty (f(x_0,u))\right\} \\\ge & {} {{\tilde{\ell }}}(x_0,u(0)) + \beta \widetilde{J}_\infty (x_u(1,x_0),u(\cdot +1)) - \left( {{\tilde{\ell }}}(x_0,u(0)) + \beta {{\widetilde{V}}}_\infty (f(x_0,u(0)))\right) \\= & {} \beta ({{\widetilde{J}}}_\infty (x_u(1,x_0),u(\cdot +1)) - \widetilde{V}_\infty (f(x_0,u(0)))). \end{aligned}$$

This implies ${{\widetilde{J}}}_\infty (x_u(1,x_0),u(\cdot +1))\le {{\widetilde{V}}}_\infty (x_u(1,x_0))+\delta /\beta$, and proceeding inductively we obtain

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_u(k,x_0),u(\cdot +k))\le {{\widetilde{V}}}_\infty (x_u(k,x_0))+\frac{\delta }{\beta ^k} \end{aligned}$$

for all $k\in {\mathbb {N}}$. This implies

$$\begin{aligned}&{{\widetilde{V}}}_\infty (x_u(k+1,x_0)) - {{\widetilde{V}}}_\infty (x_u(k,x_0)) \nonumber \\&\quad = \frac{1}{\beta }\Big (\beta {{\widetilde{V}}}_\infty (x_u(k+1,x_0)) - \beta {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big )\nonumber \\&\quad = \frac{1}{\beta }\Big (\beta {{\widetilde{V}}}_\infty (x_u(k+1,x_0)) - {{\widetilde{V}}}_\infty (x_u(k,x_0)) + (1-\beta ) {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big )\nonumber \\&\quad \le { \frac{1}{\beta }\Big (\beta {{\widetilde{J}}}_\infty (x_u(k+1,x_0),u(\cdot +k+1)) - {{\widetilde{J}}}_\infty (x_u(k,x_0),u(\cdot +k))+ \frac{\delta }{\beta ^k}}\nonumber \\&\qquad {+ (1-\beta ) {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big )}\nonumber \\&\quad = \frac{1}{\beta }\Big (\beta {{\widetilde{J}}}_\infty (x_u(k+1,x_0),u(\cdot +k+1)) - {{\widetilde{J}}}_\infty (x_u(k,x_0),u(\cdot +k))\nonumber \\&\qquad + (1-\beta ) {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big ) + \frac{\delta }{\beta ^{k+1}}\nonumber \\&\quad = \frac{1}{\beta }\Big (-{{\tilde{\ell }}}(x_u(k,x_0),u(k)) + (1-\beta ) {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big )+\frac{\delta }{\beta ^{k+1}}\nonumber \\&\quad \le \frac{1}{\beta }\Big (-\frac{1}{C} {{\widetilde{V}}}_\infty (x_u(k,x_0))+ (1-\beta ) {{\widetilde{V}}}_\infty (x_u(k,x_0))\Big ) + \frac{\delta }{\beta ^{k+1}}\nonumber \\&\quad = \frac{\kappa }{\beta } {{\widetilde{V}}}_\infty (x_u(k,x_0)) +\frac{\delta }{\beta ^{k+1}} \end{aligned}$$

(9)

where $\kappa = (1-\beta )-1/C < 0$ because of (8).

Moreover, strict discounted dissipativity implies $\widetilde{V}_\infty (x) \ge {{\tilde{\ell }}}(x,u) \ge \alpha (\Vert x-x^\beta \Vert )$. Together with the upper bound ${{\widetilde{V}}}_\infty (x) \le \alpha _V(\Vert x-x^\beta \Vert )$ from the assumption this yields that for fixed $M\in {\mathbb {N}}$ and $k\in \{0,\ldots ,M\}$ the function $\widetilde{V}_\infty$ is a practical Lyapunov function. Using [7, Theorem 2.4] restricted to $\{0,\ldots ,M\}$ and the fact that ${\mathbb {X}}_b$ is bounded we can conclude that there is a sequence $\eta _k\rightarrow 0$ (depending on ${\mathbb {X}}_b$) and a function $\gamma \in {\mathcal {K}}_\infty$ with

$$\begin{aligned} \Vert x_u(k,x_0)-x^\beta \Vert \le \eta _k + \gamma (\delta /\beta ^{k+1}) \le \eta _k + \gamma (\delta /\beta ^M) \end{aligned}$$

for all $k\in \{0,\ldots ,M\}$. This implies the desired claim by choosing $P\in {\mathbb {N}}$ (depending on $\varepsilon$ and $\eta _k$, hence on ${\mathbb {X}}_b$) such that $\eta _k < \varepsilon /2$ for all $k\ge P$ and $\delta >0$ (depending on $\beta$, $\varepsilon$ and M) such that $\gamma (\delta /\beta ^M)<\varepsilon /2$. $\square$

For an illustration of the described turnpike property we refer to Fig. 1. We note again that in the formulation of the discounted turnpike property the level $\delta$ which measures the deviation from optimality of the trajectory $x_u(\cdot ,x_0)$ depends on M. For guaranteeing the turnpike property on $\{0,\ldots ,M\}$, $\delta \rightarrow 0$ may be required if $M\rightarrow \infty$, cf. also Remark 3.5 (iv).

Fig. 1

Illustration of the set ${\mathcal {Q}}(x_0,u,\varepsilon ,M,\beta )$

The following remark discusses aspects of the assumptions of Theorem 3.4. For the turnpike property to hold, it is obviously necessary that the state of the system can be steered to $x^\beta$, at least asymptotically. This is made precise in part (i) of the remark. Part (ii) shows that if the state can be steered to $x^\beta$ fast enough, then a constant C satisfying (7) for all $\beta \in (0,1)$ holds. Finally, part (iii) of the remark discusses how inequality (7) can be relaxed if such a C cannot be found.

Remark 3.5

• (i) A necessary condition for the turnpike property to hold is that for each $\varepsilon >0$, each bounded subset ${\mathbb {X}}_b\subseteq {\mathbb {X}}$ and each $x_0\in {\mathbb {X}}_b$ there exists a control sequence $u\in {\mathbb {U}}^{P+1}(x_0)$ with $x_u(k,x_0)\in {\mathcal {B}}_\varepsilon (x^\beta )$ for some $k\le P+1$, where P is the constant from the turnpike property in Theorem 3.4. This is immediately clear, because if such a control does not exist, then the number of points $x_u(k,x_0)\not \in {\mathcal {Q}}(x_0,u,\varepsilon ,M,\beta )$ is larger than P for all u.

• (ii) If a constant C satisfying (7) for all $\beta \in (0,1)$ exists, then (8) will hold for all sufficiently large $\beta \in (0,1)$. A sufficient condition for the existence of such a C is the following exponential stabilizability assumption of the cost at the equilibrium $(x^\beta ,u^\beta )$: there are constants $\sigma ,\lambda >0$ such that for each $x_0\in {\mathbb {X}}$ there is $u\in {\mathbb {U}}^\infty (x_0)$ with

  $$\begin{aligned} {\tilde{\ell }}(x_u(k,x_0),u(k)) \le \sigma e^{-\lambda k}\inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}). \end{aligned}$$

  (10)

  Then, since ${\tilde{\ell }}\ge 0$ we obtain

  $$\begin{aligned} {{\widetilde{V}}}_\infty (x_0)&\le \sum _{k=0}^\infty \beta ^k{\tilde{\ell }}(x_u(k,x_0),u(k)) \le \sum _{k=0}^\infty {\tilde{\ell }}(x_u(k,x_0),u(k))\\&\le \sum _{k=0}^\infty \sigma e^{-\lambda k}\inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}) = \frac{\sigma }{1-e^{-\lambda }} \inf _{{\hat{u}} \in {\mathbb {U}}} {\tilde{\ell }}(x_0,{\hat{u}}), \end{aligned}$$

  implying (7) with $C=\sigma /(1-e^{-\lambda })$. We note that (10) holds in particular if the system itself is exponentially stabilizable to $x^\beta$ with exponentially bounded controls and ${\tilde{\ell }}$ is a polynomial.² Exponential stabilizability of the system, in turn, follows locally around $x^\beta$ from stabilizability of its linearization in $x^\beta$. If, in addition, the necessary condition from part (i) of this remark holds, then local exponential stabilizability implies exponential stabilizability for bounded ${\mathbb {X}}$. We refer to [5, Section 6] for a more detailed discussion on these conditions. Note that if ${\tilde{\ell }}$ is continuous, then $\inf _{u\in {\mathbb {U}}}{\tilde{\ell }}(x,u) \le \alpha _\ell (\Vert x-x^\beta \Vert )$ holds for an appropriate $\alpha _\ell \in {\mathcal {K}}_\infty$. In this case, (7) implies the assumed upper bound $\alpha _V$ on ${{\widetilde{V}}}_\infty$.

• (iii) If a C meeting (8) and (7) for all $x\in {\mathbb {X}}$ does not exist, then we may still be able to find a C satisfying (8) and (7) for all $x\in {\mathbb {X}}$ with $\vartheta \le \Vert x-x^\beta \Vert \le \Theta$, for parameters $0\le \vartheta <\Theta$. In this case we can follow the reasoning in the proof of Corollary 4.3 from [5] to conclude that we still obtain a turnpike property for $\varepsilon >\varepsilon _0$ and ${\mathbb {X}}_b={\mathcal {B}}_\Delta (x^\beta )\cap {\mathbb {X}}$, with $\varepsilon _0\rightarrow 0$ as $\vartheta \rightarrow 0$ and $\Delta \rightarrow \infty$ as $\Theta \rightarrow \infty$.

• (iv) Optimal trajectories, i.e., trajectories for which $J_\infty (x_0,u)=V_\infty (x_0)$ holds, satisfy the assumptions of Theorem 3.4 for each $\delta >0$. Hence, the assertion of the theorem holds for each $\varepsilon >0$ and each $M\in {\mathbb {N}}$, implying that $x_u(k,x_0)$ converges to $x^\beta$ as $k\rightarrow \infty$ and thus $x^\beta$ behaves similarly to an asymptotically stable equilibrium. Whenever $\delta >0$, however, the trajectory will typically move away from $x^\beta$ for large times. This is more similar to the classical turnpike phenomenon in optimal control than to asymptotic stability, which is why we prefer this term over asymptotic stability or practical asymptotic stability.

4 The Local Discounted Turnpike Property Assuming Invariance

In the previous section, we have shown that an equilibrium at which the system is globally strictly dissipative has the turnpike property. Now, we consider an equilibrium denoted by $(x^\beta _l,u^\beta _l)$ at which discounted strict dissipativity holds only locally, i.e., for all x in a neighbourhood ${\mathbb {X}}_{{\mathcal {N}}}$ of $x_l^\beta$, in the following sense.

Definition 4.1

Given a discount factor $\beta \in (0,1)$, we say that the system (1) is locally discounted strictly dissipative at an equilibrium $(x^\beta _l,u^\beta _l)$ with supply rate $s:{\mathbb {Y}}\rightarrow {\mathbb {R}}$ if there exists a storage function $\lambda :{\mathbb {X}}\rightarrow {\mathbb {R}}$ bounded from below with $\lambda (x_l^\beta )=0$ and a class ${\mathcal {K}}_\infty$-function $\alpha _\beta$ such that the inequality

$$\begin{aligned} s(x,u)+\lambda (x)-\beta \lambda (f(x,u))\ge \alpha _\beta (\Vert x-x_l^\beta \Vert ) \end{aligned}$$

(11)

holds for all $(x,u)\in {\mathbb {X}}_{\mathcal {N}}\times {\mathbb {U}}$ with $f(x,u)\in {\mathbb {X}}$.

Further, we say that system (1) is locally discounted strictly (x, u)-dissipative at the equilibrium $(x^\beta _l,u^\beta _l)$ with supply rate $s:{\mathbb {X}}\times {\mathbb {U}}\rightarrow {\mathbb {R}}$ if the same holds with the inequality

$$\begin{aligned} s(x,u)+\lambda (x)-\beta \lambda (f(x,u))\ge \alpha _\beta (\Vert (x-x_l^\beta \Vert +\Vert u-u_l^\beta )\Vert ). \end{aligned}$$

(12)

As in the global case we define the rotated stage cost by

$$\begin{aligned} {\tilde{\ell }}(x,u) := \ell (x,u)-\ell (x^\beta _l,u^\beta _l)+\lambda (x)-\beta \lambda (f(x,u)). \end{aligned}$$

(13)

We note that this definition is local since we only require it for $x\in {\mathbb {X}}_{\mathcal {N}}$. However, also in this local version we require the dissipation inequalities to hold for all $u\in {\mathbb {U}}$ with $f(x,u)\in {\mathbb {X}}$, i.e. also for control values that drive the state out of the neighbourhood ${\mathbb {X}}_{\mathcal {N}}$ of the equilibrium $x_l^\beta$. This property will be important in the proof of Lemma 5.2, below.

Obviously, with this definition Lemma 3.2 remains valid. Moreover, for $x\in {\mathbb {X}}_{{\mathcal {N}}}$ the function ${\tilde{\ell }}$ satisfies the same properties as in the globally dissipative case. This will enable us to derive a local turnpike property, provided the neighbourhood ${\mathbb {X}}_{{\mathcal {N}}}$ contains an invariant set ${\mathbb {X}}_{inv}\subset {\mathbb {X}}_{{\mathcal {N}}}$ for the optimally controlled system. The following lemma gives a consequence of this assumption for the modified optimal value function, which will be important for concluding the local turnpike property.

Lemma 4.2

Consider the optimal control problem (2) with given discount factor $\beta \in (0,1)$ and assume that the system is locally strictly dissipative in $(x^\beta _l,u^\beta _l)\in {\mathbb {X}}_{\mathcal {N}}\subset {\mathbb {X}}$. Consider a subset ${\mathbb {X}}_{inv}\subset {\mathbb {X}}_{\mathcal {N}}$ such that all optimal solutions $x^*(k,x_0)$ with $x_0\in {\mathbb {X}}_{inv}$ satisfy $x^*(k,x_0) \in {\mathbb {X}}_{inv}$ for all $k\ge 0$.

Then the modified optimal value function ${{\widetilde{V}}}_\infty$ satisfies

$$\begin{aligned} {{\widetilde{V}}}_\infty (x) \ge \alpha _\beta (\Vert x-x_l^\beta \Vert ) \end{aligned}$$

(14)

for all $x\in {\mathbb {X}}_{inv}$.

Proof

For all $x\in {\mathbb {X}}_{\mathcal {N}}$ and $u\in {\mathbb {U}}$ the modified cost satisfies ${{\tilde{\ell }}}(x,u) \ge \alpha _\beta (\Vert x-x_l^\beta \Vert )\ge 0$. This implies

$$\begin{aligned} {{\widetilde{V}}}_\infty (x_0) = \sum _{k=0}^\infty \beta ^k{{\tilde{\ell }}}(x(k,x_0),u(k)) \ge \sum _{k=0}^\infty \beta ^k\alpha _\beta (\Vert x(k,x_0)-x_l^\beta \Vert ) \ge \alpha _\beta (\Vert x_0-x_l^\beta \Vert ), \end{aligned}$$

which shows the claim. $\square$

The following now gives a local version of Theorem 3.4.

Theorem 4.3

Consider the infinite horizon optimal control problem (2) with discount factor $\beta \in (0,1)$ and assume that the system is locally strictly dissipative at $(x^\beta _l,u^\beta _l)\in {\mathbb {X}}_{\mathcal {N}}\subset {\mathbb {X}}$. Consider a subset ${\mathbb {X}}_{inv}\subset {\mathbb {X}}_{\mathcal {N}}$ such that all optimal solutions $x^*(k,x_0)$ with $x_0\in {\mathbb {X}}_{inv}$ satisfy $x^*(k,x_0) \in {\mathbb {X}}_{inv}$ for all $k\ge 0$ and suppose that the assumptions of Theorem 3.4 hold for all $x \in {\mathbb {X}}_{inv}$.

Then the optimal control problem has the following turnpike property on ${\mathbb {X}}_{inv}$:

For each $\varepsilon >0$ and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}_{inv}$ there exist a constant $P>0$ such that for each $M\in {\mathbb {N}}$ there is a $\delta >0$, such that for all $u\in {\mathbb {U}}^\infty (x_0)$ with $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$ and $x_u(k,x_0)\in {\mathbb {X}}_{inv}$ for all $k\in \{0,\ldots ,M\}$, the set ${\mathcal {Q}}(x,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}$ has at most P elements.

Proof

The proof proceeds completely identical to the proof of Theorem 3.4, using the fact that all inqualities used in this proof remain valid as long as the considered solutions stay in ${\mathbb {X}}_{inv}$ which is guaranteed by the assumptions. We note that Lemma 4.2 is needed for establishing the lower bound on ${{\widetilde{V}}}_\infty$ required from a practical Lyapunov function. $\square$

Remark 4.4

Instead of assuming the existence of the invariant set ${\mathbb {X}}_{inv}$ we could also assume (14) for all $x\in {\mathbb {X}}_{{\mathcal {N}}}$. Then by standard Lyapunov function arguments the largest sublevel set of ${{\widetilde{V}}}_\infty$ contained in ${\mathbb {X}}_{{\mathcal {N}}}$ is forward invariant for the optimal solutions and can then be used as set ${\mathbb {X}}_{inv}$. Using (9) we can even ensure that this sublevel set is also forward invariant for all solutions satisfying $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$ provided $\delta >0$ is sufficiently small. Hence, for this choice of ${\mathbb {X}}_{inv}$ the assumption that $x_u(k,x_0)\in {\mathbb {X}}_{inv}$ for all $k\in \{0,\ldots ,M\}$ in Theorem 4.3 would be automatically satisfies if $\delta$ is not too large.

5 Optimal Trajectories Stay near a Locally Dissipative Equilibrium

Theorem 4.3 shows that the local turnpike property holds if the optimal solutions stay in the neighbourhood of $x_l^\beta$ in which the strict dissipativity property holds. In this section we show that this condition is “automatically” satisfied for appropriate discount factors. This will enable us to conclude a local turnpike property from local strict dissipativity. To this end, we aim to show that there exists a range of discount factors $\beta$ for which it is more favourable to stay near the locally dissipative equilibrium than to move to other parts of the state space. The first lemma we need to this end shows a property of trajectories that move out of a neighbourhood of $x_l^\beta$. In contrast to the previous result, now we need the stronger (x, u)-dissipativity.

Lemma 5.1

Consider a discounted optimal control problem (2) subject to system (1) with continuous f. Assume local strict (x, u)-dissipativity at an equilibrium $(x^\beta _l,u^\beta _l)$ according to Definition 4.1 and let $\rho >0$ be such that ${\mathcal {B}}_\rho (x_l^\beta )\subset {\mathbb {X}}_{{\mathcal {N}}}$ holds for the neighbourhood ${\mathbb {X}}_{{\mathcal {N}}}$ from Definition 4.1. Then there exists $\eta >0$ such that for each $K\in {\mathbb {N}}$ and any trajectory $x(\cdot )$ with $x_0=x(0)\in {\mathcal {B}}_\eta (x_l^\beta )$ and $x(K)\notin {\mathcal {B}}_\rho (x_l^\beta )$ there is a $M\in \{0,\dots , K-1\}$ such that $x(0),\ldots ,x(M)\in {\mathcal {B}}_\eta (x_l^\beta )$ and either

$$\begin{aligned} \text {(i) } {x(M+1)}\in {\mathcal {B}}_\rho (x_l^\beta )\backslash {\mathcal {B}}_\eta (x_l^\beta ) \quad \text {or} \quad \text {(ii) } \Vert u(M)-u_l^\beta \Vert \ge \eta \end{aligned}$$

holds.

Proof

The continuity of f implies that there exists $\varepsilon >0$ such that $\Vert f(x,u)-x_l^\beta \Vert <\rho$ for all $(x,u)\in {\mathbb {Y}}$ with $\Vert x-x_l^\beta \Vert <\varepsilon$ and $\Vert u-u_l^\beta \Vert <\varepsilon$. We let $K_{\min }$ be minimal with $x(K_{\min })\notin {\mathcal {B}}_\rho (x_l^\beta )$, set $\eta :=\min \{\varepsilon , \rho \}$, and claim that this implies the assertion for $M=K_{\min }-1$.

We prove this claim by contradiction. To this end, we assume that for $M=K_{\min }-1$ neither assertion (i) nor assertion (ii) holds. This implies on the one hand that $\Vert x(M)-x_l^\beta \Vert <\eta$, since $x(M)\in {\mathcal {B}}_\rho (x_l^\beta )$ by minimality of $K_{\min }$ and (i) is not fulfilled. On the other hand, it implies $\Vert u(M)-u_l^\beta \Vert <\eta$, because (ii) does not hold. Then, however, since $\eta \le \varepsilon$, the continuity of f implies

$$\begin{aligned} \Vert x(K_{\min })-x_l^\beta \Vert = \Vert f(x(M),u(M))-x_l^\beta \Vert < \rho . \end{aligned}$$

This means that $x(K_{\min })\in {\mathcal {B}}_\rho (x_l^\beta )$, which is a contradiction to the choice of $K_{\min }$. Hence, either assertion (i) or assertion (ii) must hold for $M=K_{\min }-1$. $\square$

The next lemma shows that the behaviour characterized in Lemma 5.1 induces a lower bound for the rotated discounted functional ${\widetilde{J}}_\infty$ from (4) along trajectories that start in a neighbourhood of $x_l^\beta$ and leave this neighbourhood. To this end, we note that even if merely local strict dissipativity holds, the modified stage cost ${\tilde{\ell }}$ from (13) is well defined, since $\lambda$ is defined for all $x\in {\mathbb {X}}$. However, the inequality ${\tilde{\ell }}(x,u) \ge \alpha _\beta (\Vert x-x_l^\beta \Vert +\Vert u-u_l^\beta \Vert )$ and, more generally, positivity of ${\tilde{\ell }}$ are only guaranteed for $x\in {\mathbb {X}}_{{\mathcal {N}}}$.

Lemma 5.2

Let the assumptions of Lemma 5.1 hold. In addition, assume that $\lambda$ from Definition 4.1 is bounded and the stage cost $\ell$ is bounded from below. Then, there exists $\beta ^\star \in (0,1)$ with the following property: for any $\beta \in (0,\beta ^\star )$ and any $K\in {\mathbb {N}}$ there is $\sigma (\beta ,K)>0$ such that for any trajectory $x(\cdot )$ with $x_0=x(0)\in {\mathcal {B}}_\eta (x_l^\beta )$ and $x(P)\notin {\mathcal {B}}_\rho (x_l^\beta )$ for some $P\in \{1,\ldots ,K\}$ the inequality

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_0,u) \ge \sigma (\beta ,K) \end{aligned}$$

(15)

holds.

Proof

First observe that boundedness from below of $\ell$ and boundedness of $\lambda$ imply boundedness from below of ${\tilde{\ell }}$. Let ${\tilde{\ell }}_{\min } := \inf _{(x,u)\in {\mathbb {Y}}} {\tilde{\ell }}(x,u)$. Since ${\tilde{\ell }}(x^\beta ,u^\beta )=0$ it follows that ${\tilde{\ell }}_{\min }\le 0$. Moreover, local dissipativity implies that ${\tilde{\ell }}(x,u)\ge 0$ for all $x\in {\mathbb {X}}_{{\mathcal {N}}}$ and all $u\in {\mathbb {U}}$ with $f(x,u)\in {\mathbb {X}}$. We note that it is important for the remainder of the proof that this inequality holds for all these $u\in {\mathbb {U}}$ and not only when $f(x,u)\in {\mathbb {X}}_{\mathcal {N}}$.

Since the trajectory under consideration satisfies the assumptions of Lemma 5.1 with $K=P$, there exists $M\in \{0,\dots ,P\}$ such that either assertion (i) or assertion (ii) of this lemma holds. Within the strict dissipativity, in case (i), we obtain that

$$\begin{aligned} {\tilde{\ell }}(x(M),u(M)) \ge \alpha _\beta (\Vert x(M)-x_l^\beta \Vert ) \ge \alpha _\beta (\eta ) \end{aligned}$$

and in case (ii) we obtain

$$\begin{aligned} {\tilde{\ell }}(x(M),u(M)) \ge \alpha _\beta (\Vert u(M)-u_l^\beta \Vert ) \ge \alpha _\beta (\eta ). \end{aligned}$$

Hence, we get the same inequality in both cases and we abbreviate $\delta :=\alpha _\beta (\eta )>0$. In addition, Lemma 5.1 yields $x(0),\ldots ,x(M)\in {\mathcal {B}}_\eta (x_l^\beta )\subset {\mathbb {X}}_{{\mathcal {N}}}$, which implies ${\tilde{\ell }}(x(k),u(k))\ge 0$ for all $k=0,\ldots ,M-1$, and the lower bound on ${\tilde{\ell }}$ implies ${\tilde{\ell }}(x(k),u(k))\ge {\tilde{\ell }}_{\min }$ for all $k\ge M+1$. Together this yields

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_0,u) =&\sum _{k=0}^\infty \beta ^k{\tilde{\ell }}(x(k,x_0),u(k)\\ =&\sum _{k=0}^{M-1}\beta ^k\underbrace{{\tilde{\ell }}(x(k,x_0),u(k))}_{\ge 0}+\beta ^M\underbrace{{\tilde{\ell }}(x(M,x_0),u(M))}_{\ge \delta }\\&+\sum _{k=M+1}^\infty \beta ^k\underbrace{{\tilde{\ell }}(x(k,x_0),u(k))}_{\ge {\tilde{\ell }}_{\min }}\\ \ge&\beta ^M\delta +\dfrac{\beta ^{M+1}}{1-\beta }{\tilde{\ell }}_{\min } = \dfrac{\beta ^M}{1-\beta }\left( \left( {\tilde{\ell }}_{\min }-\delta \right) \beta +\delta \right) . \end{aligned}$$

We now claim that the assertion holds for $\sigma = \frac{\beta ^K\delta }{2(1-\beta )} \le \frac{\beta ^M\delta }{2(1-\beta )}$. To this end, it is sufficient to show the existence of $\beta ^\star$ with

$$\begin{aligned} \dfrac{\beta ^M}{1-\beta }\left( \left( {\tilde{\ell }}_{\min }-\delta \right) \beta +\delta \right) \ge \frac{\beta ^M\delta }{2(1-\beta )} \end{aligned}$$

for all $\beta \in (0,\beta ^\star )$. This is equivalent to

$$\begin{aligned} \dfrac{\beta ^M}{1-\beta }\left( \left( {\tilde{\ell }}_{\min }-\delta \right) \beta +\frac{\delta }{2}\right) \ge 0\;\; \Leftrightarrow \;\; \left( {\tilde{\ell }}_{\min }-\delta \right) \beta +\frac{\delta }{2}\ge 0, \end{aligned}$$

since ${\tilde{\ell }}_{\min }-\delta <0$. This inequality holds for all $\beta \in (0,\beta ^\star )$ if $\beta ^\star =\delta /(2(\delta -{\tilde{\ell }}_{\min }))$. $\square$

Remark 5.3

The choice of the fraction $\frac{1}{2}$ for $\sigma$ in the proof of Lemma 5.2 is arbitrary. We can also use a more general fraction $\frac{1}{k+1}$ with $k\in {\mathbb {N}}$. Then, with the same calculation as above we get that $\beta ^\star = \dfrac{k}{k+1}\dfrac{\delta }{\delta -{\tilde{\ell }}_{\min }}$.

Based on the estimate from Lemma 5.2 we can now conclude that near-optimal solutions starting near $x_l^\beta$ stay in ${\mathbb {X}}_{\mathcal {N}}$ for a certain amount of time.

Lemma 5.4

Consider a discounted optimal control problem (2) subject to system (1) with f continuous and stage cost $\ell$ bounded from below. Assume local strict (x, u)-dissipativity at an equilibrium $(x^\beta _l,u^\beta _l)$ according to Definition 4.1 with bounded storage function $\lambda$. Assume furthermore that there is $\gamma \in {\mathcal {K}}_\infty$ and ${\hat{\beta \in }}(0,1]$ such that ${{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )$ for all $x\in {\mathbb {X}}_{{\mathcal {N}}}$ and all $\beta \in (0,{\hat{\beta ]}}$. Then there exists $\beta _2\in (0,1)$ with the following property: for any $\beta \in (0,\beta _2)$ and any $K\in {\mathbb {N}}$ there exists a neighbourhood ${\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )$ and a threshold value $\theta (\beta ,K)>0$ such that all trajectories with $x_0\in {\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )$ and $J_\infty (x_0,u) < V_\infty (x_0) + \theta (\beta ,K)$ satisfy $x(k)\in {\mathbb {X}}_{{\mathcal {N}}}$ for all $k\in \{0,\ldots ,K\}$.

Proof

We choose $\beta _2$ as the minimum of $\beta ^\star$ from Lemma 5.2 and ${\hat{\beta }}$. We further use $\sigma (\beta ,K)>0$ from Lemma 5.2 to set $\varepsilon (\beta ,K):= \gamma ^{-1}(\sigma (\beta ,K)/2)$ and $\theta (\beta ,K):= \sigma (\beta ,K)/2$. Now consider a trajectory meeting the assumptions and observe that since $J_\infty$ and ${{\widetilde{J}}}_\infty$ differ only by a term that is independent of $u(\cdot )$, the assumption $J_\infty (x_0,u) \le V_\infty (x_0) + \theta (\beta ,K)$ together with the assumption on $x_0$ implies

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_0,u)< {{\widetilde{V}}}_\infty (x_0) + \theta (\beta ,K) < \gamma (\varepsilon (\beta ,K)) + \theta (\beta ,K). \end{aligned}$$

The definition of $\theta$ and $\varepsilon$ then implies

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_0,u) < \sigma (\beta ,K)/2 + \sigma (\beta ,K)/2 = \sigma (\beta ,K). \end{aligned}$$

Since by Lemma 5.2 any trajectory leaving ${\mathbb {X}}_{{\mathcal {N}}}$ (and thus also ${\mathcal {B}}_\rho (x_l^\beta )$) up to time K has a rotated value satisfying

$$\begin{aligned} {{\widetilde{J}}}_\infty (x_0,u) \ge \sigma (\beta ,K), \end{aligned}$$

the trajectory under consideration cannot leave ${\mathbb {X}}_{{\mathcal {N}}}$ for $k\in \{0,\ldots ,K\}$. $\square$

We note that ${{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )$ can be ensured if we can locally steer the system to $x^\beta$ fast enough and ${\tilde{\ell }}$ is continuous. For a more detailed discussion see Remark 3.5(ii).

6 The Local Discounted Turnpike Property Without Assuming Invariance

With the preparations from the previous sections, we are now able to formulate our main theorem on the existence of a local turnpike property.

Theorem 6.1

Consider a discounted optimal control problem (2) subject to system (1) with f continuous and stage cost $\ell$ bounded from below. Assume local strict (x, u)-dissipativity at an equilibrium $(x^\beta _l,u^\beta _l)$ according to Definition 4.1 with bounded storage function $\lambda$. Assume furthermore that there is $\gamma \in {\mathcal {K}}_\infty$ and ${\hat{\beta \in }}(0,1]$ such that ${{\widetilde{V}}}_\infty (x) \le \gamma (\Vert x-x_l^\beta \Vert )$ for all $x\in {\mathbb {X}}_{{\mathcal {N}}}$ and all $\beta \in (0,{\hat{\beta }})$, and that there is an interval $[\beta _1,\beta ^\star ]$ of discount rates with $\beta _1<{\hat{\beta }}$ and $\beta ^\star$ from Lemma 5.2, such that for each $\beta \in (\beta _1,\beta ^\star )$ the assumptions of Theorem 3.4 hold for all $x \in {\mathbb {X}}_{{\mathcal {N}}}$.

Then there is $\beta _2\in (0,1)$ such that for all $\beta \in (\beta _1,\beta _2)$ there exists a neighbourhood ${\mathcal {N}}$ of $x_l^\beta$ on which the system exhibits a local turnpike property in the following sense:

For each $\varepsilon >0$ there exist a constant $P>0$ such that for each $M\in {\mathbb {N}}$ there is a $\delta >0$, such that for all $x_0\in {\mathcal {N}}$ and all $u\in {\mathbb {U}}^\infty (x_0)$ with $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$, the set ${\mathcal {Q}}(x,u,\varepsilon ,M,\beta ):=\{k\in \{0,\dots , M\}\mid \Vert x_u(k,x_0)-x^\beta \Vert \ge \varepsilon \}$ has at most P elements.

Particularly, if $J_\infty (x_0,u) = V_\infty (x_0)$, i.e., if the trajectory is optimal, then for each $\varepsilon >0$ the set ${\mathcal {Q}}(x,u,\varepsilon ,\infty ,\beta ):=\bigcup _{M\in {\mathbb {N}}}{\mathcal {Q}}(x,u,\varepsilon ,M,\beta )$ has at most P elements, implying the convergence $x_u(k,x_0)\rightarrow x^\beta$ as $k\rightarrow \infty$.

Proof

The idea of the proof is to use $\beta _2$ from Lemma 5.4 and, for each $\beta \in (\beta _1,\beta _2)$, to construct a neighbourhood ${\mathcal {N}}$ of $x_l^\beta$ and a $\delta >0$ such that all trajectories starting in $x_0\in {\mathcal {N}}$ and satisfying $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$ stay in ${\mathcal {N}}$ for all future times. Then the turnpike property follows from Theorem 4.3 applied with ${\mathbb {X}}_{inv}={\mathcal {N}}$.

To this end, we take $\beta _2$ from Lemma 5.4, fix $\beta \in (\beta _1,\beta _2)$, and consider the neighbourhood ${\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )$ and the threshold value $\theta (\beta ,1)$ from Lemma 5.4 for $K=1$. We choose ${\mathcal {N}}$ as the largest sublevel set of ${{\widetilde{V}}}_\infty$ that is contained in ${\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )$ and denote the level by $\mu >0$, i.e., ${\mathcal {N}}=\{x\in {\mathbb {X}}_{{\mathcal {N}}}\,|\, {{\widetilde{V}}}_\infty (x) < {\mu }\}$. We abbreviate $\kappa = (1-\beta )-1/C$, observing that $\kappa <0$ because of (8) (cf. also the proof of Theorem 3.4), and set

$$\begin{aligned} \delta := \beta ^M\min \left\{ \theta (\beta ,1), -\frac{\kappa {\mu }}{2\beta },\frac{{ \mu }}{ 2}\right\} . \end{aligned}$$

Now let $x_0$ and u be as in the assertion, i.e., satisfying $J_\infty (x_0,u)\le V_\infty (x_0)+\delta$, and denote the corresponding trajectory by $x(\cdot )$. Then, just as in the first part of the proof of Theorem 3.4, we obtain the estimate

$$\begin{aligned} {{\widetilde{J}}}_\infty (x(k),u(\cdot +k))\le {{\widetilde{V}}}_\infty (x(k))+\frac{\delta }{\beta ^k} \end{aligned}$$

for all $k\in {\mathbb {N}}$. By definition of $\delta$ this in particular implies

$$\begin{aligned} {{\widetilde{J}}}_\infty (x(k),u(\cdot +k))\le {{\widetilde{V}}}_\infty (x(k))+ \theta (\beta ,1) \end{aligned}$$

(16)

for all $k=0,\ldots ,M$.

Now we prove by induction that $x(k)\in {\mathcal {N}}$ for all $k=0,\ldots ,M$. For $k=0$ this follows from the choice of $x_0$. For $k\rightarrow k+1$, we make the induction assumption that $x(k)\in {\mathcal {N}}$, i.e., $\widetilde{V}_\infty (x(k))<{\mu }$. Then, because of (16) and ${\mathcal {N}}\subseteq {\mathcal {B}}_{\varepsilon (\beta ,1)}(x_l^\beta )$, Lemma 5.4 (applied with initial value $x_0=x(k)$ and control $u(\cdot +k)$) implies that $x(k+1)\in {\mathbb {X}}_{{\mathcal {N}}}$. Hence, all the (in)equalities leading to inequality (9) in the proof of Theorem 3.4 are valid and, together with the definition of $\delta$, yield

$$\begin{aligned} {{\widetilde{V}}}_\infty (x(k+1)) - {{\widetilde{V}}}_\infty (x(k))&\le \frac{\kappa }{\beta } {{\widetilde{V}}}_\infty (x(k)) + \frac{\delta }{\beta ^k} \\&\le \frac{\kappa }{\beta } {{\widetilde{V}}}_\infty (x(k)) + \min \left\{ \theta (\beta ,1), -\frac{\kappa {\mu }}{2\beta },\; \frac{{\mu }}{2}\right\} . \end{aligned}$$

Now if ${{\widetilde{V}}}_\infty (x(k)) \ge { \mu }/2$, then second term in the minimum defining $\delta$ implies

$$\begin{aligned} {{\widetilde{V}}}_\infty (x(k+1)) - {{\widetilde{V}}}_\infty (x(k)) \le \frac{\kappa }{\beta } \frac{ {\mu } }{2} -\frac{\kappa {\mu }}{2\beta }=0, \end{aligned}$$

implying ${{\widetilde{V}}}_\infty (x(k+1)) \le {{\widetilde{V}}}_\infty (x(k)) <{ \mu }$ and thus $x(k+1)\in {\mathcal {N}}$.

If ${{\widetilde{V}}}_\infty (x(k)) < {\mu }/2$, then the third term in the minimum defining $\delta$ implies

$$\begin{aligned} {{\widetilde{V}}}_\infty (x(k+1)) - {{\widetilde{V}}}_\infty (x(k)) \le \underbrace{\frac{\kappa }{\beta } {{\widetilde{V}}}_\infty (x(k))}_{\le 0} + \frac{{ \mu } }{2}\le \frac{ {\mu } }{2}. \end{aligned}$$

Here we have used that ${{\widetilde{V}}}_\infty (x(k))\ge 0$. This inequality follows since $x(k)\in {\mathcal {N}}$: If the optimal trajectory starting in x(k) stays in ${\mathcal {N}}$, then it follows from strict dissipativity on ${\mathbb {X}}_{\mathcal {N}}\supset {\mathcal {N}}$ and if the optimal trajectory leaves ${\mathcal {N}}$ then it follows from Lemma 5.2. This implies ${{\widetilde{V}}}_\infty (x(k+1)) \le {{\widetilde{V}}}_\infty (x(k)) + {\mu }/2< {\mu }$, i.e., again $x(k+1)\in {\mathcal {N}}$. This proves the induction step and hence $x(k)\in {\mathcal {N}}$ for all $k=0,\ldots ,M$.

Now the turnpike property follows from Theorem 4.3 applied with ${\mathbb {X}}_{inv}={\mathcal {N}}$.$\square$

Remark 6.2

We note that the interval $(\beta _1,\beta _2)$ may be empty. This is because

• (i) the condition (7) needed for proving the turnpike property for trajectories staying near $x_l^\beta$ may require sufficiently large $\beta$ to hold

• (ii) a trajectory starting near $x_l^\beta$ will in general only stay near $x_l^\beta$ for sufficiently small $\beta$

More precisely, the lower bound in (ii) as identified at the end of the proof of Lemma 5.2 depends on the cost ${\tilde{\ell }}$ outside a neighbourhood of $x_l^\beta$ and the cost to leave this neighbourhood. The upper bound in (i), in turn, depends on the cost to reach the equilibrium $x_l^\beta$ from a neighbourhood. If this cost is high and, in addition, the cost to leave the neighbourhood and the cost outside the neighbourhood are low, then the set of discount rates for which a local turnpike behaviour occurs may be empty.

Remark 6.3

The attentive reader may have noted that we apply Lemma 5.4 with $K=1$ in this proof, rather than with $K=M$, which might appear more natural given that we want to make a statement for $\{0,\ldots ,M\}$. This is because the size of the neighbourhood ${\mathcal {B}}_{\varepsilon (\beta ,K)}(x_l^\beta )$ delivered by Lemma 5.4 depends on K. Hence, if we applied Lemma 5.4 with $K=M$ in order to construct the neighbourhood ${\mathcal {N}}$, this neighbourhood may shrink down to $\{x_l^\beta \}$ as M increases. In contrast to this, the fact that ${{\widetilde{V}}}_\infty$ is a (practical) Lyapunov function allows us to construct a neighbourhood ${\mathcal {N}}$ that does not depend on M.

7 Examples

We end our paper with a couple of examples illustrating our theoretical results. All numerical solutions were obtained using a dynamic programming algorithm as described in [6]. We start with two examples exhibiting a locally and a globally optimal equilibrium.

Fig. 2

Stage cost $\ell (x)$

Example 7.1

Consider the dynamics $f(x,u) = x+ u$ and the stage cost $\ell (x,u)=x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2$.

As visualized in Fig. 2, the stage cost $\ell$ has a local minimum in $x = \frac{3-\sqrt{905}}{32}$, a maximum in $x=0$ and a global minimum in $x = \frac{3+\sqrt{905}}{32}$. Following [12, Section 4] we can calculate the storage function $\lambda$ by using the optimality conditions for optimal equilibria. We remark that the procedure for computing global storage functions described in this reference also works for the local dissipativity in case of local convexity which is given in this example, cf. also the discussion after Example 7.2, below. Thus, by a straightforward calculation, we get the local equilibrium $(x^\beta _l,u^\beta _l)= (\frac{3-\sqrt{905}}{32},0)$ and the storage function $\lambda \equiv 0$. Inserting this, we get the rotated stage cost ${\tilde{\ell }}(x,u) = x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2 - \ell (x_l^\beta , 0)$ and local discounted strict (x, u)-dissipativity of the system $f(x,u) = x+ u$ at $x_l^\beta$ for any $\beta \in (0,1)$. Thus, the assumptions of Lemma 5.1 and Lemma 5.2 are fulfilled. Hence, following the proof of Lemma 5.2 we can estimate $\beta _2\approx 0.67$ with $\delta \approx 1$ and ${\tilde{\ell }}_{\min }\approx -0.42$. Further, since $\left\Vert {\tilde{\ell }}(x,u)\right\Vert$ is bounded for x in a neighbourhood ${\mathcal {B}}_\varepsilon (x_0)$, $\varepsilon >0$, Theorem 6.1 can be applied. For illustrating the theoretical results, we set ${\mathbb {U}}=[-0.75, 0.75]$.

On the left hand side of Fig. 3 we show the behaviour of the trajectory x and the control u for different discount factors $\beta$. On the right hand side, we can observe the optimal feedback control values $u_x$ and therefore the domain of attraction of the equilibria dependent on $\beta$. After a maximum of three time instants, the trajectory reaches the global equilibrium for $\beta$ large enough. In contrast, for $\beta \le 0.67$ we can observe that it is more favourable to stay in a neighbourhood of the local equilibrium $x_l^\beta$. We remark that is sufficient to depict $\beta = 0.8$ as a representative for all $\beta \in (0.67, 1)$ since the behaviour of the trajectory, the control and the stage cost does not change significantly. In this example $\beta _1$ can be chosen arbitrarily close to 0, because due to the absence of u-dependent terms in ${\tilde{\ell }}$ it is always cheaper to approach $x^\beta$ than to stay elsewhere in the neighbourhood of $x^\beta$.

Figure 4, for fixed $\beta = 0.7$ we consider different initial values $x_0$. As we can see, the initial value determines to which equilibrium the trajectory converges. This underpins the theoretical results of Theorem 6.1 and especially of Lemma 5.2. We note that for a completely controllable system such a behaviour cannot occur in undiscounted problems.

Fig. 3

Example 7.1 with $x_0 = -0.8$

Fig. 4

Example 7.1 with $\beta = 0.7$ (left) and $\beta = 0.6$ (right) for different start values $x_0$

The following modified example illustrates the case that the interval $(\beta _1, \beta _2)$ is empty.

Example 7.2

Consider again the system $f(x,u)=x+u$, but now with stage cost $\ell (x,u)=x^4-\frac{1}{4} x^3 - \frac{7}{4} x^2+\gamma |u|$ with $\gamma \ne 0$. As the added term has no influence on the conditions of Theorem 6.1 we can again estimate $\beta _2\approx 0.67$. Further, for $\gamma =0$ we get the same stage cost as in Example 7.1 above. In contrast to Example 7.1, now for $\gamma$ large enough we can observe that $(\beta _1, \beta _2)$ is empty. This fact is illustrated in Fig. 5 for $\gamma = 10$. For the numerical results we use the same setting as in example 7.1.

In contrast, in the graph with $\gamma = 10$ we can clearly observe that independent of the discount factor $\beta$ we do not get convergence to the local equilibrium anymore. If we introduce state constraints that restrict the optimal solutions to a neighbourhood of the local equilibrium $x_l^\beta$, then we only get convergence to $x_l^\beta$ for $\beta \approx 1$. More precisely, we have numerically determined the threshold of $\beta _1\approx 0.999$. Without such state constraints, already for $\beta \approx 0.95$ we observe convergence to the optimal equilibrium, which suggests that $\beta _2\le 0.95$ and thus $(\beta _1,\beta _2)$ is empty.

In order to examine this behaviour in more detail we illustrate the behaviour of different values of $\gamma$ for fixed discount factors $\beta$ in Fig. 6. For $\gamma >1$ and $\beta \lesssim 0.95$ we can observe that the trajectories stay near by the start value and do not move away. In contrast, for $\beta \approx 1$ the trajectories converge to the global equilibrium. Thus, we do not get convergence to the local equilibrium anymore.

Fig. 5

Example 7.2 with $\gamma = 10$ for different discount factor $\beta$

Fig. 6

Example 7.2 with $\beta = 0.7$ (left) and $\beta = 0.95$ (right) for different $\gamma$

The two examples, above, have the particular feature that the dynamics is affine and the stage cost $\ell$ is strictly convex in a neighbourhood of the optimal equilibria. In this case, similar arguments as used in the proof of Theorem 4.2 in [12] show that local strict dissipativity always holds. More precisely, we can restrict the proof of Theorem 4.2 in [12] to a bounded neighbourhood ${\mathbb {X}}_{{\mathcal {N}}}\subset {\mathbb {X}}$ of the local equilibrium $x_l^\beta$, e.g., ${\mathcal {B}}_\varepsilon (x_l^\beta )$, $\varepsilon >0$, instead of ${\mathbb {X}}$, and a local strict convex stage cost function $\ell$. Following the proof, $\mathrm {D}{\tilde{\ell }}(x^\beta _l,u^\beta _l)=0$ holds in the neighbourhood ${\mathbb {X}}_{{\mathcal {N}}}$, which by the local strict convexity of ${\tilde{\ell }}$ implies that $(x^\beta _l,u^\beta _l)$ is a strict local minimum. Together with the boundedness of ${\mathbb {X}}_{{\mathcal {N}}}$, this implies the existence of $\alpha _\beta \in {\mathcal {K}}_\infty$ and thus local discounted strict dissipativity. We remark that the calculation of $\lambda$ is the same as in the global case and yields a linear storage function. In the special case of Example 7.1, above, it yields the storage function $\lambda \equiv 0$. In conclusion, local strict dissipativity always holds if the dynamics is affine and the stage cost $\ell$ is strictly convex near the locally optimal equilibrium.

With this observation, our dissipativity based analysis provides a complementary approach to the stable manifold based analysis carried out, e.g., in [13]. Particularly, we can conclude that the model from this reference exhibits two equilibria at which the local turnpike property holds, which explains why the optimal trajectories are correctly reproduced by nonlinear model predictive control as shown in [10, Section 5.1].

Our final example demonstrates that strict convexity of $\ell$ is not needed for obtaining strict dissipativity, thus showing that a dissipativity based analysis allows for strictly weaker assumptions than strict convexity of $\ell$.

Example 7.3

Consider the 1d control system

$$\begin{aligned} x^+ = f(x,u) = 2x+u \end{aligned}$$

with state constraints ${\mathbb {X}}=[-1,1]$, control constraints ${\mathbb {U}}=[-3,3]$, and stage cost

$$\begin{aligned} \ell (x,u) = -x^2/2 + u^2. \end{aligned}$$

Obviously, the stage cost is strictly concave in x and strictly convex in u. Nevertheless, we can establish discounted strict (x, u)-dissipativity in $(x^*,u^*)=(0,0)$ (in this example even global) for $\beta \ge 3/5$ with $\lambda (x) = -x^2$. This follows from the fact that with $a=2\beta /\sqrt{1+\beta }$ and $b=\sqrt{1+\beta }$ we have

$$\begin{aligned} \ell (x,u) + \lambda (x) - \beta \lambda (f(x,u))= & {} -x^2/2 + u^2 - x^2 + \beta (2x+u)^2\\= & {} (4\beta -3/2) x^2 + 4\beta xu + (1+\beta )u^2\\= & {} (a x + b u)^2 + \left( 4\beta -\frac{3}{2} - \frac{4\beta ^2}{1+\beta }\right) x^2\\\ge & {} (a x + b u)^2, \end{aligned}$$

where the last inequality holds since the term in the large brackets is $\ge 0$ for $\beta \ge 3/5$.

Since the system is completely controllable in finite time, hence exponentially stabilizable, Theorem 3.4 in conjunction with Remark 3.5 (ii) implies that for sufficiently large $\beta$ turnpike behaviour occurs at $x^*=0$. This is confirmed for $\beta =0.7$ in the left graph in Fig. 7. In contrast to this, the right graph in Fig. 7 shows that for $\beta =0.6$ the turnpike behaviour for $x^*=0$ does not occur. Rather, the optimal solution converges to the upper bound $x=1$ of the state constraint set. In this example, the numerical computations indicate that $\beta = 3/5=0.6$ is a relatively precise estimate of the threshold for the occurrence of the turnpike property at $x^*=0$, although for $\beta$ decreasing from 0.7 to 0.6 the set of initial values around $x^*=0$ for which the turnpike behaviour can be seen shrinks down rapidly.

Fig. 7

Optimal trajectories for Example 7.3 with $\beta = 0.7$ and $x_0=1$ (left) and with $\beta = 0.59$ and $x_0=0.004$ (right)

8 Conclusion

In this paper we have shown that a local strict dissipativity assumption in conjunction with an appropriate growth condition on the optimal value function can be used in order to conclude a local turnpike property at an optimal equilibrium. The turnpike property holds for discount factors from an interval $[\beta _1,\beta _2]$, where $\beta _1$ is determined by local quantities while $\beta _2$ is also determined by properties of the optimal control problem away from the local equilibrium. Hence, local and global properties together determine whether the interval is not empty. This is in accordance with other approaches for analysing local stability of equilibria in discounted optimal control such as those based on stable and unstable manifolds [13]. In contrast to other approaches, however, the dissipativity based approach is not limited to (locally) strictly convex problems, as our last example showed.