# A Dynamic Programming Approach for Approximate Optimal Control for Cancer Therapy

- First Online:

- Received:
- Accepted:

- 772 Downloads

## Abstract

In the last 15 years, tumor anti-angiogenesis became an active area of research in medicine and also in mathematical biology, and several models of dynamics and optimal controls of angiogenesis have been described. We use the Hamilton–Jacobi approach to study the numerical analysis of approximate optimal solutions to some of those models earlier analysed from the point of necessary optimality conditions in the series of papers by Ledzewicz and Schaettler.

### Keywords

Dynamic programming*ε*-Optimal control problems

*ε*-Value function Hamilton–Jacobi inequality Cancer therapy

## 1 Introduction

The search for therapy approaches that would avoid drug resistance is of tantamount importance in medicine. Two approaches that are currently being pursued in their experimental stages are immunotherapy and anti-angiogenic treatments. Immunotherapy tries to coax the body’s immune system to take action against the cancerous growth. Tumor anti-angiogenesis is a cancer therapy approach that targets the vasculature of a growing tumor. Contrary to traditional chemotherapy, anti-angiogenic treatment does not target the quickly duplicating and genetically unstable cancer cells, but instead the genetically stable normal cells. It was observed in experimental cancer that there is no resistance to the angiogenic inhibitors [1]. For this reason, tumor anti-angiogenesis has been called a therapy “resistant to resistance” that provides a new hope for the treatment of tumor-type cancers [2]. In the last 15 years, tumor anti-angiogenesis became an active area of research not only in medicine [3, 4], but also in mathematical biology [5, 6, 7], and several models of dynamics of angiogenesis have been described, e.g., by Hahnfeldt et al. [5] and d’Onofrio [6, 7]. In a sequence of papers [8, 9, 10, 11], Ledzewicz and Schaettler completely described and solved corresponding to them, optimal control problems from a geometrical optimal control theory point of view. In most of the mentioned papers, the numerical calculations of approximate solutions are presented. However, in none of them are proved assertions that calculated numerical solutions that are really near the optimal one. The aim of this paper is an analysis of the optimal control problem from the Hamilton–Jacobi–Bellman point of view, i.e., using a dynamic programming approach, and to prove that, for calculated numerically solutions the functional considered takes an approximate value with a given accuracy.

## 2 Formulation of the Problem

*T*problem

*u*:[0,

*T*]→[0,

*a*]=

*U*that satisfy a constraint on the total amount of anti-angiogenic inhibitors to be administered

*a*in the definition of the control set

*U*=[0,

*a*] is a maximum dose at which inhibitors can be given. Note that the time

*T*does not correspond to a therapy period. However, instead the functional (1) attempts to balance the effectiveness of the treatment with cost and side effects through the positive weight

*κ*at this integral. The state variables

*p*and

*q*are, respectively, the primary tumor volume and the carrying capacity of the vasculature. Tumor growth is modelled by a Gompertzian growth function with a carrying capacity

*q*, by (3), where

*ξ*denotes a tumor growth parameter. The dynamics for the endothelial support is described by (4), where

*bp*models the stimulation of the endothelial cells by the tumor and the term \(dp^{\frac{2}{3}}q\) models the endogenous inhibition of the tumor. The coefficients

*b*and

*d*are growth constants. The terms

*μq*and

*Guq*describe, respectively, the loss to the carrying capacity through natural causes (death of endothelial cells, etc.), and the loss due to extra outside inhibition. The variable

*u*represents the control in the system and corresponds to the angiogenic dose rate, while

*G*is a constant that represents the anti-angiogenic killing parameter. More details on the descriptions and the discussion of parameters in (1)–(4) can be found in [9, 10]. They analyzed the above problem using first-order necessary conditions for optimality of a control

*u*given by the Pontryagin maximum principle and the second order: the so-called strengthened Legendre–Clebsch condition and geometric methods of optimal control theory.

## 3 Hamilton–Jacobi Approach

*y*=(

*y*

_{1},

*y*

_{2}). The true Hamiltonian is

*u*, the control may not be determined by the minimum condition (detailed discussions are in [8]). Following [8], to exclude discussions about the structure of optimal controls in regions where the model does not represent the underlying biological problem, we confine ourselves to the biologically realistic domain

*H*is defined in

*D*×

*R*

^{2}. We shall call trajectories admissible iff they satisfy (3), (4) (with control

*u*∈[0,

*a*]), and are lying in

*D*. Along any admissible trajectory, defined on [0,

*T*], satisfying necessary conditions—maximum Pontryagin principle [12], we have

*τ*∈[0,1] by

*p*(

*τ*):=

*p*(

*τ*⋅

*T*),

*q*(

*τ*):=

*q*(

*τ*⋅

*T*) and

*u*(

*τ*):=

*u*(

*τ*⋅

*T*) for the state and the control variable with respect to the new time variable

*τ*. The augmented state

*u*∈[0,

*a*]) admissible if they satisfy (5), (6), and (

*p*,

*q*) are lying in

*D*. The transformed problem (\(\tilde{\mathrm{P}}\)) on the fixed interval [0,1] falls into the category of control problems treated in [13]. Thus, we can apply the dynamic programming method developed therein.

*S*is of

*C*

^{1}, then it satisfies the Hamilton–Jacobi–Bellman equation [13]:

*ε*-value function. For any

*ε*>0, we call a function (

*τ*,

*p*,

*q*)→

*S*

_{ε}(

*τ*,

*p*,

*q*), defined in \(\tilde{D}\), an

*ε*-value function iff It is also well known that there exists a Lipschitz continuous

*ε*-value function and that it satisfies the Hamilton–Jacobi inequality: Conversely [14]:

*Each Lipschitz continuous function*

*w*(

*τ*,

*p*,

*q*)

*satisfying*

*is an*

*ε*-

*value function*;

*i.e.*,

*it satisfies*(10)–(11).

In the next section, we describe a numerical construction of a function *w* satisfying (12).

## 4 Numerical Approximation of Value Function

*w*. The essential part of the method is that we start with a quite arbitrary smooth function

*g*and then shift the Hamilton–Jacobi expression with

*g*by the piecewise constant function to get an inequality similar to that as in (12). Thus, let \(\tilde{D}\ni (\tau ,p,q)\rightarrow g(\tau ,p,q)\) be an arbitrary function of class

*C*

^{2}in \(\tilde{D}\), such that

*g*(1,

*p*,

*q*)=

*p*, (

*p*,

*q*)∈

*D*. For a given function

*g*, we define in \(\tilde{D}\times R^{+}\times U (\tau ,\tilde{p},\tilde{q},u)\rightarrow G_{g}(\tau ,\tilde{p},\tilde{q},u)\) as Next, we define the function \((\tau ,\tilde{p},\tilde{q})\rightarrow F_{g}(\tau ,\tilde{p},\tilde{q})\) as

*F*

_{g}is continuous in \(\tilde{D}\) and even Lipschitz continuous in \(\tilde{D}\), and denote its Lipschitz constant by \(M_{F_{g}}\). By the continuity of

*F*

_{g}and compactness of \(\overline{\tilde{D}}\), there exist

*k*

_{d}and

*k*

_{g}such that

*h*

^{η,g}which approximate the function

*F*

_{g}. To this effect, we need some notations and notions.

### 4.1 Definition of Covering of \(\tilde{D}\)

*η*>0 be fixed and \(\{q_{j}^{\eta }\}_{j\in \mathbb{Z}}\) be a sequence of real numbers such that \(q_{j}^{\eta }=j\eta \),

*j*∈ℤ (ℤ—set of integers). Denote

*j*∈

*J*, as follows:

*i*,

*j*∈

*J*,

*i*≠

*j*, \(P_{i}^{\eta ,g}\cap P_{j}^{\eta ,g}=\varnothing \), \(\bigcup_{j\in J}P_{j}^{\eta ,g}=\tilde{D}\) an obvious proposition.

### Proposition 4.1

*For each*\((\tau ,p,q)\in \tilde{D}\), *there exists an**ε*>0, *such that a ball with center* (*τ*,*p*,*q*) *and radius**ε**is either contained only in one set*\(P_{j}^{\eta ,g}\), *j*∈*J*, *or contained in a sum of two sets*\(P_{j_{1}}^{\eta ,g}\), \(P_{j_{2}}^{\eta ,g}\), *j*_{1},*j*_{2}∈*J*. *In the latter case*, |*j*_{1}−*j*_{2}|=1.

### 4.2 Discretization of *F*_{g}

*F*

_{g}with accuracy

*η*.

*h*

^{η,g}along any admissible trajectory by the finite sum of elements with values from the set {−

*η*,0,

*η*} multiplied by 1−

*τ*

_{i}, 0≤

*τ*

_{i}<1. Let \((\tilde{p}(\cdot ),\tilde{q}(\cdot ),u(\cdot ))\) be any admissible trajectory starting at the point \((0,p_{0},q_{0})\in \tilde{D}\). We show that there exists an increasing sequence of

*m*points {

*τ*

_{i}}

_{i=1,…,m},

*τ*

_{1}=0,

*τ*

_{m}=1, such that, for

*τ*∈[

*τ*

_{i},

*τ*

_{i+1}],

*F*

_{g}. From (17), we infer that, for each

*i*∈{1,…,

*m*−1}, if \((\tau_{i},\tilde{p}(\tau_{i}),\tilde{q}(\tau_{i}))\in P_{j}^{\eta ,g}\) for a certain

*j*∈

*J*, then we have, for

*τ*∈[

*τ*

_{i},

*τ*

_{i+1}),

*τ*∈[

*τ*

_{i},

*τ*

_{i+1}] along the trajectory \(\tilde{p}(\cdot )\), \(\tilde{q}(\cdot )\),

*i*∈{2,…,

*m*−1},

*η*or 0 or

*η*. Integrating (18), we obtain, for each

*i*∈{1,…,

*m*−1}, and in consequence, Now, we will present the expression

*h*

^{η,g}(⋅,⋅,⋅) along any trajectory \(\tilde{p}(\cdot ),\tilde{q}(\cdot )\) as a sum of finite number of values, where each value consists of a number from the set {−

*η*,0,

*η*} multiplied by

*τ*

_{m}−

*τ*

_{i}.

*B*of all trajectories \(\tilde{p}(\cdot ),\tilde{q}(\cdot )\) we can introduce an equivalence relation

**r**: We say that two trajectories \(( \tilde{p}_{1}(\cdot ),\tilde{q}_{1}(\cdot ) ) \) and \(( \tilde{p}_{2}(\cdot ),\tilde{q}_{2}(\cdot ) ) \) are equivalent iff they satisfy (22) and (23). We denote the set of all disjoint equivalence classes by

*B*

_{r}. The cardinality of

*B*

_{r}, denoted by ∥

*B*

_{r}∥, is finite and bounded from above by 3

^{m+1}.

*X*is finite.

### Remark 4.1

One can wonder whether the reduction of an infinite number of admissible trajectories to finite one makes computational sense, especially if the finite number may mean 3^{m+1}. In the theorems below, we can always take infimum and supremum over all admissible trajectories and the assertions will be still true. However, from a computational point of view, they are examples in which it is more easily and effectively to calculate infimum over finite sets.

The considerations above allow us to estimate the approximation of the value function.

### Theorem 4.1

*Let*\(( \tilde{p}(\cdot ),\tilde{q}(\cdot) ) \)

*be any admissible pair such that*

*Then we have the following estimate*:

### Proof

*τ*

_{1},

*τ*

_{m}], we get Thus, by a well-known theorem [16], Hence, we get two inequalities: and Both inequalities imply that As a consequence of the above, we get and thus the assertion of the theorem follows. □

**r**, we have

### Theorem 4.2

*Let*

*η*>0

*be given*.

*Assume that there is*

*θ*>0,

*such that*

*Then*

*is*

*εoptimal value at*(

*τ*

_{1},

*p*

_{0},

*q*

_{0})

*for*

*ε*=2

*η*+

*θ*.

### Proof

### Example 4.1

Let the total amount of anti-angiogenic inhibitors *A*=300 mg (2). Take for initial values of tumor *p*_{0}=15502 and vasculature *q*_{0}=15500, *κ*=1. With *u*=10 (we assume maximum dose at which inhibitors can be given *a*=75 mg), taking an approximate solution of (3), (4), we jump to singular arc [8], and then follow it (in discrete way) until all inhibitors available are being used up, then final *p*=9283.647 and *q*=5573.07. We construct a function *g* numerically such that *g*(0,*p*_{0},*q*_{0})=9283.647 and assumption (25) is satisfied with *θ*=0, *η*=0.1 and \(h^{\eta ,g}(\tau_{1}, \tilde{p}(\tau_{1}),\tilde{q}(\tau_{1}))=0\).

## 5 Conclusions

The paper treats the free final-time optimal control problem in cancer therapy through the *ε*-dynamic programming approach stating that every Lipschitz solution of the Hamilton–Jacobi inequality is an *ε*-optimal value of the cost of the treatment. Then a numerical construction of the *ε*-optimal value is presented. As a final result, a computational formula for the *ε*-optimal value is given.

## Notes

### Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

### References

- 1.Boehm, T., Folkman, J., Browder, T., O’Reilly, M.S.: Antiangiogenic therapy of experimental cancer does not induce acquired drug resistance. Nature
**390**, 404–407 (1997) CrossRefGoogle Scholar - 2.Kerbel, R.S.: A cancer therapy resistant to resistance. Nature
**390**, 335–336 (1997) CrossRefGoogle Scholar - 3.Kerbel, R.S.: Tumor angiogenesis: past, present and near future. Carcinogenesis
**21**, 505–515 (2000) CrossRefGoogle Scholar - 4.Klagsburn, M., Soker, S.: VEGF/VPF: the angiogenesis factor found? Curr. Biol.
**3**, 699–702 (1993) CrossRefGoogle Scholar - 5.Hahnfeldt, P., Panigrahy, D., Folkman, J., Hlatky, L.: Tumor development under angiogenic signaling: a dynamical theory of tumor growth, treatment response, and postvascular dormancy. Cancer Res.
**59**, 4770–4775 (1999) Google Scholar - 6.D’Onofrio, A.: Rapidly acting antitumoral anti-angiogenic therapies. Phys. Rev. E, Stat. Nonlinear Soft Matter Phys.
**76**(3), 031920 (2007) MathSciNetCrossRefGoogle Scholar - 7.D’Onofrio, A., Gandolfi, A.: Tumour eradication by antiangiogenic therapy: analysis and extensions of the model by Hahnfeldt et al. (1999). Math. Biosci.
**191**, 159–184 (2004) MathSciNetMATHCrossRefGoogle Scholar - 8.Ledzewicz, U., Schättler, H.: Optimal bang-bang controls for a 2-compartment model in cancer chemotherapy. J. Optim. Theory Appl.
**114**, 609–637 (2002) MathSciNetMATHCrossRefGoogle Scholar - 9.Ledzewicz, U., Schättler, H.: Anti-angiogenic therapy in cancer treatment as an optimal control problem. SIAM J. Control Optim.
**46**(3), 1052–1079 (2007) MathSciNetMATHCrossRefGoogle Scholar - 10.Ledzewicz, U., Schättler, H.: Optimal and suboptimal protocols for a class of mathematical models of tumor anti-angiogenesis. J. Theor. Biol.
**252**, 295–312 (2008) CrossRefGoogle Scholar - 11.Ledzewicz, U., Oussa, V., Schättler, H.: Optimal solutions for a model of tumor anti-angiogenesis with a penalty on the cost of treatment. Appl. Math.
**36**(3), 295–312 (2009) MathSciNetMATHGoogle Scholar - 12.Maurer, H., Oberle, J.: Second order sufficient conditions for optimal control problems with free final time: the Riccati approach. SIAM J. Control Optim.
**41**(2), 380–403 (2002) MathSciNetMATHCrossRefGoogle Scholar - 13.Fleming, W.H., Rishel, R.W.: Deterministic and Stochastic Optimal Control. Springer, New York (1975) MATHCrossRefGoogle Scholar
- 14.Nowakowski, A.:
*ε*-Value function and dynamic programming. J. Optim. Theory Appl.**138**(1), 85–93 (2008) MathSciNetMATHCrossRefGoogle Scholar - 15.Pustelnik, J.: Approximation of optimal value for Bolza problem. Ph.D. Thesis (2009). (in Polish) Google Scholar
- 16.Castaing, C., Valadier, M.: Convex Analysis and Measurable Multifunctions. Lecture Notes in Math., vol. 580. Springer, New York (1977) MATHGoogle Scholar