# Impulse Control of Standard Brownian Motion: Discounted Criterion

- 950 Downloads

## Abstract

This paper examines the impulse control of a standard Brownian motion under a discounted criterion. In contrast with the dynamic programming approach, this paper first imbeds the stochastic control problem into an infinite-dimensional linear program over a space of measures and derives a simpler nonlinear optimization problem that has a familiar interpretation. Optimal solutions are obtained for initial positions in a restricted range. Duality theory in linear programming is then used to establish optimality for arbitrary initial positions.

## Keywords

Impulse control Discounted criterion Infinite dimensional linear programming Expected occupation measures## 1 Introduction

When one seeks to control a stochastic process and every intervention incurs a strictly positive cost, one must select a sequence of separate intervention times and amounts. The resulting stochastic problem is therefore an impulse control problem in which the decision maker seeks to either maximize a reward or minimize a cost. This paper continues the examination of the impulse control of Brownian motion. It considers a discounted cost criterion while a companion paper [5] studies the long-term average criterion. The aim of the paper is to illustrate a solution approach which first imbeds the stochastic control problem into an infinite-dimensional linear program over a space of measures and then reduces the linear program to a simpler nonlinear optimization. Contrasting with the long-term average paper, the dependence of the value function on the initial position of the process requires the use of duality in linear programming to obtain a complete solution.

Impulse control problems have been extensively studied using a quasi-variational approach; now classical works include [1, 3] while the recent paper [2] examines a Brownian inventory model. This paper extends a linear programming approach used on optimal stopping problems [4]. See [5] for additional references.

*admissible*controls by \(\mathcal{A}\).

We make five important observations about impulse policies. Firstly, “\(0\)-impulses” which do not change the state only increase the cost so can be excluded from consideration. Secondly, the symmetry of the dynamics and costs means that any impulse \((\tau _k,Y_k)\) which would cause sgn\((X(\tau _k)) = - {\mathrm sgn}(X(\tau _k-))\) on a set of positive probability will have no greater cost (smaller cost when \(k_2 >0\)) by replacing the impulse with one for which \(\tilde{X}(\tau _k) = \mathrm {sgn}(X(\tau _k-))|X(\tau _k)|\). Thus we can also restrict analysis to those policies for which all impulses keep the process on the same side of \(0\). Next, any policy \((\tau ,Y)\) with \(\lim _{k\rightarrow \infty } \tau _k =:\tau _\infty < \infty \) on a set of positive probability will have infinite cost so for every admissible policy \(\tau _k \rightarrow \infty \)\(a.s.\) as \(k\rightarrow \infty \). Next let \((\tau ,Y)\) be a policy for which there is some \(k\) such that \(\tau _k = \tau _{k+1}\) on a set of positive probability. Again due to the presence of the fixed intervention cost \(k_1\), the total cost up to time \(\tau _{k+1}\) will be at least \(k_1 \mathbb {E}[e^{-\alpha \tau _k} I(\tau _k = \tau _{k+1})]\) smaller by combining these interventions into a single intervention on this set. Hence we may restrict policies to those for which \(\tau _k < \tau _{k+1}\)\(a.s.\) for each \(k\).

The final observation is similar. Suppose \((\tau ,Y)\) is a policy such that on a set \(G\) of positive probability \(\tau _k < \infty \) and \(|X(\tau _k)| > |X(\tau _k-)|\) for some \(k\). Consider a modification of this impulse policy and resulting process \(\tilde{X}\) which simply fails to implement this impulse on \(G\). Define the stopping time \(\sigma = \inf \{t >\tau _k: |X(t)| \le |\tilde{X}(t)|\}\). Notice that the running costs accrued by \(\tilde{X}\) over \([\tau _k,\sigma )\) are smaller than those accrued by \(X\). Finally, at time \(\sigma \), introduce an intervention on the set \(G\) which moves the \(\tilde{X}\) process so that \(\tilde{X}(\sigma ) = X(\sigma )\). This intervention will incur a cost which is smaller than the cost for the process \(X\) at time \(\tau _k\). As a result, we may restrict the impulse control policies to those for which every impulse decreases the distance of the process from the origin.

## 2 Restricted Problem and Measure Formulation

The solution of the impulse control problem is obtained by first considering a subclass of the admissible impulse control pairs.

### **Condition 1**

Let \(\mathcal{A}_1 \subset \mathcal{A}\) be those policies \((\tau ,Y)\) such that the resulting process \(X\) is bounded; that is, for \((\tau ,Y) \in \mathcal{A}_1\), there exists some \(M < \infty \) such that \(|X(t)| \le M\) for all \(t \ge 0\).

Note that for each \(M > 0\), any impulse control which has the process jump closer to \(0\) whenever \(|X(t-)|=M\) is in the class \(\mathcal{A}_1\) so this collection is non-empty. The bound is not required to be uniform for all \((\tau ,Y) \in \mathcal{A}_1\). The restricted impulse control problem is one of minimizing \(J(\tau ,Y;x_0)\) over all policies \((\tau ,Y) \in \mathcal{A}_1\).

### **Proposition 2**

\(V_{aux}(x_0) \le V_{lp}(x_0) \le V_1(x_0)\).

### **Remark 3**

Our analysis will also involve other auxiliary linear programs as well. One will replace the single constraint in (8) with the pair of constraints (5) while another will limit the constraints in (4) to a single function. Each auxiliary program will provide a lower bound on \(V_{lp}(x_0)\) and hence on \(V_1(x_0)\).

### 2.1 Nonlinear Optimization and Partial Solution

### **Remark 4**

*not*restricted to these policies.

### **Proposition 5**

### *Proof*

Now that the lower bound given in (9) is determined, it is important to connect an optimizing \(\mu _1^*\) with an admissible impulse control policy \((\tau ,Y) \in \mathcal{A}_1\). The existence of two minimizing pairs \((y_*,z_*)\) and \((-y_*,-z_*)\) allows many auxiliary-LP-feasible measures \(\mu _1\) to place point masses at these two points and still achieve the lower bound. This observation leads to a solution to the restricted stochastic impulse control problem.

### **Theorem 6**

### *Proof*

### 2.2 Full Solution

### **Lemma 7**

\(\widehat{V} \in C^1(\mathbb {R}) \cap C^2(\mathbb {R}\backslash \{\pm y_*\}).\)

### **Theorem 8**

The optimal value of (13) is \(\widehat{V}(x_0)\) which is achieved when \(w_* = 1\).

### *Proof*

By symmetry, it is sufficient to examine \(x,y,z \ge 0\). Notice that for \(0 \le x < y_*\), \(\alpha \widehat{V}(x) - A\widehat{V}(x) = x^2 = c_0(x) \ge 0\) and hence the dual variable \(w\) cannot exceed \(1\). The question is whether \(w=1\) is feasible for (13) so examine the rest of the constraints with \(w=1\).

We now have the following result.

### **Theorem 9**

### *Proof*

The particular choice of \((\tau ^*,Y^*)\) implies \(\widehat{V}(x_0) \le V_{lp}(x_0) \le V_1(x_0) \le J(\tau ^*,Y^*) = \widehat{V}(x_0)\).\(~\square \)

### 2.3 Solution for General Admissible Impulse Controls

The solution of Sect. 2.2 is restricted to those impulse control policies under which the process \(X\) remains bounded. It is necessary to show that no lower cost can be obtained by any policy which allows the process to be unbounded.

### **Theorem 10**

The impulse control policy \((\tau ^*,Y^*)\) of Theorem 9 is optimal in the class of all admissible policies and \(\widehat{V}(x_0)\) is the optimal value.

### *Proof*

This argument establishes that \(\widehat{V}(x_0)\) is a lower bound on \(J(\tau ,Y;x_0)\) for every admissible impulse control policy. Theorem 9 then gives the existence of an optimal policy whose cost equals the lower bound.

## References

- 1.Bensoussan, A., Lions, J.-L.: Impulse Control and Quasi-Variational Inequalities. Bordas Editions. Gauthier-Villars, Paris (1984)Google Scholar
- 2.Dai, J.G., Yao, D.: Optimal control of Brownian inventory models with convex inventory cost, Part 2: Discount-optimal controls. Stoch. Syst.
**3**(2), 500–573 (2013)CrossRefzbMATHMathSciNetGoogle Scholar - 3.Harrison, J.M., Sellke, T.M., Taylor, A.J.: Impulse control of Brownian motion. Math. Oper. Res.
**8**(3), 454–466 (1983)MathSciNetCrossRefGoogle Scholar - 4.Helmes, K., Stockbridge, R.H.: Construction of the value function and optimal rules in optimal stopping of one-dimensional diffusions. Adv. Appl. Probab.
**42**(1), 158–182 (2010)MathSciNetCrossRefGoogle Scholar - 5.Helmes, K., Stockbridge, R.H., Zhu, C.: Impulse control of standard Brownian motion: long-term average criterion. In: Helmes, K., Stockbridge, R.H., Zhu, C. (eds.) System Modeling and Optimization, vol. 443, pp. 148–157 (2014)Google Scholar