# Stochastic Linear-Quadratic Control

**DOI:**https://doi.org/10.1007/978-1-4471-5102-9_228-1

## Abstract

In this short article, we briefly review some major historical studies and recent progress on continuous-time stochastic linear-quadratic (SLQ) control and related mean-variance (MV) hedging.

## Keywords

Riccati equation Quadratic backward stochastic differential equations BMO-martingale Bellman’s quasilinearization Monotone convergence Mean-variance hedging## Introduction

A stochastic linear-quadratic (SLQ) control problem is the optimal control of a linear stochastic dynamic equation subject to an expected quadratic cost functional of the system state and control. As shown in Athans (1971), it is a typical case of optimal stochastic control both in theory and application. Due to the linearity of the system dynamics and the quadratic feature of the cost functions, the optimal control law is usually synthesized into a feedback (also called closed) form of the optimal state, and the corresponding proportional coefficients are specified by the associated Riccati equation. In what follows, we restrict our exposition within the continuous-time SLQ problem, and further, mainly for the finite-horizon case.

The initial study on the continuous-time SLQ problem seems to be due to Florentin (1961). However, his linear stochastic control system is assumed to be Gaussian. That is, the system noise is additive and has neither multiplication with the state nor with the control. Such a case is usually termed as the linear-quadratic Gaussian (LQG) problem, and in the case of complete observation, the optimal feedback law remains to be invariant when the white noise vanishes. The continuous-time partially observable case was first discussed by Potter (1964) and a more general formulation was later given by Wonham (1968a). It is proved that the optimal control can be obtained by the following two separate steps: (1) generate the conditional mean estimate of the current state using a Kalman filter and (2) optimally feed back as if the conditional mean state estimate was the true state of the system. This result is referred to as the certainty equivalence principle or the strict separation theorem. Different assumptions were discussed by Tse (1971) for the separation of control and state estimation.

*d*-dimensional Brownian motion

*W*= (

*W*

^{1},

*W*

^{2}, ⋯ ,

*W*

^{ d }):

*T*> 0,

*X*

_{ t }∈

*R*

^{ n }is the state at time

*t*, and

*u*

_{ t }∈

*R*

^{ m }is the control at time

*t*. Assume that all the coefficients

*A*,

*B*;

*C*

^{ i },

*D*

^{ i },

*i*= 1, 2,

*…*,

*d*;

*Q*,

*N*are piecewisely continuous matrix-valued (of suitable dimensions) functions of time, and

*M*,

*Q*

_{ t }are nonnegative matrices and

*N*

_{ t }is uniformly positive. Wonham (1970) gave the following Riccati equation:

*d*, and the function \(\Gamma \) is defined by

*t*∈ [0,

*T*] and any \(K \in \mathcal{S}_{+}^{n} :=\{ \mbox{ all nonnegative $n \times n$ matrices}\}\). This Riccati equation is a nonlinear ordinary differential equation (ODE). Since the nonlinear term \(\Gamma _{t}(K)(N_{t} + D_{t}^{i{\ast}}KD_{t}^{i})\Gamma _{t}(K)\) in the right-hand side is not uniformly Lipschitz in

*K*in general, the standard existence and uniqueness theorem of ODEs does not directly tell whether this Riccati equation has a unique continuous solution in \(\mathcal{S}_{+}^{n}\). To solve this issue, Wonham (1970) used Bellman’s principle of quasilinearization and constructed the following sequence of successive linear approximating matrix-valued ODEs.

*K*

^{0}≡

*M*and for

*l*= 1, 2,

*…*,

*F*

_{ t }(

*K*, ⋅) at \(\Gamma _{t}(K)\), Wonham showed that the unique nonnegative solution

*K*

^{ l }of ODE (5) is monotonically decreasing in the sequential number

*l*= 1, 2,

*…*. Using the method of monotone convergence, the sequence of solutions {

*K*

^{ l }} is shown to converge to some \(K \in \mathcal{S}_{+}^{n}\), which turns out to solve Riccati equation (1).

## The Case of Random Coefficients and Backward Stochastic Riccati Equation

*W*. When the coefficients \(A,B;C^{i},D^{i},i = 1,2,\ldots ,d;Q,N\) and

*M*may be random, with \(A,B;C^{i},D^{i},i = 1,2,\ldots ,d;Q,N\) being \(\mathcal{F}_{t}\)-adapted and essentially bounded and

*M*being \(\mathcal{F}_{T}\)-measurable and essentially bounded, Bismut (1976, 1978) used the stochastic maximum principle for optimal control and derived the following Riccati equation:

*t*∈ [0,

*T*] is defined as follows:

*L*. However, Bismut used the fixed-point method, and Peng (1992b) used Bellman’s principle of quasilinearization and the method of monotone convergence. Neither methodology works for the general case of quadratic growth in the second unknown variable

*L*in the drift of the stochastic equation. Bismut (1976, 1978) and Peng (1999) stated the general case as an open problem. By considering the stochastic equation for the inverse of

*K*

_{ t }, Kohlmann and Tang (2003a) solved some particular cases where the function \(\Psi _{t}(K,L)\) can depend on

*L*. Tang (2003) finally solved the general case, using the method of stochastic flows.

In the general case, the optimal feedback coefficient \(\Psi _{t}(K_{t},L_{t})\) at time *t* depends on *L* _{ t } in a linear manner, which is in general not essentially bounded with respect to (*t*, *ω*). Kohlmann and Tang (2003b) observed that the stochastic integral process *∫* _{0} ^{ ⋅} *L* _{ t } ^{ i } *dW* _{ t } ^{ i } is a BMO-martingale.

## Indefinite SLQ Problem

Chen (1985) contains a theory of singular (the control weighting matrix vanishing in the quadratic cost functional) LQG control, which is a particular type of indefinite SLQ problems. In the deterministic linear-quadratic (LQ) control theory, the well posedness (i.e., the value function is finite on [0, *T*] ×*R* ^{ n }) of the problem suggests that the control weighting matrix *N* in the quadratic cost functional be positive definite. In the stochastic case, when *N* _{ t } is slightly negative, the SLQ may still be well posed if the control could also increase the intensity of the system noise. Peng (1992a) used an indefinite but well-posed SLQ problem to illustrate his new second-order stochastic maximum principle. Chen et al. (1998) gave a deeper study on this feature of the SLQ problem. Yong and Zhou (1999) gave a systematic account of the progress around in the indefinite SLQ problem.

## Mean-Variance Hedging

In the theory of finance, Duffie and Richardson (1991) introduced the SLQ control model to hedge a contingent claim in an incomplete market. Schweizer (1992) developed a first framework for MV hedging, and then it was extended to a very general setting in Gouriéroux et al. (1998). Before 2000, the martingale method was used to solve the MV hedging problem. Kohlmann and Zhou (2000) began to use the standard SLQ theory to derive the optimal hedging strategy for a general contingent claim in a financial market of deterministic coefficients, and such a SLQ methodology was subsequently extended to very general settings for financial markets by Kohlmann and Tang (2002, 2003b), Bobrovnytska and Schweizer (2004), and Jeanblanc et al. (2012). See more detailed surveys on the literature by Pham (2000), Schweizer (2010), and Jeanblanc et al. (2012).

## Summary and Future Directions

In comparison to the continuous-time deterministic LQ theory, the continuous-time SLQ theory has the following two striking features: An indefinite SLQ problem may be well posed, and the optimal feedback coefficient may be unbounded due to its linear dependence on the martingale part *L* of the stochastic solution of the Riccati equation. Due to the second feature, the convergence of the sequence of successive approximations constructed via Bellman’s quasilinearization still remains to be solved in the general case. This problem partially motivates Delbaen and Tang (2010) to study the regularity of unbounded stochastic differential equations and also may help to explain the necessity of rich studies on mean-variance hedging and closedness of stochastic integrals with respect to semi-martingales (as in Delbaen et al. 1994, 1997) in various general settings.

## Cross-References

## Recommended Reading

The theory of SLQ control in various contexts is available in textbooks, monographs, or papers. Anderson and Moore (1971, 1989), Bensoussan (1992), and Chen (1985) include good accounts of the LQG control theory. Wonham (1970) includes a full introduction to the SLQ problem with deterministic piecewise continuous-time coefficients. Bismut (1978) gives a systematic and readable French introduction to SLQ problem with random coefficients. Yong and Zhou (1999) include an extensive discussion on the well-posed indefinite SLQ problem. Tang (2003) gives a complete solution of a general backward stochastic Riccati equation.

## Bibliography

- Anderson BDO, Moore JB (1971) Linear optimal control. Prentice-Hall, Englewood CliffszbMATHGoogle Scholar
- Anderson BDO, Moore JB (1989) Optimal control: linear quadratic methods. Prentice-Hall, Englewood CliffsGoogle Scholar
- Athans M (1971) The role and use of the stochastic linear-quadratic-Gaussian problem in control system design. IEEE Trans Autom Control AC-16(6):529–552CrossRefMathSciNetGoogle Scholar
- Bensoussan A (1992) Stochastic control of partially observable systems. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
- Bismut JM (1976) Linear quadratic optimal stochastic control with random coefficients. SIAM J Control Optim 14:419–444CrossRefzbMATHMathSciNetGoogle Scholar
- Bismut JM (1978) Contrôle des systems linéaires quadratiques: applications de l’intégrale stochastique. In: Dellacherie C, Meyer PA, Weil M (eds) Séminaire de probabilités XII. Lecture Notes in Math 649. Springer, Berlin, pp 180–264Google Scholar
- Bobrovnytska O, Schweizer M (2004) Mean-variance hedging and stochastic control: beyond the Brownian setting. IEEE Trans Autom Control 49:396–408CrossRefMathSciNetGoogle Scholar
- Chen H (1985) Recursive estimation and control for stochastic systems. Wiley, New York, pp 302–335Google Scholar
- Chen S, Li X, Zhou X (1998) Stochastic linear quadratic regulators with indefinite control weight costs. SIAM J Control Optim 36:1685–1702CrossRefzbMATHMathSciNetGoogle Scholar
- Delbaen F, Tang S (2010) Harmonic analysis of stochastic equations and backward stochastic differential equations. Probab Theory Relat Fields 146:291–336CrossRefMathSciNetGoogle Scholar
- Delbaen F et al (1994) Weighted norm inequalities and closedness of a space of stochastic integrals. C R Acad Sci Paris Sér I Math 319:1079–1081zbMATHMathSciNetGoogle Scholar
- Delbaen F et al (1997) Weighted norm inequalities and hedging in incomplete markets. Financ Stoch 1:181–227CrossRefzbMATHGoogle Scholar
- Duffie D, Richardson HR (1991) Mean-variance hedging in continuous time. Ann Appl Probab 1:1–15CrossRefzbMATHMathSciNetGoogle Scholar
- Florentin JJ (1961) Optimal control of continuous-time, Markov, stochastic systems. J Electron Control 10:473–488MathSciNetGoogle Scholar
- Gouriéroux C, Laurent JP, Pham H (1998) Mean-variance hedging and numéraire. Math Financ 8:179–200CrossRefzbMATHGoogle Scholar
- Jeanblanc M et al (2012) Mean-variance hedging via stochastic control and BSDEs for general semimartingales. Ann Appl Probab 22:2388–2428CrossRefzbMATHMathSciNetGoogle Scholar
- Kohlmann M, Tang S (2002) Global adapted solution of one-dimensional backward stochastic Riccati equations, with application to the mean-variance hedging. Stoch Process Appl 97:255–288CrossRefzbMATHMathSciNetGoogle Scholar
- Kohlmann M, Tang S (2003a) Multidimensional backward stochastic Riccati equations and applications. SIAM J Control Optim 41:1696–1721CrossRefzbMATHMathSciNetGoogle Scholar
- Kohlmann M, Tang S (2003b) Minimization of risk and linear quadratic optimal control theory. SIAM J Control Optim 42:1118–1142CrossRefzbMATHMathSciNetGoogle Scholar
- Kohlmann M, Zhou XY (2000) Relationship between backward stochastic differential equations and stochastic controls: a linear-quadratic approach. SIAM J Control Optim 38:1392–1407CrossRefzbMATHMathSciNetGoogle Scholar
- Peng S (1992a) New developments in stochastic maximum principle and related backward stochastic differential equations. In: Proceedings of the 31st conference on decision and control, Tucson, Dec 1992. IEEE, pp 2043–2047Google Scholar
- Peng S (1992b) Stochastic Hamilton-Jacobi-Bellman equations. SIAM J Control Optim 30:284–304CrossRefzbMATHMathSciNetGoogle Scholar
- Peng S (1999) Open problems on backward stochastic differential equations. In: Chen S, Li X, Yong J, Zhou XY (eds) Control of distributed parameter and stochastic systems, IFIP, Hangzhou. Kluwer, pp 267–272Google Scholar
- Pham H (2000) On quadratic hedging in continuous time. Math Methods Oper Res 51:315–339CrossRefzbMATHMathSciNetGoogle Scholar
- Potter JE (1964) A guidance-navigation separation theorem. Experimental Astronomy Laboratory, Massachusetts Institute of Technology, Cambridge, Rep. RE-11, 1964Google Scholar
- Schweizer M (1992) Mean-variance hedging for general claims. Ann Appl Probab 2:171–179CrossRefzbMATHMathSciNetGoogle Scholar
- Schweizer M (2010) Mean-variance hedging. In: Cont R (ed) Encyclopedia of quantitative finance. Wiley, New York, pp 1177–1181Google Scholar
- Tang S (2003) General linear quadratic optimal stochastic control problems with random coefficients: linear stochastic Hamilton systems and backward stochastic Riccati equations. SIAM J Control Optim 42:53–75CrossRefzbMATHMathSciNetGoogle Scholar
- Tse E (1971) On the optimal control of stochastic linear systems. IEEE Trans Autom Control AC-16(6):776–785Google Scholar
- Wonham WM (1967) Optimal stationary control of a linear system with state-dependent noise. SIAM J Control 5:486–500CrossRefzbMATHMathSciNetGoogle Scholar
- Wonham WM (1968a) On the separation theorem of stochastic control. SIAM J Control 6:312–326CrossRefzbMATHMathSciNetGoogle Scholar
- Wonham WM (1968b) On a matrix Riccati equation of stochastic control. SIAM J Control 6:681–697. Erratum (1969); SIAM J Control 7:365Google Scholar
- Wonham WM (1970) Random differential equations in control theory. In: Bharucha-Reid AT (ed) Probabilistic methods in applied mathematics. Academic, New York, pp 131–212Google Scholar
- Yong JM, Zhou XY (1999) Stochastic controls: Hamiltonian systems and HJB equations. Springer, New YorkCrossRefzbMATHGoogle Scholar