Optimal control in linear-quadratic stochastic advertising models with memory

This paper deals with a class of optimal control problems which arises in advertising models with Volterra Ornstein-Uhlenbeck process representing the product goodwill. Such choice of the model can be regarded as a stochastic modiﬁcation of the classical Nerlove-Arrow model that allows to incorporate both presence of uncertainty and empirically observed memory eﬀects such as carryover or distributed forgetting. We present an approach to solve such optimal control problems based on an inﬁnite dimensional lift which allows us to recover Markov properties by formulating an optimization problem equivalent to the original one in a Hilbert space. Such technique, however, requires the Volterra kernel from the forward equation to have a representation of a particular form that may be challenging to obtain in practice. We overcome this issue for H¨older continuous kernels by approximating them with Bernstein polynomials, which turn out to enjoy a simple representation of the required type. Then we solve the optimal control problem for the forward process with approximated kernel instead of the original one and study convergence. The approach is illustrated with simulations.


Introduction
The problem of optimizing advertising strategies has always been of paramount importance in the field of marketing.Starting from the pioneering works of Vidale and Wolfe [23] and Nerlove and Arrow [18], this topic has evolved into a full-fledged field of research and modeling.Realizing the impossibility of describing all existing classical approaches and results, we refer the reader to the review article of Sethi [21] (that analyzes the literature prior to 1975) and a more recent paper by Feichtinger, Hartl and Sethi [11] (covering the results up to 1994) and references therein.
It is worth noting that the Nerlove-Arrow approach, which was the foundation for numerous modern dynamic advertising models, assumed no time lag between spending on advertising and the impact of the latter on the goodwill stock.However, many empirical studies (see, for example, [15]) clearly indicate some kind of a "memory" phenomenon that is often called the "distributed lag" or "carryover" effect: the influence of advertising does not have an immediate impact but is rather spread over a period of time varying from several weeks to several months.This shortcoming of the basic Nerlove-Arrow model gave rise to many modifications of the latter aimed at modeling distributed lags.For a long time, nevertheless, the vast majority of dynamic advertising models with distributed lags had been formulated in a deterministic framework (see e.g.[21, §2.6] and [11,Section 2.3]).
In recent years, however, there have been several landmark papers that consider the Nerlove-Arrow-type model with memory in a stochastic setting.Here, we refer primarily to the series of papers [13,14] (see also a more recent work [16]), where goodwill stock is modeled via Brownian linear diffusion with delay of the form dX u (t) = α 0 X u (t) + 0 −r α 1 (s)X u (t + s)ds + β 0 u(t) + 0 −r β 1 (s)u(t + s)ds dt + σdW (t), (1.1) where X u is interpreted as the product's goodwill stock and u is the spending on advertising.The corresponding optimal control problem in this case was solved using the so-called lift approach: equation (1.1) was rewritten as a stochastic differential equation (without delay) in a suitable Hilbert space, and then infinite-dimensional optimization techniques (either dynamic programming principle or maximum principle) were applied.
In this article, we present an alternative stochastic model that also takes the carryover effect into account.Instead of the delay approach described above, we incorporate the memory into the model by means of the Volterra kernel K ∈ L 2 ([0, T ]) and consider the controlled Volterra Ornstein-Uhlenbeck process of the form where α, β, σ > 0 and X(0) ∈ R are constants (see e.g.[1, Section 5] for more details on affine Volterra processes of such type).Note that such goodwill dynamics can be regarded as the combination of deterministic lag models described in [11,Section 2.3] and the stochastic Ornstein-Uhlenbeck-based model presented by Rao [19].The main difference from (1.1) is the memory incorporated to the noise along with the drift as the stochastic environment (represented by the noise) tends to form "clusters" with time.Indeed, in reality positive increments are likely to be followed by positive increments (if conditions are favourable for the goodwill during some period of time) and negative increments tend to follow negative increments (under negative conditions).This behaviour of the noise cannot be reflected by a standard Brownian driver but can easily be incorporated into the model (1.2).Our goal is to solve an optimization problem of the form where a 1 , a 2 > 0 are given constants.The set of admissible controls for the problem (1.3), denoted by , is the space of square integrable real-valued stochastic processes adapted to the filtration generated by W .Note that the process X u is well defined for any u ∈ L 2 a since, for almost all ω ∈ Ω, the equation (1.2) treated pathwisely can be considered as a deterministic linear Volterra integral equation of the second kind that has a unique solution (see e.g.[22]).
The optimization problem (1.3) for underlying Volterra dynamics has been studied by several authors (see, e.g.[3,24] and the bibliography therein).Contrarily to most of the work in our bibliography, we will not solve such problem by means of a maximum principle approach.Even though this method allows to find necessary and sufficient conditions to obtain the optimal control to (1.3), we cannot directly apply it as we deal with low regularity conditions on the coefficients of our drift and volatility.Furthermore, such method has another notable drawback in the practice.In fact, its application is often associated with computations of conditional expectations that are substantially challenging due to the absence of Markovianity.Another possible method to solve the optimal control problem (1.3) is to get an explicit solution of the forward equation (1.2), plug it into the performance functional and try to solve the maximization problem using differential calculus in Hilbert spaces.But, even though this method seems appealing, obtaining the required explicit representation of X u in terms of u might be tedious and burdensome.Instead, we will use the approach introduced in [2,9] that is in the same spirit of the one in [13,14,16] mentioned above: we will rewrite the original forward stochastic Volterra integral equation as a stochastic differential equation in a suitable Hilbert space and then apply standard optimization techniques in infinite dimensions (see e.g.[10,12]).Moreover, the shape of the corresponding infinite-dimensional Hamilton-Jacobi-Bellman equation allows to obtain an explicit solution to the latter by exploiting the "splitting" method from [14, Section 3.3].
We notice that, while the optimization problem (1.3) is closely related to the one presented in [2], there are several important differences in comparison to our work.In particular, [2] demands the kernel to have the form where µ is a signed measure such that Although there are some prominent examples of such kernels, not all kernels K are of this type; furthermore, even if a particular K admits such a representation in theory, it may not be easy to find the explicit shape of µ.In contrast, our approach works for all Hölder continuous kernels without any restrictions on the shape and allows to get explicit approximations ûn of the optimal control û.The lift procedure presented here is also different from the one used in [2] (although they both are specific cases of the technique presented in [8]).
The lift used in the present paper was introduced in [8], then generalized in [7] for the multidimensional case, but the approach itself can be traced back to [6].It should be also emphasised that this method has its own limitations: in order to perform the lift, the kernel K is required to have a specific representation of the form K(t) = g, e tA ν H , t ∈ [0, T ], where g and ν are elements of some Hilbert space H and {e tA , t ∈ [0, T ]} is a uniformly continuous semigroup acting on H with A ∈ L(H) and, in general, it may be hard to find feasible H, g, ν and A. Here, we work with Hölder continuous kernels K and we overcome this issue by approximating the kernel with Bernstein polynomials (which turn out to enjoy a simple representation of the required type).Then we solve the optimal control problem for the forward process with approximated kernel instead of the original one and we study convergence.
The paper is organised as follows.In section 2, we present our approach in case of a liftable K (i.e.K having a representation in terms of H, g, ν and A mentioned above).Namely, we describe the lift procedure, give the necessary results from stochastic optimal control theory in Hilbert spaces as well as derive an explicit representation of the optimal control û by solving the associated Hamilton-Jacobi-Bellman equation.In section 3, we introduce a liftable approximation for general Hölder continuous kernels, give convergence results for the solution to the approximated problem and discuss some numerical aspects for the latter.In section 4, we illustrate the application of our technique with examples and simulations.
2 Solution via Hilbert space-valued lift

Preliminaries
First of all, let us begin with some simple results on the optimization problem (1.3).Namely, we notice that X u n and the optimization problem (1.3) is well defined for any u ∈ L 2 a .
1) the forward Volterra Ornstein-Uhlenbeck-type equation (1.2) has a unique solution; 2) there exists a constant C > 0 such that where Proof.Item 1) is evident since, for almost all ω ∈ Ω, the equation (1.2) treated pathwisely can be considered as a deterministic linear Volterra integral equation of the second kind that has a unique solution (see e.g.[22]).Next, it is straightforward to deduce that and hence can be represented in the form where R β is the resolvent of the corresponding Volterra integral equation and the operator L is linear and continuous.Hence J(u) can be re-written as which immediately implies that |J(u)| < ∞.

Construction of Markovian lift and formulation of the lifted problem
As anticipated above, in order to solve the optimization problem (1.3) we will rewrite X u in terms of Markovian Hilbert space-valued process Z u using the lift presented in [8] and then apply the dynamic programming principle in Hilbert spaces.We start from the description of the core idea behind the Markovian lifts in case of liftable kernels.
Definition 2.2.Let H denote a separable Hilbert space with the scalar product For examples of liftable kernels, we refer to Section 4 and to [8].Consider a controlled Volterra Ornstein-Uhlenbeck process of the form (1.2) with a liftable kernel K(t) = g, e tA ν , ν H = 1, and denote ζ 0 := X(0) g 2 H g and Using the fact that X(0) = g, ζ 0 , we can now rewrite (1.2) as follows: where Z u t := ζ 0 + t 0 e At−s νdV u (s).It is easy to check that, Z u is the unique solution of the infinite dimensional SDE and thus the process where Ā is the linear bounded operator on H such that These findings are summarized in the following theorem.

H
g and {Z u t , t ∈ [0, T ]} is the H-valued stochastic process given by Using Theorem 2.3, one can rewrite the performance functional J(u) from (1.3) as where the superscript g in J g is used to highlight dependence on the H-valued process Z u .Clearly, maximizing (2.5) is equivalent to maximizing Finally, for the sake of notation and coherence with literature, we will sometimes write our maximization problem as a minimization one by simply noticing that the maximization of the performance functional J g (u) − a 2 g, ζ 0 can be reformulated as the minimization of In other words, in case of H-liftable kernel K, the original optimal control problem (1.3) can be replaced by the following one: (2.7) Remark 2.4.The machinery described above can also be generalized for strongly continuous semigroups on Banach spaces, see e.g.[7,8].However, for our purposes it is sufficient to consider the case when A is a linear bounded operator on a Hilbert space.

Solution to the lifted problem
The optimal control problem (2.7) completely fits the framework of dynamic programming principle stated in [10,Chapter 6].More precisely, consider the Hamilton-Jacobi-Bellman (HJB) equation associated to the problem (2.7) of the form where by ∇v we denote the partial Gateaux derivative w.r.t. the spacial variable z and the Hamiltonian functional It is easy to check that the coefficients of the lifted forward equation ( 2 and the equality holds if and only if Solving (2.9), we obtain û(t), t ∈ [0, T ], which has the form (2.10) Remark 2.5.In general, [10,Theorem 6.35] does not guarantee that û exists on the initial probability space, but instead considers the weak control framework, see [10,Section 6] for more details.However, in our case optimal control exists in the strong sense and, as we will see later, û turns out to be deterministic.
Since the shape (2.10) of the optimal control û depends on ∇v(t, Z u t ), our next goal is to explicitly solve the HJB equation (2.8).The solution as well as the optimality statement are given in the next theorem.(2.12) 2. The solution û of the optimal control problem (2.7) (and thus of the problem (1.3)) has the form where Ā = A − β g, • ν.
Proof. 1.In order to solve the HJB equation (2.8), we will use the approach presented in [14, Section 3.3].Namely, we will look for the solution in the form where w(t) and c(t) are such that ∂ ∂t v and ∇v are well-defined.In this case, and, recalling that g, ζ 0 = X(0), we can rewrite the HJB equation (2.8) as Now it would be sufficient to find w and c that solve the following systems: Noticing that the first system in (2.15) has to hold for all z ∈ H, we can solve instead, which is a simple linear equation and its solution has the form (2.11).Now it is easy to see that c has the form (2.12) and 2. The result follows directly from the item 1 above and [10, Theorem 6.35].
Remark 2.8.The approach described above can be extended by lifting to Banach space-valued stochastic processes.See [9] for more details.

Approximate solution for forwards with Hölder kernels
The crucial assumption in section 2 that allowed to apply the optimization techniques in Hilbert space was the liftability of the kernel.However, in practice it is often hard to find a representation of the required type for the given kernel, and even if this representation is available, it is not always convenient from the implementation point of view.For this reason, we provide a liftable approximation for the Volterra Ornstein-Uhlenbeck process (1.2) for a general C h -kernel K, where C h ([0, T ]) denotes the set of h-Hölder continuous functions on [0, T ].This section is structured as follows: first we approximate an arbitrary C h -kernel by a liftable one in a uniform manner and introduce a new optimization problem where the forward dynamics is obtained from the original one replacing the kernel K with its liftable approximation.Afterwards, we prove that the optimal value of the approximated problem converges to the optimal value of the original problem and give an estimate for the rate of convergence.Finally, we discuss some numerical aspects that could be useful from the implementation point of view.

Remark 3.1.
In what follows, by C we will denote any positive constant the particular value of which is not important and may vary from line to line (and even within one line).By • 2 we will denote the standard L 2 (Ω × [0, T ])-norm.
(Af )(x) = f (x + 1), f ∈ H, and denote K n a Bernstein polynomial approximation for K of order n ≥ 0, i.e. where Observe that and hence K n is H-liftable as with By the well-known approximating property of Bernstein polynomials, for any ε > 0, there exist n = n(ε) ∈ N 0 such that sup Moreover, if additionally K ∈ C h ([0, T ]) for some h ∈ (0, 1), [17,Theorem 1] guarantees that for all t ∈ [0, T ] where Now, consider a controlled Volterra Ornstein-Uhlenbeck process {X u (t), t ∈ [0, T ]} of the form (1.2) with the kernel K ∈ C h ([0, T ]) satisfying (3.4).For a given admissible u define also a stochastic process {X u n (t), t ∈ [0, T ]} as a solution to the stochastic Volterra integral equation of the form where K n (t) = n k=0 κ n,k t k with κ n,k defined by (3.2), i.e. the Bernstein polynomial approximation of K of degree n.Remark 3.2.It follows from [5,Corollary 4] that both stochastic processes t 0 K(t − s)dW (s) and t 0 K n (t − s)dW (s), t ∈ [0, T ], have modifications that are Hölder continuous at least up to the order h ∧ 1 2 .From now on, these modifications will be used.Now we move to the main result of this subsection.
a , and X u , X u n are given by (1.2) and (3.5) respectively.Then there exists C > 0 which does not depend on n or u such that for any admissible u ∈ L 2 a : Proof.First, by Theorem 2.1, there exists a constant C > 0 such that sup Consider an arbitrary τ ∈ [0, T ], and denote ∆(τ Note that, by (3.3) we have that sup Moreover, since {K n , n ≥ 1} are uniformly bounded due to their uniform convergence to K it is true that Lastly, by the Ito isometry and (3.3), where C is a positive constant (recall that it may vary from line to line).The final result follows from Gronwall's inequality.

Liftable approximation of the optimal control problem
As it was noted before, our aim is to find an approximate solution to the the optimization problem (1.3) by solving the liftable problem of the form where the maximization is performed over u ∈ L 2 a .In (3.7),K n is the Bernstein polynomial approximation of K ∈ C h ([0, T ]), i.e.
K n (t) = g n , e tA ν , t ∈ [0, T ], where A ∈ L (H) acts as (Af )(x + 1), ν = 1 [0,1] and 2).Due to the liftability of K n , the problem (3.7) falls in the framework of section 2, so, by Theorem 2.6, the optimal control ûn has the form (2.13): where Ān := A − β g n , • ν.The goal of this subsection is to prove the convergence of the optimal performance in the approximated dynamics to the actual optimal, i.e.
where J is the performance functional from the original optimal control problem (1.3).
Proof.We prove only (3.9); the proof of (3.10) is the same.Let u ∈ L 2 a be fixed.For any n ∈ N denote and notice that for any t ∈ [0, T ] we have that where C > 0 is a deterministic constant that does not depend on n, t or u (here we used the fact that Now, let us prove that there exists a constant C > 0 such that First note that, by Remark 3.2, for each n ∈ N and δ ∈ 0, h 2 ∧ 1 4 there exists a random variable and whence sup Thus it is sufficient to check that sup n∈N EΥ n < ∞.It is known from [5] that one can put , where p := 1 δ and C δ > 0 is a constant that does not depend on n.Let p > p. Then Minkowski integral inequality yields dxdy. (3.12)Note that, by [17, Proposition 2], every Bernstein polynomial K n that corresponds to K is Hölder continuous of the same order h and with the same constant H, i.e.
This implies that there exists a constant C which does not depend on n such that Plugging the bound above to (3.12), we get that where C > 0 denotes, as always, a deterministic constant that does not depend on n, t, u and may vary from line to line.Therefore, there exists a constant, again denoted by C not depending on n, t or u such that and thus, by (3.11), By Gronwall's inequality, there exists C > 0 which does not depend on n such that ) and K n be its Bernstein polynomial approximation of order n.
Then there exists constant C > 0 such that Moreover, ûn is " almost optimal" for J in the sense that there exists a constant C > 0 such that Proof.First, note that for any r ≥ 0 where B r := {u ∈ L 2 a : u 2 ≤ r}.Indeed, by definitions of J, J n and Theorem 3.3, for any u ∈ B r : (3.15) In particular, this implies that there exists C > 0 that does not depend on n such that J(0) − C < J n (0), so, by Proposition 3.4, there exists r 0 > 0 that does not depend on n such that u 2 > r 0 implies J n (u) < J(0) − C < J n (0), n ∈ N.
In other words, all optimal controls ûn , n ∈ N must be in the ball B r 0 and that sup u∈L 2 a J(u) = sup u∈Br 0 J(u).This, together with uniform convergence of J n to J over bounded subsets of L 2 a and estimate (3.14), implies that there exists C > 0 not dependent on n such that Finally, taking into account (3.14) and (3.16) as well as the definition of B r 0 , which ends the proof.Proof.By (2.1), the performance functional J can be represented in a linear-quadratic form as where L: ) is a continuous linear operator.Then, by [4, Theorem 9.2.6], there exists a unique û ∈ L 2 (Ω × [0, T ]) that maximizes J and, moreover, ûn → û weakly as n → ∞.Furthermore, since all ûn are deterministic, so is û; in particular, it is adapted to filtration generated by W which implies that û ∈ L 2 a .

Algorithm for computing ûn
The explicit form of ûn given by (3.8) is not very convenient from the implementation point of view since one has to compute e (T −t) Ān ν = e (T −t) Ān 1 [0,1] , where Ān : A natural way to simplify the problem is to truncate the series for some M ∈ N.However, even after replacing e (T −t) Ān in (3.8) with its truncated version, we still need to be able to compute Āk n 1 [0,1] for the given k ∈ N.An algorithm to do so is presented in the proposition below.

Then
Āk where κ n,i are defined by (3.2) and γ(i, k) are from Proposition 3.7.
Theorem 3.8.Let n ∈ N be fixed and M ≥ (T − t) Ān L , where • L denotes the operator norm.
Then, for all t ∈ [0, T ], Proof.One has to prove the first inequality and the second one then follows.It is clear that , where we used a well-known result on tail probabilities of Poisson distribution (see e.g.[20]).

Examples and simulations
Example 4.1 (monomial kernel ).Let N ∈ N be fixed.Consider an optimization problem of the form where, as always, we optimize over u ∈ L 2 a .The kernel where (Af )(x) = f (x + 1), f ∈ H.By Theorem 2.6, the optimal control for the problem (4.1) has the form û . In this simple case, we are able to find an explicit expression for e (T −t) Ā * 1 [−i,−i+1] .Indeed, it is easy to see that, for any i ∈ N ∪ {0}, p ∈ N ∪ {0} and q = 0, 1, ..., N , and whence is the Mittag-Leffler function.This, in turn, implies that On Fig. 1, the black curve depicts the optimal û computed for the problem 4.1 with K(t) = t 2 and T = 2 using (4.2); the othere curves are the approximated optimal controls ûn,M (as in (3.17   u ∈ L 2 a , where the kernels are chosen as follows: K 1 (t) := t 0.3 (fractional kernel), K 2 (t) := t 1.1 (smooth kernel) and K 3 (t) := e −t t 0.3 (gamma kernel).In these cases, we apply all the machinery presented in section 3 to find ûn,M for each of the optimal control problems described above.In our simulations, we choose T = 2, n = 20, M = 50; the mesh of the partition for simulating sample paths of X u is set to be 0.05, σ = 1, X(0) = 0. Fig. 2 depicts approximated optimal controls for different values of α and β.Note that the gamma kernel K 3 (t) (third column) is of particularly interest in optimal advertising.This kernel, in fact, captures the peculiarities of the empirical data (see [15]) since the past dependence comes into play after a certain amount of time (like a delayed effect) and its relevance declines as time goes forward.
Remark 4.4.Note that the stochastic Volterra integral equation from (4.3) can be sometimes solved explicitly for certain kernels (e.g. via the resolvent method).For instance, the solution X u which corresponds to the fractional kernel of the type K(t) = t h , h > 0, and β = 1 has the form X u (t) = Γ(h + 1) t 0 (t − s) h E h+1,h+1 −Γ(h + 1)(t − s) h+1 (αu(s)ds + dW (s)) , t ∈ [0, T ], where E a,b again denotes the Mittag-Leffler function.Having the explicit solution, one could solve the optimization problem (4.3) by plugging in the shape of X u to the performance functional and applying the standard minimization techniques in Hilbert spaces.However, as mentioned in the introduction, this leads to some tedious calculations that are complicated to implement, whereas our approach allows to get the approximated solution in a relatively simple manner.
Acknowledgments.Authors would also like to thank Dennis Schroers for the enlightening help with one of the proofs leading to this paper as well as Giulia Di Nunno for the proofreading and valuable remarks.