On Time Inconsistent Stochastic Control in Continuous Time

In this paper, which is a continuation of a previously published discrete time paper, we study a class of continuous time stochastic control problems which, in various ways, are time inconsistent in the sense that they do not admit a Bellman optimality principle. We study these problems within a game theoretic framework, and we look for Nash subgame perfect equilibrium points. For a general controlled continuous time Markov process and a fairly general objective functional we derive an extension of the standard Hamilton-Jacobi-Bellman equation, in the form of a system of non-linear equations, for the determination for the equilibrium strategy as well as the equilibrium value function. The main theoretical result is a verification Theorem. As an application of the general theory we study a time inconsistent linear quadratic regulator. We also present a study of time inconsistency within the framework of a general equilibrium production economy of Cox-Ingersoll-Ross type.


December 12, 2016. First version January 2008
To appear in

Finance and Stochastics
Abstract In this paper, which is a continuation of the discrete time paper [4], we study a class of continuous time stochastic control problems which, in various ways, are time inconsistent in the sense that they do not admit a Bellman optimality principle. We study these problems within a game theoretic framework, and we look for Nash subgame perfect equilibrium points. For a general controlled continuous time Markov process and a fairly general objective functional we derive an extension of the standard Hamilton-Jacobi-Bellman equation, in the form of a system of non-linear equations, for the determination for the equilibrium strategy as well as the equilibrium value function. The main theoretical result is a verification Theorem. As an application of the general theory we study a time inconsistent linear quadratic regulator. We also present a study of time

Introduction
The purpose of this paper is to study a class of stochastic control problems in continuous time, which have the property of being time-inconsistent in the sense that they do not allow for a Bellman optimality principle. As a consequence of this, the very concept of optimality becomes problematic, since a strategy which is optimal given a specific starting point in time and space, may be nonoptimal when viewed from a later date and a different state. In this paper we attack a fairly general class of time inconsistent problems by using a gametheoretic approach, so instead of searching for optimal strategies we search for subgame perfect Nash equilibrium strategies. The paper presents a continuous time version of the discrete time theory developed in our previous paper [4]. Since we will build heavily on the discrete time paper, the reader is referred to that paper for motivating examples and for more detailed discussions on conceptual issues.
All the papers above deal with particular model choices, and different authors use different methods in order to solve the problems. To our knowledge, the present paper, which is the continuous time part of the working paper [3], is the first attempt to study a reasonably general (albeit Markovian) class of time inconsistent control in continuous time. We would, however, like to stress that for the present paper we have been greatly inspired by [2], [9], and [11].

Structure of the paper
The structure of the paper is roughly as follows.
• In Section 2 we present the basic setup, and in Section 3 we discuss the concept of equilibrium. This concept replaces, in our setting, the optimality concept for a standard stochastic control problem, and in Definition 3.2 we give a precise definition of the equilibrium control and the equilibrium value function.
• Since the equilibrium concept in continuous time is quite delicate, we build the continuous time theory on the discrete time theory previously developed in [4]. In Section 4 we start to study the continuous time problem by going to the limit for a discretized problem, and using the results from [4]. This leads to an extension of the standard HJB equation to a system of equations with an embedded static optimization problem. The limiting procedure described above is done in an informal manner. It is largely heuristic, and it thus remains to clarify precisely how the derived extended HJB system is related to the precisely defined equilibrium problem under consideration.
• The needed clarification is in fact delivered in Section 5. In Theorem 5.1, which is the main theoretical result of the paper, we give a precise statement and proof of a verification theorem. This theorem says that a solution to the extended HJB system does indeed deliver the equilibrium control and equilibrium value function to our original problem.
• In Section 6 the results of Section 5 are extended to a more general reward functional.
• Section 7 treats the infinite horizon case.
• In Section 8 we study a time inconsistent version of the linear quadratic regulator, to illustrate how the theory works in a concrete case.
• Section 9 is devoted to a rather detailed study of a general equilibrium model for a production economy with time inconsistent preferences.
• In Section 10 we review some remaining open problems.
For extensions of the theory as well as worked out examples such as point process models, non-exponential discounting, mean variance control, and state dependent risk, see the working paper overview [5].

The model
We now turn to the formal continuous time theory. In order to present this we need some input data.
The following objects are given exogenously.

A drift mapping
where M (n, d) denotes the set of all n × d matrices.

A control constraint mapping
We now consider, on the time interval [0, T ], a controlled SDE of the form where the state process X is n-dimensional, the Wiener process W is d-dimensional, and the control process u is k-dimensional, with the constraint u t ∈ U (t, X t ).
Loosely speaking our object is to maximize, for every initial point (t, x), a reward functional of the form This functional is not of a form which is suitable for dynamic programming, and it will be discussed in detail below, but first we need to specify our class of controls. In this paper we restrict the controls to admissible feedback control laws. Definition 2.2 An admissible control law is a map u : [0, T ] × R n → R k satisfying the following conditions:

For each initial point
has a unique strong solution denoted by X u .
The class of admissible control laws is denoted by U We will sometimes use the notation u t (x) instead of u(t, x).
We now go on to define the controlled infinitesimal generator of the SDE above. In the present paper we use the (somewhat non-standard) convention that the infinitesimal operator acts on the time variable as well as on the space variable, so it includes the term ∂ ∂t . Definition 2.3 Consider the SDE (1), and let denote matrix transpose.
• For any fixed u ∈ R k , the functions µ u , σ u and C u are defined by • For any admissible control law u, the functions µ u , σ u , C u (t, x) are defined by x, u(t, x))σ(t, x, u(t, x)) .
• For any fixed u ∈ R k , the operator A u is defined by • For any admissible control law u, the operator A u is defined by

Problem formulation
In order to formulate our problem we need an objective functional. We thus consider the two functions F and G from Definition 2.1.
Definition 3.1 For a fixed (t, x) ∈ [0, T ] × R n , and a fixed admissible control law u, the corresponding reward functional J is defined by Remark 3.1 In Section 6 we will consider a more general reward functional. The restriction to the functional (2) above is done in order to minimize the notational complexity of the derivations below, which otherwise would be somewhat messy.
In order to have a non degenerate problem we need a formal integrability assumption.
Assumption 3.1 We assume that for each initial point (t, x) ∈ [0, T ] × R n , and each admissible control law u, we have Our objective is loosely that of maximizing J(t, x, u) for each (t, x), but conceptually this turns out to be far from trivial, so instead of optimal controls we will study equilibrium controls. The equilibrium concept is made precise in Definition 3.2 below, but in order to motivate that definition we need a brief discussion concerning the reward functional above.
We immediately note that, compared to a standard optimal control problem, the family of reward functionals above are not connected by a Bellman optimality principle. The reasons for this are as follows: • The present state x appears in the function F .
• In the second term we have (even apart from the appearance of the present state x), a nonlinear function G operating on the expected value E t,x [X u T ].
Since we do not have a Bellman optimality principle it is in fact unclear what we would mean by the term "optimal", since the optimality concept would differ at different initial times t and for different initial states x. The approach of this paper is to adopt a game theoretic perspective, and look for subgame perfect Nash equilibrium points. Loosely speaking we view the game as follows: • Consider a non-cooperative game, where we have one player for each point in time t. We refer to this player as "Player t".
• For each fixed t, Player t can only control the process X exactly at time t. He/she does that by choosing a control function u(t, ·), so the action taken at time t with state X t is given by u(t, X t ).
• Gluing together the control functions for all players we thus have a feedback control law u : • Given the feedback law u, the reward to Player t is given by the reward functional A slightly naive definition of an equilibrium for this game would be to say that a feedback control lawû is a subgame perfect Nash equilibrium if, for each t, it has the following property: • If for each s > t, Player s chooses the controlû(s, ·), then it is optimal for Player t to chooseû(t, ·).
A definition like this works well in discrete time, but in continuous time this is not a bona fide definition. Since Player t can only choose the control u t exactly at time t, he only influences the control on a time set of Lebesgue measure zero, so for a controlled SDE of the form (1) the control chosen by an individual player will have no effect whatsoever on the dynamics of the process. We thus need another definition of the equilibrium concept, and we will in fact follow an approach first taken by [9] and [11]. The formal definition of equilibrium is now as follows.
Definition 3.2 Consider an admissible control lawû (informally viewed as a candidate equilibrium law). Choose an arbitrary admissible control law u ∈ U and a fixed real number h > 0. Also fix an arbitrarily chosen initial point (t, x). Define the control law u h by u h (s, y) = u(s, y), for t ≤ s < t + h, y ∈ R n , u(s, y), for all u ∈ U, we say thatû is an equilibrium control law. Corresponding to the equilibrium lawû we define the equilibrium value function V by We will sometimes refer to this as an intrapersonal equilibrium, since it can be viewed as a game between different future manifestations of your own preferences.
Remark 3.2 This is our continuous time formalization of the corresponding discrete time equilibrium concept. Note the necessity of dividing by h, since for most models we trivially would have lim We also note that we do not get a perfect correspondence with the discrete time equilibrium concept, since if the limit above equals zero for all u ∈ U, it is not clear that this corresponds to a maximum or just to a stationary point.

An informal derivation of the extended HJB equation
We now assume that there exists an equilibrium control lawû (not necessarily unique) and we go on to derive an extension of the standard Hamilton-Jacobi-Bellman (henceforth HJB) equation for the determination of the corresponding value function V . To clarify the logical structure of the derivation we outline our strategy as follows.
• We discretize (to some extent) the continuous time problem. We then use our results from discrete time theory to obtain a discretized recursion for u and we then let the time step tend to zero.
• In the limit we obtain our continuous time extension of the HJB equation.
Not surprisingly it will in fact be a system of equations.
• In the discretizing and limiting procedure we mainly rely on informal heuristic reasoning. In particular we have do not claim that the derivation is a rigorous one. The derivation is, from a logical point of view, only of motivational value.
• In Section 5 we then go on to show that our (informally derived) extended HJB equation is in fact the "correct" one, by proving a rigorous verification theorem.

Deriving the equation
In this section we will, in an informal and heuristic way, derive a continuous time extension of the HJB equation. Note again that we have no claims to rigor in the derivation, which is only motivational. To this end we assume that there exists an equilibrium lawû and we argue as follows.
• Choose an arbitrary initial point (t, x). Also choose a "small" time increment h > 0 and an arbitrary admissible control u.
• Define the control law u h on the time interval [t, T ] by • If now h is "small enough" we expect to have We now refer to the discrete time results, as well as the notation, from Theorem 3.13 of [4], with n and n + 1 replaced by t and t + h. We then obtain the inequality Here we have used the following notation from [4].
• For any fixed y ∈ R n the mapping f y : We will also, with a slight abuse of notation, denote the entire family of functions {f y ; y ∈ R n } by f .
• For any function k(t, x) the operator A u h is defined by • The function G g is defined by We now divide the inequality by h and let h tend to zero. The the operator x) requires closer investigation. From the definition of the infinitesimal operator we have the approximation and using a standard Taylor approximation for G x we obtain We thus obtain Collecting all results we arrive at our proposed extension of the HJB equation.
To stress the fact that the arguments above are largely informal we state the equation as a definition rather than as proposition.
Definition 4.1 The extended HJB system of equations for V , f , and g, is defined as follows.

The function g is defined by
We now have some comments on the extended HJB system.
• The first point to notice is that we have a system of equations (3)-(5) for the simultaneous determination of V , f and g.
• In the expressions above,û always denotes the control law which realizes the supremum in the first equation.
• The equations (4)-(5) are the Kolmogorov backward equations for the expectations • In order to solve the V -equation we need to know f and g but these are determined by the equilibrium control lawû, which in turn is determined by the sup-part of the V -equation.
• We have used the notation • The operator A u only operates on variables within parenthesis. Thus x, x). In the expression (A u f y ) (t, x) the operator does not act on the upper case index y, which is viewed as a fixed parameter. Similarly, in the expression (A u f x ) (t, x), the operator only acts on the variables t, x within the parenthesis, and does not act on the upper case index x.
• In the case when F (x, y) does not depend upon x, and there is no G term, the problem trivializes to a standard time consistent problem. The terms x) in the V -equation cancel, and the system reduces to the standard Bellman equation • We note that the g function above appears, in a more restricted framework, already in [2], [9], and [11].

Existence and uniqueness
The task of proving existence and/or uniqueness of solutions to the extended HJB system seems (at least to us) to be technically extremely difficult. We have no idea about how to proceed so we leave it for future research. It is thus very much an open problem. See Section 10 for more open problems.

A Verification Theorem
As we have noted above, the derivation of the continuous time extension of the HJB equation in the previous section was very informal. Nevertheless, it seems reasonable to expect that the system in Definition 4.1 will indeed determine the equilibrium value function V . The following two conjectures are natural.
1. Assume that there exists an equilibrium lawû and that V is the corresponding value function. Assume furthermore that V is in C 1,2 . Define f y and g by We then conjecture that V satisfies the extended HJB system and thatû realizes the supremum in the equation.

2.
Assume that V , f , and g solves the extended HJB system and that the supremum in the V -equation is attained for every (t, x). We then conjecture that there exists an equilibrium lawû, and that it is given by the maximizing u in the V -equation. Furthermore we conjecture that V is the corresponding equilibrium value function, and f and g allow for the interpretations (6)- (7).
In this paper we do not attempt to prove the first conjecture. Even for a standard time consistent control problem within an SDE framework, it is well known that this is technically quite complicated, and it typically requires the theory of viscosity solutions. It is thus left as an open problem. We will, however, prove the second conjecture. This obviously has the form of a verification result, and from standard theory we would expect that it can be proved with a minimum of technical complexity. We now give the precise formulation and proof of the verification theorem, but first we need to define a function space.
for every (t, x). In this expression h x denotes the gradient of h in the x-variable.
We can now state and prove the main result of the present paper.
x), andû(t, x) have the following properties.
1. V , f y , and g solves the extended HJB system in Definition 4.1.
2. V (t, x), and g(t, x) are smooth in the sense that they are in C 1,2 , and f (t, x, y) is in C 1,2,2 .
3. The functionû realizes the supremum in the V equation, andû is an admissible control law. 4. V , f y , g, and G g, as well as the function (t, x, x) all belong to the space L 2 T (Xû). Thenû is an equilibrium law, and V is the corresponding equilibrium value function. Furthermore, f and g can be interpreted according to (6)-(7).
Proof. The proof consists of two steps: • We start by showing that f and g have the interpretations (6)- (7) and that V is the value function corresponding toû, i.e. that V (t, x) = J(t, x,û).
• In the second step we then prove thatû is indeed an equilibrium control law.
To show that f and g have the interpretations (6)- (7) we apply the Ito formula to the processes f y (s, Xû s ) and g(s, Xû s ). Using (4)-(5) and the assumed integrability conditions for f y and g, it follows that the processes f y (s, Xû s ) and g(s, Xû s ) are martingales, so from the boundary conditions for f y and g we obtain our desired representations of f y and g as To show that V (t, x) = J(t, x,û), we use the V equation (3) to obtain: where Hûg(t, x) = G y (x, g(t, x)) · Aûg(t, x).
Since f , and g satisfies (4)-(5), we have Aûg(t, x) = 0, so (10) takes the form for all t and x. We now apply the Ito formula to the process V (s, Xû s ). Integrating and taking expectations gives us where the stochastic integral part has vanished because of the integrability condition V ∈ L 2 T (Xû). Using (11) we thus obtain Aûf (s, Xû s , Xû s ds) In the same way we obtain x)).
Using this and the boundary conditions for V , f , and g we get Plugging (8)-(9) into (12) we get so we obtain the desired result We now go on to show thatû is indeed an equilibrium law, but first we need a small temporary definition. For any admissible control law u we define f u and g u by . so, in particular we have f = fû and g = gû. For any h > 0, and any admissible control law u ∈ U, we now construct the control law u h defined in Definition 3.2. From Lemma 3.3 and Lemma 8.8 in [4], applied to the points t and t + h we obtain Since u h = u on [t, t + h], we have X u h t+h = X u t+h , and since u h =û on [t + h, T ] we have so we obtain Furthermore, from the V -equation (3) we have where we have used the notation u = u(t, x). This gives us or, after simplification, Combining this with the expression for J(t, x, u h ) above, and the fact that (as we have proved) V (t, x) = J(t, x,û), we obtain and we are done.

The general case
We now turn to the most general case of the present paper, where the functional J is given by To study the reward functional above we need a slightly modified integrability assumption.
Assumption 6.1 We assume that for each initial point (t, x) ∈ [0, T ] × R n , and each admissible control law u, we have The treatment of this case is very similar to the previous one, so we directly give the final result, which is the relevant extended HJB system. Definition 6.1 Given the objective functional (13) the extended HJB system for V is given by (14)- (19) below.

The function V is determined by
with boundary condition 2. For each fixed s and y, the function f sy (t, x) is defined by

The function g(t, x) is defined by
In the definition above,û always denotes the control law which realizes the supremum in the V equation, and we have used the notation f (t, x, s, y) = f sy (t, x), Also for this case we have a verification theorem. The proof is almost identical to that of Theorem 5.1 so we omit it. Theorem 6.1 (Verification Theorem) Assume that, for all (s, y), the functions V (t, x), f sy (t, x), g(t, x), andû(t, x) have the following properties.
1. V , f sy , and g is a solution to the extended HJB system in Definition 6.1.
2. V , f sy , and g are smooth in the sense that they are in C 1,2 .
3. The functionû realizes the supremum in the V equation, andû is an admissible control law. 4. V , f sy , g, and G g, as well as the function (t, x) −→ f (t, x, t, x) all belong to the space L 2 T (Xû). Thenû is an equilibrium law, and V is the corresponding equilibrium value function. Furthermore, f , and g have the probabilistic representations f sy (t, x) = E t,x T t H s, y, r, Xû r ,û r (Xû r ) dr + F (s, y, Xû T ) ,

Infinite horizon
The results above can easily be extended to the case with infinite horizon, i.e. when T = +∞. The natural reward functional will then have the form so the functions F and G are not present. In this case we have x, t, x). The extended HJB system is thus reduced to the following system. We also have an obvious verification theorem, where the relevant integrability condition is that, for each (s, y), the function f sy (t, x) must belong to L 2 T (Xû) for all finite T . The proof is almost identical to the earlier case.

Example: The inconsistent linear quadratic regulator
To illustrate how the theory works in a simple case, we consider a variation of the classical linear quadratic regulator. Other "quadratic" control problems are considered in [2], [6] and [8], which study mean-variance problems within the present game theoretic framework. In the papers [20] and [21], the authors study the mean-variance criterion where you are continuously rolling over instantaneously updated pre-committed strategies. The model we consider is specified as follows.
• The value functional for Player t is given by where γ is a positive constant.
• The state process X is scalar with dynamics where a, b and σ are given constants.
• The control u is scalar with no constraints. This is a time inconsistent version of the classical linear quadratic regulator. The time inconsistency stems from the fact that the target point x = X t is changing as time goes by. In discrete time this problem is studied in [4]. For this problem we have and as usual we introduce the functions f y (t, x) and f (t, x, x) by In the present case we have V (t, x) = f (t, x, x) and it is easy to see that the extended HJB system of Section 6 takes the form From the X dynamics we see that We thus obtain the following form of the HJB equation, where for shortness of notation we denote partial derivatives by lower case index so, for example, The coupled system for f y is given by The first order condition in the HJB equation gives usû(t, x) = −bf x (t, x, x) and, inspired of the standard regulator problem, we now make the Ansatz where all coefficients are deterministic functions of time. We now insert the Ansatz into the HJB system, and perform a number of extremely boring calculations. As a result of these calculations, it turns out that the variables separate in the expected way and we have the following result.
Proposition 8.1 For the time inconsistent regulator, the function f is given by (20), and the equilibrium control is given bŷ where the coefficient functions solve the following system of ODE:s. Proof. It remains to check that the technical conditions of the verification theorem are in force. Firstly we need to check that the candidate equilibrium control in (21) is admissible. Since the equilibrium control is linear, the equilibrium state dynamics are linear so admissibility is clear. Secondly we need to check that the functions V (t, x), f (t, x, x) and f y (t, x) are in L 2 T Xû . Since the state dynamics are linear with constant diffusion term, the condition for a function h(t, x) to be in L 2 T Xû is simply that x) is quadratic in x, and since the dynamics for Xû are linear, we have square integrability, so the integrability condition above is satisfied. The same argument applies to f y (t, x) for every fixed y.

Example: A Cox-Ingersoll-Ross production economy with time inconsistent preferences
In this section we apply the previously developed theory to a rather detailed study of a general equilibrium model for a production economy with time inconsistent preferences. The model under consideration is a time inconsistent analogue of the classic Cox-Ingersoll-Ross model in [7]. Our main objective is to investigate the structure of the equilibrium short rate, the equilibrium Girsanov kernel, and the equilibrium stochastic discount factor.
There are a few earlier papers on equilibrium with time inconsistent preferences. In [1] and [18] the authors study continuous time equilibrium models of a particular type of time inconsistency, namely non-exponential discounting. While [1] considers a deterministic neoclassical model of economic growth, [18] analyze general equilibrium in a stochastic endowment economy.
Our present study is much inspired by the earlier paper [15] which, in very great detail, studies equilibrium in a very general setting of an endowment economy with dynamically inconsistent preferences that is not limited to the particular case of non-exponential discounting.
Unlike the papers mentioned above, which all studies endowment models, we study a stochastic production economy of Cox-Ingersoll-Ross type.

The Model
We start with some formal assumptions concerning the production technology.
Assumption 9.1 We assume that there exists a constant returns to scale physical production technology process S with dynamics dS t = αS t dt + S t σdW t .
The economic agents can invest unlimited positive amounts in this technology, but since it is a matter of physical investment, short positions are not allowed.
From a purely formal point of view, investment in the technology S is equivalent to the possibility of investing in a risky asset with price process S, with the constraint that short selling is not allowed.
We also need a risk free asset, and this is provided by the next assumption.
Assumption 9.2 We assume that there exists a risk free asset in zero net supply with dynamics where r is the short rate process, which will be determined endogenously. The risk free rate r is assumed to be of the form where X denotes the portfolio value of the representative investor (to be defined below).
Interpreting the production technology S as above, the wealth dynamics will be given by where u is the portfolio weight on production, so 1 − u is the weight on the risk free asset. Finally we need an economic agent.

Equilibrium definitions
We now go on to study equilibrium in our model. We will in fact have two equilibrium concepts • Intrapersonal equilibrium.
The intrapersonal equilibrium is related to the lack of time consistency in preferences, whereas the market equilibrium is related to market clearing. We now discuss these concepts in more detail.

Intrapersonal equilibrium
Consider, for a given short rate function r(t, x) the control problem with reward functional and wealth dynamics where r t is shorthand for r(t, X t ). If the agent wants to maximize the reward functional for every initial point (t, x) then, because of the appearance of (t, x) in the utility function U , this is a time inconsistent control problem. In order to handle this situation we use the game theoretic setup and results developed in Sections 1-6 above. This subgame perfect Nash equilibrium concept is henceforth referred to as the intrapersonal equilibrium.

Market equilibrium
By a market equilibrium we mean a situation where the agent follows an intrapersonal equilibrium strategy, and where the market clears for the risk free asset. The formal definition is as follows.
Definition 9.1 A market equilibrium of the model is a triple of real valued functions {ĉ(t, x),û(t, x), r(t, x)} such that the following hold.
1. Given the risk free rate process r(t, x), the intrapersonal equilibrium consumption and investment are given byĉ andû respectively.
2. The market clears for the risk free asset, i.e.

Main goals of the study
As will be seen below, there will be a unique equilibrium martingale measure Q with corresponding likelihood process L = dQ/dP , where L has dynamics The process ϕ will be referred to as the equilibrium Girsanov kernel. There will also be a equilibrium short rate process r, which will be related to ϕ by the standard no arbitrage relation which says that S/B is a Q-martingale. There will also be a unique equilibrium stochastic discount factor M defined by rsds L t .
For ease of notation we will, however, only identify the stochastic discount factor M , up to a multiplicative constant, so for any arbitrage free (non dividend) price process p t we will have the pricing equation Our goal is to obtain expressions for ϕ, r and M .

The extended HJB equation
In order to determine the intrapersonal equilibrium we use the results from Section 6. At this level of generality we are not able to provide a priori conditions on the model which guarantee that the conditions of the Verification Theorem are satisfied. For the general theory below (but of course not for the concrete example presented later) we are thus forced to make the following ad hoc assumption.
Assumption 9. 4 We assume that the model under study is such that the Verification Theorem is in force.
This assumption will of course have to be checked for every concrete application. In Section 9.8.2 we consider the example of power utility, and for this case we can in fact prove that the assumption is satisfied.
In the present case we have V (t, x) = f (t, x, t, x) and it is easy to see that we can write the extended HJB equation as sup u≥0,c≥0 U (t, x, t, c) + A u,c f tx (t, x) = 0 and f sy is determined by Aû ,ĉ f sy (t, x) + U (s, y, t,ĉ(t, x)) = 0 with the probabilistic representation The term A u,c f tx (t, x) is given by where f and the derivatives are evaluated at (t, x, t, x) and where we have used the notation x, s, y), f xx (t, x, s, y) = ∂ 2 f ∂x 2 (t, x, s, y).
The first order conditions for an interior optimum are

Determining market equilibrium
In order to determine the market equilibrium we use the equilibrium condition u = 1. Plugging this into (23) we immediately obtain our first result.
Proposition 9.1 With assumptions as above the following hold.
• The equilibrium short rate is given by • The equilibrium Girsanov kernel ϕ is given by • The extended equilibrium HJB system has the form Aĉf sy (t, x) + U (s, y, t,ĉ(t, x)) = 0 (27) • The equilibrium consumptionĉ is determined by the first order condition • The term Aĉf tx (t, x) is given by The equilibrium X dynamics are given by Proof. The formula (25) follows from (24) and (22). The other results are obvious.

Recap of standard results
We can compare the results above with the standard case where the utility functional for the agent is of the time consistent form In this case we have a standard HJB equation of the form sup u∈R,c≥0 and the equilibrium quantities are given by the well known expressions We note the strong structural similarities between the old and the new formulas, but we also note important differences. Let us take the formulas for the equilibrium short rate r as an example. We recall the standard and time inconsistent formulas For the time inconsistent case we do have the relation V e (t, x) = f (t, x, t, x) (where temporarily, and for the sake of clearness, V e denotes the equilibrium value function) so it is tempting to think that we should be able to write (31) as x (t, x) which would be structurally identical to (30). This, however, turns out to be incorrect.

The stochastic discount factor
We now go on to investigate our main object of interest, namely the equilibrium stochastic discount factor M . We recall from general arbitrage theory that where L is the likelihood process L t = dQ dP on F t , with dL t = L t ϕ t dW t . From this we immediately obtain the M dynamics as so we can identify the short rate r and the Girsanov kernel ϕ from the dynamics of M . From Proposition 9.1 we know r and ϕ, so in principle we have in fact already determined M , but we now want to investigate the relation between M , the direct utility function U , and the indirect utility function f in the extended HJB equation.
We recall from standard theory that for the usual time consistent case the (non normalized) stochastic discount factor M is given by or equivalently by along the equilibrium path. In our present setting we have so a conjecture would perhaps be that the stochastic discount factor for the time inconsistent case is given by at least one of the formulas along the equilibrium path. In order to check if any of these formulas are correct we only have to compute the corresponding differential dM t and check whether it satisfies (33). It is then easily seen that none of the formulas for M are correct. The situation is in thus more complicated and we now go on to derive the correct representation of the stochastic discount factor.

A representation formula for M
We now go back to the time inconsistent case with utility of the form x, s, c s )ds .
We will, below, present a representation for the stochastic discount factor M in the production economy, but first we need to introduce some new notation.
Definition 9.2 Let X be a (possibly vector valued) semimartingale and let Y be an optional process. For a C 2 function f (x, y) we introduce the "partial stochastic differential" ∂ x by the formula The intuitive interpretation of this is that and we have the following proposition, which generalizes the standard result for the time consistent theory.
where f tx = f tx (t, x, t, x) and similarly for other derivatives,ĉ =ĉ(t, x) and U x = U x (t, x, t,ĉ(t, x)). From the extended HJB system we also recall the PDE for f sy as Differentiating this equation w.r.t. the variable y and evaluating at (t, x, t, x) andĉ(t, x) we obtain We can now plug this into (40) to obtain Plugging this into (37) we can write A as which is exactly (39).

Generalities
In the case of non exponential discounting it is natural to consider the case with infinite horizon. We will thus assume that T = ∞ so we have the functional The function f (t, x, s, y) will now be of the form f (t, x, s) and, because of the time invariance, it is natural to look for time invariant equilibria wherê Observing that f x (t, x, t) = g x (0, x) = V x (x) and similarly for second order derivatives, we may now restate proposition 9.1.
Proposition 9.2 With assumptions as above the following hold.
• The equilibrium short rate is given by • The equilibrium Girsanov kernel ϕ is given by • The extended equilibrium HJB system has the form U (ĉ) + g t (0, x) + (αx −ĉ) g x (0, x) + 1 2 x 2 σ 2 g xx (0, x) = 0, Aĉg(t, x) + β(t)U (ĉ(x)) = 0, • The function g has the representation • The equilibrium consumptionĉ is determined by the first order condition U c (ĉ) = g x (0, x) (43) • The term Aĉg(t, x) is given by Aĉg(t, x) = g t (t, x) + (αx −ĉ(x)) g x (t, x) + 1 2 • The equilibrium X dynamics are given by We see that the short rate r and the Girsanov kernel ϕ has exactly the same structural form as the standard case formulas (28)-(29). We now move to the stochastic discount factor and after some calculations we have the following version of Theorem 9.1.

Proposition 9.3
The stochastic discount factor M is determined by where g x is evaluated at (0, X t ). Alternatively, we can write M as M t = U c (ĉ t ) · exp t 0 g xt (0, X s ) g x (0, X s ) ds

Power utility
We now specialize to the case of constant relative risk aversion (CRRA) utility of the form U (c) = c 1−γ − 1 1 − γ with γ > 0, γ = 1. We make the obvious Ansatz  Using the equilibrium X-dynamics it is easy to see that so the integrability condition will indeed be satisfied. From this example with non exponential discounting we see that the risk free rate and Girsanov kernel only depend on the production opportunities in the economy. These objects are unaffected by the time inconsistency stemming from non-exponential discounting. Equilibrium consumption, however, is determined by the discounting function of the representative agent.

Conclusion and open problems
In this paper we have presented a fairly general class of time inconsistent stochastic control problems. Using a game theoretic perspective we have derived an extended HJB system of PDE:s for the determination of the equilibrium control, as well as for the equilibrium value function. We have proved a verification theorem, and we have studied a couple of concrete examples. For more examples and extensions see the working paper [5]. Some obvious open research problems are the following.
• A theorem which proving convergence of the discrete time theory to the continuous time limit. For the quadratic case, this is done in [8], but the general problem is open.
• An open and difficult problem is to provide conditions on primitives which guarantee that the functions V and f are regular enough to satisfy the extended HJB system.
• A related (hard) open problem is to prove existence and/or uniqueness for solutions of the extended HJB system.
• Another related problem is to give conditions on primitives which guarantee that the assumptions of the Verification Theorem are satisfied.
• The present theory depends critically on the Markovian structure. It would be interesting to see what can be one without this assumption.