Stochastic Differential Games and Energy-Efficient Power Control

Abstract

One of the contributions of this work is to formulate the problem of energy-efficient power control in multiple access channels (namely, channels which comprise several transmitters and one receiver) as a stochastic differential game. The players are the transmitters who adapt their power level to the quality of their time-varying link with the receiver, their battery level, and the strategy updates of the others. The proposed model not only allows one to take into account long-term strategic interactions, but also long-term energy constraints. A simple sufficient condition for the existence of a Nash equilibrium in this game is provided and shown to be verified in a typical scenario. As the uniqueness and determination of equilibria are difficult issues in general, especially when the number of players goes large, we move to two special cases: the single player case which gives us some useful insights of practical interest and allows one to make connections with the case of large number of players. The latter case is treated with a mean-field game approach for which reasonable sufficient conditions for convergence and uniqueness are provided. Remarkably, this recent approach for large system analysis shows how scalability can be dealt with in large games and only relies on the individual state information assumption.

Appendices

Appendix A: Proof of Proposition 1

Both results can be proven by using Ito’s formula (see, e.g., [16]). For the mean, from (5), one has

$$ \text{d} \mathbb{E}\bigl[\underline{h}_i(t)\bigr] = \frac{1}{2} \bigl( \underline{\mu } - \mathbb{E}\bigl[\underline{h}_i(t)\bigr]\bigr) \,\text{d}t, $$

(44)

then $\mathbb{E}[\underline{h}_{i}(t)] = \underline{\mu}(1 - e^{-\frac{t}{2}}) + \underline{h}_{i}(0)e^{-\frac{t}{2}}$. The limit when t goes to +∞ writes

$$ \lim_{t\to+ \infty} \mathbb{E}[\underline{h}_i] = \underline{\mu}. $$

(45)

For the variance, assume that for the two components x _i and y _i of h _i :

$$ \begin{array}{ll} \text{d} x_i(t) = g_x(t)\, \text{d}t + \eta\,\text{d}\mathbb{W}_x, \\\noalign{\vspace{2pt}} \text{d} y_i(t) = g_y(t)\, \text{d}t + \eta\,\text{d}\mathbb{W}_y, \end{array} $$

(46)

with $\mathbb{W}_{x}$ and $\mathbb{W}_{y}$ two independent Wiener processes of dimension 1. Then

$$ \text{d} x_i(t)^2 = \bigl(2x_i(t) g_x(t) + \eta^2\bigr) \,\text{d}t + 2 x_i(t) \eta\,\text{d}\mathbb{W}_x, $$

(47)

and

$$ \text{d} \mathbb{E}\bigl[x_i(t)^2\bigr] = \mathbb{E} \bigl[\bigl(2x_i(t) g_x(t) + \eta^2\bigr) \bigr] \,\text{d}t. $$

(48)

If g _x (t)=0, $\mathbb{E}[x_{i}(t)^{2}] = x_{i}(0)^{2} + \eta^{2} t$ and $\lim_{t \to\infty} \mathbb{E}[x_{i}(t)^{2}] = \infty$. That is the reason why a deterministic term is needed in (5). With $g_{x}(t) = \frac{1}{2}(\mu_{x} - x_{i}(t))$, one has

$$ \text{d}\mathbb{E}\bigl[x_i(t)^2\bigr] = \mathbb{E} \biggl[2 x_i(t)\frac{1}{2}\bigl(\mu_x - x_i(t) \bigr) + \eta^2\biggr]\, \text{d}t, $$

(49)

then

$$ \frac{\text{d}\mathbb{E}[x_i(t)^2]}{\text{d}t} = -\mathbb{E} \bigl[x_i(t)^2\bigr] + \mu_x \mathbb{E} \bigl[x_i(t)\bigr] + \eta^2. $$

(50)

The solution of this differential equation has the form

$$ \begin{aligned}[b] \mathbb{E}\bigl[x_i(t)^2 \bigr] &= \biggl(x_i(0)^2 + \int_0^t \bigl(\mu_x\mathbb{E}\bigl[x_i\bigl(t' \bigr)\bigr] + \eta^2 \bigr)e^{ t'}\, \text{d}t' \biggr)e^{- t}, \\ &= \bigl[ \bigl(\mu_x^2 + \eta^2 \bigr)e^{t'} + 2 \mu_x \bigl(x_i(0) - \mu_x \bigr)e^{\frac{t'}{2}} \bigr]_0^t e^{- t}, \\ &= \bigl(\mu_x^2 + \eta^2 \bigr) \bigl(1-e^{-t}\bigr) + 2 \mu_x \bigl(x_i(0) - \mu_x \bigr) \bigl(e^{-\frac{t}{2}} -e^{-t}\bigr), \end{aligned} $$

(51)

thus

$$ \lim_{t\to+ \infty} \mathbb{E}\bigl[x_i(t)^2\bigr] = \mu_x^2 + \eta^2. $$

(52)

The analogous is true for y _i . Hence, we have

$$ \lim_{t\to+ \infty} \mathbb{E}\bigl[\bigl|\underline{h}_i(t)\bigr|^2 \bigr] = |\underline{\mu}|^2 + 2\eta^2, $$

(53)

and finally

$$ \lim_{t\to+ \infty} \mathbb{E}\bigl[\bigl|\underline{h}_i(t)\bigr|^2 \bigr] - \mathbb {E}\bigl[\bigl|\underline{h}_i(t)\bigr|\bigr]^2 = 2\eta^2. $$

(54)

Regarding to the probability density functions, applying the Kolmogorov forward equation to the state $\underline{h}_{i}$ with the dynamics given in (5), one has for the component x _i

$$ \begin{aligned}[b] \frac{\partial m_x(x_i,t)}{\partial t} &= - \frac{\partial}{\partial x_i} \biggl[m_x(x_i,t)\frac{1}{2} ( \mu_x - x_i) \biggr] + \frac{\eta^2}{2} \frac{\partial^2 m_x(x_i,t)}{\partial x_i^2}, \\ &= \frac{1}{2} m_x(x_i,t) - \frac{1}{2} (\mu_x - x_i) \frac{\partial m_x(x_i,t)}{\partial x_i} + \frac{\eta^2}{2} \frac{\partial^2 m_x(x_i,t)}{\partial x_i^2}. \end{aligned} $$

(55)

The stationary case gives

$$ 0= \frac{1}{2} m_x(x_i) - \frac{1}{2} ( \mu_x - x_i) \frac{\partial m_x(x_i)}{\partial x_i} + \frac{\eta^2}{2} \frac{\partial^2 m_x(x_i)}{\partial x_i^2}. $$

(56)

One can check that $\hat{m}_{x}(x_{i}) = \frac{1}{\eta\sqrt{2 \pi}} e^{-\frac{(x_{i} - \mu_{x})^{2}}{2 \eta^{2}}}$ is a solution of (56). This is the stationary density of x _i . The analogous can also be written for y _i : $\hat{m}_{y}(y_{i}) = \frac{1}{\eta\sqrt{2 \pi}} e^{-\frac{(y_{i} - \mu_{y})^{2}}{2 \eta^{2}}}$.

Appendix B: Proof of Proposition 4

The proof follows the same principle as in chapter Risk-sensitive mean-field games in the notes Mean-field stochastic games by Tembine. Only the sketch of the proof is given here. To prove the uniqueness of the solution, we suppose that there exists two solutions (v _1,t ,m _1,t ),(v _2,t ,m _2,t ) of the above system. We want to find a sufficient condition under which the quantity $\int_{\underline{s}} (v_{2,t}(\underline{s})-v_{1,t}(\underline {s}))(m_{2,t}(\underline{s})-m_{1,t}(\underline{s}))\,\text {d}\underline{s}$ is monotone in time, which is not possible.

$$ \left \{ \begin{array}{cc} m_{1,T}(\underline{s}) = m_{2,T}(\underline{s}), \\\noalign{\vspace{2pt}} v_{1,T'}(\underline{s}) = v_{2,T'}(\underline{s}). \end{array} \right . $$

(57)

Compute the time derivative

$$ \begin{aligned}[b] S_t =& \frac{\text{d}}{\text{d}t} \biggl(\int _{\underline{s}} \bigl(v_{2,t}(\underline {s})-v_{1,t}( \underline{s})\bigr) \bigl(m_{2,t}(\underline {s})-m_{1,t}( \underline{s})\bigr)\,\text{d}\underline{s} \biggr), \\ =& \int_{\underline{s}} \biggl(\frac{\partial{v_{2,t}}}{\partial t} (\underline{s})- \frac{\partial{v_{1,t}}}{\partial t} (\underline {s})\biggr) \bigl(m_{2,t}( \underline{s})-m_{1,t}(\underline{s})\bigr)\,\text {d}\underline{s}, \\ &+ \int_{\underline{s}} \bigl(v_{2,t}(\underline{s})-v_{1,t}( \underline {s})\bigr) \biggl(\frac{\partial{m_{2,t}}}{\partial t}(\underline{s})-\frac {\partial{m_{1,t}}}{\partial t}( \underline{s})\biggr)\,\text{d}\underline{s}. \end{aligned} $$

(58)

To express the first term, the difference between the two HJBF equations is taken and multiplied by m _2,t −m _1,t :

$$ \begin{aligned} [b]&\int_{\underline{s}} \biggl( \frac{\partial{v_{2,t}}}{\partial t}-\frac {\partial{v_{1,t}}}{\partial t}\biggr) (m_{2,t}-m_{1,t})\, \text{d}\underline {s} \\ &\quad =\int_{\underline{s}}\tilde{H}\biggl(\underline{s}(t), \frac{\partial v_{1,t}}{\partial E},m_{1,t}\biggr) (m_{2,t} - m_{1,t})\, \text{d}\underline {s} - \int_{\underline{s}}\tilde{H}\biggl( \underline{s}(t),\frac {\partial v_{2,t}}{\partial E},m_{2,t}\biggr) (m_{2,t} - m_{1,t})\,\text {d}\underline{s} \\ &\qquad {}+ \int_{\underline{s}}\frac{\eta^2}{2} \frac{\partial^2 v_{1,t}}{\partial h^2}(m_{2,t} - m_{1,t})\, \text{d}\underline{s} - \int_{\underline{s}} \frac{\eta^2}{2} \frac{\partial^2 v_{2,t}}{\partial h^2}(m_{2,t} - m_{1,t})\,\text{d} \underline{s} \\ &\qquad {}+ \frac{1}{2} \int_{\underline{s}} \bigl\langle\underline{\mu}-\underline {h}(t), \nabla_{\underline{h}}v_{1,t} \bigr\rangle_{\mathbb {R}^2}(m_{2,t} - m_{1,t}) \,\text{d}\underline{s} - \frac{1}{2} \int_{\underline{s}} \bigl\langle\underline{\mu}-\underline{h}(t), \nabla_{\underline{h}}v_{2,t} \bigr\rangle_{\mathbb{R}^2}(m_{2,t} - m_{1,t})\, \text{d} \underline{s}. \end{aligned} $$

(59)

For the second term, the difference between the two FPK equations is taken and multiplied by v _2,t −v _1,t :

$$ \begin{aligned}[b] &\int_{\underline{s}} \biggl( \frac{\partial{m_{2,t}}}{\partial t} - \frac {\partial{m_{1,t}}}{\partial t}\biggr) (v_{2,t}-v_{1,t})\, \text{d}\underline {s} \\ &\quad =-\int_{\underline{s}}\frac{\partial}{\partial E}\biggl(m_{2,t} \frac {\partial}{\partial u'} \tilde{H}\biggl(\underline{s}(t),\frac{\partial v_{2,t}}{\partial E},m_{2,t} \biggr)\biggr) (v_{2,t}-v_{1,t})\,\text{d}\underline{s} \\ &\qquad {}+\int_{\underline{s}} \frac{\partial}{\partial E}\biggl(m_{1,t} \frac {\partial}{\partial u'} \tilde{H}\biggl(\underline{s}(t),\frac{\partial v_{1,t}}{\partial E},m_{1,t} \biggr)\biggr) (v_{2,t}-v_{1,t})\,\text{d}\underline{s} \\ &\qquad {}+\int_{\underline{s}} \frac{\eta^2}{2} \frac{\partial^2 m_{2,t}}{\partial h^2} (v_{2,t}-v_{1,t})\,\text{d}\underline{s}-\int _{\underline{s}} \frac{\eta^2}{2} \frac{\partial^2 m_{1,t}}{\partial h^2}(v_{2,t}-v_{1,t})\, \text{d}\underline{s} \\ &\qquad {}+\frac{1}{2} \int_{\underline{s}} \operatorname{div}_{\underline {h}}\bigl(m_{2,t} \bigl(\underline{\mu}-\underline{h}(t)\bigr)\bigr) (v_{2,t}-v_{1,t})\, \text{d}\underline{s} \\ &\qquad {}-\frac{1}{2} \int_{\underline{s}} \operatorname{div}_{\underline{h}} \bigl(m_{1,t}\bigl(\underline{\mu}-\underline{h}(t)\bigr)\bigr) (v_{2,t}-v_{1,t})\, \text{d}\underline{s} . \end{aligned} $$

(60)

By integration by parts

$$ \int_{\underline{s}} \operatorname{div}_{\underline{s}}(k) \phi\,\text{d}\underline {s} = - \int_{\underline{s}} k \operatorname{div}_{\underline{s}}(\phi)\,\text {d} \underline{s}, $$

(61)

then

(62)

The full derivative writes

(63)

We now introduce

$$ m_{\lambda,t} = (1-\lambda)m_{1,t}+\lambda m_{2,t}=m_{1,t} + \lambda (m_{2,t}-m_{1,t}), $$

(64)

and in the same way

$$ v_{\lambda,t} = (1-\lambda)v_{1,t}+\lambda v_{2,t}. $$

(65)

We study the auxiliary integral

(66)

which derivative is

$$ \begin{aligned}[b] \frac{d}{d\lambda} \biggl(\frac{C_{\lambda}}{\lambda} \biggr) &= - \int_{\underline{s}}\frac{\partial}{\partial m} \tilde {H}\biggl({ \underline{s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t}\biggr) (m_{2,t} - m_{1,t})^2 \,\text{d}\underline{s} \\ &\quad {}+\int_{\underline{s}} m_{\lambda,t} \frac{\partial^2}{\partial u'^2} \tilde{H} \biggl({\underline{s}}(t),\frac{\partial v_{\lambda ,t}}{\partial E},m_{\lambda,t}\biggr) \biggl( \frac{\partial v_{2,t}}{\partial E}-\frac{\partial v_{1,t}}{\partial E}\biggr)^2\,\text{d}\underline{s} \\ &\quad {}+\int_{\underline{s}} m_{\lambda,t} \frac{\partial^2}{\partial m \partial u'} \tilde{H} \biggl({\underline{s}}(t),\frac{\partial v_{\lambda ,t}}{\partial E},m_{\lambda,t}\biggr) (m_{2,t}-m_{1,t}) \biggl(\frac{\partial v_{2,t}}{\partial E}-\frac{\partial v_{1,t}}{\partial E} \biggr)\,\text {d}\underline{s}. \end{aligned} $$

(67)

Note that $\frac{\partial}{\partial m} \tilde{H}(\cdot)$ and $\frac {\partial^{2}}{\partial m \partial u'} \tilde{H}(\cdot)$ are functional derivatives. They are defined such that for all m′∈ℙ(ℝ³)

$$ \begin{aligned}[b] &\biggl\langle\frac{\partial}{\partial m} \tilde{H}\biggl({ \underline {s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t}\biggr), m'-m_{\lambda,t} \biggr\rangle_{\mathbb{P}(\mathbb{R}^3)} \\ &\quad =\lim_{t\to0} \frac{\tilde{H}({\underline{s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t}+t(m'-m_{\lambda,t})) - \tilde{H}({\underline{s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t})}{t}, \end{aligned} $$

(68)

and

$$ \begin{aligned}[b] &\biggl\langle\frac{\partial^2}{\partial m \partial u'} \tilde {H}\biggl({ \underline{s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t}\biggr), m'-m_{\lambda,t} \biggr\rangle_{\mathbb{P}(\mathbb {R}^3)} \\ &\quad =\lim_{t\to0} \frac{\frac{\partial\tilde{H}}{\partial u'}({\underline{s}}(t),\frac{\partial v_{\lambda,t}}{\partial E},m_{\lambda,t}+t(m'-m_{\lambda,t})) - \frac{\partial\tilde {H}}{\partial u'}({\underline{s}}(t),\frac{\partial v_{\lambda ,t}}{\partial E},m_{\lambda,t})}{t}. \end{aligned} $$

(69)

A sufficient condition for the uniqueness of the solution to the mean-field response problem is the monotonicity of the operator associated to

$$\left ( \begin{array}{c@{\quad}c} A_{11} & A_{12} \\ A_{21} & a_{22} \end{array} \right ) $$

with

$$ \begin{aligned} A_{11} &= -\frac{1}{m} \frac{\partial}{\partial m} \tilde{H}, \\ A_{12} &= A_{21}=\frac{1}{2} \frac{\partial^2}{\partial m \partial u'} \tilde{H}, \\ a_{22} &= \frac{\partial^2}{\partial u'^2} \tilde{H}. \end{aligned} $$

(70)

This is true if

$$ \begin{aligned} &A_{11} \succ0, \qquad a_{22} > 0, \\ &A_{12} - \frac{A_{12}^2}{a_{22}} \succ0. \end{aligned} $$

(71)