Model-free method for LQ mean-field social control problems with one-dimensional state space

Abstract

This paper presents a novel model-free method to solve linear quadratic (LQ) mean-field control problems with one-dimensional state space and multiplicative noise. The focus is on the infinite horizon LQ setting, where the conditions for solution either stabilization or optimization can be formulated as two algebraic Riccati equations (AREs). The proposed approach leverages the integral reinforcement learning technique to iteratively solve the drift-coefficient-dependent stochastic ARE (SARE) and other indefinite ARE, without requiring knowledge of the system dynamics. A numerical example is given to demonstrate the effectiveness of the proposed algorithm.

Appendices

Appendix A: Proof of Lemma 2

Convergence result of $k_j$ follows with convergence of $p_j$ by (11). To complete the proof, the rest is to show that $\lim _{j\rightarrow \infty }p_j=p$. To this end, we first show that for all $j\in {\mathbb {N}}_+$, $p_j$ is a unique positive solution of Eq. (9) and $k_j$ is a stabilizer.

If $j\ge 1$, suppose that $p_j$ is a unique positive solution of (9) and $k_{j-1}$ is a stabilizer. Then, by (9)–(11), we obtain

$$\begin{aligned} \begin{aligned} 2a_{j}p_j+c_j^2p_j = -q - (r+d^2p_j) (k_{j-1} - k_j)^2 - rk_j^2. \end{aligned} \end{aligned}$$

(A1)

Followed with the induction assumption and the fact $q>0$, it is immediately concluded that $K_j$ is a stabilizer by [27, Theorem 1]. Moreover, the Lyapunov function (9) with $j=j+1$, rewritten by

$$\begin{aligned} 2a_{j}p_{j+1}+c_j^2p_{j+1}+q_j=0, \end{aligned}$$

(A2)

admits a unique solution $p_{j+1}>0$, since $q_j>0$.

Then, we subtract (9) from (4) to obtain

$$\begin{aligned} 2a_{j-1}(p_j-p)+c_{j-1}^2(p_j-p)=-\upsilon (k_{j-1}-k)^2. \end{aligned}$$

(A3)

Since $k_{j-1}$ is a stabilizer and $(k_{j-1}-k)^2\ge 0$, the above equation admits a unique solution $p_j-p\ge 0$, i.e., $p_j\ge p$, $j=1,2,\ldots$.

By (A1) and (9) with $j=j+1$, we obtain

$$\begin{aligned} \begin{aligned}&2a_j(p_j-p_{j+1})+c_j^2(p_j-p_{j+1}) \\&\quad =- (r+d^2p_j) (k_{j_1}-k_{j})^2, \end{aligned} \end{aligned}$$

(A4)

followed with the obtained result that $k_j$ is a stabilizer and $(r+d^2p_j)(k_{j_1}-k_{j})^2\ge 0$, which implies $p_{j}\le p_{j+1}$, $j=1,2,\ldots$.

Therefore, $\{p_j\}_1^{\infty }$ is monotonically decreasing sequence and has a lower bound. Combing with that (p, k) satisfies (9), it can be concluded that $\lim _{j\rightarrow \infty }p_j=p$.

Appendix B: Proof of Lemma 3

Convergence result of $k_s^j$ follows with convergence of $p_j$ by (11). To complete the proof, the rest is to show that $\lim _{j\rightarrow \infty }p_j=p$. We show the result by mathematical induction.

If $j=1$, as $a-bk<0$ and $k_s^0=0$, ${\tilde{a}}_1<0$. One has

$$\begin{aligned} 2(a-bk-bk_s)s+\upsilon k_s^2-q=0. \end{aligned}$$

(B5)

We subtract the above equation from (12) with $j=1$ to obtain

$$\begin{aligned} 2{\tilde{a}}_0(s_1-s)=-\upsilon k_s^2, \end{aligned}$$

(B6)

which yields $s_1\ge s$ as $\upsilon >0$.

By (B6) and (11), we obtain

$$\begin{aligned} 2{\tilde{a}}_1(s_1-s)=-\upsilon (k_s^1)^2-\upsilon (k_s-k_s^1)^2, \end{aligned}$$

(B7)

combining with $\upsilon (k_s^1)^2+\upsilon (k_s-k_s^1)^2>0$, which yields ${\tilde{a}}_1<0$.

If $j>1$, we suppose that ${\tilde{a}}_{j-1}$ is Hurwitz. Subtracting (B5) from (12) yields

$$\begin{aligned} 2{\tilde{a}}_{j-1}(s_j-s)=-\upsilon (k_s-k_s^{j-1})^2. \end{aligned}$$

(B8)

By the induction assumption, we known that $s_j\ge s$.

The above equation can be changed into

$$\begin{aligned} 2{\tilde{a}}_{j}(s_j-s)=-\upsilon (k_s^j-k_s^{j-1})^2-\upsilon (k_s-k_s^j)^2, \end{aligned}$$

(B9)

which yields ${\tilde{a}}_{j}<0$.

By (11)–(12), we can derive

$$\begin{aligned} 2{\tilde{a}}_{j}(s_{j+1}-s_j)=-(k_s^j-k_s^{j-1})^2, \end{aligned}$$

(B10)

combining with the obtained result, which implies $s_{j+1}$ $\le s_j$.

Hence, $\{s_j\}_1^{\infty }$ is monotonically decreasing sequence and has a lower bound s, leading to $\lim \limits _{j\rightarrow \infty }s_j=s$.