An actor-critic reinforcement learning-based resource management in mobile edge computing systems

Abstract

Reinforcement learning (RL) as an effective tool has attracted great attention in wireless communication field nowadays. In this paper, we investigate the offloading decision and resource allocation problem in mobile edge computing (MEC) systems based on RL methods. Different from existing literature, our research focuses on improving mobile operators’ revenue by maximizing the amount of the offloaded tasks while decreasing the energy expenditure and time-delays. Considering the dynamic characteristics of wireless environment, the above problem is modeled as a Markov decision process (MDP). Since the action space of the MDP is multidimensional continuous variables mixed with discrete variables, traditional RL algorithms are powerless. Therefore, an actor-critic (AC) with eligibility traces algorithm is proposed to resolve the problem. The actor part introduces the parameterized normal distribution to generate the probabilities of continuous stochastic actions, and the critic part employs a linear approximator to estimate the value of states, based on which the actor part updates policy parameters in the direction of performance improvement. Furthermore, an advantage function is designed to reduce the variance of the learning process. Simulation results indicate that the proposed algorithm can find the best strategy to maximize the amount of the tasks executed by the MEC server while decreasing the energy consumption and time-delays.

Appendix 1

In the following, we discuss the obtained solution by the actor-critic algorithm is global optimal. First, we discuss the quasi-concavity of the utility function in two cases.

Case 1:${T_n}(t) + {c_n}(t){\varLambda _n}(t) > D_n^{\max }(\varDelta t) = \varDelta t{f_n}(t)L_e^{ - 1}$, we have ${D_n}(t) = D_n^{\max }(\varDelta t)$ and ${H_n}(t) = {T_n}(t) + {c_n}(t){\varLambda _n}(t) - D_n^{\max }(\varDelta t)$. The utility function is rewritten as,

$$\begin{aligned} {\varOmega _n}(t) = \varTheta _n {(t)} + {\varUpsilon _n}(t)+{\varXi _n(t)}, \end{aligned}$$

(45)

where $\varTheta _n {(t)}= ({\rho _n} - {\upsilon _n}\xi f_n^2(t){L_e} - {\omega _n}){D_n}(t)$, ${\varUpsilon _n}(t) = -{{\upsilon _n}D_n^{dl}(t)(1 - {b_n}(t))}[{p_n}(t) + p_n^{cir}]/{R_n}(t)$, and ${\varXi _n}(t) = -{\omega _n}\{ {T_n}(t) + {c_n}(t){\varLambda _n}(t)\}$.

Definition 1

A function $\mathcal F$, which maps from a convex set of real n-dimensional vectors $\mathrm I$ to a real number, is call strictly quasi-concave, if for any $x_1$, $x_2$$\in \mathrm{I}$, and ${x_1} \ne {x_2}$, $\mathcal F(\lambda {x_1} + (1 - \lambda ){x_2}) \ge \min \{ \mathcal F({x_1}),\mathcal F({x_2})\} , \forall \lambda ,0 \le \lambda \le 1$ [31].

Based on the proposition C.9 in [31], ${\varUpsilon _n}(t)$ is strictly quasi-concave if and only if the upper contour sets

$$\begin{aligned} UC({\varUpsilon _n}(t),y) = \{ p_n(t) \in \mathrm I|{\varUpsilon _n}(t)\ge y\}, \end{aligned}$$

(46)

are convex for all $y\in R$. If $y \ge 0$, the upper contour sets are empty; if $y<0$, (46) is equivalent to $UC({\varUpsilon _n}(t),y) = \{ {p_n}(t) > 0|y{R_n}(t) + {\upsilon _n}D_n^{dl}(t)(1 - {b_n}(t))[{p_n}(t) + p_n^{cir}] < 0\}$. Since $R_n(t)$ is concave in ${p_n}(t)$, the function, ${\mathcal F^*} = y{R_n}(t) + {\upsilon _n}D_n^{dl}(t)(1 - {b_n}(t))[{p_n}(t) + p_n^{cir}]$ is also concave. Therefore, the upper contour set $UC({\varUpsilon _n}(t),y)$ is convex. Based on the above analysis, ${\varUpsilon _n}(t)$ is strictly quasi-concave. Obviously, $\varTheta _n {(t)}$ and ${\varXi _n}(t)$ are strictly concave. Therefore, $\varOmega _n (t)$ is strictly quasi-concave.

Case 2: i.e., ${T_n}(t) + {c_n}(t){\varLambda _n}(t) \le D_n^{\max }(\varDelta t)$, we have ${D_n}(t) = {T_n}(t) + {c_n}(t){\varLambda _n}(t)$ and ${H_n}(t) = 0$ which means no tasks left in the buffer at the tth time slot, the utility function is rewritten as,

$$\begin{aligned} {\varOmega _n}(t) = {{\widehat{\varTheta }}}_n {(t)} + {{{\widehat{\varUpsilon }}} _n}(t), \end{aligned}$$

(47)

where ${{{\widehat{\varTheta }}} _n}(t) = {({\rho _n} - {\upsilon _n}\xi f_n^2(t){L_e})}D_n(t)$ and ${{{\widehat{\varUpsilon }}} _n}(t) = {\varUpsilon _n}(t)$. Similarly, it is easy to proof that $\varOmega _n (t)$ is strictly quasi-concave.

According to Theorem 2, the policy gradient-based actor-critic RL algorithm of this study is convergent. The actor part utilizes a parameterized Gaussian distribution to generate stochastic actions, and a local optimal policy can be obtained by updating the parameters with the stochastic gradient method [24]. Since the utility function $\varOmega _n (t)$ is strictly quasi-concave as analyzed above, according to [18, 31], the local optimal solution of strictly quasiconcave functions equals to the global optimal solution.

Therefore, the obtained solution by the RL algorithm is global optimal.