Pareto Deterministic Policy Gradients and Its Application in 6G Networks

Abstract

In this chapter, we introduce a reinforcement learning (RL)-based approach to jointly optimize cell load balance and network throughput as a potential AI/ML-based use case for sixth-generation cellular systems (6G), where inter-cell handover and massive MIMO antenna tilting are configured as the RL policy to learn. Our rationale behind using RL is to circumvent the challenges of analytically modeling user mobility and network dynamics. We integrate vector rewards into multiple value networks and conduct RL action via a separate policy network. We name this method as Pareto deterministic policy gradients (PDPG). It is an actor-critic, model-free, and deterministic policy algorithm which can handle the coupling objectives with the following two merits: (1) It solves the optimization via leveraging the degree of freedom of vector reward as opposed to choosing handcrafted scalar reward; (2) cross-validation over multiple policies can be significantly reduced. To be self-contained, an ideal static optimization-based brute-force search solver is included as the benchmark method. The comparison shows that the RL approach performs as well as this ideal strategy, though the former one is constrained with limited environment observations and lower action frequency, whereas the latter one has full access to the user mobility.

A Comparison: A Static Formulation

We now consider a static formulation of the joint optimization on load balancing and throughput maximization. To do so, we directly drop the expectation and time constraint in (11) and set $\gamma = 0$. Therefore, we have the following formulation:

$$\displaystyle \begin{aligned} {} \begin{aligned} &\max_{{\boldsymbol I}(t), {\boldsymbol b}(t) } \sum_n R_n(t) + \sum_n F_n(t)\\ &s.t. \quad \boldsymbol{I}(t)=\{I_{n, k}(t): I_{n,k}(t) \in \{0, 1\}, n \in N, k \in K\}\\ &\qquad {\boldsymbol b}(t) = \{b_{n}(t):b_{n}(t) \in \{\theta_0, \theta_1, \cdots, \theta_{M-1} \}, n \in N\}.\\ \end{aligned} \end{aligned} $$

(19)

Comparing the above formulation to (11), we can notice the following difference and practical limitations:

• To solve this static problem, it requires perfect knowledge of all ${p_{n,k}(b_n(t))}$ at every sample time which imposes a large overhead to the user feedback.

• Due to the integer constraints, the complexity becomes very high. Moreover, it requires BSs to solve out ${\boldsymbol I}(t)$ and ${\boldsymbol b}(t)$ on every time slot.

• The formulation treats user association as a variable. However, the user association cannot be directly translated to the values of the CIOs in A3 events. Therefore, it is not compatible to the handover operations in current cellular systems.

Therefore, we only consider the above formulation as an evaluation approach to our RL algorithm in the simulation. Ideally, the static way can yield the optimal solution at every time. Thus, it can serve as an upper bound for the RL algorithm.

Algorithm 1 Relaxed brute force for a small λ

1.1 A.1 Heuristic Brute-Force Solvers

Note that (19) can be equivalently written as

$$\displaystyle \begin{aligned} {} \begin{aligned} &\max_{{\boldsymbol I}(t), {\boldsymbol b}(t) } \sum_n F_n({\boldsymbol I}(t), {\boldsymbol b}(t))\\ &s.t. \quad \boldsymbol{I}(t)=\{I_{n, k}(t): I_{n,k}(t) \in \{0, 1\}, n \in N, k \in K\}\\ &\qquad {\boldsymbol b}(t) = \{b_{n}(t):b_{n}(t) \in \{\theta_0, \theta_1, \cdots, \theta_{M-1} \}, n \in N\}\\ &\qquad R_n({\boldsymbol I}(t), {\boldsymbol b}(t)) > \phi, n \in N \end{aligned} \end{aligned} $$

(20)

where $\phi$ is a parameter to avoid trivial solutions, such as all users are disconnected to BSs (in this case, all cell loads are zero). Since we do not have an analytical expression of the objective, brute-force search is considered as our primary approach. However, the number of user association combinations is prohibitively large. To narrow down the searching region, we consider a heuristic approach to approximate

$$\displaystyle \begin{aligned} \max_{{\boldsymbol I}(t) } \sum_n F_n({\boldsymbol I}(t), {\boldsymbol b}(t)) \end{aligned} $$

(21)

for a given ${\boldsymbol b}(t)$. We consider determining the user association in a round-robin manner: Each BS is allocated with an equal number of associated users, where the associated user for each BS is based on the ranking of user rates. Intuitively, this can be considered as a heuristic way to average the throughput $R_n({\boldsymbol I}(t), {\boldsymbol b}(t))$ over all cells. Accordingly, the algorithm is summarized in Algorithm 1.

Algorithm 2 Relaxed brute-force for a fair λ

Alternatively, the inner loop for user association assignment can be considered as solving

$$\displaystyle \begin{aligned} \max_{{\boldsymbol I}(t) } \sum_n R_n({\boldsymbol I}(t), {\boldsymbol b}(t)) \end{aligned} $$

(22)

A heuristic approach is assigning users to the BS with maximum transmission power. This association strategy may break the cell load balance but avoid user failure links. Therefore, we can combine the previous two heuristic association strategies and obtain Algorithm 2. Particularly, the two user association strategies are mixed through a random binary decision, where the decision threshold is proportional to the weight ratio between the two objectives, i.e., ${\lambda \over {1+\lambda }}$.