Interpreting Decision Process in Offline Reinforcement Learning for Interactive Recommendation Systems

Abstract

Recommendation systems, which predict relevant and appealing items for users on web platforms, often rely on static user interests, resulting in limited interactivity and adaptability. Reinforcement Learning (RL), while providing a dynamic and adaptive approach, brings its unique challenges in this context. Interpreting the behavior of an RL agent within recommendation systems is complex due to factors such as the vast and continuously evolving state and action spaces, non-stationary user preferences, and implicit, delayed rewards often associated with long-term user satisfaction.

Addressing the inherent complexities of applying RL in recommendation systems, we propose a framework that includes innovative metrics and a synthetic environment. The metrics aim to assess the real-time adaptability of an RL agent to dynamic user preferences. We apply this framework to LastFM datasets to interpret metric outcomes and test hypotheses regarding MDP setups and algorithm choices by adjusting dataset parameters within the synthetic environment. This approach illustrates potential applications of our framework, while highlighting the necessity for further research in this area.

Appendices

Classic Recommendation Algorithms

Recommendation systems have been extensively studied, and several classic algorithms have emerged as popular approaches for generating recommendations. In this section, we provide a brief overview of some of these algorithms, such as matrix factorization and other notable examples.

1.1 Algorithms

Matrix factorization is a widely used technique for collaborative filtering in recommendation systems. The basic idea is to decompose the user-item interaction matrix into two lower-dimensional matrices, representing latent factors for users and items. The interaction between users and items can then be approximated by the product of these latent factors. Singular Value Decomposition (SVD) and Alternating Least Squares (ALS) are common methods for performing matrix factorization. The objective function for matrix factorization can be written as:

$$\begin{aligned} \min _{U, V} \sum _{(i, j) \in \Omega } (R_{ij} - U_i^T V_j)^2 + \lambda (||U_i||^2 + ||V_j||^2), \end{aligned}$$

(7)

where $R_{ij}$ is the observed interaction between user i and item j, $U_i$ and $V_j$ are the latent factors for user i and item j, respectively, $\Omega$ is the set of observed user-item interactions, and $\lambda$ is a regularization parameter to prevent overfitting.

Besides matrix factorization, other classic recommendation algorithms include:

• User-based Collaborative Filtering: This approach finds users who are similar to the target user and recommends items that these similar users have liked or interacted with. The similarity between users can be computed using metrics such as Pearson correlation or cosine similarity.

• Item-based Collaborative Filtering: Instead of focusing on user similarity, this method computes the similarity between items and recommends items that are similar to those the target user has liked or interacted with.

• Content-based Filtering: This approach utilizes features of items and user profiles to generate recommendations, assuming that users are more likely to be interested in items that are similar to their previous interactions.

1.2 Evaluation Metrics for Recommendation Algorithms

Several evaluation metrics are commonly used to assess the performance of recommendation algorithms. We provide a brief description with equations for the metrics, which we use in our work.

Normalized Discounted Cumulative Gain (NDCG) used for measuring the effectiveness of ranking algorithms, takes into account the position of relevant items in the ranked list. First, DCG is calculated:

$$\begin{aligned} \text {DCG}@k = \sum _{i=1}^{k} \frac{\text {rel}_i}{\log _2(i+1)}, \end{aligned}$$

(8)

where k is the number of top recommendations considered, and $\text {rel}_i$ is the relevance score of the item at position i in the ranked list. Then DCG value is normalized with the ideal DCG (IDCG), which represents the highest possible DCG value:

$$\begin{aligned} \text {NDCG}@k = \frac{\text {DCG}@k}{\text {IDCG}@k}. \end{aligned}$$

(9)

Coverage measures the fraction of recommended items to the total number of items:

$$\begin{aligned} \text {Coverage} = \frac{|\text {Recommended Items}|}{|\text {Total Items}|}. \end{aligned}$$

(10)

High coverage indicates that the algorithm can recommend a diverse set of items, while low coverage implies that it is limited to a narrow subset. Hit Rate calculates the proportion of relevant recommendations out of the total recommendations provided:

$$\begin{aligned} \text {Hit Rate} = \frac{\text {Number of Hits}}{\text {Total Number of Recommendations}}, \end{aligned}$$

(11)

where a “hit” occurs when a recommended item is considered relevant or of interest to the user. A higher hit rate indicates better performance.

Fig. 5.

The results of an experiment in a highly dynamic environment when trained on suboptimal data.

Experiments in Dynamic Environment

Experiments in a high-dynamic environment for agents trained on optimal and sub-optimal data. Preference Correlation is the best metric for evaluating lower-quality datasets with negative reviews, as it assesses the agent’s understanding of user needs. However, high values can occur if the dataset has many positive ratings and the agent’s coverage is low. For higher-quality datasets with positive responses, the I-HitRate metric correlates with online evaluations but is sensitive to the agent’s coverage.

Fig. 6.

The results of an experiment in a highly dynamic environment when trained on optimal data.

Reinforcement Learning as an MDP Problem

Reinforcement learning (RL) addresses the problem of learning optimal behaviors by interacting with an environment. A fundamental concept in RL is the Markov Decision Process (MDP), which models the decision-making problem as a tuple $(S, A, P, R, \gamma )$. In this framework, S represents the state space, A is the action space, P is the state transition probability function, R denotes the reward function, and $\gamma$ is the discount factor. By formulating the recommendation problem as an MDP, RL algorithms can learn to make decisions that optimize long-term rewards. The MDP framework provides a solid foundation for designing and evaluating RL agents in recommendation systems, allowing for the development of more adaptive and effective algorithms.

Comparing Algorithms Behavior on High Dynamic Environment

Table 1. Comparison of CQL, SAC, BC, Dert4Rec algorithms in an environment with high dynamics of user mood changes.