Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling

Abstract

There is a growing interest in using reinforcement learning (RL) to personalize sequences of treatments in digital health to support users in adopting healthier behaviors. Such sequential decision-making problems involve decisions about when to treat and how to treat based on the user’s context (e.g., prior activity level, location, etc.). Online RL is a promising data-driven approach for this problem as it learns based on each user’s historical responses and uses that knowledge to personalize these decisions. However, to decide whether the RL algorithm should be included in an “optimized” intervention for real-world deployment, we must assess the data evidence indicating that the RL algorithm is actually personalizing the treatments to its users. Due to the stochasticity in the RL algorithm, one may get a false impression that it is learning in certain states and using this learning to provide specific treatments. We use a working definition of personalization and introduce a resampling-based methodology for investigating whether the personalization exhibited by the RL algorithm is an artifact of the RL algorithm stochasticity. We illustrate our methodology with a case study by analyzing the data from a physical activity clinical trial called HeartSteps, which included the use of an online RL algorithm. We demonstrate how our approach enhances data-driven truth-in-advertising of algorithm personalization both across all users as well as within specific users in the study.

Appendices

Details on the RL framework for HeartSteps

We now provide further details regarding the RL framework for the HeartSteps study discussed in Sect. 3.2 (Table 1).

Table 1 Description of state features used in the HeartSteps study

Clipping probabilities

The function h appearing in (10) is given by

$$\begin{aligned} h(p) \triangleq \textrm{min}\{ 0.8, 0.2+ \frac{0.8}{0.5} \cdot \textrm{max}\{ p-0.5,0 \} \} \quad \text {for}\quad p \in [0, 1]. \end{aligned}$$

(13)

Prior and posterior formulation

Using the notation $\theta ^\top \triangleq (\alpha _0^\top , \alpha _1^\top , \beta ^\top )$, the prior for $\theta$ was specified as $\mathcal {N}(\mu _0, \Sigma _0)$, where

$$\begin{aligned} \overline{\mu }_0 = \begin{bmatrix} \mu _{\alpha _0} \\ \mu _{\beta } \\ \mu _{\beta }\end{bmatrix} \quad \text {and}\quad \overline{\Sigma }_0 = \begin{bmatrix} \Sigma _{\alpha _0} &{}&{} \\ {} &{} \Sigma _{\beta } &{}\\ {} &{} &{}\Sigma _{\beta } \end{bmatrix} \end{aligned}$$

(14)

and $\{ \mu _{\alpha _0}, \mu _\beta , \Sigma _{\alpha _0}, \Sigma _{\beta }\}$ were computed from the prior study (see Liao et al. 2020, Sect. 6) for details on how priors were constructed and (17) for the specific values used by us; note that the HeartSteps team decided to update the priors used from those those presented in Liao et al. (2020)). Given the Gaussian prior and the Gaussian working model, the posterior for $\theta$ on the day d is also Gaussian and is given by $\mathcal {N}(\overline{\mu }_d, \overline{\Sigma }_d)$, where these posterior parameters are recursively updated as

$$\begin{aligned} \overline{\Sigma }_{d}&= \left( \frac{1}{\sigma ^2} \sum _{t=5(d-1)}^{5d} I_t \phi (S_t, A_t) \phi (S_t,A_t)^\top + \overline{\Sigma }_{d-1}^{-1} \right) ^{-1}, \quad \text {and}\quad \end{aligned}$$

(15a)

$$\begin{aligned} \overline{\mu }_{d}&= \overline{\Sigma }_d\left( \frac{1}{\sigma ^2} \sum _{t=5(d-1)}^{5d} I_t \phi (S_t, A_t) R_t + \overline{\Sigma }_{d-1}^{-1} \overline{\mu }_{d-1} \right) \end{aligned}$$

(15b)

where $\phi (S_t, A_t)^\top \triangleq [g(S_t)^\top , \pi _t f(S_t)^\top , (A_t-\pi _t)f(S_t)^\top ]$ collects all the feature vectors from the working model (8). The updates (15a) and (15b) are denoted by PosteriorUpdate in Algorithm 2. For k-dimensional $\beta$, the posterior parameters $\mu _{d, \beta }, \Sigma _{d, \beta }$ for $\beta$ are respectively given by the last k entries of $\overline{\mu }_d$ and $k\times k$ sub-matrix formed by taking the last k columns and rows of $\overline{\Sigma }_d$.

Model estimates used by ParaSim for resampling trajectories

For a user trajectory $(S_{t}, A_{t}, R_{t})_{t= 1}^{T}$ from the reward model (8), we estimate the parameters $(\alpha , \beta )$ using the updates (15) albeit without action centering. Hence the estimates $(\hat{\alpha }_T ^\top , \hat{\beta }_T^\top )$ (that inform the model parameters used by ParaSim after suitable modifications in Sect. 3.3) are given by

$$\begin{aligned} \begin{bmatrix} \hat{\alpha }_{T} \\ \hat{\beta }_{T} \end{bmatrix}&= \left( \frac{1}{\sigma ^2} \sum _{t=1}^{T} I_t \widetilde{\phi }(S_t, A_t) \widetilde{\phi }(S_t,A_t)^\top + \begin{bmatrix} \Sigma _{\alpha _0} &{}&{} \\ {} &{} \Sigma _{\beta } &{} \end{bmatrix} ^{-1}\right) ^{-1}\nonumber \\&\quad \left( \frac{1}{\sigma ^2} \sum _{t=1 }^{T} I_t \widetilde{\phi }(S_t, A_t) R_t \!+\! \begin{bmatrix} \Sigma _{\alpha _0} &{}&{} \\ {} &{} \Sigma _{\beta } &{} \end{bmatrix} ^{-1}\begin{bmatrix} \mu _{\alpha _0} \\ \mu _{\beta } \end{bmatrix} \right) \end{aligned}$$

(16)

where $\widetilde{\phi }(S_t, A_t)^\top \triangleq \left[ g(S_t)^\top , A_t f(S_t)^\top \right]$.

Priors means and variances

We now summarize the exact values of prior parameters used by the RL algorithm. For $\ell \in \mathbb {N}$, let $\textrm{diag}(a_1, \ldots , a_{\ell }) \in \mathbb {R}^{\ell \times \ell }$ denote an $\ell \times \ell$ diagonal matrix with its j-th diagonal entry equal to $a_j$ for $j=1, \ldots , \ell$. Then the mean and variance parameters used in (14) are given by

$$\begin{aligned} \begin{aligned} \mu _{\alpha _0}&= [0.82, 1.95, 3.81, -0.19, 0.76, 0.0, -0.92, 0.0]^\top \in \mathbb {R}^{7}, \\ \mu _{\beta }&= [0.47, 0.0, 0.0, 0.0, 0.0]^\top \in \mathbb {R}^{5}, \\ \Sigma _{\alpha _0}&= \textrm{diag}(14.24, 13.35, 3.24, 0.57, 19.00, 0.26, 17.00, 7.35) \in \mathbb {R}^{7\times 7}, \quad \text {and}\quad \\ \Sigma _{\beta }&= \textrm{diag}(4.93, 24.56, 4.95, 0.67, 0.82) \in \mathbb {R}^{5\times 5}, \end{aligned} \end{aligned}$$

(17)

where the features of $\alpha _0$ are ordered as (Intercept, temperature, prior 30 min step count, yesterday step count, $\texttt {dosage}$, $\texttt {engagement}$, $\texttt {location}$, $\texttt {variation}$) and that for $\beta$ and $\alpha _1$ are ordered as (Intercept, $\texttt {dosage}$, $\texttt {engagement}$, $\texttt {location}$, $\texttt {variation}$).

Details on $\texttt {Score\_int}_{}$ computation for HeartSteps

We now describe the smoothened version of $\texttt {Score\_int}_{1}$ (1) and $\texttt {Score\_int}_{2,\textsf{z}}$ (3) that we use to add stability in our HeartSteps results in Sects. 3.4 and 3.5.

At a high level, we use the following steps: (i) We use moving windows to average out the advantage forecasts and use an averaged forecast on a daily scale, i.e., for $d=\lceil {(t-1)/5}\rceil$ and not for each decision time $t\in [T]$ as in (1) and (3). (iii) While computing the interestingness scores, we omit days when the quality of data is not good due to low availability or low diversity of features. (iii) We compute the forecasts without changing any state feature from the observed data other than $\texttt {dosage}$. That is, if the observed feature value $\textsf{z} _t=1$, we do not compute a counterfactual forecast by artificially forcing $\textsf{z} _t = 0$. (iv) Finally, we do not consider a user’s interestingness score if they have only a few days of good data.

Fig. 10

Histogram of fraction of good days for the users in the original data for HeartSteps. In panel (a), the count on the vertical axis represents the number of users with the value of $\frac{1}{D} \sum _{d=1}^{D} G_{d,1 }$ on the horizontal axis. In panels (b) to (d), the count on the vertical axis represents the number of users with the value of $\frac{1}{D} \sum _{d=1}^{D} G_{d,2,\textsf{z}}$ on the horizontal axis, respectively for $\textsf{z} \in \{ \texttt {variation}, \texttt {location}, \texttt {engagement} \}$

We now describe these steps in detail for a user with total T decision times in their data trajectory (with total $D\triangleq \lfloor {T/5}\rfloor$ days of data). We note that total decision times for each user might vary.

• 1. Sliding window: For each day $d \in \{ 1, \ldots , D\}$, we define a sliding window $W_d$ using all 5 decision times on day d when computing $\texttt {Score\_int}_{1}$ and all 5 decision times on days $\{ d-1, d, d+1\}$ (total 15 decision times) when computing $\texttt {Score\_int}_{2, \textsf{z}}$. That is,

  $$\begin{aligned} W_d \triangleq {\left\{ \begin{array}{ll} \{ 5(d-1), 5(d-1)+1, \ldots , 5d\} \cap [T] &{}\quad \text {for}\quad \texttt {Score\_int}_{1}, \\ \{ 5(d-1)+1, 5(d-1)+2, \ldots , 5(d+1)\} \cap [T] &{}\quad \text {for}\quad \texttt {Score\_int}_{2, \textsf{z}}. \end{array}\right. } \end{aligned}$$

• 2. Characterizing good data day: Next, when considering $\texttt {Score\_int}_{1}$, we define an indicator variable $G_{d, 1}$ to denote a good day. It is set to 1 if the following two conditions hold: (a) if the user was available for at least 2 decision times in $W_d$, i.e., $\sum _{t\in W_d} I_t \ge 2$ and (b) the RL algorithm posterior was updated on the night of day $d-1$; we impose this additional constraint to deal with the real-time and missing data update issues. We set $G_{d, 1}=0$ in all other cases. When considering $\texttt {Score\_int}_{2, \textsf{z}}$, we define the variable $G_{d, 2, \textsf{z}}$ ta denote a good data day based on whether the user’s observed states exhibit enough diversity in the value of the variable $\textsf{z}$ for the decision times in $W_d$. In particular, we set $G_{d, \textsf{z}}=1$ when the following two conditions hold: (a) the feature $\textsf{z}$ takes values 1 and 0 at least twice out of the decisions times in $W_d$ when the user was available for randomization ($I_t=1$), i.e.,

  $$\begin{aligned} \sum _{t\in W_d} I_t \textsf{z} _t \ge 2 \quad \text {and}\quad \sum _{t\in W_d} I_t (1-\textsf{z} _t) \ge 2, \end{aligned}$$

  where $\textsf{z} _t$ denotes the value of the variable $\textsf{z}$ for the user at decision time t, and (b) the RL algorithm posterior was updated on the night of at least one of the days in $\{ d-1, d, d+1\}$. In all other cases, we set $G_{d,\textsf{z}}=0$.

• 3. Interestingness score for a user trajectory: We consider a user for interestingness only if the fraction of good days is greater than a certain threshold, i.e.,

  $$\begin{aligned} \frac{1}{D} \sum _{d=1}^{D} G_{d,1 } \ge (1-\gamma ) \ \text {for}\ \texttt {Score\_int}_{1} \quad \text {or}\quad \frac{1}{D} \sum _{d=1}^{D} G_{d, 2, \textsf{z}} \ge (1-\gamma ) \ \text {for}\ \texttt {Score\_int}_{2, \textsf{z}}, \end{aligned}$$

  (18)

  for a suitable $\gamma \in (0, 1)$. (Note that increasing the value of $\gamma$ lowers the cutoff for a user to become eligible for being considered for interestingness.) For such a user, we define the interestingness scores as follows:

  $$\begin{aligned} \texttt {Score\_int}_{1} (\mathcal {U})&\triangleq \frac{1}{\sum _{d=1}^{D} G_{d, 1}} \sum _{d=1}^{D} G_{d, 1} \varvec{1}\left( \frac{\sum _{t\in W_d} I_t \hat{\Delta }_t(S_t) }{\sum _{t\in W_d}I_t}> 0 \right) \\ \texttt {Score\_int}_{2, \textsf{z}} (\mathcal {U})&\triangleq \frac{1}{\sum _{d=1}^{D} G_{d, 2, \textsf{z}}} \sum _{d=1}^{D} G_{d, 2, \textsf{z}} \varvec{1}\left( \frac{\sum _{t\in W_d} I_t \textsf{z} _t \hat{\Delta }_t(S_t) }{\sum _{t\in W_d}I_t \textsf{z} _t} > \frac{\sum _{t\in W_d}I_t (1-\textsf{z} _t) \hat{\Delta }_t(S_t) }{\sum _{t\in W_d}I_t (1-\textsf{z} _t)} \right) , \end{aligned}$$

  where we multiply by indicators $G_{d, 1}$ and $G_{d, 2, \textsf{z}}$ to include only “good days” in our score computations. Note that $\frac{\sum _{t\in W_d} I_t \textsf{z} _t \hat{\Delta }_t(S_t) }{\sum _{t\in W_d}I_t \textsf{z} _t}$ is the stable proxy (without counterfactual imputation) for the quantity $\hat{\Delta }_{t}(S_t(\textsf{z} =1))$ in (3).

Remark 3

Note that we omit all users from our results who do not satisfy the good day requirement (18). The number of omitted users depends on the value of $\gamma$; see Fig. 10 for the histogram of $\sum _{d}G_{d, 1}/D$ and $\sum _{d}G_{d, 2, \textsf{z}}/D$ across the 91 users in HeartSteps. For the results in Sects. 3.4 and 3.5, we use $\gamma =0.4$, that allows 63, 60, 12, and 43 to be considered respectively for interestingness of type 1, and of type 2 for feature $\texttt {variation}$, $\texttt {location}$, and $\texttt {engagement}$.

Another look at user 2’s advantage forecasts

Figure 11a reproduces the advantage forecasts for user 2 from Fig. 1b. In addition, panels (b) and (c) of Fig. 11 show the analogs of the panel (a), where user 2’s standardized advantage forecasts are color-coded based on the values of $\texttt {location}$ and $\texttt {engagement}$ respectively. Overall, we observe from the three panels in Fig. 11 that user 2 does not appear interesting of type 2 for $\textsf{z} \in \{ \texttt {location}, \texttt {engagement} \}$ since the standardized advantage forecasts are not well separated when $\textsf{z} = 0$ versus $\textsf{z} = 1$ like that for $\textsf{z} = \texttt {variation}$ in panel (a) (or equivalently in Fig. 1b). In particular, for this user, we have $\texttt {Score\_int}_{2, \texttt {location}} = 0.38$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.38$, while $\texttt {Score\_int}_{2,\texttt {variation}} =0$.

Fig. 11

User 2’s standardized advantage forecasts from Fig. 1b, color-coded by values of $\textsf{z}$ = $\texttt {variation}$, $\texttt {location}$, and $\texttt {engagement}$ in panels (a), (b), and (c) respectively. The value on the vertical axis represents the RL algorithm’s forecast of the standardized advantage of sending an activity message for the user if the user was available for sending a message on the day marked on the horizontal axis. (Note each day has 5 decision times.) The forecasts are marked as blue circles based on $\textsf{z}$ = 1 and red triangles if $\textsf{z}$ = 0 at the decision time. Panels (a) to (c) exhibit, respectively, $\texttt {Score\_int}_{2, \texttt {variation}} =0$, $\texttt {Score\_int}_{2, \texttt {location}} =0.38$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.38$. Note that we reproduced Fig. 1b in panel (a) for the reader’s convenience, and the three panels plot the same data and differ only in the color coding (Color figure online)

Deeper dive into interestingness of type 1 for HeartSteps

To further refine the conclusions from Fig. 3, we consider one-sided variants of the definition (2) for the number of interesting users. For reader’s convenience, we reproduce Fig. 3 in panel (a) Fig. 12, besides the corresponding results with one-sided interesting user counts, defined by counting the users with $\texttt {Score\_int}_{1} \ge 0.9$ and $\texttt {Score\_int}_{1} \le 0.1$ separately in panels (b) and (c).

Fig. 12

Histogram of the number of interesting users of type 1, $\texttt {\#User\_int}_{1}$ (2) and its one-sided variant across 500 trials. Here panel (a) with $\texttt {\#User\_int}_{1} \triangleq \sum _{i=1}^n \varvec{1}(|\texttt {Score\_int}_{1} -0.5|\ge 0.4)$ reproduces Fig. 3 for the reader’s convenience. In panels (b) and (c), the one-sided interesting users of type 1 are defined as $\texttt {\#User\_int}_{1} ^+ \triangleq \sum _{i=1}^n \varvec{1}(\texttt {Score\_int}_{1} \ge 0.9)$ and $\texttt {\#User\_int}_{1} ^- \triangleq \sum _{i=1}^n \varvec{1}(\texttt {Score\_int}_{1} \le 0.4)$ respectively. The trial data is the same as that in Fig. 3. That is, each is composed of 63 resampled trajectories. The trajectories are generated such that the true advantage is zero. The proportions on the vertical axis represent the fraction of the 500 trials, with the value of $\texttt {\#User\_int}_{1}, \texttt {\#User\_int}_{1} ^+,$ and $\texttt {\#User\_int}_{1} ^-$ on the horizontal axis. The vertical blue dashed line in each panel marks the corresponding interesting user count observed in the original data

From Fig. 12b, we find that in the original data 17 users exhibit $\texttt {Score\_int}_{1} \ge 0.9$; we denote this user count by $\texttt {\#User\_int}_{1} ^+$. However, the value of $\texttt {\#User\_int}_{1} ^+$ is always significantly smaller than 17 across the 500 trials with resampled trajectories. In Table 2, we denote this analysis as Type $1^+$.

Table 2 Summary of results from our resampling-based exploratory data analyses for HeartSteps data

On the other hand, Fig. 12c shows that one user exhibits $\texttt {Score\_int}_{1} \le 0.1$ in the original data; we denote this count by $\texttt {\#User\_int}_{1} ^-$. We also find that all 500 trials have $\texttt {\#User\_int}_{1} ^->1$. In Table 2, we denote this analysis as Type $1^-$.

Overall, we conclude that the data presents evidence in favor that the RL algorithm is potentially personalizing by learning that many users benefit from sending an activity message. However, many users might exhibit $\texttt {Score\_int}_{1} \le 0.1$ and it might appear that sending the message is less beneficial than not sending for these users, just due to algorithmic stochasticity. Consequently, the value of $\texttt {\#User\_int}_{1}$—the number of interesting users with $|\texttt {Score\_int}_{1}-0.5| \ge 0.4$, which is also equal to $\texttt {\#User\_int}_{1} ^+ + \texttt {\#User\_int}_{1} ^-$—can be as high as 18 (the observed value in the original data) due to algorithmic stochasticity.

Stability of conclusions with respect to the choice of $(\delta , \gamma )$

Next, we investigate the stability of the claims made above for $\texttt {Score\_int}_{1}$ and the one-sided variants to the choice of hyper-parameters $\delta$ and $\gamma$, appearing in (2) and (18), respectively. Note that for a given definition of $\texttt {Score\_int}_{}$, increasing $\gamma$ in (18) for a fixed $\delta$ in (2) allows more users to become eligible for being considered as interesting, both in original data and the resampled trials. Similarly, decreasing $\delta$ in (2) for a fixed $\gamma$ in (18) would typically lead to more number of interesting users, both in original data and the resampled trials.

The results of this exploration for the choices $\delta \in \{ 0.35, 0.40, 0.45\}$, and $\gamma \in \{ 0.65, 0.70, 0.75\}$ are presented in Fig. 14. For a given panel, the value in the cell corresponding to the value of $\delta$ on the horizontal axis and $\gamma$ on the vertical axis is equal to the fraction of the 500 trials for which the number of interesting users $\texttt {\#User\_int}_{}$ computed using those hyperparameter choices was as at least as large that in was at least as large as that in the original data. Looking at Fig. 14b, c, we find the conclusions drawn from Fig. 12b, c with $(\delta , \gamma )=(0.4, 0.75)$ about $\texttt {\#User\_int}_{1} ^+$ and $\texttt {\#User\_int}_{1} ^-$ remains stable even if we slightly perturb the values of $\delta$ and $\gamma$. In particular, across the $3\times 3$ choices for $\delta$ and $\gamma$, the value of $\texttt {\#User\_int}_{1} ^+$ would not appear as high as the observed value in the original data just by chance. On the other hand, the value of $\texttt {\#User\_int}_{1} ^-$ might appear higher than the observed value in the original data simply due to algorithmic stochasticity. Given the competing nature of these two quantities and the fact that $\texttt {\#User\_int}_{1} = \texttt {\#User\_int}_{1} ^+ + \texttt {\#User\_int}_{1} ^-$, the resulting fraction of trials with a count at least as high as $\texttt {\#User\_int}_{1}$ in the original data is quite sensitive to the particular choice of $(\delta , \gamma )$ as Fig. 13a illustrates.

Fig. 13

Stability of conclusions from Figs. 3 and 12 for interestingness of type 1 with respect to the choice of hyperparameters $(\delta , \gamma )$. Panels (a) to (c) respectively plot the results for the number of interesting users of type 1, $\texttt {\#User\_int}_{1} \triangleq \sum _{i=1}^n \varvec{1}(|\texttt {Score\_int}_{1} -0.5|\ge \delta )$, and its one-sided variants, namely $\texttt {\#User\_int}_{1} ^+\triangleq \sum _{i=1}^n \varvec{1}(\texttt {Score\_int}_{1} \ge 0.5+\delta )$, and $\texttt {\#User\_int}_{1} ^-\triangleq \sum _{i=1}^n \varvec{1}(\texttt {Score\_int}_{1} \le 0.5-\delta )$. For a given panel, the value in the cell corresponding to $\delta$ on the horizontal axis and $\gamma$ on the vertical axis is equal to the fraction of the 500 trials for which the number of interesting users was at least as large as that in the original data. Recall that the full histogram of $\texttt {\#User\_int}_{1},\texttt {\#User\_int}_{1} ^+,$ and $\texttt {\#User\_int}_{1} ^-$ across these 500 trials in Figs. 3 and 12 correspond to the choice of $\delta =0.4$ and $\gamma =0.75$

Stability of HeartSteps results for interestingness of type 2

We perform stability analysis for $\texttt {\#User\_int}_{2,\textsf{z}}$ with respect to the choice of $(\delta , \gamma )$ similarly to that done for $\texttt {\#User\_int}_{1}$ above in Fig. 13 and provide the results in Fig. 14.

Panels (a), (b), and (c) display the results, respectively, for $\textsf{z} = \texttt {variation}, \texttt {location}$, and $\texttt {engagement}$. In a given panel, the value in the cell corresponding to the value of $\delta$ on the horizontal axis and $\gamma$ on the vertical axis is equal to the fraction of the 500 trials for which the number of interesting users $\texttt {\#User\_int}_{2, \textsf{z}}$ computed using those hyperparameter choices was as at least as large that in the original data. Across the $3\times 3$ choices for $\delta$ and $\gamma$, we notice that for interestingness of type 2 for the features $\texttt {variation}$, $\texttt {location}$, and $\texttt {engagement}$, the fraction remains stable around 0, 0, and 1, same as the fraction in Fig. 7 for $(\delta , \gamma )=(0.4, 0.75)$.

Fig. 14

Stability of conclusions from Fig. 7 for interestingness of type 2 with respect to the choice of hyperparameters $(\delta , \gamma )$. Panels (a) to (c) respectively plot the results for $\texttt {\#User\_int}_{2,\textsf{z}}$ for interestingness of type 2 for feature $\textsf{z} \in \{ \texttt {variation}, \texttt {location}, \texttt {engagement} \}$. For a given panel, the value in the cell corresponding to $\delta$ on the horizontal axis and $\gamma$ on the vertical axis is equal to the fraction of the 500 trials for which the number of interesting users $\texttt {\#User\_int}_{2, \textsf{z}}$ was at least as large as that in the original data. Recall that the full histogram of $\texttt {\#User\_int}_{2,\textsf{z}}$ across these 500 trials in Fig. 7 corresponds to the choice of $\delta =0.4$ and $\gamma =0.75$

Interesting users of type 2 for $\texttt {location}$ and $\texttt {engagement}$

We now demonstrate the analysis (like in Figs. 1b and  8) for two different users, who exhibit potential interestingness of type 2 for $\texttt {location}$ and $\texttt {engagement}$.

A potentially interesting user of type 2 for $\texttt {location}$

Fig. 15 displays the advantage forecasts for a user, who we call user 3, to distinguish them from the two users associated with Fig. 1. The three panels in Fig. 15 plot user 3’s advantages color-coded by the value of the three features for that user. We find that this user admits $\texttt {Score\_int}_{2, \texttt {variation}} =0.68$, $\texttt {Score\_int}_{2, \texttt {location}} =0$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.43$—so that this user would be deemed potentially interesting of type 2 for $\texttt {location}$ (and not other features) as per our definition (4).

Fig. 15

Standardized advantage forecasts of user 3, an interesting user of type 2 for $\texttt {location}$, color-coded by $\textsf{z}$ = $\texttt {variation}$, $\texttt {location}$, and $\texttt {engagement}$ in panels (a), (b), and (c) respectively. The value on the vertical axis represents the RL algorithm’s forecast of the standardized advantage of sending an activity message for the user if the user was available for sending a message on the day marked on the horizontal axis. (Note each day has 5 decision times.) The forecasts are marked as blue circles based on $\textsf{z}$ = 1 and red triangles if $\textsf{z}$ = 0 at the decision time. Panels (a) to (c) exhibit, respectively, $\texttt {Score\_int}_{2, \texttt {variation}} =0.68$, $\texttt {Score\_int}_{2, \texttt {location}} =0$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.43$. Note that the three panels plot the same data and differ only in the color coding

Next, we evaluate how likely the user graph in Fig. 15b would appear just by chance. Panels (a) and (b) of Fig. 16 visualize two resampled trajectories of user 3 (chosen uniformly at random from user 3’s 500 resampled trajectories) generated under the generative model that there is no differential advantage of sending a message based on the value of $\texttt {location}$. The color coding is as in Fig. 15b, namely, the forecasts are marked in red triangles if $\texttt {location}$ = 1 and blue circles if $\texttt {location}$ = 0. In panel (c) of Fig. 16, we plot the histogram for the $\texttt {Score\_int}_{2, \texttt {location}}$ for this user across all 500 resampled trajectories and denote the observed value in the original data as a vertical dotted line.

Figure 16a, b show that the resampled trajectories do not appear interesting of type 2 for $\texttt {location}$ as in Fig. 15b; the two trajectories, respectively, have $\texttt {Score\_int}_{2,\texttt {variation}} =$ 0.50, and 0.42. Moreover, panel (c) shows that the interestingness score of 0, which was observed for user 3 in the original data (Fig. 15b), never appears across any of the resampled trajectories. Thus we can conclude that the data presents evidence that the RL algorithm potentially personalized for user 3 by learning to treat the user based on $\texttt {location}$ differentially, and this personalization would not likely arise simply due to algorithmic stochasticity.

Fig. 16

Resampling results for user 3 considered in Fig. 15, whose original trajectory exhibits interestingness of type 2 for $\texttt {location}$. Panels (a) and (b) plot two randomly chosen (out of 500) resampled trajectories generated with zero advantage; the two trajectories, respectively, have $\texttt {Score\_int}_{2,\texttt {location}} =$ 0.50, and 0.42. In panel (c), the vertical axis represents the fraction of the 500 resampled trajectories for this user with the value of $\texttt {Score\_int}_{2,\texttt {location}}$ on the horizontal axis; and the vertical blue dashed line marks the observed $\texttt {Score\_int}_{2,\texttt {location}}$ (value 0) for this user

A potentially interesting user of type 2 for $\texttt {engagement}$

Fig. 17 displays the advantage forecasts for a user, who we call user 4, to distinguish them from the three users associated with Figs. 1 and 15. The three panels in Fig. 17 plot user 4’s advantages color-coded by the three features; which admit $\texttt {Score\_int}_{2, \texttt {variation}} =0.65$, $\texttt {Score\_int}_{2, \texttt {location}} =0.9$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.037$, respectively. Thus based on our definition (4), this user is potentially interesting of type 2 for $\texttt {engagement}$, but not for $\texttt {variation}$. The user does not qualify the criterion ($\gamma = 0.75$ in (18)) for being considered as a potentially interesting user for $\texttt {location}$ due to a lack of diversity in the values taken by its $\texttt {location}$ feature.

Fig. 17

Standardized advantage forecasts of user 4, an interesting user of type 2 for $\texttt {engagement}$, color-coded by $\textsf{z}$ = $\texttt {variation}$, $\texttt {location}$, and $\texttt {engagement}$ in panels (a), (b), and (c) respectively. The value on the vertical axis represents the RL algorithm’s forecast of the standardized advantage of sending an activity message for the user if the user was available for sending a message on the day marked on the horizontal axis. (Note each day has 5 decision times.) The forecasts are marked as blue circles based on $\textsf{z}$ = 1 and red triangles if $\textsf{z}$ = 0 at the decision time. Panels (a) to (c) exhibit, respectively, $\texttt {Score\_int}_{2, \texttt {variation}} =0.65$, $\texttt {Score\_int}_{2, \texttt {location}} =0.9$, and $\texttt {Score\_int}_{2, \texttt {engagement}} =0.037$. Note that the three panels plot the same data and differ only in the color coding

Next, we evaluate how likely the user graph in Fig. 17c would appear just by chance. Panels (a) and (b) of Fig. 18 visualize two resampled trajectories of user 4 (chosen uniformly at random from user 4’s 500 resampled trajectories) generated under the generative model that there is no differential advantage of sending a message based on the value of $\texttt {engagement}$. The color coding is as in Fig. 17c, namely, the forecasts are marked in red triangles if $\texttt {engagement}$ = 1 and blue circles if $\texttt {engagement}$ = 0. In panel (c) of Fig. 18, we plot the histogram for the $\texttt {Score\_int}_{2, \texttt {engagement}}$ for this user across all 500 resampled trajectories and denote the observed value in the original data as a vertical dotted line.

Figure 18a, b show that the resampled trajectories do not appear interesting of type 2 for $\texttt {engagement}$ as in Fig. 17c; the two trajectories, respectively, have $\texttt {Score\_int}_{2,\texttt {engagement}} =$ 0.94, and 0.41. However, panel (c) shows that the interestingness score of 0.037, which was observed for user 4 in the original data (Fig. 17c), appears for around 20% of the resampled trajectories. Thus we can conclude that the data presents evidence that user 4’s interestingness score for $\texttt {engagement}$ might appear extreme simply due to algorithmic stochasticity.

Fig. 18

Resampling results for user 4 considered in Fig. 17, whose original trajectory exhibits interestingness of type 2 for $\texttt {engagement}$. Panels (a) and (b) plot two randomly chosen (out of 500) resampled trajectories generated with zero advantage; the two trajectories, respectively, have $\texttt {Score\_int}_{2,\texttt {engagement}} =$ 0.94, and 0.41. In panel (c), the vertical axis represents the fraction of the 500 resampled trajectories for this user with the value of $\texttt {Score\_int}_{2,\texttt {engagement}}$ on the horizontal axis; and the vertical blue dashed line marks the observed $\texttt {Score\_int}_{2,\texttt {engagement}}$ (value 0.037) for this user (Color figure online)