Q-Learning: Flexible Learning About Useful Utilities

Abstract

Dynamic treatment regimes are fast becoming an important part of medicine, with the corresponding change in emphasis from treatment of the disease to treatment of the individual patient. Because of the limited number of trials to evaluate personally tailored treatment sequences, inferring optimal treatment regimes from observational data has increased importance. Q-learning is a popular method for estimating the optimal treatment regime, originally in randomized trials but more recently also in observational data. Previous applications of Q-learning have largely been restricted to continuous utility end-points with linear relationships. This paper is the first attempt at both extending the framework to discrete utilities and implementing the modelling of covariates from linear to more flexible modelling using the generalized additive model (GAM) framework. Simulated data results show that the GAM adapted Q-learning typically outperforms Q-learning with linear models and other frequently-used methods based on propensity scores in terms of coverage and bias/MSE. This represents a promising step toward a more fully general Q-learning approach to estimating optimal dynamic treatment regimes.

Appendix: Derivation of the True Dynamic Regime Parameters

In the following, we calculate the true values of the first-interval decision rule parameters ψ ₁₀ and ψ ₁₁ in terms of γ’s and δ’s, the parameters of the generative model following the calculations in [2].

1.1 A.1 Bernoulli Utility

We begin with the derivations for the case of a Bernoulli utility. Let M=γ ₀+γ ₁ C ₁+γ ₂ O ₁+γ ₃ A ₁+γ ₄ O ₁ A ₁+γ ₅ C ₂+f ₁(C ₁)+f ₂(C ₂), and define μ=M+γ ₆ A ₂+γ ₇ O ₂ A ₂+γ ₈ A ₁ A ₂. It follows that

$$\begin{aligned} &\max_{a_2} Q_2(H_2, a_2) \\ &\quad{}= \operatorname{expit} \bigl(M + |\gamma_6 + \gamma_7O_2 + \gamma_8A_1| \bigr) \\ &\quad{}= \operatorname{expit} \biggl(M + \frac{1}{4}(1+O_2) (1+A_1)|f_1| + \frac{1}{4}(1+O_2) (1-A_1)|f_2| \\ &\qquad{} + \frac{1}{4}(1-O_2) (1+A_1)|f_3| + \frac{1}{4}(1-O_2) (1-A_1)|f_4| \biggr) , \end{aligned}$$

where f ₁=γ ₆+γ ₇+γ ₈, f ₂=γ ₆+γ ₇−γ ₈, f ₃=γ ₆−γ ₇+γ ₈, and f ₄=γ ₆−γ ₇−γ ₈. Further,

$$\begin{aligned} E(O_2\mid O_1,A_1) =& \frac{\exp(\delta_1O_1 +\delta_2A_1) -1}{\exp(\delta_1O_1 + \delta_2A_1)+1}, \\ 1 + E(O_2\mid O_1,A_1) =& 2 \operatorname{expit}(\delta_1O_1 + \delta_2A_1), \\ 1 - E(O_2\mid O_1,A_1) =& 2 \bigl(1- \operatorname{expit}(\delta_1O_1 + \delta_2A_1) \bigr). \end{aligned}$$

Therefore,

$$\begin{aligned} Q_1(H_1, A_1) =& E \Bigl[\max _{a_2} Q_2(H_2, a_2) \mid H_1, A_1 \Bigr] \\ =& \operatorname{expit} \biggl(M + \frac{1}{2}(1+A_1)|f_3| + \frac{1}{2}(1-A_1)|f_4| \\ &{} + \frac{1}{2}\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1+A_1) \bigl( |f_1| - |f_3| \bigr) \\ &{} + \frac{1}{2}\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1-A_1) \bigl( |f_2| - |f_4| \bigr) \biggr). \end{aligned}$$

Furthermore,

$$\begin{aligned} 4\operatorname{expit}(\delta_1O_1 + \delta_2A_1) =& (1+O_1) (1+A_1) \operatorname{expit}(\delta_1+\delta_2) \\ &{} + (1+O_1) (1-A_1)\operatorname{expit}( \delta_1-\delta_2) \\ &{} + (1-O_1) (1+A_1)\operatorname{expit}(- \delta_1+\delta_2) \\ &{} + (1-O_1) (1-A_1)\operatorname{expit}(- \delta_1-\delta_2). \end{aligned}$$

Since A ₁∈{−1,1}, we have $(1-A_{1}^{2})=0$, (1+A ₁)²=2(1+A ₁) and (1−A ₁)²=2(1−A ₁). It may therefore be deduced that

$$\begin{aligned} 2\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1+A_1) =& (1+O_1) (1+A_1)\operatorname{expit}(\delta_1+ \delta_2) \\ &{} + (1-O_1) (1+A_1)\operatorname{expit}(- \delta_1+\delta_2); \end{aligned}$$

and

$$\begin{aligned} 2\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1-A_1) =& (1+O_1) (1-A_1)\operatorname{expit}(\delta_1- \delta_2) \\ &{} + (1-O_1) (1-A_1)\operatorname{expit}(- \delta_1-\delta_2). \end{aligned}$$

Let $k_{1}=\frac{1}{4}\operatorname{expit}(\delta_{1}+\delta_{2})$, $k_{2}=\frac{1}{4}\operatorname{expit}(-\delta_{1}+\delta_{2})$, $k_{3}=\frac{1}{4}\operatorname{expit}(\delta_{1}-\delta_{2})$, $k_{4}=\frac{1}{4}\operatorname{expit}(-\delta_{1}-\delta_{2})$. Therefore,

$$\begin{aligned} &Q_1(H_1, A_1) \\ &\quad{}= \operatorname{expit} \biggl(M + \frac{1}{2}\bigl(|f_3| + |f_4|\bigr) + \frac{1}{2}\bigl(|f_3|-|f_4|\bigr)A_1 \\ &\qquad{} + (1+O_1) (1+A_1)k_1 \bigl( |f_1| - |f_3| \bigr) + (1-O_1) (1+A_1)k_2 \bigl( |f_1| - |f_3| \bigr) \\ &\qquad{} + (1+O_1) (1-A_1)k_3 \bigl( |f_2| - |f_4| \bigr) + (1-O_1) (1-A_1)k_4 \bigl( |f_2| - |f_4| \bigr) \biggr). \end{aligned}$$

Applying a logit transformation to Q ₁(H ₁,A ₁) gives that the coefficient of A ₁ in the above expression for $\operatorname{logit}(Q_{1}(H_{1}, A_{1}))$ is

$$\psi_{10} = \gamma_3 + (k_1+k_2) |f_1| - (k_3+k_4) |f_2| + (k_3+k_4) |f_3| - (k_1+k_2) |f_4|, $$

and the coefficient of O ₁ A ₁ in the expression for Q ₁ is

$$\psi_{11} = \gamma_4 + (k_1-k_2)|f_1| - (k_3-k_4)|f_2| - (k_1-k_2)|f_3| + (k_3-k_4)|f_4|. $$

1.2 A.2 Poisson Utility

In this section, we derive the correspondence between the Q-function model parameters and the parameters from the data-generating models when the utility is given by a Poisson count. As above, let M=γ ₀+γ ₁ C ₁+γ ₂ O ₁+γ ₃ A ₁+γ ₄ O ₁ A ₁+γ ₅ C ₂+f ₁(C ₁)+f ₂(C ₂), and μ=M+γ ₆ A ₂+γ ₇ O ₂ A ₂+γ ₈ A ₁ A ₂. Then

$$\begin{aligned} \max_{a_2} Q_2(H_2, a_2) =& \exp \biggl(M + \frac{1}{4}(1+O_2) (1+A_1)|f_1| + \frac{1}{4}(1+O_2) (1-A_1)|f_2| \\ &{} + \frac{1}{4}(1-O_2) (1+A_1)|f_3| + \frac{1}{4}(1-O_2) (1-A_1)|f_4| \biggr) , \end{aligned}$$

where f ₁, f ₂, f ₃, and f ₄ are as defined above. Thus,

$$\begin{aligned} &Q_1(H_1, A_1) \\ &\quad{}= \exp \biggl(M + \frac{1}{2}(1+A_1)|f_3| + \frac{1}{2}(1-A_1)|f_4| \\ &\qquad{} + \frac{1}{2}\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1+A_1) \bigl( |f_1| - |f_3| \bigr) \\ &\qquad{} + \frac{1}{2}\operatorname{expit}(\delta_1O_1 + \delta_2A_1) (1-A_1) \bigl( |f_2| - |f_4| \bigr) \biggr) \\ &\quad{}= \exp \biggl(M + \frac{1}{2}\bigl(|f_3| + |f_4|\bigr) + \frac{1}{2}\bigl(|f_3|-|f_4|\bigr)A_1 \\ &\qquad{} + (1+O_1) (1+A_1)k_1 \bigl( |f_1| - |f_3| \bigr) + (1-O_1) (1+A_1)k_2 \bigl( |f_1| - |f_3| \bigr) \\ &\qquad{} + (1+O_1) (1-A_1)k_3 \bigl( |f_2| - |f_4| \bigr) + (1-O_1) (1-A_1)k_4 \bigl( |f_2| - |f_4| \bigr) \biggr) \end{aligned}$$

where k ₁, k ₂, k ₃, k ₄ are as above. Therefore, the coefficients of A ₁ and O ₁ A ₁ in the above expression take the same form as in the case of a Bernoulli utility:

$$\begin{aligned} \psi_{10} =& \gamma_3 + (k_1+k_2) |f_1| - (k_3+k_4) |f_2| + (k_3+k_4) |f_3| - (k_1+k_2) |f_4|, \\ \psi_{11} =& \gamma_4 + (k_1-k_2)|f_1| - (k_3-k_4)|f_2| - (k_1-k_2)|f_3| + (k_3-k_4)|f_4|. \end{aligned}$$