Estimating the optimal treatment regime for student success programs

Abstract

We expand methods for estimating an optimal treatment regime (OTR) from the personalized medicine literature to educational data mining applications. As part of this development, we detail and modify the current state-of-the-art, assess the efficacy of the approaches for student success studies, and provide practitioners the machinery to apply the methods in their specific problems. Our particular interest is to estimate an optimal treatment regime for students enrolled in an introductory statistics course at San Diego State University (SDSU). The available treatments are combinations of three programs SDSU implemented to foster student success in this large enrollment, bottleneck STEM course. We leverage tree-based reinforcement learning approaches based on either an inverse probability-weighted purity measure or an augmented probability-weighted purity measure. The thereby deduced OTR promises to significantly increase the average grade in the introductory course and also reveals the need for program recommendations to students as only very few, on their own, selected their optimal treatment.

Appendices

Proofs

Lemma 1

1If Assumptions 1through 4hold, we have

$$\begin{aligned} \pi ^{\text {opt}}(X) = \text {arg max}_{\pi \in \varPi } {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] . \end{aligned}$$

Proof

Recall the definition of the OTR in Eq. (1)

$$\begin{aligned} \pi ^{\text {opt}} = \text {arg max}_{\pi \in \varPi } {\mathbb {E}}[Y^*\{\pi (X)\}]. \end{aligned}$$

Hence, it is sufficient to show that

$$\begin{aligned} {\mathbb {E}}[Y^*\{\pi (X)\}] = {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] . \end{aligned}$$

We begin using the law of iterated expectation to get

$$\begin{aligned} {\mathbb {E}}[Y^*\{\pi (X)\}]&={\mathbb {E}}[{\mathbb {E}}\{Y^*\{\pi (X)\}|X]]\nonumber \\&= {\mathbb {E}}\left[ {\mathbb {E}}\left[ \sum _{a\in {\mathcal {A}}}Y^*(a){\mathbb{1}}_{\{\pi (X)=a\}}\bigg |X\right] \right] . \end{aligned}$$

(13)

Now, we leverage Assumption 3, stating that—conditioned on the characteristics X—all potential outcomes $Y^*(a)$ are independent of the treatment A to add the condition in (13) that we assign treatment according to a treatment regime $\pi (X)$ which gives us

$$\begin{aligned} {\mathbb {E}}[Y^*\{\pi (X)\}]&={\mathbb {E}}\left[ {\mathbb {E}}\left[ \sum _{a\in {\mathcal {A}}}Y^*(a){\mathbb{1}}_{\{\pi (X)=a\}}|X, A=\pi (X)\right] \right] \nonumber \\&={\mathbb {E}}\left[ \frac{{\mathbb {E}}\left[ \sum _{a\in {\mathcal {A}}}Y^*(a){\mathbb{1}}_{\{\pi (X)=a\}}{\mathbb{1}}_{\{A=\pi (X)\}}|X\right] }{P\{A=\pi (X)|X\}}\right] , \end{aligned}$$

(14)

where we used in the last step the definition of the conditional expectation. Note that the expression in Eq. (14) is well defined as Assumption 4 guarantees $P\{A=\pi (X)|X\}>0$. As

$$\begin{aligned} {\mathbb{1}}_{\{\pi (X)=a\}}{\mathbb{1}}_{\{A=\pi (X)\}}&=\left\{ \begin{array}{ll} 1 &{} \, \text {if} \, \pi (X)=a \, \text {and} \, A=\pi (X) \\ 0 &{} \, \text {otherwise}\\ \end{array} \right. \nonumber \\&=\left\{ \begin{array}{ll} 1 &{} \, \text {if} \, \pi (X)=a=A\\ 0 &{} \, \text {otherwise} \\ \end{array} \right. \nonumber \\&=\left\{ \begin{array}{ll} 1 &{} \, \text {if} \, A=a \, \text {and} \, A=\pi (X)\\ 0 &{} \, \text {otherwise} \\ \end{array} \right. \nonumber \\&={\mathbb{1}}_{\{A=a\}}{\mathbb{1}}_{\{A=\pi (X)\}}, \end{aligned}$$

(15)

we can rewrite Eq. (14) as

$$\begin{aligned} {\mathbb {E}}[Y^*\{\pi (X)\}]&={\mathbb {E}}\left[ \frac{{\mathbb {E}}\left[ \sum _{a\in {\mathcal {A}}}Y^*(a){\mathbb{1}}_{\{A=a\}}{\mathbb{1}}_{\{A=\pi (X)\}}|X\right] }{P\{A=\pi (X)|X\}}\right] \nonumber \\&={\mathbb {E}}\left[ \frac{{\mathbb {E}}\left[ \left\{ \sum _{a\in {\mathcal {A}}}Y^*(a){\mathbb{1}}_{\{A=a\}}\right\} {\mathbb{1}}_{\{A=\pi (X)\}}|X\right] }{P\{A=\pi (X)|X\}}\right] \nonumber \\&={\mathbb {E}}\left[ \frac{{\mathbb {E}}\left[ Y{\mathbb{1}}_{\{A=\pi (X)\}}|X\right] }{P\{A=\pi (X)|X\}}\right] \nonumber \\&={\mathbb {E}}\left[ {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\bigg | X\right] \right] \nonumber \\&={\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] , \end{aligned}$$

(16)

where we used Assumption 2 in Eq. (16) and that $P\{A=\pi (X)|X\}$ is a $\sigma (X)$-measurable function, where $\sigma (X)$ is the $\sigma$-algebra generated by X. This ends our proof. $\square$

Lemma 2

If Assumptions 1through 4hold, we have

$$\begin{aligned} \pi ^{\text {opt}}(X) = \text {arg max}_{\pi \in \varPi } {\mathbb {E}}\left[ \frac{\{Y-g(X)\}{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] . \end{aligned}$$

for any arbitrary function $g:{\mathbb {R}}^p\mapsto {\mathbb {R}}$ (Laber and Zhao 2015).

Proof

Let $g:{\mathbb {R}}^p\mapsto {\mathbb {R}}$ be an arbitrary function to define

$$\begin{aligned} L_g\{\pi (X)\}=\frac{\{Y-g(X)\}{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}. \end{aligned}$$

(17)

Then, it holds that

$$\begin{aligned} {\mathbb {E}}[L_g\{\pi (X)\}]&= {\mathbb {E}}\left[ \frac{\{Y-g(X)\}{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] \\&= {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ \frac{g(X){\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] \\&= {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ {\mathbb {E}}\left[ \frac{g(X){\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\bigg |X\right] \right] \\&={\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ \frac{g(X)}{P\{A=\pi (X)|X\}}{\mathbb {E}}[{\mathbb{1}}_{\{A=\pi (X)\}}|X]\right] \\&={\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ \frac{g(X)}{P\{A=\pi (X)|X\}}P\{A=\pi (X)|X\}\right] \\&={\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ g(X)\right] , \end{aligned}$$

and therefore

$$\begin{aligned} \text {arg max}_{\pi \in \varPi } {\mathbb {E}}[L_g\{\pi (X)\}]&= \text {arg max}_{\pi \in \varPi } {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] -{\mathbb {E}}\left[ g(X)\right] \\&= \text {arg max}_{\pi \in \varPi } {\mathbb {E}}\left[ \frac{Y{\mathbb{1}}_{\{A=\pi (X)\}}}{P\{A=\pi (X)|X\}}\right] \\&=\pi ^{\text {opt}}(X). \end{aligned}$$

$\square$

Adjusted R code of Tao et al. (2018)

1.1 Function DTRtree to grow tree with ${\mathcal {P}}^{\text {Tao}}$

1.2 Function LZtree to grow tree with ${\mathcal {P}}^{LZ}$

1.3 Example

This subsection provides the R-code of the file 03 Example Application to demonstrate how to deploy the developed methods. The functions DTRtree and LZtree are available in the file 01 TRL Functions—along with other functions.

The example considers simulated data in the file 02 Example Data which presents a similar structure to the student success data, but is not based on actual student data. For each student, an SAT Math Score, HSGPA, Age, Gender, and URM were simulated along with an overall grade. Three treatments were assumed to be available to every student: no program (encoded with 1), MSLC (encoded with 2), and SI (encoded with 3).

To estimate an OTR for the given data either using ${\mathcal {P}}^{\text {Tao}}$ or ${\mathcal {P}}^{LZ}$, we need to choose at first a maximal tree depth, a minimal purity gain $\lambda$, and a minimal node size $\gamma$ following our algorithm in Sect. 2.3.

The remaining steps of the algorithm are then performed by the functions DTRtree for ${\mathcal {P}}^{\text {Tao}}$ and LZtree for ${\mathcal {P}}^{LZ}$.

The output taotree is given as a matrix:

For example, the first node is split with the help of the third covariate—which is age—at a value of 18. The purity ${\mathcal {P}}^{\text {Tao}}$ associated with this split is 608. No treatment is assigned as node 1 is not a terminal node. All students who are at most 18 are sent to node 2 which assigns treatment 2 (MSLC) to each student. All students who are older than 18 are differentiated according to their SAT Math score—which is the first covariate in the dataset. If they achieved an SAT Math Score of at most 580, they are recommended to attend SI (encoded as 3); otherwise, they should not attend any program (encoded as 1). Figure 5 displays the graphical representation of the output taotree.

Fig. 5

Example tree grown with the help of the purity measure ${\mathcal {P}}^{\text {Tao}}$