Skip to main content
SpringerLink
Log in

Optimal Dynamic Treatment Strategies with Protection Against Missed Decision Points

  • Published:
Statistics in Biosciences Aims and scope Submit manuscript

Abstract

We review methods for determination of optimal dynamic treatment strategies and consider the consequences of patients missing scheduled clinic visits. We describe a Markov chain Monte Carlo procedure for parameter estimation in the presence of incomplete data. We propose an optimal dynamic fixed-dose treatment allocation rule that accommodates the possibility of patients missing future scheduled visits. We compare our strategy with a globally optimal strategy through simulations and an application on control of blood clotting time for patients on long-term anticoagulation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Almirall D, Ten Have T, Murphy SA (2010) Structural nested mean models for assessing time-varying effect moderation. Biometrics 66:131–139

    Article  MATH  MathSciNet  Google Scholar 

  2. Barrett JK, Henderson R, Rosthøj S (2013) Doubly robust estimation of optimal dynamic treatment regimes. Stat Biosci. doi:10.1007/s12561-013-9097-6

    Google Scholar 

  3. Cole SR, Frangakis CE (2009) The consistency statement in causal inference: a definition or an assumption? Epidemiology 20:3–5

    Article  Google Scholar 

  4. Hirsh J, Dalen JE, Anderson DR, Poller L, Bussey H, Ansell J, Deykin D (2001) Oral anticoagulants: mechanism of action, clinical effectiveness, and optimal therapeutic range. Chest 119:8S–21S

    Article  Google Scholar 

  5. Henderson R, Ansell P, Alshibani D (2010) Regret-regression for optimal dynamic treatment regimes. Biometrics 66:1192–1201

    Article  MATH  MathSciNet  Google Scholar 

  6. Lok JJ (2008) Statistical modeling of causal effects in continuous time. Ann Stat 36:1464–1507

    Article  MATH  MathSciNet  Google Scholar 

  7. Little RJA, Rubin DB (2002) Statistical analysis with missing data, 2nd edn. Wiley, New York

    Book  MATH  Google Scholar 

  8. Moodie EMM, Richardson TS, Stephens DA (2007) Demystifying optimal dynamic treatment regimes. Biometrics 63:447–455

    Article  MATH  MathSciNet  Google Scholar 

  9. Murphy SA (2003) Optimal dynamic treatment regimes. J R Stat Soc Ser B 65:331–355

    Article  MATH  Google Scholar 

  10. Orellana L, Rotnitzky A, Robins JM (2010) Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes, part I: main content. Int J Biostat 2:1–47

    Google Scholar 

  11. Pearl J (2009) Causality: models, reasoning and inference, 2nd edn. Cambridge University Press, Cambridge

    Book  Google Scholar 

  12. Petersen ML, Deeks SG, Martin JN, van der Laan MJ (2007) History-adjusted marginal structural models for estimating time-varying effect modification. Am J Epidemiol 166(9):985–993

    Article  Google Scholar 

  13. Robert C, Casella G (2004) Monte Carlo statistical methods, 2nd edn. Springer, New York

    Book  MATH  Google Scholar 

  14. Robins JM (1994) Correcting for non-compliance in randomized trials using structural nested mean models. Commun Stat, Theory Methods 23:2379–2412

    Article  MATH  MathSciNet  Google Scholar 

  15. Robins JM, Rotnitzky A, Zhao LP (1994) Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc 89:846–866

    Article  MATH  MathSciNet  Google Scholar 

  16. Robins JM (1997) Causal inference from complex longitudinal data. In: Berkane M (ed) Latent variable modeling and applications to causality. Springer, New York, pp 69–117

    Chapter  Google Scholar 

  17. Robins JM (2004) Optimal structural nested models for optimal sequential decisions. In: Lin DY, Heagerty P (eds) Proceedings of the second symposium on biostatistics. Springer, New York, pp 189–326

    Chapter  Google Scholar 

  18. Robins JM, Hernán MA, Rotnitzky A (2007) Invited commentary: effect modification by time-varying covariates. Am J Econ 166:994–1002

    Google Scholar 

  19. Robins JM, Orellana L, Rotnitzky A (2008) Estimation and extrapolation of optimal treatment and testing strategies. Stat Med 27:4678–4721

    Article  MathSciNet  Google Scholar 

  20. Rosendaal FR, Cannegieter SC, Van Der Meer FJ, Briët E et al. (1993) A method to determine the optimal intensity or oral anticoagulant therapy. Thromb Haemost 69:236–239

    Google Scholar 

  21. Rosthøj S, Fullwood C, Henderson R, Stewart S (2006) Estimation of optimal dynamic anticoagulation regimes from observational data: a regret-based approach. Stat Med 25:4197–4215

    Article  MathSciNet  Google Scholar 

  22. Rosthøj S, Keiding N, Schmiegelow K (2012) Estimation of dynamic treatment strategies for maintenance therapy of children with acute lymphoblastic leukemia: an application of history-adjusted marginal structural models. Stat Med 31:470–488

    Article  MathSciNet  Google Scholar 

  23. Rubin DB (1978) Bayesian inference for causal effects: the role of randomization. Ann Stat 6:34–58

    Article  MATH  Google Scholar 

  24. Schafer JL (1997) Analysis of incomplete multivariate data. Chapman & Hall, London

    Book  MATH  Google Scholar 

  25. van der Laan MJ, Petersen ML, Joffe MM (2005) History-adjusted marginal structural models and statically-optimal dynamic treatment regimens. Int J Biostat 1:Article 4

  26. Welch PD, Heidelberger P (1983) Simulation run length control in presence of an initial transient. Oper Res 31:1109–1144

    Article  MATH  Google Scholar 

  27. Zhang B, Tsiatis AA, Laber EB, Davidian M (2012) A robust method for estimating optimal treatment regimes. Biometrics 68:1010–1018

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

S.R. was supported by Public Health Services Grant 5 R01 CA 54706-11 from the National Cancer Institute. J.B. was supported by the Medical Research Council Grant number G0902100 We thank Riema Ali, Department of Biostatistics, University of Copenhagen, for assistance with the simulation studies. We are grateful for the helpful comments of two anonymous referees.

Author information

Authors and Affiliations

  1. Department of Biostatistics, Institute of Public Health, University of Copenhagen, Copenhagen, Denmark

    Susanne Rosthøj

  2. School of Mathematics and Statistics, University of Newcastle, Newcastle upon Tyne, UK

    Robin Henderson

  3. MRC Biostatistics Unit, Institute of Public Health, University Forvie Site, Cambridge, UK

    Jessica K. Barrett

Authors
  1. Susanne Rosthøj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robin Henderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jessica K. Barrett
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susanne Rosthøj.

Appendix

Appendix

To simplify notation, we will in the following supress the dependence of the OD and the ODFD strategy on the state history and parameters, writing \(d^{\mathrm{OD}}_{j}(A_{j-1})\) and \(d_{j}^{\mathrm{ODFD}}(A_{j-1})\) instead of \(d_{j}^{\mathrm{OD}}(\bar{S}_{j},\bar{A}_{j-1};\psi)\) and \(d_{j}^{\mathrm{ODFD}}(\bar{S}_{j},\bar{A}_{j-1};\psi)\), respectively.

At the third time point, the OD and the ODFD strategy are equal, \(d_{3}^{\mathrm{ODFD}}=d_{3}^{\mathrm{OD}}\), such that the fixed-dose contrast at time point 3 can be written

$$\begin{aligned} \nu_3(a_3\mid \bar{S}_3,\bar{A}_2) =& -\mu_3(a_3\mid \bar{S}_3,\bar{A}_2) + \mu_3(A_2\mid \bar{S}_3,\bar{A}_2) \\ =& -\psi_1\bigl( a_3 - d_3^{\mathrm{OD}}(A_2) \bigr)^2 + \psi_1 \bigl(A_2-d_3^{\mathrm{OD}}(A_2) \bigr)^2 \\ =& \psi_1 \bigl( \bigl(A_2^2 - a_3^2\bigr) - 2(A_2-a_3) d_3^{\mathrm{OD}}(A_2) \bigr) \\ =& \psi_1 (A_2-a_3) \bigl( (A_2+a_3) \ - \ 2 d_3^{\mathrm{FD}}(A_2) \bigr). \end{aligned}$$

In the calculation of the expected future regrets at time point 1 and 2 we need the conditional moments of the states,

$$\begin{aligned} E(S_j\mid \bar{S}_{j-1},\bar{A}_{j-1}) =& \theta_1+\theta_2S_{j-1}+\theta_3A_{j-1} \\ E\bigl(S_j^2\mid \bar{S}_{j-1},\bar{A}_{j-1}\bigr) =& \sigma_Z^2 + ( \theta_1+\theta _2S_{j-1}+\theta_3A_{j-1})^2. \end{aligned}$$

At the second time point, we need the expected future regret at time point 3, assuming the same dose is assigned at time point 2 and 3. The difference of the expected regrets assigning unchanged dose A 1 versus new dose a 2 is

$$\begin{aligned} &{-} \mu_3^{-1} (a_2\mid \bar{S}_2,A_1 ) + \mu_3^{-1}(A_1\mid \bar{S}_2,A_1) \\ &\quad = -E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)} \psi_1\bigl(a_2-d_3^{\mathrm {OD}}(a_2) \bigr)^2 + E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)} \psi_1\bigl(A_1-d_3^{\mathrm {OD}}(A_2) \bigr)^2 \\ &\quad = \psi_1 \bigl\{ \bigl(A_1^2-a_2^2 \bigr) + \bigl( E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}d_3^{\mathrm{OD}}(A_1)^2 - E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}d_3^{\mathrm{OD}}(A_2)^2 \bigr) \\ &\qquad {} - 2 \bigl( A_1 E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}d_3^{\mathrm{OD}}(A_1) - a_2 E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}d_3^{\mathrm{OD}}(a_2) \bigr) \bigr\} . \end{aligned}$$

The second term on the right hand side of the above equation can be written

$$\begin{aligned} & \bigl(E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)} d_3^{\mathrm{OD}}(A_1)^2 - E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}d_3^{\mathrm{OD}}(A_2)^2 \bigr) \\ &\quad = E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)} (\psi_2S_3+ \psi_3S_2+\psi _4A_1)^2 \\ &\qquad {}- E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)} (\psi_2S_3+ \psi_3S_2+\psi _4a_2)^2 \\ &\quad = E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)} \bigl((\psi_2S_3)^2 + \psi _4^2A_1^2 + 2 \psi_3\psi_4S_2A_1 +2 \psi_2S_3(\psi_3S_2+ \psi_4A_1)\bigr) \\ &\qquad {} -E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)} \bigl((\psi _2S_3)^2 + \psi_4^2a_2^2 + 2 \psi_3\psi_4S_2a_2 +2 \psi_2S_3(\psi_3S_2+ \psi_4a_2)\bigr) \\ &\quad = \psi_4^2\bigl(A_1^2-a_2^2 \bigr) + 2\psi_3\psi_4S_2(A_1-a_2) \\ &\qquad {}+ \psi_2^2 \bigl(E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}S_3^2-E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}S_3^2 \bigr) \\ &\qquad {}+2\psi_2\psi_3S_2 (E_{S_3\mid (\bar{S}_2,A_1),\mathrm {do}(A_1)}S_3-E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}S_3) \\ &\qquad {} +2\psi_2\psi_4(A_1E_{S_3\mid (\bar{S}_2,A_1),\mathrm {do}(A_1)}S_3-a_2E_{ S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}S_3) \\ &\quad = \psi_4^2\bigl(A_1^2-a_2^2 \bigr) + 2\psi_3\psi_4S_2(A_1-a_2) + \psi_2^2\theta_3^2 \bigl(A_1^2-a_2^2\bigr) \\ &\qquad {}+ 2 \psi_2^2\theta_3(A_1-a_2) (\theta_1+\theta_2S_2) \\ &\qquad {} +2\psi_2\psi_3S_2 \theta_3(A_1-a_2) +2\psi_2 \psi_4 \bigl((A_1-a_2) ( \theta_1+\theta_2S_2) \\ &\qquad {}+\bigl(A_1^2-a_2^2 \bigr)2\psi_2\psi_4\theta_3\bigr) \\ &\quad = \bigl(\psi_4^2 + \psi_2^2 \theta_3^2+2\psi_2\psi4\theta_3 \bigr) \bigl(A_1^2-a_2^2\bigr) \\ &\qquad {} + \bigl(2\psi_3\psi_4 S_2 +2 \psi_2^2\theta_3(\theta_1+ \theta_2S_2)+2\psi_2\psi_3 \theta_3S_2 \\ &\qquad {}+2\psi_2\psi_4( \theta_1+\theta_2S_2)\bigr) (A_1-a_2). \end{aligned}$$

Similarly the third term can be written

$$\begin{aligned} &{-} 2 \bigl( A_1 E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}d_3^{\mathrm{OD}}(A_1) - a_2 E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}d_3^{\mathrm{OD}}(a_2) \bigr) \\ &\quad =- 2 \bigl( A_1(\psi_2E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}S_3+ \psi_3S_2+\psi_4A_1) \\ &\qquad {} - a_2 \psi_2E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}S_3+ \psi_3S_2+\psi _4a_2\bigr) \\ &\quad = -2 \bigl( \psi_4 \bigl(A_1^2-a_2^2 \bigr) + \psi_3(A_1-a_2)S_2 + \psi _2(A_1E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(A_1)}S_3 \\ &\qquad{} - a_2E_{S_3\mid (\bar{S}_2,A_1),\mathrm{do}(a_2)}S_3) \bigr) \\ &\quad = -2 \bigl( \psi_4 \bigl(A_1^2-a_2^2 \bigr) + \psi_3(A_1-a_2)S_2 \\ &\qquad {}+ \psi_2\bigl( (A_1-a_2) (\theta_1+ \theta_2S_2) + \bigl(A_1^2-a_2 \bigr)^2\theta_3 \bigr) \bigr) \\ &\quad = -2 \bigl( (\psi_4+\psi_2\theta_3) \bigl(A_1^2-a_2^2\bigr) + (A_1-a_2) \bigl(\psi_2\theta_1+( \psi_3+\psi_2\theta_2)S_2\bigr) \bigr). \end{aligned}$$

The fixed-dose contrast at the second time point is

$$\begin{aligned} &\nu_2(a_2\mid \bar{S}_2,A_1) \\ &\quad = - \mu_3^{-1}(a_2\mid \bar{S}_2,A_1 ) - \mu_2(a_2\mid \bar{S}_2,A_1) + \mu_2(A_1\mid \bar{S}_2,A_1) + \mu_3^{-1}(A_1\mid \bar{S}_2,A_1) \\ &\quad = - \mu_3^{-1}(a_2\mid \bar{S}_2,A_1 ) - \mu_3(a_2\mid \bar{S}_2,A_1) + \mu_3(A_1\mid \bar{S}_2,A_1) + \mu_3^{-1}(A_1\mid \bar{S}_2,A_1) \\ &\quad = - \mu_3^{-1}(a_2\mid \bar{S}_2,A_1 ) + \nu_3(a_2\mid \bar{S}_2,A_1) + \mu_3^{-1}(A_1\mid \bar{S}_2,A_1) \end{aligned}$$

since the regrets μ 2 and μ 3 share parameters and only depend on current and previous state and previous dose. Now adding the above terms we obtain

$$\begin{aligned} \nu_2(a_2\mid \bar{S}_2,A_1) =& f_1(\psi,\theta) \bigl(A_1^2-a_2^2 \bigr) + f_2(\bar{S}_2,A_1,\psi,\theta) (A_1-a_2) \end{aligned}$$

with

$$\begin{aligned} f_1(\psi,\theta) =& \psi_1\bigl(2+\bigl( \psi_4^2 + \psi_2^2 \theta_3^2+2\psi_2\psi4\theta_3 \bigr)-2 (\psi_4+\psi_2\theta_3) \bigr) \\ f_2(\bar{S}_2,A_1,\psi,\theta) =& 2 \psi_1 \bigl(f_3(\psi,\theta)+f_4(\psi , \theta)S_2+f_5(\psi,\theta)S_1+f_6( \psi,\theta)A_1\bigr) \end{aligned}$$

such that the first term f 1 does not depend on the observed history \((\bar{S}_{2},A_{1})\) whereas the second term f 2 is linear in history. Here

$$\begin{aligned} f_3(\psi,\theta) =& \psi_2^2 \theta_3\theta_1+\psi_2\psi_4 \theta_1-\psi _2\theta_1 \\ f_4(\psi,\theta) =& -\psi_2 + \psi_3 \psi_4+\psi_2^2\theta_2 \theta_3 +\psi_2\psi_3\theta_3+ \psi_2\psi_4\theta_2-(\psi_3+ \psi_2\theta_2) \\ f_5(\psi,\theta) =& \psi_3 \\ f_6(\psi,\theta) =& \psi_4. \end{aligned}$$

Defining ϕ j0=f 1(ψ,θ), ϕ j1=f 3(ψ,θ)/η 1, ϕ j2=f 4(ψ,θ)/η 1, ϕ j3=ψ 3/η 1 and ϕ j4=ψ 4/η 1 we can write the fixed-dose contrast on the claimed form, namely

$$\begin{aligned} &\nu_2\bigl(a_2\mid \bar{S}_2,A_1;( \phi_{20},\phi_{2})\bigr) \\ &\quad = \phi_{20} (A_1-a_2) \bigl( (A_1+a_2) - 2 (\phi_{21}+\phi_{22}S_2+\phi_{23}S_1+ \phi_{24}A_1) \bigr) \end{aligned}$$

such that the optimal dynamic fixed-dose strategy at the second time point is linear in the same terms as the optimal dynamic strategy. The parameters are different and there is an intercept term. For the parameters in the simulation study we find ϕ 20=10, ϕ 21=0, ϕ 22=0.595, ϕ 23=0.075 and ϕ 24=−0.1.

Similar calculations for the first time point give

$$\begin{aligned} \nu_1\bigl(a_1\mid {S}_1;( \phi_{10},\phi_{1})\bigr) =& \phi_{10} (a_0-a_1) \bigl( (a_0+a_1) - 2 (\phi_{11}+\phi_{12}S_1) \bigr) \end{aligned}$$

with ϕ 10=32.2725, ϕ 11=0 and ϕ 12=0.4165 for the chosen parameter values.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rosthøj, S., Henderson, R. & Barrett, J.K. Optimal Dynamic Treatment Strategies with Protection Against Missed Decision Points. Stat Biosci 6, 261–289 (2014). https://doi.org/10.1007/s12561-013-9107-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12561-013-9107-8

Keywords

Search

Navigation