Urn Model-Based Adaptive Multi-arm Clinical Trials: A Stochastic Approximation Approach

Abstract

This paper presents the link between stochastic approximation and multi-arm clinical trials based on randomized urn models investigated in Bai et al. (J. Multivar. Anal. 81(1):1–18, 2002) where the urn updating depends on the past performances of the treatments. We reformulate the dynamics of the urn composition, the assigned treatments and the successes of assigned treatments as standard stochastic approximation (SA) algorithms with remainder. Then, we derive the a.s. convergence of the normalized procedure under less stringent assumptions by calling upon the ODE and a new asymptotic normality result (Central Limit Theorem CLT) by calling upon the SDE methods.

Appendix: Basic Tools of Stochastic Approximation

Consider the following recursive procedure defined on a filtered probability space $(\varOmega,{\mathcal{A}},({\mathcal{F}}_{n})_{n\geq0},{\mathbb{P}})$

$$ \forall n\ge n_0,\quad\theta_{n+1}= \theta_n-\gamma_{n+1}h(\theta _n)+ \gamma_{n+1} (\Delta M_{n+1}+r_{n+1} ), $$

(3.16)

where $h:{\mathbb{R}}^{d}\rightarrow {\mathbb{R}}^{d}$ is a locally Lipschitz continuous function, $\theta_{n_{0}}$ an ${\mathcal{F}}_{n_{0}}$-measurable finite random vector and, for every n≥n ₀, ΔM _n+1 is an ${\mathcal{F}}_{n}$-martingale increment and r _n is an ${\mathcal{F}}_{n}$-adapted remainder term.

Theorem 3.3

(A.s. Convergence with ODE Method, see e.g. [7, 8, 10, 12, 17])

Assume that h is locally Lipschitz, that

$$r_n\overset{a.s.}{\underset{n\rightarrow\infty}{\longrightarrow}}0 \quad\textit{and} \quad\sup_{n\geq n_0}{\mathbb{E}}\bigl[\Vert \Delta M_{n+1}\Vert ^2\,\big|\,{\mathcal{F}}_n \bigr]<+\infty\quad a.s., $$

and that (γ _n ) _n≥1 is a positive sequence satisfying

$$\sum_{n\geq1}\gamma_n=+\infty\quad \textit{and} \quad\sum_{n\geq1}\gamma _n^2<+ \infty. $$

Then the set Θ ^∞ of its limiting values as n→+∞ is a.s. a compact connected set, stable by the flow of

$$\mathit{ODE}_h\equiv\dot{\theta}=-h(\theta). $$

Furthermore if θ ^∗∈Θ ^∞ is a uniformly stable equilibrium on Θ ^∞ of ODE _h , then

$$\theta_n\overset{a.s.}{\underset{n\rightarrow\infty}{ \longrightarrow }}\theta^*. $$

Comments

By uniformly stable we mean that

$$\sup_{\theta\in\varTheta^{\infty}}\bigl \vert \theta(\theta_0,t)- \theta^*\bigr \vert \longrightarrow0\quad\mbox{as} \ t\rightarrow+\infty, $$

where $\theta(\theta_{0},t)_{\theta_{0}\in\varTheta^{\infty},t\in {\mathbb{R}}_{+}}$ is the flow of ODE _h on Θ ^∞.

Theorem 3.4

(Rate of Convergence see [10], Theorem 3.III.14, p.131 (for CLT see also e.g. [8, 17]))

Let θ ^∗ be an equilibrium point of {h=0}. Assume that the function h is differentiable at θ ^∗ and all the eigenvalues of Dh(θ ^∗) have positive real parts. Assume that for some δ>0,

$$ \sup_{n\geq n_0}{\mathbb{E}}\bigl[\Vert \Delta M_{n+1}\Vert ^{2+\delta}\,\big|\,{\mathcal{F}}_n \bigr]<+\infty\quad \textit{a.s.}, \quad {\mathbb{E}}\bigl[\Delta M_{n+1}\Delta M_{n+1}^t \,\big|\,{\mathcal{F}}_n \bigr]\overset{a.s.}{\underset{n\rightarrow\infty}{ \longrightarrow}}\varGamma, $$

(3.17)

where Γ is a deterministic symmetric definite positive matrix and for an ϵ>0,

(3.18)

Specify the gain parameter sequence as follows: for every n≥1, $\gamma_{n}=\frac{1}{n}$. If $\varLambda:=\Re e(\lambda_{{\rm min}})>\frac {1}{2}$, where $\lambda_{{\rm min}}$ denotes the eigenvalue of Dh(θ ^∗) with the lowest real part, then, the above a.s. convergence is ruled on the convergence set {θ _n →θ ^∗} by the following Central Limit Theorem

$$\sqrt{n} \bigl(\theta_n-\theta^* \bigr)\overset{{\mathcal{L}}}{ \underset {n\rightarrow\infty}{\longrightarrow}} {\mathcal{N}} \biggl(0, \frac{\varSigma }{2\varLambda-1} \biggr) $$

with

$$\varSigma:= \int_0^{+\infty} \bigl(e^{- (Dh(\theta^*)-\frac {I_d}{2} )u} \bigr)^t\varGamma e^{- (Dh(\theta^*)-\frac {I_d}{2} )u}du. $$