Characterization of optimality in classes of “truncatable” stopping rules

Abstract

Let $X_1,X_2,\ldots ,X_n,\ldots$ be a discrete-time stochastic process. The following optimal stopping problem is considered. We observe $X_1,X_2,\ldots$ on the one-by-one basis getting successively the data $x_1,x_2,\ldots$. At each stage $n$, $n=1,2,\ldots$, after the data $x_1,\ldots ,x_n$ have been observed, we may stop, and if we stop, we gain $g_n(x_1,\ldots ,x_n)$. In this article, we characterize the structure of all optimal (randomized) stopping times $\tau$ that maximize the average gain value $G(\tau )=E g_\tau (X_1,\ldots , X_\tau )$ in some natural classes of stopping times $\tau$ we call truncatable: $\tau$ is called truncatable if $G(\tau \wedge N)\rightarrow G(\tau )$ as $N\rightarrow \infty$. It is shown that under some additional conditions on the structure of $g_n$ (suitable for statistical applications) every finite (with probability 1) stopping time is truncatable.

Appendix: Proofs

1.1 Proof of Theorem 1

Let

$$\begin{aligned} B_kv=B_kv(x_1,\ldots ,x_k)=\int v(x_1,\ldots ,x_k,x_{k+1})d\mu (x_{k+1}), \end{aligned}$$

where $v=v(x_1,\ldots , x_{k+1})$ is any $\mu ^{k+1}$-integrable function, $k=0,1,2,\ldots$.

The proof of Theorem 1 is based on the following simple lemma (see also Lemma A.2 in [14]):

Lemma 5

Let $k$ be any natural number, and let

$$\begin{aligned} v=v(x_1,x_2,\ldots ,x_{k+1}) \end{aligned}$$

be any $\mu ^{k+1}$-integrable function. Then

$$\begin{aligned} \int s_k^\psi l_k \mathrm{d}\mu ^k +\int t_{k+1}^\psi v \mathrm{d}\mu ^{k+1}\le \int t_{k}^\psi v_k\mathrm{d}\mu ^{k}, \end{aligned}$$

(4.1)

where

$$\begin{aligned} v_k=v_{k}(x_1,\ldots ,x_k)=\max \{l_{k}(x_1,\ldots ,x_k),B_kv(x_1,\ldots ,x_k)\}, \end{aligned}$$

is a $\mu ^k$-integrable function. There is an equality in (4.1) if and only if

$$\begin{aligned} I_{\{l_{k}> B_kv\}}\le \psi _{k}\le I_{\{l_{k}\ge B_kv\}} \end{aligned}$$

(4.2)

$\mu ^{k}$-almost everywhere on $T_{k}^\psi =\{(x_1,\ldots ,x_{k}): t_{k}^\psi (x_1,\ldots ,x_{k})>0\}$.

By the Fubini theorem, the left-hand side of (4.1) is equal to

$$\begin{aligned} \int s_k^\psi l_{k}\mathrm{d}\mu ^{k}+\int t_{k+1}^\psi \left( \int v \mathrm{d}\mu (x_{k+1})\right) \mathrm{d}\mu ^{k} \end{aligned}$$

$$\begin{aligned} =\int t_k^\psi [\psi _{k}l_{k}+ (1-\psi _{k})\int v\mathrm{d}\mu (x_{k+1})]\mathrm{d}\mu ^{k}. \end{aligned}$$

(4.3)

Applying Lemma A.1 [14] (for $\max$ instead of $\min$) we see that (4.3) is lesser than or equal to

$$\begin{aligned} \int t_k^\psi \left[ \max \{l_{k},\int v\mathrm{d}\mu (x_{k+1})\}\right] \mathrm{d}\mu ^{k}=\int t_k^\psi v_{k}\mathrm{d}\mu ^{k}, \end{aligned}$$

(4.4)

by the definition of $v_{k}$.

Moreover, by the same Lemma A.1 [14], (4.3) is equal to (4.4) if and only if (4.2) holds $\mu ^k$-almost everywhere on $T_k^\psi$.$\Box$

Let now for any natural $1\le n\le N$

$$\begin{aligned} \widetilde{G}_n^N(\psi )=\sum _{i=1}^{n-1} Es_i^\psi g_i+\int t_n^\psi V_{n}^N \mathrm{d}\mu ^n. \end{aligned}$$

It is obvious that $G_N(\psi )=\widetilde{G}_N^N(\psi )$ (see (2.2)) and that $\widetilde{G}_1^N(\psi )=Q_0^N$ (see (2.4)). Applying Lemma 5 initially to $v\equiv V_N^N$ with $k+1=N$ we have

$$\begin{aligned} G_{N}(\psi ) \le \sum _{n=1}^{N-2} Es_n^\psi g_n+\int t_N^\psi V_{N-1}^N \mathrm{d}\mu ^{N-1}=\widetilde{G}_{N-1}^N(\psi ), \end{aligned}$$

where the equality is attained if and only if

$$\begin{aligned} I_{\left\{ l_{N-1}> Q_{N-1}^N\right\} }\le \psi _{N-1}\le I_{\left\{ l_{N-1}\ge Q_{N-1}^N\right\} } \end{aligned}$$

$\mu ^{N-1}$-almost everywhere on $T_{N-1}^\psi$.

It is not difficult to see, by induction, using Lemma 5 again, that for all $n=N-1,\ldots ,1$

$$\begin{aligned} \widetilde{G}_{n+1}^N(\psi )\le \widetilde{G}_{n}^N(\psi ), \end{aligned}$$

(4.5)

with an equality if and only if

$$\begin{aligned} I_{\left\{ l_{n}> Q_{n}^N\right\} }\le \psi _{n}\le I_{\left\{ l_{n}\ge Q_{n}^N\right\} } \end{aligned}$$

(4.6)

$\mu ^n$-almost everywhere on $T_n^\psi$ for all $n=1,\ldots , N-1$. Thus, we have

$$\begin{aligned} G_N(\psi )=\widetilde{G}_N^N(\psi )\le \widetilde{G}_1^N(\psi )= Q_0^N \end{aligned}$$

with an equality if and only if (4.6) holds $\mu ^n$-almost everywhere on $T_n^\psi$ for all $n=1,\ldots ,N-1$. $\Box$

1.2 Proof of Lemma 3

Let us denote

$$\begin{aligned} U=\sup _{\psi \in \fancyscript{F}}G(\psi ),\quad U_N=\sup _{\psi \in {\fancyscript{F}}^N}G(\psi ). \end{aligned}$$

By Theorem 1, for all $N=1,2,\ldots$

$$\begin{aligned} U_N=Q_0^N. \end{aligned}$$

Obviously, $U_N\le U$ for every $N=1,2,\ldots$, so

$$\begin{aligned} \lim _{N\rightarrow \infty }U_N\le U. \end{aligned}$$

(4.7)

Let us show first that there is an equality in (4.7).

Suppose first that $U<\infty$.

Let us suppose that there is a strict inequality in (4.7), i.e. that $\lim _{N\rightarrow \infty }U_N= U-3\epsilon$, with some $\epsilon >0$. We immediately have from this that

$$\begin{aligned} U_N\le U-3\epsilon \end{aligned}$$

(4.8)

for all $N$.

On the other hand, by the definition of $U$ there exists a $\psi \in \fancyscript{F}$ such that $U-\epsilon \le G(\psi )\le U$.

By the condition of the Lemma, $\lim _{N\rightarrow \infty }G_N(\psi )= G(\psi )$, so we have that

$$\begin{aligned} G_N(\psi )\ge U-2\epsilon \end{aligned}$$

(4.9)

for all sufficiently large $N$. Because, by definition, $G_N(\psi )\le U_N$, we have from (4.9) that

$$\begin{aligned} U_N\ge U-2\epsilon \end{aligned}$$

for all sufficiently large $N$, which contradicts (4.8).

In a very similar way we also get a contradiction in case $U=\infty$ in (4.7). The contrary to the equality in (4.7) is $\lim _{N\rightarrow \infty } U_N=k<\infty$. Substituting, in the above proof $k$, $2k$ and $3k$, respectively, for $U-3\epsilon$, $U-2\epsilon$, and $U-\epsilon$, we easily get to a contradiction in the same way as before.

Therefore, in either case

$$\begin{aligned} U=\lim _{N\rightarrow \infty }U_N=\lim _{N\rightarrow \infty }Q_0^N= Q_0. \end{aligned}$$

1.3 Proof of Theorem 2

Let for any $\psi \in \fancyscript{F}$

$$\begin{aligned} \widetilde{G}_n(\psi )=\sum _{i=1}^{n-1} Es_i^\psi g_i+\int t_n^\psi V_{n} \mathrm{d}\mu ^n. \end{aligned}$$

Passing to the limit, as $N\rightarrow \infty$, on both sides of (4.5) we have

$$\begin{aligned} \widetilde{G}_{n+1}(\psi )\le \widetilde{G}_{n}(\psi ), \end{aligned}$$

(4.10)

and passing to the limit, as $N\rightarrow \infty$, in (2.3), we obtain

$$\begin{aligned} V_n=\max \{l_n,Q_n\}, \end{aligned}$$

where, $Q_n=Q_n(x_1,\ldots ,x_n)=\lim _{N\rightarrow \infty }Q_n^N(x_1,\ldots ,x_n)$ satisfies the equation

$$\begin{aligned} Q_n(x_1,\ldots ,x_n)=\displaystyle {\int \limits _{\displaystyle \mathfrak {X}}} V_{n+1}(x_1,x_2,\ldots , x_n,x_{n+1})\mathrm{d}\mu (x_{n+1}), \end{aligned}$$

$\mu ^n$-almost everywhere, $n=0,1,2,\ldots$. Obviously, $\widetilde{G}_1(\psi )=Q_0$, so it follows from (4.10) that for any fixed natural $n$

$$\begin{aligned} G(\psi )\le \widetilde{G}_{n}(\psi )\le \widetilde{G}_{n-1}(\psi )\le \cdots \le \widetilde{G}_1=Q_0. \end{aligned}$$

(4.11)

Let us suppose that there exists some stopping rule $\psi \in \fancyscript{F}$ such that $G(\psi )=Q_0<\infty$. Then there are equalities in all of the inequalities in (4.11). Taking into account the necessary condition in Lemma 5, we see that this is only possible when

$$\begin{aligned} I_{\{l_{n}> Q_{n}\}}\le \psi _{n}\le I_{\{l_{n}\ge Q_{n}\}} \end{aligned}$$

(4.12)

$\mu ^n$-almost everywhere on $T_n^\psi$, for all natural $n$. In addition, we have that for all natural $n$

$$\begin{aligned} \widetilde{G}_n(\psi )=\sum _{i=1}^{n-1} Es_i^\psi g_i+\int t_n^\psi V_{n} \mathrm{d}\mu ^n=Q_0. \end{aligned}$$

(4.13)

It follows from (4.13), in particular that

$$\begin{aligned} G_n(\psi )+\int t_n^\psi (V_{n}-l_n) \mathrm{d}\mu ^n=Q_0. \end{aligned}$$

(4.14)

Since $\psi \in \fancyscript{F}$, by supposition, we have that $\lim _{n\rightarrow \infty }G_n(\psi )= G(\psi ) =Q_0$. Thus, (3.6) follows from (4.14).

On the other hand, suppose now that a stopping rule $\psi \in \fancyscript{F}$ satisfies (4.12) $\mu ^n$-almost everywhere on $T_n^\psi$, for all natural $n$, and that (3.6) holds.

Then, by Lemma 5, there are equalities in all of the inequalities (4.11) except, probably, for the first one. In particular, we have that (4.13) (and consequently (4.14)) holds for all natural $n$. Then it follows from (4.14) and (3.6) that $G_n(\psi )\rightarrow Q_0$, as $n\rightarrow \infty$. Because, by supposition, $G(\psi )=\lim _{n\rightarrow \infty }G_n(\psi )=Q_0$, we have from Lemma 3

$$\begin{aligned} G(\psi )=\sup _{\psi ^\prime \in \fancyscript{F}}G(\psi ^\prime ). \end{aligned}$$

1.4 Proof of Lemma 4

By Assumption 2, for every $\psi \in \fancyscript{F}_1$ $G^+(\psi )=\sum _{n=1}^\infty E s_n^\psi g_n^+\le E\sup _{n\ge 1}g_n^+=M<\infty$, so Condition (a) of Definition 1 is satisfied. Conditions (b) and (c) of Definition 1 are satisfied in an obvious way.

To complete the proof, let us show that Condition (d) of Definition 1 is also satisfied.

Let $\psi \in \fancyscript{F}_1$ be a stopping rule such that $G^-(\psi )<\infty$. Thus, by Assumption 1,

$$\begin{aligned} \sum _{n=N}^\infty Es_n^\psi z_n\rightarrow 0,\quad \text{ as }\quad N\rightarrow \infty . \end{aligned}$$

(4.15)

Because $z_n\ge z_N$ for all $n\ge N$, it follows from (4.15) that

$$\begin{aligned} Et_N^\psi z_N=\sum _{n=N}^\infty Es_n^\psi z_N\le \sum _{n=N}^\infty Es_n^\psi z_n \rightarrow 0,\quad \text{ as }\quad N\rightarrow \infty ; \end{aligned}$$

thus,

$$\begin{aligned} Et_N^\psi c_N^+\le Et_N^\psi z_N\rightarrow 0,\quad \text{ as }\quad N\rightarrow \infty . \end{aligned}$$

(4.16)

We now have

$$\begin{aligned} G(\psi )-G_N(\psi )&= \sum _{n=N}^\infty Es_n^\psi g_n^+-\sum _{n=N}^\infty Es_n^\psi g_n^--Et_N^\psi g_N^++Et_N^\psi g_N^- \\&\le \sum _{n=N}^\infty Es_n^\psi g_n^++Et_N^\psi g_N^-\nonumber \end{aligned}$$

(4.17)

$$\begin{aligned}&\le \sum _{n=N}^\infty Es_n^\psi g_n^++Et_N^\psi r_N^- +Et_N^\psi c_N^+. \end{aligned}$$

(4.18)

The first summand on the right-hand side of (4.18) tends to 0 as $N\rightarrow \infty$, because

$$\begin{aligned} \sum _{n=N}^\infty Es_n^\psi g_n^+\le \sum _{n=N}^\infty Es_n^\psi \sup _{i\ge 1}g_i^+= Et_N^\psi \sup _{i\ge 1}g_i^+\rightarrow 0, \end{aligned}$$

(4.19)

because $Et_N^\psi =P(\tau _\psi \ge N)\rightarrow 0$ as $N\rightarrow \infty$, and $E\sup _{n\ge 1}g_n^+<\infty$.

In a similar way, the second summand on the right-hand side of (4.18) tends to 0 as $N\rightarrow \infty$, because, by Assumption 1, $\{r_n^-,n\ge 1\}$ is uniformly integrable, and $Et_n^\psi \rightarrow 0$ as $n\rightarrow \infty$.

At last, the third summand on the right-hand side of (4.18) tends to 0 as $N\rightarrow \infty$ by virtue of (4.16).

Thus,

$$\begin{aligned} G(\psi )\le \liminf _{N\rightarrow \infty } G_N(\psi ). \end{aligned}$$

(4.20)

It follows from (4.17) that

$$\begin{aligned} G(\psi )-G_N(\psi )\ge -\sum _{n=N}^\infty Es_n^\psi g_n^--Et_N^\psi g_N^+, \end{aligned}$$

(4.21)

where the first summand converges to 0, as $N\rightarrow \infty$, because $G^-(\psi )<\infty$. The second summand on the right-hand side of (4.21) tends to 0 as $N\rightarrow \infty$ by virtue of (4.19).

This implies that $G(\psi )\ge \limsup _{N\rightarrow \infty }G_N(\psi )$; thus (see also (4.20)) $\lim _{N\rightarrow \infty }G_N$ $(\psi )=G(\psi )$ follows.

1.5 Proof of Theorem 3

The necessity part of Theorem 3 follows from that of Theorem 2.

To prove the sufficiency part we only need to verify the condition (3.6) of Theorem 2. As $0\le V_n-l_n\le V_n^++g_n^-f^n$, it suffices to show that

$$\begin{aligned} \int \limits _{T_n^\psi } t_n^\psi V_n^+\mathrm{d}\mu ^n\rightarrow 0 \end{aligned}$$

(4.22)

as $n\rightarrow \infty$ and that ${ \int } t_n^\psi g_n^-f^n\mathrm{d}\mu ^n\rightarrow 0$ as $n\rightarrow \infty$. The latter follows from the proof of Lemma 4 in Section 4.4.

Let us prove (4.22).

Let $g_N^*=\sup _{1\le n\le N}g_n^+(X_1,\ldots ,X_n)$ and $g^*=\sup _{N\ge 1} g_N^*$. By Assumption 2, $Eg^*<\infty$. It is not difficult to see, by induction, that

$$\begin{aligned} (V_n^N)^+(X_1,\ldots ,X_n)\le E\{g_N^*|X_1,\ldots ,X_n\}f^n(X_1,\ldots ,X_n) \end{aligned}$$

with probability 1, for all $n\le N$, for all natural $N$ (see the definitions in (2.2), (2.3) and (2.4)). Thus,

$$\begin{aligned} V_n^+(X_1,\ldots ,X_n)\le E\{g^*|X_1,\ldots ,X_n\}f^n(X_1,\ldots ,X_n) \end{aligned}$$

with probability 1 for all natural $n$. Because of this, for every $\psi$ such that $P(\tau _\psi <\infty )=1$

$$\begin{aligned} \int \limits _{T_n^\psi } t_n^\psi V_n^+P\mathrm{d}\mu ^n\le Et_n^\psi g^*=Eg^*I_{\{\tau _\psi \ge n\}}\rightarrow 0, \end{aligned}$$

as $n\rightarrow \infty$.

It follows from (4.13) that

$$\begin{aligned} \sum _{i=1}^{n-1} Es_i^\psi (r_i-c_i)+\int t_n^\psi W_{n} d\mu ^n-E t_n^\psi c_{n} =Q_0. \end{aligned}$$

(4.23)

Applying (4.29) on the left-hand side of (4.23), we have $ER^* -E t_n^\psi c_{n} \ge Q_0,$ or

$$\begin{aligned} E t_n^\psi c_{n} \le ER^* - Q_0. \end{aligned}$$

(4.24)

Let $k$ be any positive number. Then it follows from (4.24) that

$$\begin{aligned} kE t_n^\psi I_{\{c_n>k\}}\le E t_n^\psi c_{n} \le ER^* - Q_0. \end{aligned}$$

(4.25)

Because

$$\begin{aligned} E t_n^\psi = E t_n^\psi I_{\{ c_n>k\}}+E t_n^\psi I_{\{ c_n\le k\}} \end{aligned}$$

(4.26)

and the second summand by virtue of (4.27) tends to 0, as $n\rightarrow \infty$, we have that the difference between the first summand on the right-hand side of (4.26) and the left-hand side of it, goes to 0 as $n\rightarrow \infty$. Thus, from (4.25), we have that

$$\begin{aligned} \lim _{n\rightarrow \infty }E t_n^\psi =\lim _{n\rightarrow \infty }P(\tau _\psi \ge n)= P (\tau _\psi =\infty )<(ER^*-Q_0)/k. \end{aligned}$$

Since $k>0$ is arbitrary, $P (\tau _\psi =\infty )=0$ follows.

1.6 Proof of Corollary 1

It is easy to see from the proof of the Corollary of Theorem 4.4 in [4] that Assumption 3 implies that Assumption 1 is fulfilled (with $r_n^*$ and $c_n^*$ in place of $r_n$ and $c_n$ in Assumption 1, respectively). Assumption 2 follows from Assumption 3 as well, because $g_n^+\le r_n^+$ (see Assumption 3 (a)). Thus, under Assumption 3 the assertions of Theorem 2 (with $\fancyscript{F}$ defined by (3.7) ) and of Theorem 3 hold. Therefore, Corollary 1 follows.

The only thing to show is that the condition $c_n\rightarrow \infty$, $n\rightarrow \infty$, is sufficient for almost sure finiteness of all stopping times corresponding to (3.9). So let us suppose that

$$\begin{aligned} \lim _{n\rightarrow \infty }P(c_n<k)=0\quad \text{ for } \text{ every }\quad k>0 \end{aligned}$$

(4.27)

(which is equivalent to the almost sure convergence $c_n\rightarrow \infty$, $n\rightarrow \infty$), and let us show that any $\psi$ satisfying (3.9) $\mu ^n$-almost everywhere on $T_n^\psi$ for all $n=1,2,\ldots$ has the property $P(\tau _\psi <\infty )=1$.

For any $n=1,2,\ldots$ let us define $W_n^N=V_n^N+c_nf^n$, $N\ge n$, and $W_n=V_n+c_nf^n$. It follows from (2.3) that

$$\begin{aligned} W_{n}^N=\max \left\{ r_nf^n,\int W_{n+1}^N\mathrm{d}\mu (x_{n+1})-\left( \int c_{n+1}f^{n+1}\mathrm{d}\mu (x_{n+1})-c_nf^n\right) \right\} \nonumber \\ \end{aligned}$$

(4.28)

with $W_N^N=(g_N+c_N)f^N=r_Nf^N$, $N=1,2,\ldots$, $n=1,2,\ldots ,N-1$. Let $R_N^*=\max _{1\le n\le N}(r_n^N)^+$, $N=1,2,\ldots$, and $R^*=\sup _{N\ge 1}R_N^*$. Because $\int c_{n+1}f^{n+1}\mathrm{d}\mu (x_{n+1})-c_nf^n\ge 0$ for all $n=1,2,\ldots$, it is easy to see by induction, using (4.28), that

$$\begin{aligned} (W_n^N)^+\!\le \! E\{R_N^*|X_1,\ldots ,X_n\}f^n(X_1,\ldots ,X_n)\!\le \! E\{R^*|X_1,\ldots ,X_n\}f^n(X_1,\ldots ,X_n) \end{aligned}$$

almost surely for all $n=1,\ldots , N-1$, $N=1,2,\ldots$, and hence

$$\begin{aligned} W_n^+\le E\{R^*|X_1,\ldots ,X_n\}f^n(X_1,\ldots ,X_n) \end{aligned}$$

(4.29)

almost surely for all natural $n$.

It follows from (4.13) that

$$\begin{aligned} \sum _{i=1}^{n-1} Es_i^\psi (r_i-c_i)+\int t_n^\psi W_{n}\mathrm{d}\mu ^n-E t_n^\psi c_{n} =Q_0. \end{aligned}$$

(4.30)

Applying (4.29) on the left-hand side of (4.30), we have $ER^* -E t_n^\psi c_{n} \ge Q_0,$ or

$$\begin{aligned} E t_n^\psi c_{n} \le ER^* - Q_0. \end{aligned}$$

(4.31)

Let $k$ be any positive number. Then it follows from (4.31) that

$$\begin{aligned} kE t_n^\psi I_{\{c_n>k\}}\le E t_n^\psi c_{n} \le ER^* - Q_0. \end{aligned}$$

(4.32)

Because

$$\begin{aligned} E t_n^\psi = E t_n^\psi I_{\{ c_n>k\}}+E t_n^\psi I_{\{ c_n\le k\}} \end{aligned}$$

(4.33)

and the second summand by virtue of (4.27) tends to 0, as $n\rightarrow \infty$, we have that the difference between the first summand on the right-hand side of (4.33) and the left-hand side of it, goes to 0 as $n\rightarrow \infty$. Thus, from (4.32), we have that

$$\begin{aligned} \lim _{n\rightarrow \infty }E t_n^\psi =\lim _{n\rightarrow \infty }P(\tau _\psi \ge n)= P (\tau _\psi =\infty )<(ER^*-Q_0)/k. \end{aligned}$$

Since $k>0$ is arbitrary, $P (\tau _\psi =\infty )=0$ follows.