On the role of Volterra integral equations in self-consistent, product-limit, inverse probability of censoring weighted, and redistribution-to-the-right estimators for the survival function

Abstract

This paper reconsiders several results of historical and current importance to nonparametric estimation of the survival distribution for failure in the presence of right-censored observation times, demonstrating in particular how Volterra integral equations help inter-connect the resulting estimators. The paper begins by considering Efron’s self-consistency equation, introduced in a seminal 1967 Berkeley symposium paper. Novel insights provided in the current work include the observations that (i) the self-consistency equation leads directly to an anticipating Volterra integral equation whose solution is given by a product-limit estimator for the censoring survival function; (ii) a definition used in this argument immediately establishes the familiar product-limit estimator for the failure survival function; (iii) the usual Volterra integral equation for the product-limit estimator of the failure survival function leads to an immediate and simple proof that it can be represented as an inverse probability of censoring weighted estimator; (iv) a simple identity characterizes the relationship between natural inverse probability of censoring weighted estimators for the survival and distribution functions of failure; (v) the resulting inverse probability of censoring weighted estimators, attributed to a highly influential 1992 paper of Robins and Rotnitzky, were implicitly introduced in Efron’s 1967 paper in its development of the redistribution-to-the-right algorithm. All results developed herein allow for ties between failure and/or censored observations.

Appendix

1.1 Proof of Proposition 1

Proof

To prove (8), we begin by noting that $t < X_{(m)}$ guarantees that $Y(u+) > 0$ for $u \le t$. Hence, using results briefly summarized in Gill and Johansen (1990, p. 1526) for general forms of anticipating Volterra integral equations, the unique solution to (7) is given by

(22)

Here, the finite product form given in the last equality on the right-hand side arises because $C(\cdot )$ can only jump at a finite set of times. To see that (22) reduces to (8), observe that straightforward algebra yields

$$\begin{aligned} 1/\left\{ 1 + \frac{\Delta C(u)}{Y(u+)} \right\} = \frac{Y(u+)}{Y(u+) + \Delta C(u)} = 1 - \frac{\Delta C(u)}{Y(u+) + \Delta C(u)}. \end{aligned}$$

Now, we can write $Y(u)-Y(u+) = \Delta N(u) + \Delta C(u);$ hence, $Y(u+) + \Delta C(u) = Y(u) - \Delta N(u) = Y^{\dagger }(u)$ and it follows immediately that

(23)

establishing (8).

To see that result (9) follows under the assumption that $\Delta N(u) \Delta C(u) = 0$ for $u< X_{(m)}$, we simply note

$$\begin{aligned} \widehat{K}(t) = \prod _{(0,t]} \left\{ 1 - \frac{\Delta C(u)}{Y^{\dagger }(u)} \right\} = \prod _{(0,t]} \left\{ 1 - \frac{\Delta C(u)}{Y(u)- \Delta N(u)} \right\} = \prod _{(0,t]} \left\{ 1 - \frac{\Delta C(u)}{Y(u)} \right\} , \end{aligned}$$

which is equivalent to (9). This latter form is also the known unique solution to the integral equation for ${\widehat{K}}(t)$ stated in the theorem for the case where $\Delta N(u) \Delta C(u) = 0$ for $u< X_{(m)};$ see, for example, Theorem II.6.1 in Andersen et al. (1993, Sec. II.6). $\square$

1.2 Proof of Proposition 2

Proof

The result (11) follows immediately for $t = 0$ and $t \ge X_{(m)}$ by previous arguments; hence, we must only show that ${\widehat{S}}_{ SC }(t)$ equals ${\widehat{S}}_{ PL }(t)$ given in (10) for $t \in (0,X_{(m)}).$

Suppose ${\widehat{S}}(t)$ solves (4), equivalently (3), for $t \in (0,X_{(m)}).$ Proposition 1 and the definition of ${\widehat{K}}(t)$ in (6) then imply that the solution ${\widehat{S}}(t)$ to (4) necessarily must also satisfy

$$\begin{aligned} \widehat{S}(t) \times \prod _{(0,t]} \left\{ 1 - \frac{\Delta C(u)}{Y^{\dagger }(u)} \right\} = \frac{Y(t+)}{n}. \end{aligned}$$

The crux of the proof thus involves determining a suitable relationship between $n^{-1} Y(t+)$ and the product-limit form of $\widehat{K}(t)$ in Proposition 1.

We begin by finding a comparable finite-product representation for $n^{-1} Y(t+)$. Using a telescoping product and the fact that $Y(0+) = n$, we can write

$$\begin{aligned} \frac{Y(t+)}{n} = \frac{Y(t+)}{Y(0+)} = \prod _{0< u \le t} \frac{Y(u+)}{Y(u)} = \prod _{0 < u \le t} \left\{ 1 - \frac{\Delta N(u)}{Y(u)} - \frac{\Delta C(u)}{Y(u)} \right\} , \end{aligned}$$

where the last equality follows from $Y(u+) = Y(u) - \Delta N(u) - \Delta C(u)$. Now, since $Y^{\dagger }(u) = Y(u) - \Delta N(u),$ observe that

$$\begin{aligned} \frac{\Delta C(u)}{Y(u)} = \frac{Y^{\dagger }(u)}{Y(u)} \frac{\Delta C(u)}{Y^{\dagger }(u)} = \left\{ 1 - \frac{\Delta N(u)}{Y(u)} \right\} \frac{\Delta C(u)}{Y^{\dagger }(u)}; \end{aligned}$$

hence,

$$\begin{aligned} 1 - \frac{\Delta N(u)}{Y(u)} - \frac{\Delta C(u)}{Y(u)} = \left\{ 1 - \frac{\Delta N(u)}{Y(u)} \right\} \left\{ 1 - \frac{\Delta C(u)}{Y^{\dagger }(u)} \right\} . \end{aligned}$$

Taking the product of both sides over all jumps, we see that

(24)

Therefore, since ${\widehat{K}}(t) {\widehat{S}}(t) = n^{-1} Y(t+),$ we have

the latter being given in (10) and proving the desired result. We note here that the relationship derived in (24) is that originally given in Gill (1980, p. 36); to see this, note that (i) $Y^{\dagger }(t) = Y(t) - \Delta N(t);$ and, (ii) $Y(t+) = Y(t) - \Delta N(t) - \Delta C(t)$ gives $\Delta C(t) = Y(t) - Y(t+) - \Delta N(t).$ $\square$

1.3 Proof of Proposition 3

Proof

The result is trivial for $t =0$. For $t > 0,$ previous results imply that the relation (6) can be rewritten as $\widehat{K}(u) \widehat{S}_{ PL }(u) = Y(u+)/n;$ as seen in (24), this relation is in fact valid for $u > 0.$ Taking the limit of this equation from the left, it follows that $\widehat{K}(u-) \widehat{S}_{ PL }(u-) = Y(u)/n$ for $u > 0$. Since $Y(u) > 0$ for $u \le X_{(m)}$, we have $\widehat{K}(u-) > 0$ and $\widehat{S}_{ PL }(u-) > 0$ for $u \le X_{(m)}$. Substitution of $\widehat{S}_{ PL }(u-) = n^{-1} Y(u) / \widehat{K}(u-)$ into the right-hand side of the integral Eq. (12) then gives

$$\begin{aligned} \widehat{F}_{ PL }(t) = 1 - \widehat{S}_{ PL }(t) = \frac{1}{n} \int _{(0,t]} \frac{\textrm{d}N(u)}{\widehat{K}(u-)}. \end{aligned}$$

This last expression remains well-defined for $t > X_{(m)}$ since $\widehat{K}(X_{(m)}-) > 0$ and $\Delta N(u) = 0$ for $u > X_{(m)}$. The desired result now follows immediately upon evaluating the integral. $\square$

1.4 Proof of Proposition 4

Proof

The proof of Proposition 3 and (14) imply

$$\begin{aligned} \widehat{S}_{ PL }(X_{(m)}) = 1 - \frac{1}{n} \sum _{i=1}^n \frac{D_i }{\widehat{K}(X_i-)} - \widetilde{S}_{ IPCW }(X_{(m)}); \end{aligned}$$

since $X_{(m)}$ is the largest observation time and $\widehat{K}(X_i-) >0$ for each i,  it is trivially true that $\widetilde{S}_{ IPCW }(X_{(m)}) = 0.$ Consequently,

$$\begin{aligned} \widehat{S}_{ PL }(X_{(m)}) = 1 - \frac{1}{n} \sum _{i=1}^n \frac{D_i }{\widehat{K}(X_i-)}, \end{aligned}$$

and (16) therefore holds if and only if

$$\begin{aligned} \widehat{S}_{ PL }(X_{(m)}) = \frac{\Delta C(X_{(m)})}{n \widehat{K}(X_{(m)}-)}. \end{aligned}$$

Towards this end, note that

$$\begin{aligned} \widehat{S}_{ PL }(X_{(m)}) = \widehat{S}_{ PL }(X_{(m)}-) \left( 1 - \frac{\Delta N(X_{(m)})}{Y(X_{(m)})} \right) = \widehat{S}_{ PL }(X_{(m)}-) \frac{\Delta C(X_{(m)})}{Y(X_{(m)})}, \end{aligned}$$

the last equality following from the fact that $Y(X_{(m)}) = \Delta N(X_{(m)}) + \Delta C(X_{(m)})$. Since $\widehat{K}(u-) \widehat{S}_{ PL }(u-) = Y(u)/n$ for $u > 0,$ we have for $u = X_{(m)}$ that

$$\begin{aligned} \frac{\widehat{S}_{ PL }(X_{(m)}-)}{Y(X_{(m)})} = \frac{1}{n \widehat{K}(X_{(m)}-)}, \end{aligned}$$

proving (16). In view of the fact that $Y(X_{(m)}) = \Delta N(X_{(m)}) + \Delta C(X_{(m)}),$ we also see that (15) holds if and only if $\Delta C(X_{(m)}) = 0$ $\Leftrightarrow$ $Y(X_{(m)}) = \Delta N(X_{(m)}).$ Finally, it is immediate from (14) that $\widetilde{S}_{ IPCW }(t) = 1- \widehat{F}_{ IPCW }(t)$ for $t \ge 0$ if and only if (15) holds. $\square$

1.5 Proof of Proposition 5

Proof

Because the RTTR algorithm starts with a total mass of one, the preservation of mass through redistribution implies that

$$\begin{aligned} \sum _{k=1}^{m-1} J_{(k)} \Delta N(X_{(k)}) + Y(X_{(m)}) J_{(m)} = 1. \end{aligned}$$

(25)

Hence, using (21), we find that

$$\begin{aligned} {\widehat{S}}_{ RTTR }(t)= & {} 1- \sum _{r=1}^{k_t} \left( J_{(r)} \Delta N(X_{(r)}) I\{ r< m \} + J_{(m)} Y(X_{(m)}) I\{ r = m \} \right) \\= & {} {\left\{ \begin{array}{ll} 1 - \sum _{r=1}^{m-1} J_{(r)} \Delta N(X_{(r)}) - J_{(m)} Y(X_{(m)}) &{} t \ge X_{(m)} \\ 1 - \sum _{r=1}^{k_t} J_{(r)} \Delta N(X_{(r)}) &{} t< X_{(m)} \end{array}\right. } \\= & {} {\left\{ \begin{array}{ll} 0 &{} t \ge X_{(m)} \\ 1 - \sum _{r=1}^{k_t} \frac{\Delta N(X_{(r)})}{n {\widehat{K}}(X_{(r)}-)}&{} t < X_{(m)} \end{array}\right. }, \end{aligned}$$

the last representation respectively following from (25) and (20). Focusing on the case where $t < X_{(m)},$ it is easily seen that

$$\begin{aligned} \sum _{r=1}^{k_t} J_{(r)} \Delta N(X_{(r)})= \sum _{i=1}^n I\{ X_i \le t \} \frac{D_i}{n {\widehat{K}}(X_i-)} = {\widehat{F}}_{ IPCW }(t), \end{aligned}$$

where ${\widehat{F}}_{ IPCW }(t)$ is defined in (13). Therefore,

$$\begin{aligned} {\widehat{S}}_{ RTTR }(t) = {\left\{ \begin{array}{ll} 0 &{} t \ge X_{(m)} \\ 1 - {\widehat{F}}_{ IPCW }(t) &{} t< X_{(m)} \end{array}\right. } \, = \, {\left\{ \begin{array}{ll} 0 &{} t \ge X_{(m)} \\ {\widehat{S}}_{ PL }(t) &{} t < X_{(m)} \end{array}\right. }, \end{aligned}$$

the last equality following by Proposition 3 and (14). The result (11) in Proposition 2 now establishes the desired equivalence, that is, ${\widehat{S}}_{ RTTR }(t) = {\widehat{S}}_{ SC }(t)$ for $t \ge 0.$

Moreover, these same calculations prove that ${\widehat{F}}_{ RTTR }(t) = 1 - {\widehat{S}}_{ RTTR }(t) ={\widehat{F}}_{ IPCW }(t)$ for $t < X_{(m)};$ by Proposition 4, this relationship between the IPCW and RTTR estimator holds for $t \ge 0$ under the additional condition that $\Delta N(X_{(m)}) = Y(X_{(m)})$ (i.e., all observations at the last observation time are failures), in which case ${\widehat{S}}_{ RTTR }(t) = {\widetilde{S}}_{ IPCW }(t)$ for $t \ge 0.$

The arguments above are predicated on the validity of (25); we now provide independent verification that (25) holds. Recalling the fact that $Y(X_{(m)}) = \Delta N(X_{(m)}) + \Delta C(X_{(m)}),$ we can rewrite the left-hand side of (25) as

$$\begin{aligned} \sum _{k=1}^{m} J_{(k)} \Delta N(X_{(k)}) + \Delta C(X_{(m)}) J_{(m)}. \end{aligned}$$

Using the definition of $J_{(k)},$ it is easy to show that

$$\begin{aligned} \sum _{k=1}^{m} J_{(k)} \Delta N(X_{(k)}) = \frac{1}{n} \sum _{i=1}^n \frac{D_i}{\widehat{K}(X_i-)} \end{aligned}$$

and

$$\begin{aligned} \Delta C(X_{(m)}) J_{(m)} = \frac{\Delta C(X_{(m)})}{n {\widehat{K}}(X_{(m)}-)}; \end{aligned}$$

hence, the left-hand side of (25) reduces to

$$\begin{aligned} \frac{1}{n} \sum _{i=1}^n \frac{D_i}{\widehat{K}(X_i-)} + \frac{\Delta C(X_{(m)})}{n {\widehat{K}}(X_{(m)}-)}, \end{aligned}$$

which is indeed equal to one by (16). $\square$