The Wally plot approach to assess the calibration of clinical prediction models

Abstract

A prediction model is calibrated if, roughly, for any percentage x we can expect that x subjects out of 100 experience the event among all subjects that have a predicted risk of x%. Typically, the calibration assumption is assessed graphically but in practice it is often challenging to judge whether a “disappointing” calibration plot is the consequence of a departure from the calibration assumption, or alternatively just “bad luck” due to sampling variability. We propose a graphical approach which enables the visualization of how much a calibration plot agrees with the calibration assumption to address this issue. The approach is mainly based on the idea of generating new plots which mimic the available data under the calibration assumption. The method handles the common non-trivial situations in which the data contain censored observations and occurrences of competing events. This is done by building on ideas from constrained non-parametric maximum likelihood estimation methods. Two examples from large cohort data illustrate our proposal. The ‘wally’ R package is provided to make the methodology easily usable.

Appendices

A Constrained Kaplan–Meier estimates

Let us assume that we observe the data $\big \{\big (\widetilde{T}_i,\varDelta _i\big ),i=1,\ldots ,n\big \}$, where $\widetilde{T}_i=\min (T_i,C_i)$ and $\varDelta _i={\mathbb {1}}\{ T_i \le C_i \}$. The constrained Kaplan–Meier estimate $\widehat{S}(u)$ of $S(u)=\mathbb {P}(T>u)$, with constraint $\widehat{S}(t)=p$, for a given value $p \in ]0,1[$, is defined as

$$\begin{aligned} \widehat{S}(u)&= \prod _{i \, : \, \widetilde{T}_i \le u} \left( 1 - \frac{d_i}{n_i + \lambda }\right) \quad \text{ for } \text{ all } \quad u \le t, \\ \text{ and } \quad \widehat{S}(u)&= \left\{ \prod _{i: \, \widetilde{T}_i \le t} \left( 1 - \frac{d_i}{n_i + \lambda }\right) \right\} \left\{ \prod _{i \, : \, t< \widetilde{T}_i \le u } \left( 1 - \frac{d_i}{n_i }\right) \right\} \quad \text{ for } \text{ all } \quad t < u, \end{aligned}$$

where $d_i=\sum _{j=1}^n {\mathbb {1}}\{ \widetilde{T}_j = \widetilde{T}_i \}\varDelta _j$ is the number of observed events at time $\widetilde{T}_i$, where $n_i=\sum _{j=1}^n {\mathbb {1}}\{ \widetilde{T}_j \ge \widetilde{T}_i \}$ is number of subjects at risk at time $\widetilde{T}_i$ and with $\lambda \in \mathbb {R}$ such that $\widehat{S}(t)=p$ (Thomas and Grunkemeier 1975). The usual Kaplan–Meier estimator corresponds to the above formulas in the special case where $\lambda =0$.

B Constrained Aalen–Johansen estimates

Let us assume that we observe the data $\big \{\big (\widetilde{T}_i,\widetilde{\eta }_i\big ),i=1,\ldots ,n\big \}$ where $\widetilde{\eta }_i=\varDelta _i\eta _i$. For for sake of clarity, we further assume that there is no ties in the sample $\big \{\widetilde{T}_i, i=1,\ldots ,n\big \}$. Therefore, without loss of generality, for the formulas below we assume to observe $0< \widetilde{T}_1< \cdots < \widetilde{T}_n$. In particular, this implies $n_i=\sum _{j=1}^n {\mathbb {1}}\{ \widetilde{T}_j \ge \widetilde{T}_i \}=n-(i-1)$ for all $i=1,\ldots ,n$.

The constrained Aalen–Johansen estimates $\widehat{F}_{k}^{(1)}(u)$ of the cumulative incidence functions of event $k=1,2$ at time u, that is $F_{k}(u)=\mathbb {P}(T \le u, \eta =k)$, with constraint $\widehat{F}_{1}^{(1)}(t)=p$, for a given $p \in ]0,1[$, is defined as

$$\begin{aligned} \widehat{F}_{k}^{(1)}(u)&= \sum _{i \, : \, \widetilde{T}_i \le u}\left\{ \prod _{j=1}^{i-1} \left( 1 - \widehat{a}_{1,j}^{(1)} - \widehat{a}_{2,j}^{(1)} \right) \widehat{a}_{k,i}^{(1)} \right\} \end{aligned}$$

where

$$\begin{aligned} \widehat{a}_{1,i}^{(1)}&= \frac{ {\mathbb {1}}\{ \widetilde{\eta }_i=1 \} }{ n - (i-1) - \lambda \left\{ 1 - \widehat{F}_{2}^{(1)}(\widetilde{T}_{i-1}) - p \right\} } \quad \text{ if } \quad i \quad \text{ is } \text{ such } \text{ that } \quad \widetilde{T}_{i} \le t, \\ \text{ and } \qquad \widehat{a}_{1,i}^{(1)}&= \frac{ {\mathbb {1}}\{ \widetilde{\eta }_i=1 \} }{ n - (i-1) } \qquad \qquad \qquad \qquad \qquad \qquad \ \text{ if } \quad i \quad \text{ is } \text{ such } \text{ that } \quad \widetilde{T}_{i} > t, \nonumber \end{aligned}$$

and

$$\begin{aligned} \widehat{a}_{2,i}^{(1)}&= \frac{ {\mathbb {1}}\{ \widetilde{\eta }_i=2 \} }{ n - (i-1) - \lambda \left\{ \widehat{F}_{1}^{(1)}(\widetilde{T}_{i-1}) - p \right\} } \quad \text{ if } \quad i \quad \text{ is } \text{ such } \text{ that } \quad \widetilde{T}_{i} \le t, \\ \text{ and } \qquad \widehat{a}_{2,i}^{(1)}&= \frac{ {\mathbb {1}}\{ \widetilde{\eta }_i=2 \} }{ n - (i-1) } \qquad \qquad \qquad \qquad \qquad \ \ \text{ if } \quad i \quad \text{ is } \text{ such } \text{ that } \quad \widetilde{T}_{i} > t. \nonumber \end{aligned}$$

In the above equations, we define $\widetilde{T}_{0}=0$, $\widehat{F}_{1}^{(1)}(0)=\widehat{F}_{2}^{(1)}(0)=0$ and $\lambda \in \mathbb {R}$ such that $\widehat{F}_{1}^{(1)}(t)=p$. The superscript $^{(1)}$ refers to the fact that the constraint relates to the cumulative incidence function of event 1. Following similar ideas to those used to derive the formulas of “Appendix A” by Thomas and Grunkemeier (1975), these formulas were derived from maximizing the following non-parametric likelihood $L= \prod _{i=1}^n a_{1,i}^{{\mathbb {1}}\{ \widetilde{\eta }_i=1 \}} a_{2,i}^{{\mathbb {1}}\{ \widetilde{\eta }_i=2 \}} \big ( 1 - a_{1,i} - a_{2,i} \big )^{n-i}$, under the constraint $\widehat{F}_{1}^{(1)}(t)=p$, using the Lagrange multiplier technique. Extensions of the above formulas can also be derived to account for ties in $\big \{\widetilde{T}_i, i=1,\ldots ,n\big \}$ (Blanche 2017). The usual Aalen–Johansen estimator corresponds to the above formulas in the special case where $\lambda =0$.