Internal validation of models with several interventions

Abstract

In cost-effectiveness analyses, models are used typically to synthesize the best available data and/or extrapolate beyond clinical trial data. Ideally, models should be validated both internally and externally. The purpose of this paper is to suggest a test for internal validation of models where several interventions for the same clinical indication are compared. To the best of our knowledge, such a specific test does not yet exist. There are four versions of the test, which consider the relationship between incremental downstream costs and effects in the case of a single or several endpoints. We apply two versions of the validation test to published cost-effectiveness analyses of physical activity programs and demonstrate internal validity of the model in one study and lack of internal validity of the model in the other study.

Appendix

Proof of version 3

Consider four interventions A, B, C, and D, which are compared in the short term in terms of the ratio of incremental downstream costs to incremental downstream effects:

$$ \begin{gathered} \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{D\;{\text{vs}} .\;C, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } + \sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{D\;{\text{vs}} .\;C, \, i}} \times \alpha_{i} \times C_{{{\text{survival}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{D{\text{ vs}} . { }C, \, i \, }} \times w_{i} } }} + \theta = \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta \text{CE}_{{C\;{\text{vs}} .\;B, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } + \sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{C\;{\text{vs}} .\;B, \, i \, }} \times \alpha_{i} \times C_{{{\text{survival}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{C\;{\text{vs}} .\;B, \, i \, }} \times w_{i} } }} + \vartheta = \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{B\;{\text{vs}} .\;A, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } + \sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{B\;{\text{vs}} .\;A, \, i \, }} \times \alpha_{i} \times C_{{{\text{survival}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{B\;{\text{vs}} .\;A, \, i \, }} \times w_{i} } }} \end{gathered} $$

(9)

where $\theta$ and $\vartheta$ are two constants used to equalize the ratios of incremental downstream costs to incremental downstream effects. The ratios for the different interventions are not the same unless clinical events do not differ in terms of costs and severity.

When all interventions prevent death by preventing clinical events over the period of effectiveness (condition 2), the resulting costs of survival until the next period are the same for each clinical event prevented, i.e., $C_{\text{survival}}$ is identical for all interventions. Hence, the ratio of life extension costs to the weighted-average effect is the same for all interventions and thus can be denoted by a constant ($K_{3}$):

$$ \begin{gathered} \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{D\;{\text{vs}} .\;C, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{D\;{\text{vs}} .\;C, \, i \, }} \times w_{i} } }} + K_{3} + \theta \hfill \\ = \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{C{\text{ vs}} . { }B, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{C\;{\text{vs}} .\;B, \, i \, }} \times w_{i} } }} + K_{3} + \vartheta \hfill \\ = \frac{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{B\;{\text{vs}} .\;A, \, i \, }} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {\Updelta {\text{CE}}_{{B\;{\text{vs}} .\;A, \, i \, }} \times w_{i} } }} + K_{3} \hfill \\ \end{gathered} $$

(10)

Deleting $K_{3}$ and considering that the number of CEs avoided is a function of incidence (Inc) and relative risk reduction (RRR) yields:

$$ \begin{gathered} \frac{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D{\text{ vs}} . { }C}} \times (1 - \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D{\text{ vs}} . { }C}} \times w_{i} } }} + \theta \hfill \\ = \frac{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C{\text{ vs}} . { }B}} \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C{\text{ vs}} . { }B}} \times w_{i} } }} + \vartheta \hfill \\ = \frac{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B{\text{ vs}} . { }A}} \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B{\text{ vs}} . { }A}} \times w_{i} } }} \hfill \\ \end{gathered} $$

(11)

When determining the relative change in cost-effectiveness beyond the trial period over the time horizon k we need to consider the relative change in incidence, death rate, and costs of clinical events:

$$ \begin{gathered} \frac{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D\;{\text{vs}} .\;C}} \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D\;{\text{vs}} .\;C}} \times w_{i} } }} + \frac{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \theta }}{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta Inc_{{t_{n} - t_{n - 1} }} } \right)} }} \hfill \\ = \frac{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C\;{\text{vs}} .\;B}} \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C\;{\text{vs}} .\;B}} \times w_{i} } }} + \frac{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{CE, \, t_{n} - t_{n - 1} }} } \right)} \times \vartheta }}{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta Inc_{{t_{n} - t_{n - 1} }} } \right)} }} \hfill \\ = \frac{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B\;{\text{vs}} .\;A}} \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {{\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B\;{\text{vs}} .\;A}} \times w_{i} } }} \hfill \\ \end{gathered} $$

(12)

When deleting $\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right)}$ from the denominator of each term, each term is multiplied with $\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \, \times \, \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)}$. Hence, differences between interventions in terms of the ratio of incremental downstream costs to incremental downstream effects change proportional to $\coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta {\text{Inc}}_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 - \Updelta \alpha_{{t_{n} - t_{n - 1} }} } \right) \times \left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)}$, thus proving version 3 of the test.

Proof of version 4

When extrapolating cost-effectiveness beyond the trial period over the time horizon k, we introduce factors k ₁ and k ₂ to describe that the relative change in incidence of each clinical event (${\text{Inc}}_{i}$) and the relative change in death rate of each CE ($1 - \Updelta \alpha_{i}$) stay constant over k periods. Using Eq. 11 as a basis we obtain Eq. 13.

$$ \begin{gathered} \frac{{k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D\;{\text{vs}} .\;C}} \times \left( {1 - \Updelta \alpha_{{{\text{death}},i}} } \right) \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, D\;{\text{vs}} .\;C}} \times w_{i} } }} + \frac{{k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \left( {1 - \Updelta \alpha_{{{\text{death}},i}} } \right) \times \theta }}{{k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right)} }} \hfill \\ = \frac{{k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C\;{\text{vs}} .\;B}} \times \left( {1 - \Updelta \alpha_{{{\text{death}},i}} } \right) \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, C\;{\text{vs}} .\;B}} \times w_{i} } }} + \frac{{k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \left( {1 - \Updelta \alpha_{{{\text{death}},i}} } \right) \times \vartheta }}{{k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta Inc_{i} } \right)} }} \hfill \\ = \frac{{k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B\;{\text{vs}} .\;A}} \times \left( {1 - \Updelta \alpha_{{{\text{death}},i}} } \right) \times (1 \, - \, \alpha_{{{\text{death}},i}} ) \times C_{{{\text{CE}}, \, i}} } }}{{k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right) \times {\text{Inc}}_{i} \times {\text{RRR}}_{{i, \, B\;{\text{vs}} .\;A}} \times w_{i} } }} \hfill \\ \end{gathered} $$

(13)

When deleting $k_{1} \times \sum\nolimits_{i = 1}^{m} {\left( {1 + \Updelta {\text{Inc}}_{i} } \right)}$ from the denominator of each term it is easy to see that each term is multiplied with $k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)}$. Hence, differences between interventions in terms of the ratio of incremental downstream costs to incremental downstream effects change proportional to $k_{1} \times k_{2} \times \coprod\nolimits_{n = 1}^{k} {\left( {1 + \Updelta C_{{{\text{CE}}, \, t_{n} - t_{n - 1} }} } \right)}$, thus proving version 4 of the test.