Power calculation in multiply imputed data

Abstract

Multiple imputation (MI) has been proven an effective procedure to deal with incomplete datasets. Compared with complete case analysis (CCA), MI is more efficient since it uses the information provided by incomplete cases which are simply discarded in CCA. A few simulation studies have shown that statistical power can be improved when MI is used. However, there is a lack of knowledge about how much power can be gained. In this article, we build a general formula to calculate the statistical power when MI is used. Specific formulas are given for several different conditions. We demonstrate our finding through simulation studies and a data example.

Appendix

1.1 Compare type I error of different methods

To evaluate the comparison between CCA and MI, we also provide the proportion of rejecting null hypothesis if the null hypothesis is true ($\alpha$). We are showing the results in Tables 3 and 4. ‘P’ represents the percentage of missing values. ‘Complete’ represents the complete data results, ‘MI’ represents the MI results, and ‘CCA’ represents the CCA results.

Table 3 Proportion of rejecting $H_0$ under $H_0$, small $B_E$

Table 4 Proportion of rejecting $H_0$ under $H_0$, large $B_E$

Fig. 6

Simulations under different alternative hypothesis with $d = 1$. The two plots are correspondingly representing the results from setup 1 (0 vs 1) and setup 2 (1 vs 2). In both of the setups, $d = 1$

Fig. 7

Simulations under different alternative hypothesis with $d = 0.5$. The two plots are correspondingly representing the results from setup 1 (0 vs 0.5) and setup 2 (1 vs 1.5). In both of the setups, $d = 0.5$

In terms of the type I error, there is not much differences between the three methods no matter the size of $B_E$. All methods reach a type I error rate close to the predetermined value 0.05.

1.2 Explore the effect of changing alternative hypothesis, sample size n and number of imputations m

As requested by reviewers, we explored the effect of changing the simulation parameters and evaluated the bias between simulated power and calculated power under different setups.

To explore if different alternative hypothesis could affect the results, we had four different setups shown in Figs. 6 and 7. In the first two setups, we set $n=100$, $m=10$, and $Q_1 - Q_0 = 1$. The only difference between the first two setups is that in setup 1, $Q_0 = 0$, and in setup 2, $Q_0 = 1$. Similarly, the last two setups have same sample size and number of imputation, $Q_1 - Q_0 = 0.5$. In setup 3, $Q_0 = 0$, and in setup 4, $Q_0 = 1$. All four setups used the same random seed and the number of Monte Carlo samples (iterations) was 10000 across all setups.

The simulation results shows that if d, the difference between null hypothesis and alternative hypothesis is constant, the simulation result won’t change. In other words, it is not the alternative hypothesis, but the difference between two hypothesis that determined the simulation results.

We examined the performance of the power calculation method by comparing the simulated and theoretical power under different combinations of m and n. In the simulation, we use $Q_0 = 1$, $Q_1 = 1.5$, and number of Monte Carlo samples (i.e., the number of iterations) is 10,000. The results are shown in Fig. 8.

In addition, we calculated the average of absolute difference between simulated and calculated power, see Table 5.

The simulation shows that while there exists a minimal bias. The bias could be either positive or negative, and due to the its small amount, it may not of big problem.

Fig. 8

Simulations under different different combinations of m and n, d = 0.5

Table 5 Average absolute difference (abs diff) between the simulated and theoretical powers under different combinations of m and n

1.3 When the population and MI imputation model have different distributions

To examine how the method perform if the population follows a different distribution, we examined the situation where the population follows a t distribution with degree of freedom as 3. In the simulation shown in Fig. 9, we have the random error follow a t-distribution instead of a standard normal distribution; In the simulations shown in Fig. 10, we keep the variance of Y to be 20. For the large $B_E$ setup, the correlation coefficient between X and Y is about 0.32. For the small $B_E$ setup that is about 0.92.

Fig. 9

Comparison between theoretical and simulated power under MCAR, one dimensional X, under t distribution. In the simulation, $\mathbf {X} \sim N(0, 4^{2})$, ${\varvec{\beta ^*}} = (1, 1)$, ${\varvec{\epsilon }} \sim {\mathrm{t}}_{3}$, and $m = 10$. Each point in the figure represents the theoretical power and the simulated power for one population. Ideally, the points should be clustered along the forty-five-degree line

Fig. 10

MCAR, different p, under t distribution. The population follows a t distribution with degree of freedom is 3. Besides that, we keep the variance of Y to be 20. For the large $B_E$ setup, the correlation coefficient between X and Y is about 0.32. For the small $B_E$ setup that is about 0.92

It can be seen from Fig. 9 that while the population follows a t distribution instead, the simulated power now is slightly greater than the calculated power, meaning our method now generates a small negative bias. In Table 6. Such phenomenon can also be observed from Fig. 10.

Table 6 Average difference (diff) and average squared difference (MSE) between the simulated and theoretical powers