Power Considerations in Designed Experiments

Abstract

Designed experiments for ANOVA studies are ubiquitous across all areas of scientific endeavor. An important decision facing experimenter is that of the experiment run size. Often the run size is chosen to meet a desired level of statistical power. The conventional approach in doing so uses the lower bound on statistical power for a given experiment design. However, this minimum power specification is conservative and frequently calls for larger experiments than needed in many settings. At the very least, it does not give the experimenter the entire picture of power across competing arrangements of the factor effects. In this paper, we propose to view the unknown effects as random variables, thereby inducing a distribution on statistical power for an experimental design. The power distribution can then be used as a new way to assess experimental designs. It turns out that using the proposed expected power criterion often recommends smaller, less costly, experimental designs.

Appendices

Appendix

A Proof of Theorem 4

Proof

By definition, the expected value of $\eta (D,{\tau }^a)$ satisfies

$$\begin{aligned} 1 - E\left[ \eta (D,{\tau }^a)\right]= & {} \int P(F_{k-1,n_e,\phi }\le c|{\tau }^a)\cdot f_{\tau }(\tau ^a)\mathrm{d}{\tau }^a \\= & {} \int \sum \limits _{s=0}^\infty \frac{e^{-\phi /2}(\phi /2)^s}{ s!} \cdot I\left( \frac{(k-1)c}{n_e+(k-1)c}\bigg |\frac{k-1}{2}+s, \frac{n_e}{2}\right) \displaystyle {f_{\tau }(\tau ^a)\mathrm{d}{\tau }^a}. \end{aligned}$$

For any design D and value of $\tau ^a$, $\eta (D,{\tau }^a)$ as a power function is bounded between 0 and 1. Therefore, $1 - E\left[ \eta (D,{\tau }^a)\right]$ is also bounded between 0 and 1. As both the cdf of F distribution and $f_{\tau }(\cdot )$ are nonnegative, by Fubini’s theorem, the integration and summation in the above equation are interchangeable. We have

$$\begin{aligned} 1 - E\left[ \eta (D,{\tau }^a)\right]= & {} \sum \limits _{s=0}^\infty I\left( \frac{(k-1)c}{n_r+(k-1)c}\bigg |\frac{k-1}{2}+s,\frac{n_e}{2}\right) \cdot \int \frac{e^{-\phi /2}(\phi /2)^s}{ s!}\cdot \displaystyle {f_{\tau }(\tau ^a)\mathrm{d}{\tau }^a} \nonumber \\= & {} \sum _s^{\infty }I\left( \frac{(k-1)c}{n_e+(k-1)c}\bigg |\frac{k-1}{2} +s,\frac{n_e}{2}\right) \cdot g(s). \end{aligned}$$

$\square$

B Multivariate Truncated Normal Distribution

One example for $f_{\tau }(\cdot )$ that is employed in this work for illustration purposes is the truncated normal distribution with truncation range $(-\Delta /2,\Delta /2)$, which implies that $\tau _1=\Delta /2$ and $\tau _2=-\Delta /2$. This is done to maintain effect size of $\Delta$. If we further assume that $\tau _i,\ i=3,\ldots ,k,$ are mutually independent and have expected value of 0, we can write the joint probability density function for $\tau$ as

$$\begin{aligned} f_\tau (\mathbf{\tau })= & {} \displaystyle {\prod _{i=3}^k \frac{ \frac{1}{\sigma _\tau }f_z\left( \frac{\tau _i}{\sigma _\tau }\right) \cdot I_{[-\Delta /2\le \tau _i\le \Delta /2]}}{\Phi \left( \frac{\Delta }{2\sigma _\tau }\right) - \Phi \left( -\frac{\Delta }{2\sigma _\tau }\right) }}\nonumber \\= & {} \left[ \sigma _\tau \left( 2\Phi \left( \frac{\Delta }{2\sigma _\tau }\right) -1\right) \right] ^{2-k} \prod _{i=3}^k f_z\left( \frac{\tau _i}{\sigma _\tau }\right) \cdot I_{[-\Delta /2\le \tau _i\le \Delta /2]}, \end{aligned}$$

(9)

where $\sigma _\tau$ is the scale parameter of the distribution, and $f_z(\cdot )$ is the pdf of standard normal distribution.

C Considerations for Numerical Approximations

As discussed earlier, truncation of an infinite sum and numerical method are employed to obtain the expected power in (6). That is

$$\begin{aligned} E\left[ \eta (D,\mathbf{\tau }^a)\right] = 1 - \sum _{s=0}^\infty I^*\left( s\right) \cdot g(s) \approx 1 - \sum _{s=0}^M I^*\left( s\right) \cdot g(s). \end{aligned}$$

Here, we provide some theoretical justification and a numerical illustration on the accuracy of the above approximation. First, we derive an upper bound of the approximation error. Note that the incomplete beta function is bounded above by the corresponding beta function, that is to say,

$$\begin{aligned} I\left( \frac{(k-1)c}{n_r+(k-1)c}\bigg |\frac{k-1}{2}+s,\frac{n_e}{2}\right) \le B\left( \frac{k-1}{2}+s,\frac{n_e}{2}\right) \le \frac{1}{\left( \frac{k-1}{2}+s\right) \cdot \frac{n_e}{2}}. \end{aligned}$$

Furthermore, g(s) can be viewed as the expected value of a Poisson probability mass function with parameter $\phi /2$ as

$$\begin{aligned} g(s) = \int \frac{e^{-\phi /2}(\phi /2)^s}{ s!}\cdot \displaystyle {f_{\tau }(\tau ^a)\mathrm{d}{\tau }^a} = E_{\tau ^a}\left[ \frac{e^{-\phi /2}(\phi /2)^s}{s!}\right] . \end{aligned}$$

Denote by $\phi _{\mathrm{max}}$ the maximum $\phi$ (which corresponds to the maximum power) given $f_{\tau }(\cdot )$, $\sigma$ and the experimental setting, we can then assert that for $s>\lceil \phi /2 \rceil$,

$$\begin{aligned} g(s) \le \frac{e^{-\phi _{\mathrm{max}}/2} (\phi _{\mathrm{max}}/2)^s}{s!}, \end{aligned}$$

by property from the Poisson distribution. Therefore, each term in the summation in equation (6) is bounded above by

$$\begin{aligned} \frac{1}{\left( \frac{k-1}{2}+s\right) \cdot \frac{n_e}{2}} \cdot \frac{e^{-\phi _{\mathrm{max}}/2} (\phi _{\mathrm{max}}/2)^s}{s!}. \end{aligned}$$

It follows that the truncation error $\gamma$ which is $1 - \sum _{s=0}^M I^*\left( s\right) \cdot g(s) - E\left[ \eta (D,\mathbf{\tau }^a)\right]$ satisfies

$$\begin{aligned} \gamma&= \sum _{s=M+1}^\infty I^*\left( s\right) \cdot g(s)\\&\le \sum _{s=M+1}^\infty \frac{2}{1+s}\cdot \frac{e^{-\phi _{\mathrm{max}}/2} (\phi _{\mathrm{max}}/2)^s}{s!}\\&= 2(\phi _{\mathrm{max}}/2)^{-1} \sum _{s=M+2}^\infty \frac{e^{-\phi _{\mathrm{max}}/2} (\phi _{\mathrm{max}}/2)^s}{s!}\\&\le 2(\phi _{\mathrm{max}}/2)^{-1}\cdot \left( 1-\frac{\phi _{\mathrm{max}}/2}{M+3}\right) \cdot \frac{e^{-\phi _{\mathrm{max}}/2} (\phi _{\mathrm{max}}/2)^{M+2}}{(M+2)!} \\&\textit{(upper bound for Poisson tail probability)} \end{aligned}$$

With this upper bound and some acceptable numeric error level $\gamma$, one can choose a large enough M ( $M > \lceil \phi _{\mathrm{max}}/2 \rceil$) to construct a finite sum as approximation for the infinite sum, such that the numeric error of such approximation is within $[0,\gamma )$.

Fig. 4

Values of $I^*\left( s\right) \cdot g(s)$ (upper) and $\sum _0^M I^*\left( s\right) \cdot g(s)$ (lower) for $s\le 30$ and $M\le 30$. The distribution for ${\tau }^a$ is assumed to be truncated normal with standard deviation $\sigma _\tau =0.33$ and $\mu _\tau =0$ (left), truncated normal with standard deviation $\sigma _\tau =0.33$ and $\mu _\tau =0.5$ (second to the left), uniform (third to the left) and mixture of normals (right), respectively

In the material example, factor A has $\phi _{\mathrm{max}} = 24$ with $\Delta _A = 2$. It can be easily calculated that when $M = \lceil \phi _{\mathrm{max}}/2 \rceil = 12$, the approximation error $\gamma = 0.002$ (relative error $\gamma /0.8074 \times 100\% = 0.25\%$), and when $M = 20$, $\gamma$ drops to 0.00024 (relative error $\gamma /0.8074 \times 100\% = 0.03\%$). In Fig. 4, $I^*(s)\cdot g(s)$ and $\sum _{t=0}^M I^*(s)\cdot g(s)$ are plotted with $s \le 30$ and $M\le 30$, for each factor in the material example with the four illustrative distributions. It can be seen that when s is large enough, $I^*(s)\cdot g(s)$ decreases and approaches zero, which means that $\sum _{t=0}^s I^*(t)\cdot g(t)$ starts to become more steady and converge to its limit, $\sum _{t=0}^\infty I^*(t)\cdot g(t)$.

The computational resources required to complete numerical approximation of the expected power for the material example are presented in Table 9, together with approximation accuracy measures. Results are produced on a computer with 2.5 GHz Intel Core i7 processor and 6 GB 1600 MHz DDR3 memory. Different sizes of Monte Carlo in approximating g(s) are considered for each factor. It can be seem that within reasonable computation time (no longer than two seconds), satisfactory approximation (standard deviation no larger than 0.005) of expected power can be achieved.

Table 9 Computation time and error