Valid p-values and expectations of p-values revisited

Abstract

We focus on valid definitions of p-values. A valid p-value (VpV) statistic can be used to make a prefixed level-$\alpha$ decision. In this context, Kolmogorov–Smirnov goodness-of-fit tests and the normal two-sample problem are considered. We examine an issue regarding the goodness-of-fit testability based on a single observation. We exemplify constructions of new test procedures, advocating practical reasons to implement VpV mechanisms. The VpV framework induces an extension of the conventional expected p-value (EPV) tool for measuring the performance of a test. Associating the EPV concept with the receiver operating characteristic (ROC) curve methodology, a well-established biostatistical approach, we propose a Youden’s index-based optimality to derive critical values of tests. In these terms, the significance level $\alpha = 0.05$ is suggested. We introduce partial EPV’s to characterize properties of tests including their unbiasedness. We provide the intrinsic relationship between the Bayes Factor (BF) test statistic and the BF of test statistics.

Appendix

Proof of Proposition 1

Consider, for $u > 0$,

$$ \begin{aligned} D(u) & = \left| {1 - \exp ( - \theta u) - I\left( {X_{1} \le u} \right)} \right| = \left| {1 - \exp ( - \theta u)} \right|I\left( {X_{1} > u} \right) + \left| { - \exp ( - \theta u)} \right|I\left( {X_{1} \le u} \right) \\ & = \left\{ {1 - \exp ( - \theta u)} \right\}I\left( {X_{1} > u} \right) + \exp ( - \theta u)I\left( {X_{1} \le u} \right), \\ \end{aligned} $$

where the function $1 - \exp ( - \theta u)$ increases and the function $\exp ( - \theta u)$ decreases with respect to $u > 0$. Then, the function $D(u)$ increases, for $u < X_{1}$, and decreases, for $u \ge X_{1}$. Thus,

$$ D_{1} (\theta ) = \sup_{0 < u < \infty } D\left( u \right) = \hbox{max} \left\{ {1 - \exp ( - \theta X_{1} ),\exp ( - \theta X_{1} )} \right\}. $$

Assume $1 - \exp ( - \theta X_{1} ) < \exp ( - \theta X_{1} )$. In this case, $\theta < \log \left( 2 \right)/X_{1}$ and $D_{1} (\theta ) = \exp ( - \theta X_{1} )$ that is a decreasing function with respect to $\theta .$ Assume $1 - \exp ( - \theta X_{1} ) \ge \exp ( - \theta X_{1} )$. In this case, $\theta \ge \log \left( 2 \right)/X_{1}$ and $D_{1} (\theta ) = 1 - \exp ( - \theta X_{1} )$ that is an increasing function with respect to $\theta .$ Thus, $D_{1} (\theta )$ decreases, for $\theta < \log \left( 2 \right)/X_{1}$, and increases, for $\theta \ge \log \left( 2 \right)/X_{1}$. That is, we conclude that $\inf D_{1} (\theta ) = D_{1} \left( {\log (2)/X_{1} } \right) = 0.5$. By virtue of (8), the proof is complete.□

Proof of Proposition 2

Define the notation $H_{0} (\theta_{0} )$ to indicate the hypothesis $H_{0}$ when the true value of $\theta$ is $\theta_{0}$. Now, we will obtain bounds related to the interval $C_{\beta }$. The function $u^{ - 1} \exp \left( {u - 1} \right),u > 0,$ has a global minimum at $u = 1$. Then, the threshold $A_{\beta }$ satisfies $A_{\beta } > 1$, in order to provide a solution of $\Pr_{{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\left( {\theta_{0} X_{1} } \right)^{ - 1} \exp \left( {\theta_{0} X_{1} - 1} \right) < A_{\beta } } \right\} = 1 - \beta$. Let $0 < u_{0} < 1 < u_{1}$ be roots of the equation $u^{ - 1} \exp \left( {u - 1} \right) = A_{\beta }$. The roots $0 < u_{0} < 1 < u_{1}$ exist, since $A_{\beta } > 1$ and the function $u^{ - 1} \exp \left( {u - 1} \right)$ monotonically decreases, for $0 < u \le 1$, and increases, for $u > 1$. This behavior of the function $u^{ - 1} \exp \left( {u - 1} \right)$ can be used to show that

$$ \begin{aligned} \beta & = {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\left( {\theta_{0} X_{1} } \right)^{ - 1} \exp \left( {\theta_{0} X_{1} - 1} \right) > A_{\beta } } \right\} \\ & = {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\left( {\theta_{0} X_{1} } \right)^{ - 1} \exp \left( {\theta_{0} X_{1} - 1} \right) > A_{\beta } ,\theta_{0} X_{1} \le 1} \right\} \\&\quad+ {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\left( {\theta_{0} X_{1} } \right)^{ - 1} \exp \left( {\theta_{0} X_{1} - 1} \right) > A_{\beta } ,\theta_{0} X_{1} > 1} \right\} \\ & = {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\theta_{0} X_{1} < u_{0} ,\theta_{0} X_{1} \le 1} \right\} + {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\theta_{0} X_{1} > u_{1} ,\theta_{0} X_{1} > 1} \right\} \\ & = {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\theta_{0} X_{1} < u_{0} } \right\} + {\Pr}{_{H_{0} \left( {\theta_{0} } \right)}} \left\{ {\theta_{0} X_{1} > u_{1} } \right\} \\&= F_{{X_{1} \sim\theta_{0} \exp \left( { - \theta_{0} x} \right),0}} \left( {u_{0} /\theta_{0} } \right) + 1 - F_{{X_{1} \sim\theta_{0} \exp \left( { - \theta_{0} x} \right),0}} \left( {u_{1} /\theta_{0} } \right) \\ & = 1 - \exp \left( { - u_{0} } \right) + \exp \left( { - u_{1} } \right). \\ \end{aligned} $$

This defines the system of equations

$$ 1 - \exp \left( { - u_{0} } \right) + \exp \left( { - u_{1} } \right) = \beta \;{\text{and}}\;\left( {u_{0} } \right)^{ - 1} \exp \left( { - u_{0} } \right) = \left( {u_{1} } \right)^{ - 1} \exp \left( { - u_{1} } \right). $$

(11)

Then, given $\beta$, one can derive values of $u_{0}$ and $u_{1}$ that do not depend on values of $\theta$ and provide $\Pr_{{H_{0} \left( {\theta_{0} } \right)}} \left\{ {u_{0} < \theta_{0} X_{1} < u_{1} } \right\} = 1 - \beta$. Figure 1 presents numerical solutions of (11), depending on $\beta \in (0,1)$. Then, we have $\log (2) \in (u_{0} ,u_{1} )$, for $\beta \le 0.75$. According to the proof of Proposition 1, $\inf_{0 < \theta < \infty } D_{1} (\theta ) = D_{1} \left( {\log (2)/X_{1} } \right) = 0.5$. That is, $\inf_{{\theta \in C_{\beta } }} D_{1} (\theta )$ $= \inf_{{u_{0} < \theta X_{1} < u_{1} }} D_{1} (\theta )$ $= D_{1} \left( {\log (2)/X_{1} } \right) = 0.5$, for $\beta \le 0.75$. By virtue of (8), this completes the proof.□

Fig. 1

The values of $u_{0}$ (solid lines) and $u_{1}$ (dashed lines), which satisfy (11), plotted against $\beta \in (0,1)$. The dotted lines show $\log (2)$

Proof of Proposition 3

It is clear that

$$ D_{1} (\theta ) = \hbox{max} \left\{ {F_{{X_{1} ,0}} \left( {X_{1} } \right),1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)} \right\}, \, F_{{X_{1} ,0}} \left( u \right) = \int_{ - \infty }^{u} {\exp \left( { - (z - \theta )^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} . $$

Assume $F_{{X_{1} ,0}} \left( {X_{1} } \right) \ge 1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)$, i.e., $F_{{X_{1} ,0}} \left( {X_{1} } \right) \ge 1/2$. In this case, $F_{{X_{1} ,0}} \left( {X_{1} } \right) = \left( {2\pi } \right)^{ - 1/2} \int_{ - \infty }^{{X_{1} - \theta }} {\exp \left( { - z^{2} /2} \right){\text{d}}z}$, where $\theta \le X_{1}$, and then $D_{1} (\theta ) = F_{{X_{1} ,0}} \left( {X_{1} } \right)$ is a decreasing function with respect to $\theta .$ Assume $F_{{X_{1} ,0}} \left( {X_{1} } \right) < 1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)$, i.e., $F_{{X_{1} ,0}} \left( {X_{1} } \right) < 1/2$. In this case, $\theta > X_{1}$ and $D_{1} (\theta ) = 1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)$ increases with respect to $\theta .$ Thus, $\inf_{ - \infty < \theta < \infty } D_{1} (\theta ) = D_{1} \left( {X_{1} } \right) = 0.5$. The point $\theta = X_{1}$ belongs to the interval $C_{\beta } = \left\{ {\theta :\,\exp \left( {\left( {X_{1} - \theta } \right)^{2} /2} \right) < A_{\beta } } \right\}$. Therefore, $\inf_{{\theta \in C_{\beta } }} D_{1} (\theta ) = D_{1} \left( {X_{1} } \right) = 0.5$. The proof is complete.□

Proof of Proposition 4

We have

$$ D_{1} (\theta ) = \hbox{max} \left\{ {F_{{X_{1} ,0}} \left( {X_{1} } \right),1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)} \right\}, \, F_{{X_{1} ,0}} \left( u \right) = \int_{ - \infty }^{u/\theta } {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} . $$

If $X_{1} > 0$, then $F_{{X_{1} ,0}} \left( {X_{1} } \right) > 1/2$ and $F_{{X_{1} ,0}} \left( {X_{1} } \right) > 1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)$. In this case, since $D_{1} (\theta ) = F_{{X_{1} ,0}} \left( {X_{1} } \right)$ is a decreasing function with respect to $\theta > 0$, $\inf_{0 < \theta < \infty } D_{1} (\theta ) = D_{1} \left( \infty \right) = 0.5$.

If $X_{1} < 0$, then $D_{1} (\theta ) = 1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)$ is a decreasing function with respect to $\theta > 0$, and $\inf_{0 < \theta < \infty } D_{1} (\theta ) = D_{1} \left( \infty \right) = 0.5$.

Now, we consider $p_{C}$. Note that since the function $u^{ - 1/2} \exp \left( {u/2 - 1/2} \right),u > 0,$ has a global minimum at $u = 1$, in order to provide a solution of $\Pr \left\{ {\eta^{ - 1/2} \exp \left( {\eta /2 - 1/2} \right) > A_{\beta } } \right\} = \beta$, where $\eta \sim\chi_{1}^{2}$, the threshold $A_{\beta }$ should satisfy $A_{\beta } > 1$. Thus, we have $0 < u_{0} < 1 < u_{1}$ that are roots of the equation $u^{ - 1} \exp \left( {u^{2} /2 - 1/2} \right) = A_{\beta }$ and

$$ C_{\beta } = \left\{ {\theta > 0 :\,\theta \left| {X_{1} } \right|^{ - 1} \exp \left( {\left| {X_{1} } \right|^{2} /\left( {2\theta^{2} } \right) - 0.5} \right) < A_{\beta } } \right\} = \left\{ {\theta > 0 :\,u_{0} < \left| {X_{1} } \right|/\theta < u_{1} } \right\}. $$

According to the above proof scheme, $D_{1} (\theta )$ is a decreasing function with respect to $\theta > 0$ and then we obtain $p_{C} = 1 - F_{{KS_{n} ,0}}^{{}} \left( {D_{1} \left( {\left| {X_{1} } \right|/u_{0} } \right)} \right) + \beta$, for $\theta \in C_{\beta }$, where

$$ \begin{aligned} D_{1} \left( {\left| {X_{1} } \right|/u_{0} } \right) & = \int_{ - \infty }^{{u_{0} }} {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} I\left( {X_{1} \ge 0} \right) \\ & \quad + \left\{ {1 - \int_{ - \infty }^{{ - u_{0} }} {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} } \right\}I\left( {X_{1} < 0} \right), \\ \end{aligned} $$

since $D_{1} (\theta ) = \hbox{max} \left\{ {F_{{X_{1} ,0}} \left( {X_{1} } \right),1 - F_{{X_{1} ,0}} \left( {X_{1} } \right)} \right\}$. The distribution function $\int_{ - \infty }^{u} {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} = 1 - \int_{ - \infty }^{ - u} {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2}$ is symmetric. This implies

$$ D_{1} \left( {\left| {X_{1} } \right|/u_{0} } \right) = \int_{ - \infty }^{{u_{0} }} {\exp \left( { - z^{2} /2} \right){\text{d}}z} /\left( {2\pi } \right)^{1/2} . $$

Now, one can easily use a simple R Code (R Development Core Team 2002) to compute the accurate Monte Carlo approximations to $p_{C} = 1 - F_{{KS_{n} ,0}}^{{}} \left( {D_{1} \left( {\int_{ - \infty }^{{u_{0} }} {\exp (- z^{2} /2 ) {\text{d}}z} /(2\pi )^{1/2} } \right)} \right) + \beta$, showing that $p_{C} \ge 1$ increases when $\beta$ increases. The proof is complete.□

Proof of Proposition 5

Consider, for non-random variables $u$ and $s$, the probability

$$ \begin{aligned} & \int {\Pr \left\{ {u - s \le B_{n} \le u|H_{1} } \right\}} \pi (\theta ){\text{d}}\theta = \int {E\left[ {I\left\{ {u - s \le B_{n} \le u} \right\}|H_{1} } \right]} \pi (\theta ){\text{d}}\theta \\ & \quad = \int {\left[ {\int {I\left\{ {u - s \le B_{n} \le u} \right\}} f_{1} (x_{1} , \ldots ,x_{n} ;\,\theta ){\text{d}}x_{1} \ldots {\text{d}}x_{n} } \right]} \pi (\theta ){\text{d}}\theta \\ & \quad = \int {I\left\{ {u - s \le B_{n} \le u} \right\}\left[ {\int {f_{1} (x_{1} , \ldots ,x_{n} ;\,\theta )\pi (\theta ){\text{d}}\theta } } \right]} {\text{d}}x_{1} \ldots {\text{d}}x_{n} \\ & \quad = \int {I\left\{ {u - s \le B_{n} \le u} \right\}\left[ {\int {\frac{{f_{1} (x_{1} , \ldots ,x_{n} ;\,\theta )}}{{f_{0} (x_{1} , \ldots ,x_{n} )}}\pi (\theta ){\text{d}}\theta } } \right]} f_{0} (x_{1} , \ldots ,x_{n} ){\text{d}}x_{1} \ldots {\text{d}}x_{n}\\ & = \int I \left\{ {u - s \le B_{n} \le u} \right\}\left( {B_{n} } \right)f_{0} . \\ \end{aligned} $$

This implies the inequalities

$$ \begin{aligned} & \int {\Pr \left\{ {u - s \le B_{n} \le u|H_{1} } \right\}} \pi (\theta ){\text{d}}\theta \le \int I \left\{ {u - s \le B_{n} \le u} \right\}\left( u \right)f_{0} \\&\quad= u\Pr \left\{ {u - s \le B_{n} \le u|H_{0} } \right\}\;{\text{and}} \\ & \int {\Pr \left\{ {u - s \le B_{n} \le u|H_{1} } \right\}} \pi (\theta ){\text{d}}\theta \ge \int I \left\{ {u - s \le B_{n} \le u} \right\}\left( {u - s} \right)f_{0} \\&\quad= \left( {u - s} \right)\Pr \left\{ {u - s \le B_{n} \le u|H_{0} } \right\}. \\ \end{aligned} $$

Dividing these inequalities by s and employing $s \to 0$, we obtain Proposition 5.□

Proof of Proposition 6

Define the power function $g(u) = \Pr (p{\text{-value}} < u|H_{1} )$. We have $\int_{0}^{\alpha } {g(u)} {\text{d}}u \ge \alpha^{2} /2$, where $\int_{0}^{\alpha } {g(u)} {\text{d}}u = \left. {g(u)u} \right|_{u = 0}^{u = \alpha } - \int_{0}^{\alpha } {uw(u)} {\text{d}}u$, $w(u) = {\text{d}}g(u)/{\text{d}}u$. Since $g(u) = \Pr \left\{ {1 - F_{T,0} (T) < u|H_{1} } \right\} = 1 - \Pr \left\{ {T < F_{T,0}^{ - 1} (1 - u)|H_{1} } \right\} = 1 - F_{T,1} \left( {F_{T,0}^{ - 1} (1 - u)} \right)$, we obtain $w(u) = f_{T,1} (C_{u} )/f_{T,0} (C_{u} )$ with $C_{u} = F_{T,0}^{ - 1} (1 - u)$. It is clear that when $u \nearrow$, the corresponding critical values $C_{u} \searrow$ and then the likelihood ratio $f_{T,1} (C_{u} )/f_{T,0} (C_{u} ) \searrow$. This implies $\alpha^{2} /2 \le \int_{0}^{\alpha } {g(u)} {\text{d}}u = g(\alpha )\alpha - \int_{0}^{\alpha } {uw(u)} {\text{d}}u \le g(\alpha )\alpha - w(\alpha )\int_{0}^{\alpha } u {\text{d}}u$ that completes the proof.□