Tree based diagnostic procedures following a smooth test of goodness-of-fit

Abstract

This paper introduces a statistical procedure, to be applied after a goodness-of-fit test has rejected a null model, that provides diagnostic information to help the user decide on a better model. The procedure goes through a list of departures, each being tested by a local smooth test. The list is organized into a hierarchy by seeking answers to the questions “Where is the problem?” and “What is the problem there?”. This hierarchy allows to focus on finer departures as the data becomes more abundant. The procedure controls the family-wise Type 1 error rate. Simulations show that the procedure can succeed in providing useful diagnostic information.

Appendix: Proof of theorems

We prove Theorem 1and concentrate on the more complex ${\mathcal {R}}_{M}^{{\hat{Q}}}$. To avoid trivial problems, assume that ${\hat{b}}-{\hat{a}}>0$. With ${\hat{X}}_{i}^{*}=F_{0}^{{\hat{Q}}}(X_{i})$ and $X_{i}^{*}=F_{0}^{Q}(X_{i})$, consider

$$\begin{aligned} \frac{1}{\sqrt{N^{{\hat{Q}}}}}\underset{X_{i}\in {\mathcal {E}}^{{\hat{Q}}}}{\sum }L_{m}\left( {\hat{X}}_{i}^{*}\right) =\left( \frac{\sqrt{n}}{\sqrt{N^{{\hat{Q}}}}}\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}L_{m}^{*}\left( {\hat{X}}_{i}^{*}\right) . \end{aligned}$$

Now, $L_{m}^{*}({\hat{X}}_{i}^{*})-L_{m}^{*}(X_{i}^{*})=C(X_{i})+D(X_{i}),$ where $C(x)={\mathbb {I}}\{ x\in Q\} [L_{m}(F_{0}^{{\hat{Q}}}(x))-L_{m}(F_{0}^{Q}(x))]$ and $D(x)=L_{m}(F_{0}^{{\hat{Q}}}(x))[{\mathbb {I}}\{ x\in {\hat{Q}}\} -{\mathbb {I}}\{ x\in Q\}].$ Hence,

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}L_{m}^{*}\left( {\hat{X}}_{i}^{*}\right) =\frac{1}{\sqrt{n}}\sum _{i=1}^{n}L_{m}^{*}\left( X_{i}^{*}\right) +\frac{1}{\sqrt{n}}\sum _{i=1}^{n}C(X_{i})+\frac{1}{\sqrt{n}}\sum _{i=1}^{n}D(X_{i}).\qquad \end{aligned}$$

(7.1)

Consider the second term on the right hand side of this equation. $F_{0}^{Q}(x)$ is differentiable in a, b except at $a=x$ et $b=x$. Because $X_{i}\ne a,b$ with probability 1, we can Taylor expand $L_{m}(F_{0}^{{\hat{Q}}}(x))-L_{m}(F_{0}^{Q}(x))$ about (a, b) for $x\in (a,b)$ to get

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}C(X_{i}) \!= & {} \! \left[ \frac{1}{n}\underset{i=1}{\overset{n}{\sum }}{\mathbb {I}}\left\{ X_{i}\!\in \! Q\right\} \left( \begin{array}{c} \frac{\partial }{\partial a}L_{m}\left( F_{0}^{Q}\left( X_{i}\right) \right) \\ \frac{\partial }{\partial b}L_{m}\left( F_{0}^{Q}\left( X_{i}\right) \right) \end{array}\right) \right] ^{T}\sqrt{n}\left( \begin{array}{c} {\hat{a}}-a\\ {\hat{b}}-b \end{array}\right) \!+\!o_{p}\left( 1\right) . \end{aligned}$$

By the law of large numbers, the term in bracket converges to its expectation. Because ${\mathbb {E}}_{H_{0}^{Q}}({\mathbb {I}}\{ X_{i}\in Q\} L_{m}(F_{0}^{Q}(X_{i})))=0$, Leibniz’s integral rule yields

$$\begin{aligned} {\mathbb {E}}_{H_{0}^{Q}}\left( {\mathbb {I}}\left\{ X_{i}\in Q\right\} \left( \begin{array}{c} \frac{\partial }{\partial a}L_{m}\left( F_{0}^{Q}\left( X_{i}\right) \right) \\ \frac{\partial }{\partial b}L_{m}\left( F_{0}^{Q}\left( X_{i}\right) \right) \end{array}\right) \right) =\left( \begin{array}{c} L_{m}(0)\\ -L_{m}(1) \end{array}\right) . \end{aligned}$$

Hence

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}C(X_{i})&= \left( \begin{array}{c} L_{m}(0)\\ -L_{m}(1) \end{array}\right) ^{T}\sqrt{n}\left( \begin{array}{c} {\hat{a}}-a\\ {\hat{b}}-b \end{array}\right) +o_{p}(1). \end{aligned}$$

(7.2)

Now, regarding the term involving the $D(X_{i})$ in (7.1), it is easy to see that

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}D(X_{i})= & {} \left( L_{m}(0)+o_{p}(1)\right) \text {sgn}\left( a-{\hat{a}}\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ X_{i}\in \left\langle a,{\hat{a}}\right\rangle \right\} \\&+\,\left( L_{m}(1)+o_{p}(1)\right) \text {sgn}\left( {\hat{b}}-b\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ X_{i}\in \left\langle b,{\hat{b}}\right\rangle \right\} , \end{aligned}$$

where $\langle a,{\hat{a}}\rangle = (\min (a,{\hat{a}}),\max (a,{\hat{a}}))$ and similarly for $\langle b,{\hat{b}}\rangle$. Moreover

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ X_{i}\in \left\langle a,{\hat{a}}\right\rangle \right\} =\text {sgn}\left( {\hat{a}}-a\right) \left( \alpha _{n}\left( {\hat{a}}\right) -\alpha _{n}(a)\right) +\text {sgn}\left( {\hat{a}}-a\right) \sqrt{n}\left( {\hat{a}}-a\right) , \end{aligned}$$

where $\alpha _{n}(\cdot )$ is the stochastic process $\sqrt{n}({\hat{F}}_{n}(\cdot )-\cdot )$. From the Komlos et al. (1976) strong approximation, there exists a sequence of Brownian bridges $B_{n}(\cdot )$ uniformly approximating $\alpha _{n}(\cdot )$ to the order $O\left( \frac{\log n}{\sqrt{n}}\right)$. Thus

$$\begin{aligned} \text {sgn}\left( a-{\hat{a}}\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ x\in \left\langle a,{\hat{a}}\right\rangle \right\} \!=\!\left( B_{n}(a)-B_{n}\left( {\hat{a}}\right) \right) \!-\!\sqrt{n}\left( {\hat{a}}-a\right) +O\left( \frac{\log n}{\sqrt{n}}\right) . \end{aligned}$$

The term $B_{n}(a)-B_{n}(\hat{a)}=o_{p}(1)$ by the continuity of Brownian bridges. Hence,

$$\begin{aligned} \left( L_{m}(0)\!+\!o_{p}(1)\right) \text {sgn}\left( a-{\hat{a}}\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ X_{i}\!\in \!\left\langle a,{\hat{a}}\right\rangle \right\} \!= & {} \!-L_{m}(0)\sqrt{n}\left( {\hat{a}}-a\right) \!+\!o_{p}(1),\\ \left( L_{m}(1)+o_{p}(1)\right) \text {sgn}\left( {\hat{b}}-b\right) \frac{1}{\sqrt{n}}\sum _{i=1}^{n}{\mathbb {I}}\left\{ X_{i}\in \left\langle b,{\hat{b}}\right\rangle \right\}= & {} L_{m}(1)\sqrt{n}({\hat{b}}-b)+o_{p}(1). \end{aligned}$$

Regrouping, we get:

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}D(X_{i})=\left( \begin{array}{c} -L_{m}(0)\\ L_{m}(1) \end{array}\right) ^{T}\sqrt{n}\left( \begin{array}{c} {\hat{a}}-a\\ {\hat{b}}-b \end{array}\right) +o_{p}(1), \end{aligned}$$

(7.3)

Going back to (7.1), we find after combining (7.2) and (7.3)

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^{n}L_{m}^{*}\left( {\hat{X}}_{i}^{*}\right) =\frac{1}{\sqrt{n}}\underset{i=1}{\overset{n}{\sum }}L_{m}^{*}\left( X_{i}^{*}\right) +o_{p}(1). \end{aligned}$$

Finally, as a byproduct of the above, $\frac{N^{{\hat{Q}}}}{n}=\frac{N^{Q}}{n}+o_{p}(1)=F(Q)+o_{p}(1)$ because $N^{Q}$ has the binomial distribution B(n, F(Q)). Hence the asymptotic behavior of the ${\mathcal {L}}_{m}^{{\hat{Q}}}$ are the same as that of the ${\mathcal {L}}_{m}^{Q}$ which are easily shown to independent $\chi _{1}^{2}$ . $\square$

Next, we prove Theorem 2. Suppose for simplicity that $Q_{S}$ are single intervals. We consider the more complex case where $Q_{S}$ are estimated by ${\hat{Q}}_{S}$. Define the adjusted p value as $\pi _{S}^{(adj)}=\frac{1}{{\mathbb {P}}_{H_{0}}[{\hat{Q}}_{S}]}\pi _{S}$, where $\pi _{S}$ pertains to ${\mathcal {R}}_{M}^{\hat{_{Q}}_{S}}$. Let ${\mathcal {T}}_{0}$ be as in Theorem 2. Define the hierarchical adjusted p value as

$$\begin{aligned} \pi _{S}^{(h,adj)}=\underset{S\subseteq C\in {\mathcal {T}}}{\max }\pi _{C}^{(adj)}. \end{aligned}$$

(7.4)

Let ${\mathcal {T}}_{rej}=\{ S\in {\mathcal {T}};H_{0}^{Q_{S}} \text{ is } \text{ rejected } \text{ by } \text{ the } \text{ rule: } \pi _{S}^{(h,adj)}<\alpha \}$. It is easy to see that the null hypotheses rejected using the hierarchical adjusted p value coincide with those rejected using the $\pi _{S}^{(adj)}$. In particular, no null hypothesis gets rejected if its parent has not been rejected because $\pi _{S}^{(h,adj)}\ge \pi _{pa(S)}^{(h,adj)}$. The probability of a family-wise Type 1 error can be written as

$$\begin{aligned} {\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ {\mathcal {T}}_{rej}\cap {\mathcal {T}}_{0}\ne \emptyset \right] ={\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ \exists S\in {\mathcal {T}}_{0}: \pi _{S}^{(h,adj)}<\alpha \right] . \end{aligned}$$

Let $\widetilde{{\mathcal {T}}}_{0}$ be a subset of ${\mathcal {T}}$ maximal in the sense that $\widetilde{{\mathcal {T}}}_{0}:=\{ S\in {\mathcal {T}}_{0}: \not \exists C\in {\mathcal {T}}_{0}\text { with }S\subset C\}$. Obviously $\widetilde{{\mathcal {T}}}_{0}\subseteq {\mathcal {T}}_{0}$. Also the definition of $\pi _{S}^{(h,adj)}$ implies that a falsely rejected $S\in {\mathcal {T}}_{0}-\widetilde{{\mathcal {T}}}_{0}$, implies a falsely rejected $S^{\prime }\in \widetilde{{\mathcal {T}}}_{0}$, where $S\subset S^{\prime }$. Thus, we need only to look at the probability of committing a Type 1 error in $\widetilde{{\mathcal {T}}}_{0}$. But because $\pi _{S}^{(h,adj)}\geqslant \pi _{S}^{(adj)}$,

$$\begin{aligned} {\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ \exists S\in {\mathcal {T}}_{0}:\,\pi _{S}^{(h,adj)}<\alpha \right] \le \underset{S\in \widetilde{{\mathcal {T}}}_{0}}{\sum }{\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ \pi _{S}^{(adj)}<\alpha \right] , \end{aligned}$$

by Bonferroni’s inequality. Notice that ${\mathbb {P}}_{H_{0}}[{\hat{Q}}_{S}] ={\mathbb {E}}_{H_{0}}({\hat{b}}-{\hat{a}}) = {\mathbb {P}}_{H_{0}}[Q_{S}]+o(1)$. Now writing $G_{n}^{S}(\cdot )$ and $G_{\infty }^{S}(\cdot )$ for the exact and asymptotic CDF under $\tilde{{\mathcal {T}}}_{0}$ of test statistic ${\mathcal {R}}_{M}^{\hat{Q_{S}}}$, we have

$$\begin{aligned} {\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ \pi _{S}^{(adj)}<\alpha \right]&={\mathbb {P}}_{{\mathcal {T}}_{0}}\left[ \pi _{S}<\alpha {\mathbb {P}}_{H_{0}}[Q_{S}]+o(1)\right] \\&=1-G_{n}^{S}\left( G_{\infty }^{S^{-1}}\left( 1-\alpha P_{H_{0}}[Q_{S}]+o(1)\right) \right) \\&=\alpha {\mathbb {P}}_{H_{0}}[Q_{S}]+o(1). \end{aligned}$$

Hence, $\sum \nolimits _{S\in \widetilde{{\mathcal {T}}}_{0}}{\mathbb {P}}_{{\mathcal {T}}_{0}}[\pi _{S}^{(adj)}<\alpha ] <\alpha \sum \nolimits _{S\in \widetilde{{\mathcal {T}}}_{0}}{\mathbb {P}}_{H_{0}}[Q_{S}]+o(1).$ It only remains to show that $\sum _{S\in \widetilde{{\mathcal {T}}}_{0}}{\mathbb {P}}_{H_{0}}[Q_{S}] \leqslant 1$. But by the construction of $\widetilde{{\mathcal {T}}}_{0}$, $\forall S\ne S^{\prime }\in \widetilde{{\mathcal {T}}}_{0}: S\cap S^{\prime }=\emptyset$. Hence $\bigcup _{S\in \widetilde{{\mathcal {T}}}_{0}}S\subseteq \{ 1,\ldots ,K\}$. Because ${\mathbb {P}}_{H_{0}}[Q_{S}]={\mathbb {P}}_{H_{0}}[X\in \bigcup _{k\in S}P_{k}]$, we have

$$\begin{aligned} \underset{S\in \widetilde{{\mathcal {T}}}_{0}}{\sum }{\mathbb {P}}_{H_{0}}[Q_{S}]\leqslant {\mathbb {P}}_{H_{0}}\left[ X\in \bigcup _{k=1}^{K}P_{k}\right] =1, \end{aligned}$$

and this completes the proof. $\square$

Finally, we prove Theorem 3 with given $Q_{S}$ for simplicity. From the above, it suffices to show that $\sum \nolimits _{S\in \tilde{{\mathcal {T}}}_{0}}{\mathbb {P}}_{H_{0}}^{eff}[Q_{S}] \le 1$. For simplicity, assume there exists only one $S^{*}\in \widetilde{{\mathcal {T}}}_{0}$ such that ${\mathbb {P}}_{H_{0}}^{eff}[Q_{S^{*}}]>{\mathbb {P}}_{H_{0}}[Q_{S^{*}}]$. Because the tree is binary,

$$\begin{aligned} \underset{S\in \widetilde{{\mathcal {T}}}_{0}}{\sum }{\mathbb {P}}_{H_{0}}^{\text {eff }}[Q_{S}]&=\underset{S\in \widetilde{{\mathcal {T}}}_{0}{\setminus } S^{*}}{\sum }{\mathbb {P}}_{H_{0}}[Q_{S}]+{\mathbb {P}}_{H_{0}}[Q_{S^{*}}]+{\mathbb {P}}_{H_{0}}\left[ si(Q_{S^{*}})\right] . \end{aligned}$$

Now, identifiability along with the relation ${\mathbb {P}}_{H_{0}}^{\text {eff }}[S^{*}]>{\mathbb {P}}_{H_{0}}[S^{*}]$ imply that $si(S^{*})\notin \widetilde{{\mathcal {T}}}_{0}$, for otherwise $pa(S^{*})\in \widetilde{{\mathcal {T}}}_{0}$, which in turn implies $S^{*}\notin \widetilde{{\mathcal {T}}}_{0}$. The conclusion follows from

$$\begin{aligned} \underset{S\in \widetilde{{\mathcal {T}}}_{0}{\setminus } S^{*}}{\sum }{\mathbb {P}}_{H_{0}}[Q_{S}]+{\mathbb {P}}_{H_{0}}\left[ Q_{S^{*}}\right] \le 1-{\mathbb {P}}_{H_{0}}\left[ si\left( Q_{S^{*}}\right) \right] .\quad \quad \quad \quad \square \end{aligned}$$