Wavelet-Based Estimation of Generalized Discriminant Functions

Abstract

In this work we propose a wavelet-based classifier method for binary classification. Basically, based on a training data set, we provide a classifier rule with minimum mean square error. Under mild assumptions, we present asymptotic results that provide the rates of convergence of our method compared to the Bayes classifier, ensuring universal consistency and strong universal consistency. Furthermore, in order to evaluate the performance of the proposed methodology for finite samples, we illustrate the approach using Monte Carlo simulations and real data set applications. The performance of the proposed methodology is compared with other classification methods widely used in the literature: support vector machine and logistic regression model. Numerical results showed a very competitive performance of the new wavelet-based classifier.

Appendix: Proofs

In this appendix we provide the proofs of the theorems presented in Section 3. The proof of Theorem 1 is partially based on the proof of Theorem 1 in Montoril, Morettin and Chiann (2014) and the proof of Theorem 1 of Huang and Shen (2004). We use two lemmas and one proposition, which are given below. For the sake of simplicity, we will use hereafter the symbol≲ to represent the magnitude order O, i.e., we will write a_n ≲ b_n to represent a_n = O(b_n), for the two positive sequences a_n and b_n. Hereafter, we will use the abbreviation a.s. to denote “almost surely”.

Lemma 1.

Under assumptions of the model in Section 3,

Lemma 2.

Under assumptions of the model in Section 3, there are 0 < A ≤ B < ∞such that all the eigenvalues of$\frac {1}{n} \boldsymbol {Z}^{\top } \boldsymbol {Z}$fallin [A, B] a.s., asn → ∞, whereZis an × 2^Jmatrix such that itsi-th line corresponds toϕ_Jk(X_i), k = 0, 1,…, 2^J − 1.

Proposition 1.

Following the notations in the proof of Lemma 1, for anyk, l = 0, 1,…, 2^J − 1, η > 0, s ≥ 3, 2r < γ < 1 and 0 < δ < γ − 2r,

Proof of Theorem 1.

Observe that

where $f_{J}(x) = {\sum }_{k = 0}^{2^{J}-1} c_{Jk} \phi _{Jk}(x)$ is the orthogonal projection of f on V_J. Since $\| f_{J} - f \|^{2} = {{\rho }_{J}^{2}}$, it will be enough to verify that .

Note that the least squares estimator of the wavelet coefficients c_J can be written as

$$\hat{\boldsymbol{c}}_{J} = (\boldsymbol{Z}^{\top} \boldsymbol{Z})^{-1} \boldsymbol{Z}^{\top} \boldsymbol{Y}^{*}, $$

whereZ is a n × 2^J matrix such that its i-th line corresponds to ϕ_Jk(X_i), k = 0, 1,…, 2^J − 1, and $\boldsymbol {Y}^{*} = ({Y}_{1}^{*}, \ldots , {Y}_{n}^{*})^{\top }$, with ${Y}_{i}^{*} = 2Y_{i} - 1$, as in model (3.1). Denote $\bar {\boldsymbol {Y}}^{*} = ({\bar {Y}}_{1}^{*}, \ldots , {\bar {Y}}_{n}^{*})^{\top }$, where ${\bar {Y}}_{i}^{*} = f(X_{i})$, and define $\bar {\boldsymbol {c}}_{J} = (\boldsymbol {Z}^{\top } \boldsymbol {Z})^{-1} \boldsymbol {Z}^{\top } \bar {\boldsymbol {Y}}^{*}$. Thus,

$$\boldsymbol{c}_{J} - \bar{\boldsymbol{c}}_{J} = (\boldsymbol{Z}^{\top} \boldsymbol{Z})^{-1} \boldsymbol{Z}^{\top} \boldsymbol{\epsilon}, $$

where 𝜖 = (𝜖₁,…,𝜖_n)^⊤, with 𝜖_i defined as in the model (3.1).

Since the errors are assumed to be iid and independent of the covariates . This implies that

Then, by Lemma 2,

Hence, by Parseval’s identity,

Again, applying the Parseval’s identity and the Lemma 2,

Once $\boldsymbol {Z} \bar {\boldsymbol {c}}_{J} = \boldsymbol {Z}(\boldsymbol {Z}^{\top } \boldsymbol {Z})^{-1} \boldsymbol {Z}^{\top } \bar {\boldsymbol {Y}}^{*}$ is an orthogonal projection, by (i) and (ii),

which implies that

The desired result follows from the fact that , by (A.1) and (A.2).

Proof of the Corolary 1.

By Corollary 6.2 in Devroye et al. (1996) and by assumption (ii),

Thus, applying the expectation in both sides, the result follows from Theorem 1,

Proof of Theorem 2..

By Corollary 6.2 in Devroye et al. (1996), we see that

$$L(g_{J}) - L^{*} \leq \sqrt{ \int (\hat{f}_{J}(x) - f(x)^{2} \mu(dx) }. $$

In order to verify the convergence of the right hand side of the inequality above, observe that the integral above can be written as

Combining the last two terms above and applying the assumption (ii),

Now observe that

where the class of functions $\mathcal {T}$ is defined by

$$\mathcal{T} = \left\{ h(x,y) = (f(x)-y^{*})^{2} \colon \: f \in V_{J} \right\}. $$

Since y^∗ = − 1or y^∗ = 1,|y^∗| = 1. Furthermore, since |ϕ(x)|≤ W, for some positive constant W, we have that

$$\begin{array}{@{}rcl@{}} 0 &\leq& h(x, y^{*}) = (f(x) - y^{*})^{2} = \left( \sum\limits_{k = 0}^{2^{J} - 1} c_{Jk} \phi_{Jk}(x) - y^{*} \right)^{2}\\ & \leq& \left( \sum\limits_{k = 0}^{2^{J} - 1} |c_{Jk}| |\phi_{Jk}(x)| + |y^{*}| \right)^{2} \\ & \leq& \left( W 2^{J/2}\sum\limits_{k = 0}^{2^{J} - 1} |c_{Jk}| + 1 \right)^{2} \\ & \leq& (W_{1} 2^{J} + 1 )^{2} \leq C 2^{2J}, \end{array} $$

for some positive constant W₁. Observe that we used the fact that there exists a positive constant C such that ${\sum }_{k = 0}^{2^{J} - 1} |c_{Jk}| \leq C 2^{J/2}$.

By Theorem 29.1 in Devroye et al. (1996) and (A.4),

where ${{Z}_{1}^{n}} = (X_{1}, Y_{1}), \ldots , (X_{n},Y_{n})$, and $\mathcal {N} (\epsilon , \mathcal {T}({{z}_{1}^{n}}))$ is the ℓ₁-covering number of $\mathcal {T}({{z}_{1}^{n}})$, as in Definition 29.1 in Devroye et al. (1996).

For fixed ${{z}_{1}^{n}}$, one can estimate $\mathcal {N} \left (\frac {\epsilon }{16}, \mathcal {T}({{z}_{1}^{n}})\right )$. For arbitrary functions f₁, f₂ ∈ V_J, denote h₁(x, y^∗) = (f₁(x) − y^∗)² and h₂(x, y^∗) = (f₂(x) − y^∗)². Then, for any probability measure ν on

for some positive constant C₁. Thus, for any ${{z}_{1}^{n}} = (x_{1}, {y}_{1}^{*}), \ldots , (x_{n}, {y}_{n}^{*})$ and 𝜖,

$$\mathcal{N} (\epsilon, \mathcal{T}({{z}_{1}^{n}})) \leq \mathcal{N} \left( \frac{\epsilon}{C_{1} 2^{J}},V_{J}({{x}_{1}^{n}})\right), $$

where$V_{J}({{x}_{1}^{n}}) = \{ (f(x_{1}), \ldots , f(x_{n})): f \in V_{J} \}$. Thus, it suffices to estimate the covering number corresponding toV_J, which is a subspace of a linear space of functions. Then, following the Definitions 12.1 and 12.3, and Theorem 13.9 in Devroye et al. (1996), $V_{V_{J}^{+}} \leq 2^{J}$. Hence, by Corollary 29.2 in the same reference,

$$\begin{array}{@{}rcl@{}} \mathcal{N} \left( \frac{\epsilon/16}{C_{1} 2^{J}}, V_{J}({x_{1}^{n}}) \right) \!\!\!\!&\leq&\!\!\!\! \left[ \frac{ 4 e W 2^{J} }{ \epsilon/(16 C_{1} 2^{J}) } \log \left( \frac{ 2 e W 2^{J} }{ \epsilon/(16 C_{1} 2^{J}) } \right) \right]^{2^{J}}\\ &=&\!\!\!\! [C_{2} 2^{2J} \log (C_{2} 2^{2J} )]^{2^{J}}\\ &\leq&\!\!\!\! (C_{3} 2^{4J})^{2^{J}}, \end{array} $$

(A.6)

for some positive constant C₃.

By (A.5) and (A.6),

where C₃ and C₄ are positive constants, and the convergence follows from assumption (iii). The fact that ρ_J = o(1), (A.3) and (A.7) ensure that g_J is universally consistent, as already stated in the Corollary 1.

In order to verify that $\hat {g}_{J}$ is strongly universal consistent, it suffices, by the Borel-Cantelli lemma and Eq. A.7, to verify the convergence of the series

$$\underset{n}{\sum} (C_{3} 2^{4J} )^{2^{J}} e^{-n\epsilon^{2}/(512(C 2^{2J})^{2})} \equiv \underset{n}{\sum} a_{n}. $$

Based on assumption (iii), there are positive constants B, C and D such that

$$a_{n} = \left( \frac{C n^{B}}{e^{D n^{1-5r}}} \right)^{n^{r}}. $$

It is easy to see that there existsa natural n₀ such that, for all n ≥ n₀,

$$\frac{C n^{B}}{e^{D n^{1-5r}}} \leq \frac{1}{n}. $$

Furthermore, for every n ≥ 2^1/r,

$$\left( \frac{1}{n} \right)^{n^{r}} \leq \left( \frac{1}{n} \right)^{2}. $$

Then, there exist anatural m such that, for all n > m,

$$\underset{n \geq 1}{\sum} a_{n} \leq \sum\limits_{n = 1}^{m} a_{n} + \sum\limits_{n > m} \frac{1}{n^{2}} < \infty, $$

which ensures the desired result.

Proof of Theorem 3.

By the Parseval’s identity

Since 0 ≤ λ_jk ≤ 1, then $0 \leq \lambda _{jk}^{2} \leq 1$ and 0 ≤ (1 − λ_jk)² ≤ 1. Thus, the second term of the right hand side of the inequality above can be bounded by

because ${\sum }_{j = J_{0}}^{J - 1} {\sum }_{k} d_{jk}^{2} = {\rho }_{J_{0}}^{2} - {{\rho }_{J}^{2}}$.

Furthermore, we have observed in the proof of Theorem 1 that

Hence,

which yields the desired result, because $\rho _{J_{0}} \asymp \rho _{J}$ (due to the fact that J₀ and J have the same order of convergence).

The results of universal consistency and strong universal consistency can be analogously derived as in the proofs of the Corollary 1 and the Theorem 2, respectively.

Proof of Lemma 1.

Denote ${E}_n(Z.) = \frac {1}{n} {\sum }_{i = 1}^{n} Z_{i}$and, where Z₁, Z₂,… is a stationary series. For any f ∈ V_J there is a vector $\boldsymbol {c} = (c_{J0}, \ldots , c_{J,2^{J}-1})^{\top }$, |c_J|² < ∞, such that $f(x) = {\sum }_{k = 0}^{2^{J}-1} c_{Jk} \phi _{Jk}(x)$. Fix η > 0. If |(E_n − E)ϕ_Jk(X.)ϕ_Jl(X.)|≤ η, then

$$\begin{array}{@{}rcl@{}} |({E}_n - {E}) [f(X.)]^{2}| \!\!\!\!&=&\!\!\!\! \left| \underset{k,l}{\sum} c_{Jk} c_{Jl} ({E}_n - {E}) \phi_{Jk}(X.) \phi_{Jl}(X.) \right|\\ &\leq&\!\!\!\! \eta \underset{k,l}{\sum} |c_{Jk}| |c_{Jl}| I_{k,l}, \end{array} $$

where I_{k, l} is one, if the supports of ϕ_Jk and ϕ_Jl overlap, and zero, otherwise.

It is easy to see that ${\sum }_{k} I_{k,l} \leq C$ and${\sum }_{l} I_{k,l} \leq C$, for some positive constant C. Thus, applying the Cauchy-Schwarz inequality in the first and second inequalities below, and by the Parseval’s identity,

$$\begin{array}{@{}rcl@{}} \sum\limits_{k,l} |c_{Jk}| |c_{Jl}| I_{k,l} &\leq& \underset{k}{\sum} |c_{Jk}| \left\{ \underset{l}{\sum} {c}_{Jl}^{2} I_{k,l} \right\}^{2} C^{1/2}\\ &\leq& \left\{ \underset{k}{\sum} {c}_{Jk}^{2} \right\}^{1/2} \left\{ \underset{k}{\sum} \underset{l}{\sum} {c}_{Jl}^{2} I_{k,l} \right\}^{1/2} C^{1/2}\\ &\leq& C \left\{ \underset{k}{\sum} {c}_{Jk}^{2} \right\}^{1/2} \left\{ \underset{l}{\sum} {c}_{Jl}^{2} \right\}^{1/2}\\ &=& C |\boldsymbol{c}_{J}|^{2} = C \| f \|^{2}. \end{array} $$

This implies

$$|({E}_n - {E}) [ f(X.)]^{2}| \leq \eta C \| f \|^{2}. $$

Then, since ${\sum }_{k} {\sum }_{l} I_{k,l} \lesssim 2^{J} \asymp n^{r}$,

By Proposition 1 and using $\delta = \frac {2r-\gamma }{2}$, we have that, for every 2r < γ < 1 and s ≥ 3,

Observe that, whenever 2r < γ < 1 and $r + 1 + \frac {sr}{2s + 1} - \frac {2s \alpha (1 - \gamma )}{2s + 1} < -1$,

Hence, since η > 0 is arbitrary, we have by Borel-Cantelli lemma that

$$ \sup_{f \in V_{J}} \frac{|({E}_n - {E}) [ f(X.)]^{2}|}{\| f \|^{2}} \overset{a.s.}{\longrightarrow} 0, \quad \text{as} \quad n \to \infty. $$

(A.10)

The fact that

$$\underset{s \to \infty}{\underset{\gamma \to 2r}{\lim}} r + 1 + \frac{sr}{2s + 1} - \frac{2s \alpha (1 - \gamma)}{2s + 1} = \frac{3}{2}r + 1 - \alpha (1 - 2r) $$

ensures that it isalways possible to find 0 < γ < 1 and s ≥ 3 satisfying

$$2r < \gamma < 1 \quad \text{and} \quad r + 1 + \frac{sr}{2s + 1} - \frac{2s \alpha (1 - \gamma)}{2s + 1} < -1. $$

Thus the desired result follows from (A.10), because assumption (ii) ensures .

Proof of Lemma 2.

Let $\boldsymbol {c}_{J} = (c_{J0}, \ldots , c_{J,2^{J}-1})^{\top }$ be a vector such that |c_J|² < ∞, and denote f(x; c_J). Thus, by Lemma 1,

Hence, by assumption (ii) and the Parseval’s identity,

which ensures the desired result.

Proof of Proposition 1.

From Theorem 1.4 of Bosq (1998),

where c is a positive constant, q ∈ [1,n/2],

Observe that, since any compactly supported orthonormal wavelet atom has the order ϕ_Jk(x) ≲ 2^J/2, then by assumption (iii),

$$m_{s} \lesssim 2^{J} \lesssim n^{r}, $$

which implies that

$${{m}_{s}^{v}} \lesssim n^{vr}, \quad v, r = 2, 3, \ldots. $$

Let q = n^γ, γ ∈ (0, 1). Thus, it is easy to see that

$$ a_{1} \lesssim n^{1-\gamma}, $$

(A.11)

$$ \exp \left( - \frac{n^{\gamma} \eta^{2}}{25 {{m}_{2}^{2}} + 5 c \eta} \right) = o (\exp(- n^{\gamma - 2r - \delta}) ), \quad 2r < \gamma < 1, \quad 0 < \delta < 2r - \gamma, $$

(A.12)

and

$$ a_{2}(s) \lesssim n^{1 + \frac{sr}{2s + 1}}. $$

(A.13)

By assumption (iv),

$$ \alpha\left( \left[ \frac{n}{q + 1} \right] \right)^{\frac{2s}{2s + 1}} \lesssim n^{-\frac{2s \alpha (1-\gamma)}{2s + 1}}. $$

(A.14)

Then, the desired result follows from (A.11)–(A.14).