A New Classifier for Imbalanced Data Based on a Generalized Density Ratio Model

Abstract

Achieving higher true positive rate when decreasing false positive rate is always a great challenge to the imbalance learning community. This work combines penalized empirical likelihood method, lower bound algorithm and Nyström method and applies these techniques along with kernel method to density ratio model. The resulting classifier, density ratio classifier (DRC), is a combination of kernelization, regularization, efficient implementation and threshold moving, all of which are critical to enable DRC to be an effective and powerful method for solving difficult imbalance problems. Compared with other methods, DRC is competitive in that it is widely applicable and it is simple and easy to use without additional imbalance handling skills. In addition, the convergence rate of the estimate of log density ratio is discussed as well. And the results of numerical analysis also show that DRC outperforms other methods in AUC and G-mean score.

Appendices

A Proof of Theorems

1.1 A.1 Proof of Theorem 2.1

Consider the quadratic functional

$$\begin{aligned} \frac{1}{n}\sum _{i = 1}^{n}u(h^{*}({\varvec{z}}_{i});y_{i})h({\varvec{z}}_{i}) + \frac{1}{2}V(h-h^{*}) + \frac{\lambda }{2}J(h), \end{aligned}$$

(4.1)

and we add a multiplier $\frac{1}{2}$ in the last term for convenience, which has no effect on our technical development. Since V is completely continuous with respect to J and $J(h)<\infty$, h can be expressed as a Fourier series expansion $h = \sum _{\mu }h_{\mu }\psi _{\mu }$, where $\psi _{\mu }$ are eigenfunctions of J with respect to V and $h_{\mu } = V(h,\psi _{\mu }) = \int _{{\mathcal {X}}}h(z)\psi _{\mu }(z)w(h^{*}(z))f(z)dz$ are Fourier coefficients. Plugging Fourier series expansions $h = \sum _{\mu }h_{\mu }\psi _{\mu }$ and $h^{*} = \sum _{\mu }h_{\mu }^{*}\psi _{\mu }$ into (4.1), it is easy to show that the minimizer $\tilde{h}$ of (4.1) has Fourier coefficients

$$\begin{aligned} \tilde{h}_{\mu } = (\beta _{\mu }+h_{\mu }^{*})/(1+\lambda \xi _{\mu }), \end{aligned}$$

where $\beta _{\mu } = n^{-1}\sum _{i= 1}^{n}u(h^{*}({\varvec{z}}_{i});y_{i})\psi _{\mu }({\varvec{z}}_{i})$.

Since $E(\beta _{\mu }) = 0$ and $E(\beta _{\mu }^{2}) = \frac{1}{n}$, we have

$$\begin{aligned} E(V(\tilde{h}-h^{*}))&= E\left( \sum _{\mu }(\tilde{h}_{\mu }-h_{\mu }^{*})^{2}) = E(\sum _{\mu }\frac{\beta _{\mu }^{2}-2\beta _{\mu }\lambda \xi _{\mu }h_{\mu }^{*}+\lambda ^{2}\xi _{\mu }^{2}h_{\mu }^{*2}}{(1+\lambda \xi _{\mu })^{2}}\right) \nonumber \\&= \frac{1}{n}\sum _{\mu }\frac{1}{(1+\lambda \xi _{\mu })^{2}}+\lambda ^{p}\sum _{\mu }\frac{(\lambda \xi _{\mu })^{2-p}}{(1+\lambda \xi _{\mu })^{2}}\xi _{\mu }^{p}h_{\mu }^{*2}, \end{aligned}$$

(4.2)

$$\begin{aligned} E(\lambda J(\tilde{h}-h^{*}))&= E\left( \sum _{\mu }\lambda \xi _{\mu }(\tilde{h}_{\mu }-h_{\mu }^{*})^{2}\right) \nonumber \\&= E\left( \sum _{\mu }\lambda \xi _{\mu }\frac{\beta _{\mu }^{2}-2\beta _{\mu }\lambda \xi _{\mu }h_{\mu }^{*}+\lambda ^{2}\xi _{\mu }^{2}h_{\mu }^{*2}}{(1+\lambda \xi _{\mu })^{2}}\right) \nonumber \\&= \frac{1}{n}\sum _{\mu }\frac{\lambda \xi _{\mu }}{(1+\lambda \xi _{\mu })^{2}}+\lambda ^{p}\sum _{\mu }\frac{(\lambda \xi _{\mu })^{3-p}}{(1+\lambda \xi _{\mu })^{2}}\xi _{\mu }^{p}h_{\mu }^{*2}. \end{aligned}$$

(4.3)

When $\lambda \rightarrow 0$ and condition 2 holds, Lemma 8.1 in [21] says that

$$\begin{aligned} \sum _{\mu }\frac{\lambda \xi _{\mu }}{(1+\lambda \xi _{\mu })^{2}}&= O(\lambda ^{-1/r}),\nonumber \\ \sum _{\mu }\frac{1}{(1+\lambda \xi _{\mu })^{2}}&= O(\lambda ^{-1/r}),\nonumber \\ \sum _{\mu }\frac{1}{1+\lambda \xi _{\mu }}&= O(\lambda ^{-1/r}). \end{aligned}$$

(4.4)

So, combining (4.2)–(4.4) and assuming condition 3, we immediately have

$$\begin{aligned} V(\tilde{h}-h^{*}) = O_{p}(n^{-1}\lambda ^{-1/r}+\lambda ^{p}) \end{aligned}$$

and

$$\begin{aligned} \lambda J(\tilde{h}-h^{*}) = O_{p}(n^{-1}\lambda ^{-1/r}+\lambda ^{p}) . \end{aligned}$$

Therefore,

$$\begin{aligned} (V+\lambda J)(\tilde{h}-h^{*}) = O_{p}(n^{-1}\lambda ^{-1/r}+\lambda ^{p}). \end{aligned}$$

1.2 A.2 Proof of Theorem 2.2

We first characterize the rate of ${\hat{h}}-\tilde{h}$ and then turn to ${\hat{h}}-h^{*}$. Define

$$\begin{aligned} A_{g,h}(\eta ) = \frac{1}{n}\sum _{i=1}^{n}l((g+\eta h)({\varvec{z}}_{i});y_{i})+\frac{\lambda }{2}J(g+\eta h) \end{aligned}$$

(4.5)

and

$$\begin{aligned} B_{g,h}(\eta ) = \frac{1}{n}\sum _{i=1}^{n}u(h^{*}({\varvec{z}}_{i});y_{i})(g+\eta h)({\varvec{z}}_{i})+\frac{1}{2}V(g+\eta h-h^{*})+\frac{\lambda }{2}J(g+\eta h). \end{aligned}$$

(4.6)

Then, it can be easily shown that

$$\begin{aligned} \frac{dA_{g,h}}{d\eta }|_{\eta = 0} = \frac{1}{n}\sum _{i=1}^{n}u(g({\varvec{z}}_{i});y_{i})h({\varvec{z}}_{i})+ \lambda J(g,h) \end{aligned}$$

(4.7)

and

$$\begin{aligned} \frac{dB_{g,h}}{d\eta }|_{\eta = 0} = \frac{1}{n}\sum _{i=1}^{n}u(h^{*}({\varvec{z}}_{i});y_{i})h({\varvec{z}}_{i})+V(g-h^{*},h) + \lambda J(g,h). \end{aligned}$$

(4.8)

If we set $g = {\hat{h}}, h = {\hat{h}} - \tilde{h}$ in (4.7), we have

$$\begin{aligned} \frac{1}{n}\sum _{i=1}^{n}u({\hat{h}}({\varvec{z}}_{i});y_{i})({\hat{h}} - \tilde{h})({\varvec{z}}_{i})+ \lambda J({\hat{h}},{\hat{h}} - \tilde{h}) = 0. \end{aligned}$$

(4.9)

And setting $g = \tilde{h}, h = {\hat{h}} - \tilde{h}$ in (4.8), we have

$$\begin{aligned} \frac{1}{n}\sum _{i=1}^{n}u(h^{*}({\varvec{z}}_{i});y_{i})({\hat{h}} - \tilde{h})({\varvec{z}}_{i})+V(\tilde{h}-h^{*},{\hat{h}}-\tilde{h})+ \lambda J(\tilde{h},{\hat{h}} - \tilde{h}) = 0. \end{aligned}$$

(4.10)

Subtracting (4.10) from (4.9), we can immediately get

$$\begin{aligned}&\frac{1}{n}\sum _{i=1}^{n}(u({\hat{h}}({\varvec{z}}_{i});y_{i})-u(\tilde{h}({\varvec{z}}_{i});y_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i})) + \lambda J({\hat{h}} - \tilde{h})\nonumber \\&\quad =V(\tilde{h}-h^{*},{\hat{h}}-\tilde{h})-\frac{1}{n}\sum _{i=1}^{n}(u(\tilde{h}({\varvec{z}}_{i});y_{i})-u(h^{*}({\varvec{z}}_{i});y_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i})). \end{aligned}$$

(4.11)

Using the mean value theorem, we can get

$$\begin{aligned} u({\hat{h}}({\varvec{z}}_{i});y_{i})-u(\tilde{h}({\varvec{z}}_{i});y_{i})= & {} w(h_{1}({\varvec{z}}_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i})), \end{aligned}$$

(4.12)

$$\begin{aligned} u(\tilde{h}({\varvec{z}}_{i});y_{i})-u(h^{*}({\varvec{z}}_{i});y_{i})= & {} w(h_{2}({\varvec{z}}_{i}))(\tilde{h}({\varvec{z}}_{i})-h^{*}({\varvec{z}}_{i})), \end{aligned}$$

(4.13)

where $h_{1}$ is a convex combination of ${\hat{h}}$ and $\tilde{h}$, and $h_{2}$ is that of $\tilde{h}$ and $h^{*}$. Substituting (4.12) and (4.13) back to (4.11), we have

$$\begin{aligned}&\frac{1}{n}\sum _{i=1}^{n}w(h_{1}({\varvec{z}}_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i}))^{2}+\lambda J({\hat{h}}-\tilde{h}) \nonumber \\&\quad = V(\tilde{h}-h^{*},{\hat{h}}-\tilde{h})-\frac{1}{n}\sum _{i=1}^{n}w(h_{2}({\varvec{z}}_{i}))(\tilde{h}({\varvec{z}}_{i})-h^{*}({\varvec{z}}_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i})). \end{aligned}$$

(4.14)

Under conditions 1, 2, 5 and assuming $\lambda \rightarrow 0,n\lambda ^{r/2}\rightarrow \infty ,$ lemma 8.17 in [21] indicates that

$$\begin{aligned} \frac{1}{n}\sum _{i=1}^{n}w(h^{*}({\varvec{z}}_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i}))^{2}= V({\hat{h}}-\tilde{h})+o_{p}((V+\lambda J)({\hat{h}}-\tilde{h})) \end{aligned}$$

(4.15)

and

$$\begin{aligned}&\frac{1}{n}\sum _{i=1}^{n}w(h^{*}({\varvec{z}}_{i}))(\tilde{h}({\varvec{z}}_{i})-h^{*}({\varvec{z}}_{i}))({\hat{h}}({\varvec{z}}_{i})-\tilde{h}({\varvec{z}}_{i}))\nonumber \\&\quad =V({\hat{h}}-\tilde{h},\tilde{h}-h^{*})+o_{p}(\{(V+\lambda J)({\hat{h}}-\tilde{h})(V+\lambda J)(\tilde{h}-h^{*})\}^{1/2}). \end{aligned}$$

(4.16)

Combining (4.14)–(4.16) and assuming condition 4, for some $c\in [c_{1},c_{2}]$, we have

$$\begin{aligned}&(c_{1}V+\lambda J)({\hat{h}}-\tilde{h})(1+o_{p}(1)) \nonumber \\&\quad \le \{(|1-c|V+\lambda J)({\hat{h}}-\tilde{h})\}^{1/2}O_{p}(\{(|1-c|V+\lambda J)(\tilde{h}-h^{*})\}^{1/2}), \end{aligned}$$

(4.17)

which together with Theorem 2.1 implies that

$$\begin{aligned} (V+\lambda J)({\hat{h}}-\tilde{h}) = O_{p}(n^{-1}\lambda ^{-1/r}+\lambda ^{p}), \end{aligned}$$

and

$$\begin{aligned} (V+\lambda J)({\hat{h}}-h^{*}) = O_{p}(n^{-1}\lambda ^{-1/r}+\lambda ^{p}). \end{aligned}$$

B Details for Creation of Objective Function

Let us first recall the log empirical likelihood

$$\begin{aligned} \ell _{n} = \sum _{i=1}^{n}\log (p_{i}) + \sum _{j=1}^{n_{1}}h({\varvec{x}}_{1j}), \end{aligned}$$

(5.1)

where

$$\begin{aligned} \sum _{i=1}^{n}p_{i} = 1, \quad p_{i}\ge 0, \quad \sum _{i=1}^{n}p_{i}\mathrm {exp}(h({\varvec{z}}_{i})) = 1. \end{aligned}$$

(5.2)

The maximization employs the method of Lagrange multipliers, i.e.,

$$\begin{aligned} \ell _{n} + \gamma _{1}\left( \sum _{i=1}^{n}p_{i}\mathrm {exp}(h({\varvec{z}}_{i}))-1\right) +\gamma _{0}(\sum _{i=1}^{n}p_{i}-1), \end{aligned}$$

(5.3)

where $\gamma _{0},\gamma _{1}$ are Lagrange multipliers. By profiling $\ell _{n}$ subject to (5.2) with respect to h first, we can express each $p_{i}$ in terms of h as

$$\begin{aligned} p_{i} = \{n- \gamma _{1}[\mathrm {exp}(h({\varvec{z}}_{i}))-1]\}^{-1}, \end{aligned}$$

(5.4)

where $\gamma _{0},\gamma _{1}$ satisfy $\gamma _{0}+\gamma _{1}+n = 0$ and they solve

$$\begin{aligned} \sum _{i=1}^{n}p_{i}[\mathrm {exp}(h({\varvec{z}}_{i}))-1] = 0. \end{aligned}$$

(5.5)

Substituting $p_{i}$s back into $\ell _{n}$, the profile log empirical likelihood can be obtained as

$$\begin{aligned} \tilde{\ell }_{n} = -\sum _{i=1}^{n}log(n-\gamma _{1}[\mathrm {exp}(h({\varvec{z}}_{i}))-1])+\sum _{j=1}^{n_{1}}h(x_{1j}). \end{aligned}$$

(5.6)

Actually, we are solving a constrained maximum likelihood problem, so we can define our optimization problem as

$$\begin{aligned} -\frac{1}{n}\tilde{\ell }_{n} + \lambda \Vert P_{1}h \Vert ^{2}, \end{aligned}$$

(5.7)

With the help of Theorem 2 in [43], we know that the minimizer of the penalized negative log-likelihood has the form as

$$\begin{aligned} h(\cdot )=\sum _{v=1}^{M}d_{v}\phi _{v}(\cdot )+\sum _{i=1}^{n}\alpha _{i}R(\cdot ,{\varvec{z}}_{i}). \end{aligned}$$

(5.8)

Suppose $\phi _{1}(\cdot ) = 1$, to get $\gamma s$, we differentiate $\ell _{n}$ with respect to $d_{1}$ and apply the constraints (5.2).

$$\begin{aligned} \frac{\partial \tilde{\ell _{n}}}{\partial d_{1}} = \sum _{i=1}^{n}\frac{\gamma _{1}\mathrm {exp}(h({\varvec{z}}_{i}))}{n-\gamma _{1}[\mathrm {exp}(h({\varvec{z}}_{i})-1]}+n_{1} =\sum _{i = 1}^{n}\gamma _{1}p_{i}\mathrm {exp}(h({\varvec{z}}_{i}) + n_{1} = 0. \end{aligned}$$

(5.9)

We have $\gamma _{1} = -n_{1}$. So,

$$\begin{aligned} p_{i} = n^{-1}\{1-{\hat{\rho }}+{\hat{\rho }}\mathrm {exp}(h({\varvec{z}}_{i}))\}^{-1}, \end{aligned}$$

(5.10)

and

$$\begin{aligned} \tilde{\ell }_{n} = -\sum _{i=1}^{n}\log n(1-{\hat{\rho }}+{\hat{\rho }}\mathrm {exp}(h({\varvec{z}}_{i})))+\sum _{j=1}^{n_{1}}h(x_{1j}). \end{aligned}$$

(5.11)

C Details for Lower Bound Optimization Method

Let

$$\begin{aligned} r_{i} = \frac{{\hat{\rho }}\mathrm {exp}({\varvec{T}}_{i}^{'}{\varvec{d}}+{\varvec{R}}_{i}^{'}\varvec{\alpha })}{1-{\hat{\rho }}+{\hat{\rho }}\mathrm {exp}({\varvec{T}}_{i}^{'}{\varvec{d}}+{\varvec{R}}_{i}^{'}\varvec{\alpha })}, \end{aligned}$$

(6.1)

${\varvec{r}} = (r_{1},\ldots ,r_{n})^{'}$, ${\varvec{y}} = (y_{1},\ldots ,y_{n})^{'}$, ${\varvec{G}} = -{\varvec{y}}+{\varvec{r}}$, ${\varvec{Q}} = ({\varvec{T}},{\varvec{R}})$ and

$$\begin{aligned} {\varvec{K}}^{*} = \begin{pmatrix} {\varvec{O}}_{M\times M} &{} {\varvec{O}}_{M\times n}\\ {\varvec{O}}_{n\times M} &{} {\varvec{R}} \end{pmatrix}, \end{aligned}$$

(6.2)

where ${\varvec{O}}_{M\times n}$ is zero matrix with dimension $M\times n$. Define ${\varvec{W}} = ({\varvec{0}}^{'}, \varvec{\alpha }^{'}{\varvec{R}})^{'}$, where ${\varvec{0}}$ is M-dimensional vector with elements all 0. The first-order partial derivative and Hessian matrix of objective function with respect to $\varvec{\theta }$ are

$$\begin{aligned} {\varvec{G}}^{*} = {\varvec{Q}}^{'}{\varvec{G}}+ {\varvec{W}} \end{aligned}$$

(6.3)

and

$$\begin{aligned} {\varvec{H}}^{*} = \sum _{i=1}^{n}{\varvec{Q}}_{i}{\varvec{Q}}_{i}^{'}r_{i}(1-r_{i})+2\lambda {\varvec{K}}^{*}, \end{aligned}$$

(6.4)

where ${\varvec{Q}}_{i}$ is the ith row of ${\varvec{Q}}$. So, the iterative equation of Newton’s method is

$$\begin{aligned} \varvec{\theta }^{(t+1)}= \varvec{\theta }^{(t)}- ({\varvec{H}}^{*(t)})^{-1}{\varvec{G}}^{*(t)}, \end{aligned}$$

(6.5)

where (t) indicates quantities evaluated at tth Newton–Raphson iteration. According to (6.5), we need to calculate the inverse of Hessian matrix every iterative step, which is extremely inefficient. The core idea of LB is to find an upper bound ${\varvec{B}}$ of ${\varvec{H}}^{*}$ such that

$$\begin{aligned} \varvec{\theta }^{(t+1)}= \varvec{\theta }^{(t)}- {\varvec{B}}^{-1}{\varvec{G}}^{*(t)}, \end{aligned}$$

(6.6)

where ${\varvec{B}}$ is a symmetric, positive definite matrix and does not depend on $\varvec{\theta }$. Using ${\varvec{C}}(\varvec{\theta })$ for the objective function, this corresponds to a quadratic approximation to ${\varvec{C}}(\varvec{\theta })-{\varvec{C}}(\varvec{\theta }_{0})$ of the following form

$$\begin{aligned} (\varvec{\theta }-\varvec{\theta }_{0})^{'}\nabla {\varvec{C}}(\varvec{\theta })+(\varvec{\theta }-\varvec{\theta }_{0})^{'}{\varvec{B}}(\varvec{\theta }-\varvec{\theta }_{0})/2. \end{aligned}$$

(6.7)

The properties of LB can be found in Theorem 4.1 in [4]. As stated in [4], substituting all $r_{i}s$ with 1/2 in ${\varvec{H}}^{*}$ we will do the job. Actually, we have

$$\begin{aligned} {\varvec{H}}^{*} \preccurlyeq \frac{1}{4}\sum _{i=1}^{n}{\varvec{Q}}_{i}{\varvec{Q}}_{i}^{'}+2\lambda {\varvec{K}}^{*} := {\varvec{B}}. \end{aligned}$$

(6.8)

Here, $C \preccurlyeq D$ means that $D-C$ is nonnegative definite.