Training Effective Node Classifiers for Cascade Classification

Abstract

Cascade classifiers are widely used in real-time object detection. Different from conventional classifiers that are designed for a low overall classification error rate, a classifier in each node of the cascade is required to achieve an extremely high detection rate and moderate false positive rate. Although there are a few reported methods addressing this requirement in the context of object detection, there is no principled feature selection method that explicitly takes into account this asymmetric node learning objective. We provide such an algorithm here. We show that a special case of the biased minimax probability machine has the same formulation as the linear asymmetric classifier (LAC) of Wu et al. (linear asymmetric classifier for cascade detectors, 2005). We then design a new boosting algorithm that directly optimizes the cost function of LAC. The resulting totally-corrective boosting algorithm is implemented by the column generation technique in convex optimization. Experimental results on object detection verify the effectiveness of the proposed boosting algorithm as a node classifier in cascade object detection, and show performance better than that of the current state-of-the-art.

Appendices

Appendix A: Proof of Theorem 1

Before we present our results, we introduce an important proposition from (Yu et al. 2009). Note that we have used different notation.

Proposition 2

For a few different distribution families, the worst-case constraint

$$\begin{aligned} \left[ \inf _\mathbf{x \sim ( {\varvec{\mu }}, {\varvec{\varSigma }}) } \Pr \left\{ \mathbf{w }^{\!\top }\mathbf x \le b \right\} \right] \ge \gamma , \end{aligned}$$

(26)

can be written as:

• 1. if $\mathbf x \sim ({\varvec{\mu }}, {\varvec{\varSigma }})$, i.e., $\mathbf x$ follows an arbitrary distribution with mean ${\varvec{\mu }}$ and covariance ${\varvec{\varSigma }}$, then

  $$\begin{aligned} b \ge \mathbf w ^{\!\top }{\varvec{\mu }}+ \sqrt{ \tfrac{ \gamma }{ 1 - \gamma } } \cdot \sqrt{ \mathbf w ^{\!\top }{\varvec{\varSigma }}\mathbf w }; \end{aligned}$$

  (27)

• 2. if $\mathbf x \sim ({\varvec{\mu }}, {\varvec{\varSigma }})_\mathrm{S},$ ⁶then we have

  $$\begin{aligned} {\left\{ \begin{array}{ll} b \!\ge \mathbf w ^{\!\top }{\varvec{\mu }}\!+\! \sqrt{ \tfrac{ 1 }{ 2 (1 - \gamma ) } } \cdot \sqrt{ \mathbf w ^{\!\top }{\varvec{\varSigma }}\mathbf w },&\text{ if}\quad \gamma \in (0.5,1);\nonumber \\ b \!\ge \mathbf w ^{\!\top }{\varvec{\mu }},&\text{ if} \quad \gamma \in (0,0.5]; \end{array}\right.}\nonumber \\ \end{aligned}$$

  (28)

• 3. if $\mathbf x \sim ({\varvec{\mu }}, {\varvec{\varSigma }})_\mathrm{SU}$, then

  $$\begin{aligned} {\left\{ \begin{array}{ll} b \!\ge \mathbf w ^{\!\top }{\varvec{\mu }}\!+\! \frac{2}{3} \sqrt{ \tfrac{ 1 }{ 2 (1 - \gamma ) } } \cdot \sqrt{ \mathbf w ^{\!\top }{\varvec{\varSigma }}\mathbf w },&\text{ if} \quad \gamma \in (0.5,1);\nonumber \\ b \!\ge \mathbf w ^{\!\top }{\varvec{\mu }},&\text{ if} \quad \gamma \in (0,0.5]; \end{array}\right.}\nonumber \\ \end{aligned}$$

  (29)

• 4. if $\mathbf x$ follows a Gaussian distribution with mean ${\varvec{\mu }}$ and covariance ${\varvec{\varSigma }}$, i.e., $\mathbf x \sim \mathcal{G}( {\varvec{\mu }}, {\varvec{\varSigma }})$, then

  $$\begin{aligned} b \ge \mathbf w ^{\!\top }{\varvec{\mu }}+ \Phi ^{-1} ( \gamma ) \cdot \sqrt{ \mathbf w ^{\!\top }{\varvec{\varSigma }}\mathbf w }, \end{aligned}$$

  (30)

  where $\Phi (\cdot )$ is the cumulative distribution function (c.d.f.) of the standard normal distribution $\mathcal{G} (0,1)$, and $\Phi ^{-1} (\cdot )$ is the inverse function of $\Phi (\cdot ).$ Two useful observations about $\Phi ^{-1} (\cdot )$ are: $\Phi ^{-1} ( 0.5) = 0$; and $\Phi ^{-1} ( \cdot )$ is a monotonically increasing function in its domain.

We omit the proof of Proposition 2 here and refer the reader to Yu et al. (2009) for details. Next we begin to prove Theorem 1:

Proof

The second constraint of (2) is simply

$$\begin{aligned} b \ge \mathbf w ^{\!\top }{\varvec{\mu }}_2. \end{aligned}$$

(31)

The first constraint of (2) can be handled by writing $\mathbf w ^{\!\top }\mathbf x _1 \ge b$ as $- \mathbf w ^{\!\top }\mathbf x _1 \le - b$ and applying the results in Proposition 2. It can be written as

$$\begin{aligned} - b + \mathbf w ^{\!\top }{\varvec{\mu }}_1 \ge \varphi ( \gamma ) \sqrt{ \mathbf w ^{\!\top }\varvec{{{\varvec{\varSigma }}_1}} \mathbf w }, \end{aligned}$$

(32)

with (6).

Let us assume that $\Sigma _1$ is strictly positive definite (if it is only positive semidefinite, we can always add a small regularization to its diagonal components). From (32) we have

$$\begin{aligned} \varphi ( \gamma ) \le \frac{ -b + \mathbf w ^{\!\top }{\varvec{\mu }}_1 }{ \sqrt{ \mathbf w ^{\!\top }\varvec{{{\varvec{\varSigma }}_1}} \mathbf w }}. \end{aligned}$$

(33)

So the optimization problem becomes

$$\begin{aligned} \max _\mathbf{w , b, \gamma } \, \gamma , \quad \text{ s.t.} \quad (31)\quad \text{ and}\quad (33). \end{aligned}$$

(34)

The maximum value of $\gamma$ (which we label $\gamma ^\star$) is achieved when (33) is strictly an equality. To illustrate this point, let us assume that the maximum is achieved when

$$\begin{aligned} \varphi ( \gamma ^\star ) < \frac{ -b + \mathbf w ^{\!\top }{\varvec{\mu }}_1 }{ \sqrt{ \mathbf w ^{\!\top }\varvec{{{\varvec{\varSigma }}_1}} \mathbf w }}. \end{aligned}$$

Then a new solution can be obtained by increasing $\gamma ^\star$ with a positive value such that (33) becomes an equality. Notice that the constraint (31) will not be affected, and the new solution will be better than the previous one. Hence, at the optimum, (5) must be fulfilled.

Because $\varphi (\gamma )$ is monotonically increasing for all the four cases in its domain $(0,1)$ (see Fig. 9), maximizing $\gamma$ is equivalent to maximizing $\varphi (\gamma )$ and this results in

$$\begin{aligned} \max _\mathbf{w , b } \, \frac{ -b + \mathbf w ^{\!\top }{\varvec{\mu }}_1 }{ \sqrt{ \mathbf w ^{\!\top }\varvec{{{\varvec{\varSigma }}_1}} \mathbf w } }, \quad \text{ s.t.} \quad b \ge \mathbf w ^{\!\top }{\varvec{\mu }}_2. \end{aligned}$$

(35)

As in Lanckriet et al. (2002) and Huang et al. (2004), we also have a scale ambiguity: if $( \mathbf w ^\star , b^\star )$ is a solution, $( t \mathbf w ^\star , t b^\star )$ with $t > 0$ is also a solution.

An important observation is that the problem (35) must attain the optimum at (4). Otherwise if $b > \mathbf w ^{\!\top }{\varvec{\mu }}_2$, the optimal value of (35) must be smaller. So we can rewrite (35) as an unconstrained problem (3).

We have thus shown that, if $\mathbf x _1$ is distributed according to a symmetric, symmetric unimodal, or Gaussian distribution, the resulting optimization problem is identical. This is not surprising considering the latter two cases are merely special cases of the symmetric distribution family.

At optimality, the inequality (33) becomes an equality, and hence $\gamma ^\star$ can be obtained as in (5). For ease of exposition, let us denote the fours cases in the right side of (6) as $\varphi _\mathrm{gnrl} ( \cdot )$, $\varphi _\mathrm{S} ( \cdot )$, $\varphi _\mathrm{SU} ( \cdot )$, and $\varphi _\mathcal{G} ( \cdot ) .$ For $\gamma \in [0.5, 1)$, as shown in Fig. 9, we have $\varphi _\mathrm{gnrl} ( \gamma ) > \varphi _\mathrm{S} ( \gamma ) > \varphi _\mathrm{SU} ( \gamma ) > \varphi _\mathcal{G} ( \gamma ).$ Therefore, when solving (5) for $\gamma ^\star$, we have $\gamma ^\star _\mathrm{gnrl} < \gamma ^\star _\mathrm{S} < \gamma ^\star _\mathrm{SU} < \gamma ^\star _\mathcal{G}.$ That is to say, one can get better accuracy when additional information about the data distribution is available, although the actual optimization problem to be solved is identical.

Fig. 9

The function $\varphi (\cdot )$ in (6). The four curves correspond to the four cases. They are all monotonically increasing in $(0,1)$

Appendix B: Proof of Theorem 2

Let us assume that in the current solution we have selected $n$ weak classifiers and their corresponding linear weights are $\mathbf w = [ w_1, \ldots , w_n ] .$ If we add a weak classifier $h^{\prime }(\cdot )$ that is not in the current subset, the corresponding $w$ is zero, then we can conclude that the current weak classifiers and $\mathbf w$ are the optimal solution already. In this case, the best weak classifier that is found by solving the subproblem (20) does not contribute to solving the master problem.

Let us consider the case that the optimality condition is violated. We need to show that we are able to find such a weak learner $h^{\prime }(\cdot )$, which is not in the set of current selected weak classifiers, that its corresponding coefficient $w > 0$ holds. Again assume $h^{\prime }(\cdot )$ is the most violated weak learner found by solving (20) and the convergence condition is not satisfied. In other words, we have

$$\begin{aligned} \sum \limits _{i=1}^m u_i y_i h^{\prime }( \mathbf x _i ) \ge r. \end{aligned}$$

(36)

Now, after this weak learner is added into the master problem, the corresponding primal solution $w$ must be non-zero (positive because we have the nonnegative-ness constraint on $\mathbf w$).

If this is not the case, then the corresponding $w = 0 .$ This is not possible because of the following reason. From the Lagrangian (17), at optimality we have $\partial L / \partial \mathbf w = {\varvec{0}}$, which leads to

$$\begin{aligned} r - \sum _{i=1}^m u_i y_i h^{\prime }( \mathbf x _i ) = q > 0. \end{aligned}$$

(37)

Clearly (36) and (37) contradict.

Thus, after the weak classifier $h^{\prime }(\cdot )$ is added to the primal problem, its corresponding $w$ must have a positive solution. This is to say, one more free variable is added into the problem and re-solving the primal problem (16) must reduce the objective value. Therefore a strict decrease in the objective is obtained. In other words, Algorithm 1 must make progress at each iteration. Furthermore, the primal optimization problem is convex, there are no local optimal points. The column generation procedure is guaranteed to converge to the global optimum up to some prescribed accuracy.

Appendix C: Exponentiated Gradient Descent

Exponentiated gradient descent (EG) is a very useful tool for solving large-scale convex minimization problems over the unit simplex. Let us first define the unit simplex $\varDelta _n = \left\{ \mathbf{w } \in \mathbb{R }^n : \mathbf{1 } ^ {\!\top }\mathbf w = 1, \mathbf w \succcurlyeq \mathbf{0 } \right\} .$ EG efficiently solves the convex optimization problem

$$\begin{aligned} \min _\mathbf w \quad f(\mathbf w ), \quad \mathrm{s.t.} \quad \mathbf w \in \varDelta _n, \end{aligned}$$

(38)

under the assumption that the objective function $f(\cdot )$ is a convex Lipschitz continuous function with Lipschitz constant $L_f$ w.r.t. a fixed given norm $||\cdot ||.$ The mathematical definition of $L_f$ is that $| f(\mathbf w ) -f (\mathbf z ) | \le L_f ||\mathbf x - \mathbf z ||$ holds for any $\mathbf x , \mathbf z$ in the domain of $f(\cdot ).$ The EG algorithm is very simple:

• 1. Initialize with $\mathbf w ^0 \in \text{ the} \text{ interior} \text{ of} \varDelta _n$;

• 2. Generate the sequence $\left\{ \mathbf{w }^k \right\}$, $k=1,2,\ldots$ with:

  $$\begin{aligned} \mathbf w ^k_j = \frac{ \mathbf w ^{k-1}_j \exp [ - \tau _k {f_{j}^{\prime }} ( \mathbf w ^{k-1} ) ] }{ \sum _{j=1}^n \mathbf w ^{k-1}_j \exp [ - \tau _k {f_{j}^{\prime }} ( \mathbf w ^{k-1} ) ] }. \end{aligned}$$

  (39)

  Here $\tau _k$ is the step-size. $f^{\prime }( \mathbf w ) = [ f_{1}^{\prime }(\mathbf w ), \dots , f_{n}^{\prime }(\mathbf w ) ] ^{\!\top }$ is the gradient of $f(\cdot )$;

• 3. Stop if some stopping criteria are met.

The learning step-size can be determined by

$$\begin{aligned} \tau _k = \frac{ \sqrt{ 2\log n } }{ L_f } \frac{1}{ \sqrt{ k } }, \end{aligned}$$

following Beck and Teboulle (2003). In Collins et al. (2008), the authors have used a simpler strategy to set the learning rate.

In EG there is an important parameter $L_f$, which is used to determine the step-size. $L_f$ can be determined by the $\ell _\infty$-norm of $| f^{\prime } (\mathbf w ) | .$ In our case $f^{\prime } (\mathbf w )$ is a linear function, which is trivial to compute. The convergence of EG is guaranteed; see Beck and Teboulle (2003) for details.