Transfer Learning Decision Forests for Gesture Recognition

Abstract

Decision forests are an increasingly popular tool in computer vision problems. Their advantages include high computational efficiency, state-of-the-art accuracy and multi-class support. In this paper, we present a novel method for transfer learning which uses decision forests, and we apply it to recognize gestures and characters. We introduce two mechanisms into the decision forest framework in order to transfer knowledge from the source tasks to a given target task. The first one is mixed information gain, which is a data-based regularizer. The second one is label propagation, which infers the manifold structure of the feature space. We show that both of them are important to achieve higher accuracy. Our experiments demonstrate improvements over traditional decision forests in the ChaLearn Gesture Challenge and MNIST data set. They also compare favorably against other state-of-the-art classifiers.

Editors: Isabelle Guyon, Vassilis Athitsos, and Sergio Escalera

Appendix A

We prove Theorem 1. First, we prove ${\mathbb {E}}(\mathcal{H}(S_K)) + \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( 1 + \frac{1 - p_{\mathbf {y}}}{K p_{\mathbf {y}}} \right) \le \mathcal{H}({P})$

By definition of the empirical entropy and linearity of the expectation, we have:

$$\begin{aligned} {\mathbb {E}}(\mathcal{H}(S_K)) = - {\mathbb {E}}\left[ \sum _{\mathbf {y}\in \mathcal{Y}} \hat{p}_{S_K}(\mathbf {y}) \log (\hat{p}_{S_K}(\mathbf {y})) \right] = - \sum _{\mathbf {y}\in \mathcal{Y}} {\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \log (\hat{p}_{S_K}(\mathbf {y})) \right] \end{aligned}$$

Using the definitions of the empirical histogram $\hat{p}_{S_K}(\mathbf {y})$ and the expectation:

$$\begin{aligned} - \sum _{\mathbf {y}\in \mathcal{Y}} {\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \log (\hat{p}_{S_K}(\mathbf {y})) \right] = - \sum _{\mathbf {y}\in \mathcal{Y}} \sum _{j=0}^K {P}\left( \hat{p}_{S_K}(\mathbf {y}) = \frac{j}{K}\right) \frac{j}{K} \log \frac{j}{K} \end{aligned}$$

Assuming that the samples are iid, then:

$$\begin{aligned} = - \sum _{\mathbf {y}\in \mathcal{Y}} \sum _{j=0}^K {K \atopwithdelims ()j} p_{\mathbf {y}}^j (1-p_{\mathbf {y}})^{K-j} \frac{j}{K} \log \frac{j}{K} \end{aligned}$$

Note that, in this equation, $p_{\mathbf {y}}$ is the true probability of distribution ${P}$. After some algebraic manipulations, we obtain the following:

$$\begin{aligned}&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \sum _{j=0}^{K-1} {K-1 \atopwithdelims ()j} p_{\mathbf {y}}^j (1-p_{\mathbf {y}})^{K-1-j} \log \frac{j+1}{K} \\&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \sum _{j=0}^{K-1} {P}\left( \hat{p}_{S_K}(\mathbf {y}) = \frac{j}{K}\right) \log \frac{j+1}{K} \end{aligned}$$

Applying Jensen’s inequality for the convex function $-\log (x)$, we obtain:

$$\begin{aligned}&\ge - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( \sum _{j=0}^{K-1} {P}\left( \hat{p}_{S_K}(\mathbf {y}) = \frac{j}{K}\right) \frac{j+1}{K} \right) \\&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \frac{(K-1)p_{\mathbf {y}} + 1}{K}\\&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( p_{\mathbf {y}} + \frac{1 - p_{\mathbf {y}}}{K}\right) \\&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( p_{\mathbf {y}} \left( 1 + \frac{1 - p_{\mathbf {y}}}{K p_{\mathbf {y}}}\right) \right) \\&= - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log p_{\mathbf {y}} - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( 1 + \frac{1 - p_{\mathbf {y}}}{K p_{\mathbf {y}}} \right) \\&= \mathcal{H}({P}) - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log \left( 1 + \frac{1 - p_{\mathbf {y}}}{K p_{\mathbf {y}}} \right) \end{aligned}$$

Now we prove $\mathcal{H}({P}) \le {\mathbb {E}}(\mathcal{H}(S_K))$.

By definition of the empirical entropy and linearity of the expectation, we have:

$$\begin{aligned} {\mathbb {E}}(\mathcal{H}(S_K)) = - {\mathbb {E}}\left[ \sum _{\mathbf {y}\in \mathcal{Y}} \hat{p}_{S_K}(\mathbf {y}) \log (\hat{p}_{S_K}(\mathbf {y})) \right] = - \sum _{\mathbf {y}\in \mathcal{Y}} {\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \log (\hat{p}_{S_K}(\mathbf {y})) \right] \end{aligned}$$

Applying Jensen’s inequality for the convex function $x \log x$, we obtain the following:

$$\begin{aligned} \le - \sum _{\mathbf {y}\in \mathcal{Y}} {\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \right] \log ({\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \right] ) \end{aligned}$$

Since ${\mathbb {E}}\left[ \hat{p}_{S_K}(\mathbf {y}) \right] = p_{\mathbf {y}}$, we have:

$$\begin{aligned} = - \sum _{\mathbf {y}\in \mathcal{Y}} p_{\mathbf {y}} \log (p_{\mathbf {y}}) = \mathcal{H}({P}) \end{aligned}$$