A Joint Gaussian Process Model for Active Visual Recognition with Expertise Estimation in Crowdsourcing

Abstract

We present a noise resilient probabilistic model for active learning of a Gaussian process classifier from crowds, i.e., a set of noisy labelers. It explicitly models both the overall label noise and the expertise level of each individual labeler with two levels of flip models. Expectation propagation is adopted for efficient approximate Bayesian inference of our probabilistic model for classification, based on which, a generalized EM algorithm is derived to estimate both the global label noise and the expertise of each individual labeler. The probabilistic nature of our model immediately allows the adoption of the prediction entropy for active selection of data samples to be labeled, and active selection of high quality labelers based on their estimated expertise to label the data. We apply the proposed model for four visual recognition tasks, i.e., object category recognition, multi-modal activity recognition, gender recognition, and fine-grained classification, on four datasets with real crowd-sourced labels from the Amazon Mechanical Turk. The experiments clearly demonstrate the efficacy of the proposed model. In addition, we extend the proposed model with the Predictive Active Set Selection Method to speed up the active learning system, whose efficacy is verified by conducting experiments on the first three datasets. The results show our extended model can not only preserve a higher accuracy, but also achieve a higher efficiency.

Appendices

Appendix 1: Derivation of the Normalization Factor $Z_i$

$$\begin{aligned} Z_i= & {} \int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) \{ p(+1|s_i, \xi _g) \times p(+1|s_i, \epsilon _j) \nonumber \\&+\, p(-1|s_i, \xi _g) \prod _{j} p(t_{ij}|-1, \epsilon _j) \} ds_i\nonumber \\= & {} \int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) p(+1|s_i, \xi _g) \nonumber \\&\qquad \times \prod _{j} p(t_{ij}|+1, \epsilon _j)ds_i\nonumber \\&+ \, \int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) p(-1|s_i, \xi _g) \nonumber \\&\qquad \times \prod _{j} p(t_{ij}|-1, \epsilon _j)ds_i\nonumber \\= & {} \int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) \pounds (\xi _g, s_i) \nonumber \\&\qquad \times \prod \limits _{t_{ij} \in \{1,-1\}} \pounds (\epsilon _j, t_{ij}) ds_i \nonumber \\&+\,\int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) \pounds (\xi _g, -s_i) \nonumber \\&\qquad \times \prod \limits _{t_{ij} \in \{1,-1\}} \pounds (\epsilon _j, -t_{ij}) ds_i \nonumber \\= & {} \int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) \pounds (\xi _g, s_i) \nonumber \\&\qquad \times \prod \limits _{t_{ij} = 1} {\epsilon }_j \prod \limits _{t_{ij} = -1} (1-{\epsilon }_j) ds_i \nonumber \\&+ \,\int _{-\infty }^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) \pounds (\xi _g, -s_i) \nonumber \\&\qquad \times \prod \limits _{t_{ij} = 1} (1-{\epsilon }_j) \prod \limits _{t_{ij} = -1} {\epsilon }_j ds_i \nonumber \\= & {} \int _{-\infty }^{0} \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) C_1 ds_i \nonumber \\&+\,\int _{0}^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}}) C_2 ds_i, \end{aligned}$$

(37)

where

$$\begin{aligned} \pounds (\alpha , \beta )= & {} (2\alpha -1)\varTheta (\beta ) + (1-\alpha ),\end{aligned}$$

(38)

$$\begin{aligned} C_1= & {} (1-\xi _g)\prod \limits _{t_{ij} = -1} (1-{\epsilon }_j)\nonumber \\&\times \prod \limits _{t_{ij} = 1} {\epsilon }_j + \xi _g\prod \limits _{t_{ij} = -1} {\epsilon }_j \prod \limits _{t_{ij} = 1} (1-{\epsilon }_j), \end{aligned}$$

(39)

$$\begin{aligned} C_2= & {} \xi _g\prod \limits _{t_{ij} = -1} (1-{\epsilon }_j) \nonumber \\&\times \prod \limits _{t_{ij} = 1} {\epsilon }_j \!+\! (1-\xi _g)\prod \limits _{t_{ij} \!=\! -1} {\epsilon }_j \prod \limits _{t_{ij} = 1} (1-{\epsilon }_j).\quad \end{aligned}$$

(40)

Consider that

$$\begin{aligned}&\int _{0}^{+\infty } \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}})d s_i \nonumber \\&\quad = \int _{-\infty }^{\frac{m_{-i}^{\text {old}}}{\sqrt{v_{-i}^{\text {old}}}}} \mathcal{N}(\tau ; 0, 1) d\tau , = \varPhi \left( \frac{ m_{-i}^{\text {old}}}{\sqrt{v_{-i}^{\text {old}}}}\right) \end{aligned}$$

(41)

$$\begin{aligned}&\int _{-\infty }^{0} \mathcal{N}(s_i| m_{-i}^{\text {old}}, v_{-i}^{\text {old}})d s_i \nonumber \\&\quad = 1 - \varPhi \left( \frac{ m_{-i}^{\text {old}}}{\sqrt{v_{-i}^{\text {old}}}}\right) , \end{aligned}$$

(42)

and we can rewrite the eqnarray as:

$$\begin{aligned} Z_i = (C_2-C_1) \varPhi \left( \frac{ m_{-i}^{\text {old}}}{\sqrt{v_{-i}^{\text {old}}}}\right) + C_1. \end{aligned}$$

(43)

Appendix 2: Moment Matching in the Expectation Propagation Algorithm

Minimizing $KL[Q_{-i}(s_i)p(\mathbf{t}_i|s_i)||Q_{-i}(s_i){\tilde{F}}_i(s_i)]$ to obtain an update for the approximation ${\tilde{F}}_i(s_i)$, we recompute the parameters according to the normalized constant presented in Appendix 1, i.e.,

$$\begin{aligned} Z_i = (C_2 - C_1)\varPhi (z_i) + C_1, \end{aligned}$$

(44)

where

$$\begin{aligned} z_i = \frac{m_{-i}^{\text {old}}}{\sqrt{v_{-i}^{\text {old}}}} , \varPhi (x) = \int _{-\infty }^{x} \mathcal{N}(\tau ; 0, 1) d\tau . \end{aligned}$$

By moment matching (Minka 2001), we have

$$\begin{aligned} m_{-i}^{\text {new}} = m_{-i}^{\text {old}} + v_{-i}^{\text {old}} \alpha , \end{aligned}$$

(45)

where $\alpha = \frac{1}{\sqrt{v_{-i}^{\text {old}}}} \cdot \frac{(C_2 - C_1) \mathcal{N}(z_i;0,1)}{Z_i}$. Hence we can obtain a new ${\tilde{F}}_i(s_i)$ by recomputing its parameters $A_i$, $\tilde{m}_i$, and $v_i$ as

$$\begin{aligned} v_i= & {} v_{-i}^{\text {old}} \left( \frac{1}{m_{-i}^{\text {new}} \alpha } - 1\right) ,\end{aligned}$$

(46)

$$\begin{aligned} \tilde{m}_i= & {} m_{-i}^{\text {new}} + v_i\alpha ,\end{aligned}$$

(47)

$$\begin{aligned} A_i= & {} Z_i \sqrt{1+ {v_i}^{-1} v_{-i}^{\text {old}}} \exp { \left( \frac{v_{-i}^{\text {old}} \alpha }{2 m_{-i}^{\text {new}}}\right) }. \end{aligned}$$

(48)

Appendix 3: Gradients of the Lower Bound

The gradients of the lower bound with respect to the parameters $\varvec{\varepsilon }$ are as follows

(49)

(50)

where

(51)

$$\begin{aligned} \mathfrak {R}(y_i, i, j)= & {} p(y_i|m_{s_i}, \xi _g)\prod \limits _{ j^{\prime } \ne j} p(t_{i j^{\prime }}|y_i,\epsilon _{j^{\prime }}). \end{aligned}$$

(52)

And the gradient with respect to a kernel parameter $\theta \in \varvec{\vartheta }$ is

$$\begin{aligned} \frac{\partial F_{\varvec{\vartheta }}}{\partial {\theta }}= & {} -\frac{1}{2} tr(\mathbf{K}^{-1} \frac{\partial \mathbf{K}}{\partial {\theta }} \nonumber \\&+\, \frac{1}{2}E_q[\mathbf{S}_L]^T \mathbf{K}^{-1} \frac{\partial \mathbf{K}}{\partial {\theta }} \mathbf{K}^{-1} E_q[\mathbf{S}_L] \nonumber \\&+\,\frac{1}{2}tr(\mathbf{K}^{-1} \frac{\partial \mathbf{K}}{\partial {\theta }} \mathbf{K}^{-1} Cov[\mathbf{S}_L]) . \end{aligned}$$

(53)