Facial expression recognition using ${l}_{p}$-norm MKL multiclass-SVM

Abstract

Automatic recognition of facial expressions is an interesting and challenging research topic in the field of pattern recognition due to applications such as human–machine interface design and developmental psychology. Designing classifiers for facial expression recognition with high reliability is a vital step in this research. This paper presents a novel framework for person-independent expression recognition by combining multiple types of facial features via multiple kernel learning (MKL) in multiclass support vector machines (SVM). Existing MKL-based approaches jointly learn the same kernel weights with $l_{1}$-norm constraint for all binary classifiers, whereas our framework learns one kernel weight vector per binary classifier in the multiclass-SVM with $l_{p}$-norm constraints $(p \ge 1)$, which considers both sparse and non-sparse kernel combinations within MKL. We studied the effect of $l_{p}$-norm MKL algorithm for learning the kernel weights and empirically evaluated the recognition results of six basic facial expressions and neutral faces with respect to the value of “$p$”. In our experiments, we combined two popular facial feature representations, histogram of oriented gradient and local binary pattern histogram, with two kernel functions, the heavy-tailed radial basis function and the polynomial function. Our experimental results on the CK$+$, MMI and GEMEP-FERA face databases as well as our theoretical justification show that this framework outperforms the state-of-the-art methods and the SimpleMKL-based multiclass-SVM for facial expression recognition.

Appendices

Appendices

1.1 Appendix A. Description of MKL-based SVM in multiple kernel spaces

To explain the functionality of MKL-based SVM, we utilize the $p=1$ case as an example. In this case, the optimized kernel combination weights follow the constraint ${\sum _{i=1}^{M}{d_{m}}} = 1,d_{m} \ge 0$. Given a test example $x_{0} \in {\mathbb {R}}^{D}$, we rewrite Eq. 5 as follows.

$$\begin{aligned}&y_{0} = {{\text {sgn}}}\left[ {\sum }_{i=1}^{N}{{\sum }_{m=1}^{M}{{\alpha }_{i}y_{i}d_{m}k_{m}(x_{i},x_{0})+w_{0}}}\right] \\& = {{\text {sgn}}}\left[ {\sum }_{m=1}^{M}{d_{m}\underbrace{\left( {\sum }_{i=1}^{N}{{\alpha }_{i}y_{i}k_{m}(x_{i},x_{0})}+w_{0}\right) }_{\text {single kernel with single feature}}}\right] \end{aligned}$$

From the above equation, the formulation within the under bracket is the discriminant function used for classifying new samples in canonical binary SVM. In other words, using MKL-based SVM the label of a sample is determined based on weighted summation of the results obtained from each RKHS, which enhances the discriminant power for classification.

1.2 Appendix B. Proof of the superiority of MKL-based SVM over canonical binary SVM with single kernel and single type of features

Without loss of generality, our proof is pursed in the case of $1<p<2$. We transform the object function of Eq. 3 based on the Lemma 26 in [82] as:

$$\begin{aligned} \mathop {\min }_{d,\Vert d\Vert _{r} \le 1} \mathop {\min }_{w,w_{0},\xi } \quad J(d,w,w_{0},\xi ) = \frac{1}{2}\sum _{m=1}^{M}{\frac{\Vert w_{m}\Vert ^{2}_{2}}{d_{m}}}+C\sum _{i=1}^{N}{\xi _{i}} \end{aligned}$$

where $r=p/(2-p)$.

As described in Sect. 3.3, this convex optimization problem is solved by the two-step method, where two nested iterations are equipped in each loop of the method. In the outer iteration, the kernel combination weights are updated by fixing the parameters of SVM, whereas in the inner iteration the optimization problem of canonical SVM is solved by fixing the updated kernel combination weights. Let $N_{f}$ be the number of features extracted from each sample and $N_{k}$ the number of kernel functions used in the $l_{p}$-norm MKL-based SVM. We denote the updated kernel combination vector in the $t$th loop of the two-step method as follows.

$$\begin{aligned} d^{(t)}= & {} \left[ \underbrace{d_{1}^{(t)}, \ldots , d_{N_{k}}^{(t)}}_{\text {the } 1{\mathrm{st}} \text { feature}}, \ldots , \underbrace{d_{(i-1)N_{k}+1}^{(t)}, \ldots , d_{i \cdot N_{k}}^{(t)}}_{\text {the } i{\mathrm{th}} \text { feature}}, \ldots ,\right. \\&\quad \left. \underbrace{d_{(N_{f}-1)N_{k}+1}^{(t)}, \ldots , d_{N_{f}N_{k}}^{(t)}}_{\text {the } {N_{f}}{\mathrm{th}}} \text {feature}\right] ^{T} \\ d^{(t)}\in & {} {{\mathbb {R}}_{+}^{\star N_{f}N_{K}}},\quad {\Vert d^{(t)}\Vert }_{r}=1 \end{aligned}$$

In addition, the SVM discriminant hyperplane obtained in the outer iteration of the $t$th loop is denoted based on $w^{(t)}$ and $w_{0}^{(t)}$.

For the canonical binary SVM, we suppose that the $i$th feature with the $j$th kernel function is utilized. Then, the canonical SVM becomes a special case in the framework of MKL-based SVM, and its corresponding kernel combination vector can be defined as follows.

$$\begin{aligned} \hat{d} = \left[ \underbrace{0,0, \ldots ,0, \ldots , 0}_{{d_{1} \sim d_{(i-1)N_{k}+j-1}}}, 1, \underbrace{0,0, \ldots ,0, \ldots , 0}_{{d_{(i-1)N_{k}+j+1} \sim d_{N_{f}N_{k}}}}\right] ^{T} \end{aligned}$$

Further, the learned discriminant hyperplane of canonical SVM is defined based on ${\hat{w}}^{\star }$ and ${\hat{w}}_{0}^{\star }$.

By assuming that in the first loop of the two-step method $d^{(1)}$ is initialized as $\hat{d}$ in the outer iteration, we obtain that in the inner iteration of the first loop the learned $w^{(1)}={\hat{w}}^{\star }$ and $w_{0}^{(1)} = {\hat{w}}_{0}^{\star }$. Thereafter, our proof is formulated as follows,

$$\begin{aligned} {\hat{J}}^{\star }&= J({\hat{d}}, {\hat{w}}^{\star },{\hat{w}}_{0}^{\star }) = J(d^{(1)},w^{(1)},w_{0}^{(1)})\\&\ge J(d^{(2)},w^{(1)},w_{0}^{(1)}) \ge J(d^{(2)},w^{(2)},w_{0}^{(2)}) \\&\ge \cdots \ge J(d^{\star },w^{\star },w_{0}^{\star }) = J^{\star } \end{aligned}$$

where $J^{\star }$ is the learned minimum of the objective function in $l_{p}$-norm MKL-based SVM with its corresponding optimum $d^{\star },w^{\star },w_{0}^{\star }$, and ${\hat{J}}^{\star }$ with ${\hat{w}}^{\star },{\hat{w}}_{0}^{\star }$ is for canonical binary SVM.

Based on the above justification, we can naturally extend the conclusion to a more general case. That is:

Suppose $\exists$ a set of basis kernel functions $S$ ($S \ne \emptyset$) and a set of features $F$ ($F \ne \emptyset$). Then $\forall S^{\prime } \subseteq S$ ($S^{\prime } \ne \emptyset$) and $F^{\prime } \subseteq F$ ($F^{\prime } \ne \emptyset$), we obtain that $J^{\star }_{S \times F} \le J^{\star }_{S^{\prime } \times F^{\prime }}$, since $d^{\star }_{S^{\prime } \times F^{\prime }}$ can be seen as a special case of $d_{S \times F}$. The subscripts $S \times F$ and $S^{\prime } \times F^{\prime }$ denote the kernels and features in use.

To be more specific, we conclude that MKL-based SVM with multiple kernels and features performs better or at least equally than those with multiple kernels and single feature or with single kernel and multiple features.

1.3 Appendix C. Proof of the superiority of our proposed MKL-based multiclass-SVM over the SimpleMKL-based multiclass-SVM

To be consistent with the SimpleMKL-based multiclass-SVM, we set $p=1$ in our framework. Then, the only difference between the two methods are the ways of updating the kernel combination vectors for multiclass classification tasks as mentioned in Eqs. 6 and 7. The superiority of our proposed MKL framework for multiclass-SVM lies in the fact that its minimized objective function preserves the lower boundary of the one obtained using SimpleMKL-based multiclass-SVM. That is, the derived hyperplanes from our method perform better or at least equally among the training data.

Suppose that ${\hat{L}}^{\star }$ is the optimal value of the objective function in Eq. 7, and $d_{u}^{\star }$ is the learned optimum for each binary classifier in our framework. $L^{\star }$ and $d^{\star }$ are the corresponding notations for the SimpleMKL-based multiclass-SVM in Eq. 6. Our proof is as follows

$$\begin{aligned} {\hat{L}}^{\star } = \sum _{u \in \varPhi }L_{u}(d_{u}^{\star }) \le \sum _{u \in \varPhi }L_{u}({d}^{\star }) = L^{\star } \end{aligned}$$

since $L_{u}(d_{u}^{\star }) \le L_{u}({d}^{\star }), \forall u \in \varPhi$.