Weighted Decoding ECOC for Facial Action Unit Classification

  • Terry Windeatt
Part of the Studies in Computational Intelligence book series (SCI, volume 245)


There are two approaches to automating the task of facial expression recognition, the first concentrating on what meaning is conveyed by facial expression and the second on categorising deformation and motion into visual classes. The latter approach has the advantage that the interpretation of facial expression is decoupled from individual actions as in FACS (Facial Action Coding System). In this chapter, upper face action units (aus) are classified using an ensemble of MLP base classifiers with feature ranking based on PCA components. When posed as a multi-class problem using Error-Correcting-Output-Coding (ECOC), experimental results on Cohn-Kanade database demonstrate that error rates comparable to two-class problems (one-versus-rest) may be obtained. The ECOC coding and decoding strategies are discussed in detail, and a novel weighted decoding approach is shown to outperform conventional ECOC decoding. Furthermore, base classifiers are tuned using the ensemble Out-of-Bootstrap estimate, for which purpose, ECOC decoding is modified. The error rates obtained for six upper face aus around the eyes are believed to be among the best for this database.


Code Word Random Code Recursive Feature Elimination Facial Action Code System Multiclass Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allwein, E.L., Schapire, R.E., Singer, Y.: Reducing multi-class to binary: a unifying approach for margin classifiers. J. Mach. Learn. Res. 1, 113–141 (2000)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: Proc. IEEE Conf. Syst., Man and Cybernetics, The Hague, The Netherlands, pp. 592–597. IEEE Comp. Soc., Los Alamitos (2004)Google Scholar
  3. 3.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Fully automatic facial action recognition in spontaneous behavior. In: Proc. 7th IEEE Conf. Automatic Face and Gesture Recogn., Southampton, UK, pp. 223–238. IEEE Comp. Soc., Los Alamitos (2006)CrossRefGoogle Scholar
  4. 4.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1997)Google Scholar
  5. 5.
    Bylander, T.: Estimating generalisation error two-class datasets using out-of-bag estimate. Mach. Learn. 48(1-3), 287–297 (2002)MATHCrossRefGoogle Scholar
  6. 6.
    Cover, T.M.: Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans. Electronic Comp. 14(3), 326–334 (1965)MATHCrossRefGoogle Scholar
  7. 7.
    Crammer, K., Singer, Y.: On the learnability and design of output codes for multiclass problems. Mach. Learn. 47(2-3), 201–233 (2002)MATHCrossRefGoogle Scholar
  8. 8.
    Dietterich, T.G., Bakiri, G.: Error-correcting output codes: a general method for improving multiclass inductive learning programs. In: Proc. 9th Natl. Conf. Artif. Intell., Anaheim, CA, pp. 572–577. AAAI/MIT Press, Cambridge (1991)Google Scholar
  9. 9.
    Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Research 2, 263–286 (1995)MATHGoogle Scholar
  10. 10.
    Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  11. 11.
    Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Patt. Analysis Mach. Intell. 21(10), 974–989 (1999)CrossRefGoogle Scholar
  12. 12.
    Escalara, S., Tax, D.M.J., Pujol, O., Radeva, P., Duin, R.W.: Subclass problem-dependent design for error-correcting output codes. IEEE Trans. Patt. Analysis Mach. Intell. 30(8), 1041–1054 (2008)CrossRefGoogle Scholar
  13. 13.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalisation of on-line learning and application to boosting. J. Comp. Syst. Sci. 55(1), 119–139 (1997)MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Guyon, I., Weston, J., Barnhill, S., Vapnik, V.: Gene selection for cancer classification using support vector machines. Mach. Learn. 46(1-3), 389–422 (2002)MATHCrossRefGoogle Scholar
  15. 15.
    Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Trans. Patt. Analysis Mach. Intell. 12(10), 993–1001 (1990)CrossRefGoogle Scholar
  16. 16.
    Hastie, T., Tibshirani, R.: Classification by pairwise coupling. The Annals of Stat. 26(2), 451–471 (1998)MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Haykin, S.: Neural Networks: a Comprehensive Foundation. Prentice Hall, Upper Saddle River (1999)MATHGoogle Scholar
  18. 18.
    James, G.M., Hastie, T.: The error coding method and PiCT. Computational and Graphical Stat. 7(3), 377–387 (1998)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proc. 4th Int. Conf. Automatic Face and Gesture Recogn., Grenoble, France, pp. 46–53. IEEE Comp. Soc., Los Alamitos (2000)Google Scholar
  20. 20.
    Kong, E.B., Diettrich, T.G.: Error-correcting output coding corrects bias and variance. In: Prieditis, A., Russell, S.J. (eds.) Proc. 12th Int. Conf. Mach. Learn., Tahoe City, CA, pp. 313–321. Morgan Kaufmann, San Francisco (1995)Google Scholar
  21. 21.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles. Mach. Learn. 51(2), 181–207 (2003)MATHCrossRefGoogle Scholar
  22. 22.
    Merz, C.J., Murphy, P.M.: UCI repository of machine learning databases (1998),
  23. 23.
    Peterson, W.W., Weldon, J.R.: Error-Correcting Codes. MIT Press, Cambridge (1972)MATHGoogle Scholar
  24. 24.
    Schapire, R.E.: Using output codes to boost multiclass learning problems. In: Fisher, D.H. (ed.) Proc. 14th Int. Conf. Mach. Learn., Learn., Nashville, TN, pp. 313–321. Morgan Kaufmann, San Francisco (1997)Google Scholar
  25. 25.
    Sejnowski, T.J., Rosenberg, C.R.: Parallel networks that learn to pronounce English text. Complex Systems 1(1), 145–168 (1987)MATHGoogle Scholar
  26. 26.
    Silapachote, P., Karuppiah, D.R., Hanson, A.R.: Feature selection using Adaboost for face expression recognition. In: Villanueva, J.J. (ed.) Proc. 4th IASTEAD Int. Conf. Visualization, Imaging and Image Proc., Marbella, Spain, pp. 84–89. ACTA Press, Calgary (2004)Google Scholar
  27. 27.
    Tian, Y., Kanade, T., Cohn, J.F.: Recognising action units for facial expression analysis. IEEE Trans. Patt. Analysis Mach. Intell. 23(2), 97–115 (2001)CrossRefGoogle Scholar
  28. 28.
    Tian, Y., Kanade, T., Cohn, J.F.: Evaluation of Gabor-based facial action unit recognition in image sequences of increasing complexity. In: Proc. 5th Int. Conf. Automatic Face and Gesture Recogn., Washington, DC, pp. 229–234. IEEE Comp. Soc., Los Alamitos (2002)Google Scholar
  29. 29.
    Tou, J.T., Gonzales, R.C.: Pattern Recognition Principles. Addison-Wesley, Reading (1974)MATHGoogle Scholar
  30. 30.
    Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Proc. IEEE Int. Conf. Comp. Vision Patt. Recogn., Maui, HW, pp. 586–591. IEEE Comp. Soc., Los Alamitos (1991)CrossRefGoogle Scholar
  31. 31.
    Valentini, G., Dietterich, T.G.: Bias-variance analysis of support vector machines for the development of SVM-based ensemble methods. J. Mach. Learn. Res. 5, 725–775 (2004)MathSciNetGoogle Scholar
  32. 32.
    Wang, W., Jones, P., Partridge, D.: Assessing the impact of input features in a feedforward neural network. Neural Computing Appl. 9(2), 101–112 (2000)CrossRefGoogle Scholar
  33. 33.
    Wilson, C.L., Grother, P.J., Barnes, C.S.: Binary decision clustering for neural network-based optical character recognition. Patt. Recogn. 29(3), 425–437 (1996)CrossRefGoogle Scholar
  34. 34.
    Windeatt, T.: Diversity measures for multiple classifier system analysis and design. Inf. Fusion 6(1), 21–36 (2004)CrossRefGoogle Scholar
  35. 35.
    Windeatt, T.: Spectral measure for multi-class problems. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 184–193. Springer, Heidelberg (2004)Google Scholar
  36. 36.
    Windeatt, T.: Accuracy/diversity and ensemble classifier design. IEEE Trans. Neural Networks 17(5), 287–297 (2006)CrossRefGoogle Scholar
  37. 37.
    Windeatt, T.: Ensemble MLP classifier design. In: Jain, L.C., Sato-Ilic, M., Virvou, M., Tsihrintzis, G.A., Balas, V.E., Abeynayake, C. (eds.) Computational Intelligence Paradigms. Studies in Computational Intelligence, vol. 137, pp. 133–147. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  38. 38.
    Windeatt, T., Ghaderi, R.: Multi-class learning and error-correcting code sensitivity. Electronics Letters 36(19), 1630–1632 (2000)CrossRefGoogle Scholar
  39. 39.
    Windeatt, T., Ghaderi, R.: Coding and decoding strategies for multiclass learning problems. Inf. Fusion 4(1), 11–21 (2003)CrossRefGoogle Scholar
  40. 40.
    Windeatt, T., Prior, M.: Stopping criteria for ensemble-based feature selection. In: Haindl, M., Kittler, J., Roli, F. (eds.) MCS 2007. LNCS, vol. 4472, pp. 271–281. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  41. 41.
    Windeatt, T., Dias, K.: Feature-ranking ensembles for facial action unit classification. In: Prevost, L., Marinai, S., Schwenker, F. (eds.) ANNPR 2008. LNCS (LNAI), vol. 5064, pp. 267–279. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  42. 42.
    Windeatt, T., Prior, M., Effron, N., Intrator, N.: Ensemble-based feature selection criteria. In: Perner, P. (ed.) Poster Proc. 5th Int. Conf. Mach. Learn. Data Mining in Patt. Recogn., Leipzig, Germany, pp. 168–182. IBaI Publishing, Leipzig (2007)Google Scholar
  43. 43.
    Windeatt, T., Smith, R.S., Dias, K.: Weighted decoding ECOC for facial action unit classification. In: Okun, O., Valentini, G. (eds.) Proc. 2nd Workshop Supervised and Unsupervised Ensemble Methods and Their Appl., Patras, Greece, pp. 26–30 (2007)Google Scholar
  44. 44.
    Yu, L., Liu, H.: Efficient feature selection via analysis of relevance and redundancy. J. Mach. Learn. Res. 5, 1205–1224 (2004)MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Terry Windeatt
    • 1
  1. 1.CVSSPUniversity of Surrey, University, GuildfordSurreyUK

Personalised recommendations