Simultaneous Feature and Dictionary Learning for Image Set Based Face Recognition

  • Jiwen Lu
  • Gang Wang
  • Weihong Deng
  • Pierre Moulin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8689)


In this paper, we propose a simultaneous feature and dictionary learning (SFDL) method for image set based face recognition, where each training and testing example contains a face image set captured from different poses, illuminations, expressions and resolutions. While several feature learning and dictionary learning methods have been proposed for image set based face recognition in recent years, most of them learn the features and dictionaries separately, which may not be powerful enough because some discriminative information for dictionary learning may be compromised in the feature learning stage if they are applied sequentially, and vice versa. To address this, we propose a SFDL method to learn discriminative features and dictionaries simultaneously from raw face images so that discriminative information can be jointly exploited. Extensive experimental results on four widely used face datasets show that our method achieves better performance than state-of-the-art image set based face recognition methods.


Face recognition image set feature learning dictionary learning simultaneous learning 


  1. 1.
    Aharon, M., Elad, M., Bruckstein, A.: Ksvd: An algorithm for designing overcomplete dictionaries for sparse representation. TSP 54(11), 4311–4322 (2006)Google Scholar
  2. 2.
    Arandjelovic, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: CVPR, pp. 581–588 (2005)Google Scholar
  3. 3.
    Cevikalp, H., Triggs, B.: Face recognition based on image sets. In: CVPR, pp. 2567–2573 (2010)Google Scholar
  4. 4.
    Chen, Y.-C., Patel, V.M., Phillips, P.J., Chellappa, R.: Dictionary-based face recognition from video. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 766–779. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  5. 5.
    Chen, Y.C., Patel, V.M., Phillips, P.J., Chellappa, R.: Dictionary-based face recognition from video. Tech. rep., University of Maryland (2013)Google Scholar
  6. 6.
    Chen, Y.C., Patel, V.M., Shekhar, S., Chellappa, R., Phillips, P.J.: Video-based face recognition via joint sparse representation. In: FG, pp. 1–8 (2013)Google Scholar
  7. 7.
    Chin, T.J., Schindler, K., Suter, D.: Incremental kernel svd for face recognition with image sets. In: FG, pp. 461–466 (2006)Google Scholar
  8. 8.
    Cui, Z., Shan, S., Zhang, H., Lao, S., Chen, X.: Image sets alignment for video-based face recognition. In: CVPR, pp. 2626–2633 (2012)Google Scholar
  9. 9.
    Deng, W., Hu, J., Guo, J.: Extended src: Undersampled face recognition via intraclass variant dictionary. PAMI 34(9), 1864–1870 (2012)CrossRefGoogle Scholar
  10. 10.
    Deng, W., Hu, J., Lu, J., Guo, J.: Transform-invariant pca: A unified approach to fully automatic face alignment, representation, and recognition. PAMI 36(6), 1275–1284 (2014)CrossRefGoogle Scholar
  11. 11.
    Fan, W., Yeung, D.: Locally linear models on face appearance manifolds with application to dual-subspace based classification. In: CVPR, pp. 1384–1390 (2006)Google Scholar
  12. 12.
    Fitzgibbon, A., Zisserman, A.: Joint manifold distance: a new approach to appearance based clustering. In: CVPR, pp. 26–33 (2003)Google Scholar
  13. 13.
    Gross, R., Shi, J.: The cmu motion of body (mobo) database. Tech. rep., Carnegie Mellon University (2001)Google Scholar
  14. 14.
    Guo, H., Jiang, Z., Davis, L.S.: Discriminative dictionary learning with pairwise constraints. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part I. LNCS, vol. 7724, pp. 328–342. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  15. 15.
    Hadid, A., Pietikainen, M.: From still image to video-based face recognition: an experimental analysis. In: FG, pp. 813–818 (2004)Google Scholar
  16. 16.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)CrossRefzbMATHGoogle Scholar
  17. 17.
    Hu, Y., Mian, A., Owens, R.: Sparse approximated nearest points for image set classification. In: CVPR, pp. 121–128 (2011)Google Scholar
  18. 18.
    Hu, Y., Mian, A.S., Owens, R.: Face recognition using sparse approximated nearest points between image sets. PAMI 34(10), 1992–2004 (2012)CrossRefGoogle Scholar
  19. 19.
    Huang, L., Lu, J., Tan, Y.P.: Co-learned multi-view spectral clustering for face recognition based on image sets. IEEE Signal Processing Letters 21(7), 875–879 (2014)CrossRefGoogle Scholar
  20. 20.
    Jiang, Z., Lin, Z., Davis, L.S.: Learning a discriminative dictionary for sparse coding via label consistent k-svd. In: CVPR, pp. 1697–1704 (2011)Google Scholar
  21. 21.
    Kan, M., Shan, S., Xu, D., Chen, X.: Side-information based linear discriminant analysis for face recognition. In: BMVC, pp. 1–12 (2011)Google Scholar
  22. 22.
    Kim, M., Kumar, S., Pavlovic, V., Rowley, H.: Face tracking and recognition with visual constraints in real-world videos. In: CVPR, pp. 1–8 (2008)Google Scholar
  23. 23.
    Kim, T., Kittler, J., Cipolla, R.: Discriminative learning and recognition of image set classes using canonical correlations. PAMI 29(6), 1005–1018 (2007)CrossRefGoogle Scholar
  24. 24.
    Kong, S., Wang, D.: A dictionary learning approach for classification: Separating the particularity and the commonality. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 186–199. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  25. 25.
    Le, Q.V., Karpenko, A., Ngiam, J., Ng, A.: Ica with reconstruction cost for efficient overcomplete feature learning. In: NIPS, pp. 1017–1025 (2011)Google Scholar
  26. 26.
    Lee, H., Battle, A., Raina, R., Ng, A.: Efficient sparse coding algorithms. In: NIPS, pp. 801–808 (2006)Google Scholar
  27. 27.
    Lee, K., Ho, J., Yang, M., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. In: CVPR, pp. 313–320 (2003)Google Scholar
  28. 28.
    Lin, T., Liu, S., Zha, H.: Incoherent dictionary learning for sparse representation. In: ICPR, pp. 1237–1240 (2012)Google Scholar
  29. 29.
    Lu, J., Tan, Y.P., Wang, G., Yang, G.: Image-to-set face recognition using locality repulsion projections and sparse reconstruction-based similarity measure. TCSVT 23(6), 1070–1080 (2013)Google Scholar
  30. 30.
    Lu, J., Wang, G., Moulin, P.: Image set classification using multiple order statistics features and localized multi-kernel metric learning. In: ICCV, pp. 1–8 (2013)Google Scholar
  31. 31.
    Ma, L., Wang, C., Xiao, B., Zhou, W.: Sparse representation for face recognition based on discriminative low-rank dictionary learning. In: CVPR, pp. 2586–2593 (2012)Google Scholar
  32. 32.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Discriminative learned dictionaries for local image analysis. In: CVPR, pp. 1–8 (2008)Google Scholar
  33. 33.
    Shakhnarovich, G., Fisher III, J.W., Darrell, T.: Face recognition from long-term observations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part III. LNCS, vol. 2352, pp. 851–865. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  34. 34.
    Tosic, I., Frossard, P.: Dictionary learning. IEEE Signal Processing Magazine 28(2), 27–38 (2011)CrossRefGoogle Scholar
  35. 35.
    Viola, P., Jones, M.: Robust real-time face detection. IJCV 57(2), 137–154 (2004)CrossRefGoogle Scholar
  36. 36.
    Wang, R., Chen, X.: Manifold Discriminant Analysis. In: CVPR, pp. 1–8 (2009)Google Scholar
  37. 37.
    Wang, R., Guo, H., Davis, L., Dai, Q.: Covariance discriminative learning: A natural and efficient approach to image set classification. In: CVPR, pp. 2496–2503 (2012)Google Scholar
  38. 38.
    Wang, R., Shan, S., Chen, X., Gao, W.: Manifold-manifold distance with application to face recognition based on image set. In: CVPR, pp. 1–8 (2008)Google Scholar
  39. 39.
    Wang, X., Wang, B., Bai, X., Liu, W., Tu, Z.: Max-margin multiple-instance dictionary learning. In: ICML, pp. 846–854 (2013)Google Scholar
  40. 40.
    Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: CVPR, pp. 529–534 (2011)Google Scholar
  41. 41.
    Yamaguchi, O., Fukui, K., Maeda, K.: Face recognition using temporal image sequence. In: FG, pp. 318–323 (1998)Google Scholar
  42. 42.
    Yang, M., Zhang, L., Feng, X., Zhang, D.: Fisher discrimination dictionary learning for sparse representation. In: ICCV, pp. 543–550 (2011)Google Scholar
  43. 43.
    Yang, M., Zhang, L., Yang, J., Zhang, D.: Metaface learning for sparse representation based face recognition. In: ICIP, pp. 1601–1604 (2010)Google Scholar
  44. 44.
    Yang, M., Zhu, P., Van Gool, L., Zhang, L.: Face recognition based on regularized nearest points between image sets. In: FG, pp. 1–7 (2013)Google Scholar
  45. 45.
    Zhang, Q., Li, B.: Discriminative k-svd for dictionary learning in face recognition. In: CVPR, pp. 2691–2698 (2010)Google Scholar
  46. 46.
    Zuo, Z., Wang, G.: Learning discriminative hierarchical features for object recognition. IEEE Signal Processing Letters 21(9), 1159–1163 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Jiwen Lu
    • 1
  • Gang Wang
    • 1
    • 2
  • Weihong Deng
    • 3
  • Pierre Moulin
    • 1
    • 4
  1. 1.Advanced Digital Sciences CenterSingapore
  2. 2.Nanyang Technological UniversitySingapore
  3. 3.Beijing University of Posts and TelecommunicationsBeijingChina
  4. 4.University of Illinois at Urbana-ChampaignUSA

Personalised recommendations