Skip to main content
SpringerLink
Log in

A discriminative multiple-manifold network for image set classification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Because the distinct advantages of manifold-learning methods for feature extraction, Riemannian manifolds have been used extensively in image recognition tasks in recent years. However, large intra-class variation is a major problem for researchers. Although recent studies have alleviated this problem by extending nonlinear learning to Riemannian manifolds, important classification information may be lost when describing image sets with a single manifold. To address this problem, we design a novel discriminative multiple-manifold network (DMMNet) for image recognition. The Grassmann and symmetric positive definite manifold are first used simultaneously to model each image set. Then, the symmetric positive definite manifold and Grassmann manifold nonlinear learning blocks are devised to transform the data into low-dimensional manifold matrices to enhance their discriminability. A feature fusion layer is designed to fuse the complementary features of different manifolds, and a point-to-point hidden layer is devised to focus on more important feature maps. To acquire the importance of the manifold feature maps, a feature selection method based on manifold feature importance is designed to learn the corresponding weight coefficients. Finally, an output layer is used to map the optimal feature subset into the Euclidean space, such that the kernel discriminant analysis and K-nearest neighbor algorithm are applied. The proposed DMMNet is compared with state-of-the-art techniques on four datasets. Extensive experimental results demonstrate the effectiveness of the proposed DMMNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request

References

  1. Huang S, Cai G, Wang T, Ma T (2021) Amplitude-phase information measurement on riemannian manifold for motor imagery-based bci. IEEE Signal Process Lett 28:1310–1314

    Article  Google Scholar 

  2. Gao Y, Sun X, Meng M, Zhang Y (2022) Eeg emotion recognition based on enhanced spd matrix and manifold dimensionality reduction. Comput Biol Med 146(105):606

    Google Scholar 

  3. Xie X, Zou X, Yu T, Tang R, Hou Y, Qi F (2022) Multiple graph fusion based on riemannian geometry for motor imagery classification. Appl Intell 52(8):9067–9079

    Article  Google Scholar 

  4. Harandi M, Salzmann M, Hartley R (2018) Dimensionality reduction on spd manifolds: The emergence of geometry-aware methods. IEEE Trans Pattern Anal Mach Intell 40(1):48–62

    Article  Google Scholar 

  5. Liu Z, Xiang L, Shi K, Zhang K, Wu Q (2020) Robust manifold embedding for face recognition. IEEE Access 8:101,224–101,234

  6. Chen KX, Ren JY, Wu XJ, Kittler J (2020) Covariance descriptors on a gaussian manifold and their application to image set classification. Pattern Recognit 107(107):463

    Google Scholar 

  7. Ren J, Wu XJ (2020) Probability distribution-based dimensionality reduction on riemannian manifold of spd matrices. IEEE Access 8:153,881–153,890

  8. Ding C, Liu K, Cheng F, Belyaev E (2021) Spatio-temporal attention on manifold space for 3d human action recognition. Appl Intell 51:560–570

    Article  Google Scholar 

  9. Huang Z, Van Gool L (2017) A riemannian network for spd matrix learning. In: AAAI conference on artificial intelligence, pp 2036–2042

  10. Wang R, Wu XJ, Xu T, Hu C, Kittler J (2022) Deep metric learning on the spd manifold for image set classification. IEEE Trans Circuits Syst Video Technol

  11. Wang R, Wu XJ, Chen KX, Kittler J (2020) Multiple riemannian manifold-valued descriptors based image set classification with multi-kernel metric learning. IEEE Trans Big Data 8(3):753–769

    Article  Google Scholar 

  12. Harandi MT, Sanderson C, Shirazi S, Lovell BC (2011) Graph embedding discriminant analysis on grassmannian manifolds for improved image set matching. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 2705–2712

  13. Wang R, Guo H, Davis LS, Dai Q (2012) Covariance discriminative learning: A natural and efficient approach to image set classification. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 2496–2503

  14. Arsigny V, Fillard P, Pennec X, Ayache N (2007) Geometric means in a novel vector space structure on symmetric positive-definite matrices. SIAM J Matrix Anal Appl 29(1):328–347

    Article  MathSciNet  MATH  Google Scholar 

  15. Hamm J, Lee DD (2008) Grassmann discriminant analysis: a unifying view on subspace-based learning. In: 2008 25th international conference on Machine learning, pp 376–383

  16. Pennec X, Fillard P, Ayache N (2006) A riemannian framework for tensor computing. Int J Comput Vision 66:41–66

    Article  MATH  Google Scholar 

  17. Gao W, Ma Z, Xiong C, Gao T (2022) Dimensionality reduction of spd data based on riemannian manifold tangent spaces and local affinity. Appl Intell pp 1–25

  18. Fang H, Jin J, Daly I, Wang X (2022) Feature extraction method based on filter banks and riemannian tangent space in motor-imagery bci. IEEE J Biomed Health Inf 26(6):2504–2514

    Article  Google Scholar 

  19. Zou J, Zhang Y, Liu H, Ma L (2022) Monogenic features based single sample face recognition by kernel sparse representation on multiple riemannian manifolds. Neurocomputing 504:82–98

    Article  Google Scholar 

  20. Li X, Yang Y, Hu N, Cheng Z, Shao H, Cheng J (2022) Maximum margin riemannian manifold-based hyperdisk for fault diagnosis of roller bearing with multi-channel fusion covariance matrix. Adv Eng Inf 51(101):513

    Google Scholar 

  21. Gao Z, Wu Y, Harandi M, Jia Y (2019) A robust distance measure for similarity-based classification on the spd manifold. IEEE Trans Neural Netw Learn Syst 31(9):3230–3244

    Article  MathSciNet  Google Scholar 

  22. Feng W, Wang Z (2022) Multi-view multi-manifold learning with local and global structure preservation. Appl Intell pp 1–17

  23. Feng S, Hua X, Zhu X (2020) Matrix information geometry for spectral-based spd matrix signal detection with dimensionality reduction. Entropy 22(9):914

    Article  MathSciNet  Google Scholar 

  24. Huang Z, Wang R, Shan S, Chen X (2015) Face recognition on large-scale video in the wild with hybrid euclidean-and-riemannian metric learning. Pattern Recognit 48(10):3113–3124

    Article  Google Scholar 

  25. Wang R, Wu XJ, Chen KX, Kittler J (2018) Multiple manifolds metric learning with application to image set classification. In: 2018 24th International conference on pattern recognition, IEEE, pp 627–632

  26. Sun T, Ding S, Guo L (2022) Low-degree term first in resnet, its variants and the whole neural network family. Neural Networks 148:155–165

    Article  Google Scholar 

  27. Hu M, Wang H, Wang X, Yang J, Wang R (2019) Video facial emotion recognition based on local enhanced motion history image and cnn-ctslstm networks. J Visual Commun Image Represent 59:176–185

    Article  Google Scholar 

  28. Zhang J, Zou X, Kuang LD, Wang J, Sherratt RS, Yu X (2022) Cctsdb 2021: a more comprehensive traffic sign detection benchmark. Hum -centric Comput Inf Sci 12

  29. Brooks D, Schwander O, Barbaresco F, Schneider JY, Cord M (2019) Riemannian batch normalization for spd neural networks. Adv Neural Inf Process Syst 32

  30. Liu X, Ma Z (2020) Kernel-based subspace learning on riemannian manifolds for visual recognition. Neural Process Lett 51:147–165

    Article  Google Scholar 

  31. Zhuang R, Ma Z, Feng W, Lin Y (2020) Spd data dictionary learning based on kernel learning and riemannian metric. IEEE Access 8:61,956–61,972

  32. Wang R, Wu XJ, Kittler J (2020) Graph embedding multi-kernel metric learning for image set classification with grassmannian manifold-valued features. IEEE Trans Multimedia 23:228–242

    Article  Google Scholar 

  33. Hu WB, Wu XJ, Xu TY (2022) One-step kernelized sparse clustering on grassmann manifolds. Multimedia Tools Appl 81(21):31,017–31,038

  34. Lu J, Wang G, Moulin P (2013) Image set classification using holistic multiple order statistics features and localized multi-kernel metric learning. In: 2013 IEEE international conference on computer vision, pp 329–336

  35. Chen Z, Xu T, Wu XJ, Wang R, Kittler J (2021) Hybrid riemannian graph-embedding metric learning for image set classification. IEEE Trans Big Data

  36. Huang Z, Wang R, Shan S, Li X, Chen X (2015) Log-euclidean metric learning on symmetric positive definite manifold with application to image set classification. In: Int Conf Mach Learn, PMLR, pp 720–729

  37. Wei D, Shen X, Sun Q, Gao X, Ren Z (2022) Neighborhood preserving embedding on grassmann manifold for image-set analysis. Pattern Recognit 122(108):335

    Google Scholar 

  38. Zhang J, Zheng Z, Xie X, Gui Y, Kim GJ (2022) Reyolo: A traffic sign detector based on network reparameterization and features adaptive weighting. J Ambient Intell Smart Environ 14:317–334

    Article  Google Scholar 

  39. Zhang J, Huang H, Jin X, Kuang LD, Zhang J (2023) Siamese visual tracking based on criss-cross attention and improved head network. Multimedia Tools Appl pp 1–27

  40. Wang R, Wu XJ, Kittler J (2021) Symnet: A simple symmetric positive definite manifold deep learning method for image set classification. IEEE Trans Neural Netw Learn Syst 33(5):2208–2222

    Article  MathSciNet  Google Scholar 

  41. Wang R, Wu XJ (2020) Grasnet: A simple grassmannian network for image set classification. Neural Process Lett 52(1):693–711

    Article  Google Scholar 

  42. Wang R, Wu XJ, Chen Z, Xu T, Kittler J (2022) Learning a discriminative spd manifold neural network for image set classification. Neural Networks 151:94–110

    Article  Google Scholar 

  43. Nguyen XS, Brun L, Lézoray O, Bougleux S (2019) A neural network based on spd manifold learning for skeleton-based hand gesture recognition. In: 2019 IEEE Conference on Computer Vision and Pattern Recognition, pp 12,036–12,045

  44. Zhang T, Zheng W, Cui Z, Zong Y, Li C, Zhou X, Yang J (2020) Deep manifold-to-manifold transforming network for skeleton-based action recognition. IEEE Trans Multimedia 22(11):2926–2937

    Google Scholar 

  45. Chakraborty R, Bouza J, Manton JH, Vemuri BC (2020) Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Trans Pattern Anal Mach Intell 44(2):799–810

    Article  Google Scholar 

  46. Sra S (2012) A new metric on the manifold of kernel matrices with application to matrix geometric means. Adv Neural Inf Process Syst 25

  47. Moakher M, Batchelor PG (2006) Symmetric positive-definite matrices: From geometry to applications and visualization. Visualization and processing of tensor fields pp 285–298

  48. Kulis B, Sustik M, Dhillon I (2006) Learning low-rank kernel matrices. In: 2006 23rd international conference on Machine learning, pp 505–512

  49. Huang Z, Wang R, Shan S, Chen X (2015) Projection metric learning on grassmann manifold with application to video based face recognition. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp 140–149

  50. Niijima S, Okuno Y (2008) Laplacian linear discriminant analysis approach to unsupervised feature selection. IEEE/ACM Trans Comput Biol Bioinf 6(4):605–614

  51. Dornaika F (2020) Multi-layer manifold learning with feature selection. Appl Intell 50(6):1859–1871

    Article  Google Scholar 

  52. Zhang L, Zheng X, Pang Q, Zhou W (2021) Fast gaussian kernel support vector machine recursive feature elimination algorithm. Appl Intell 51:9001–9014

    Article  Google Scholar 

  53. Hamm J, Lee DD (2008) Grassmann discriminant analysis: a unifying view on subspace-based learning. In: 2008 25th international conference on Machine learning, pp 376–383

  54. Wang R, Wu XJ, Xu T, Hu C, Kittler J (2023) U-spdnet: An spd manifold learning-based neural network for visual classification. Neural Networks 161:382–396

    Article  Google Scholar 

  55. Li C, Li S, Gao Y, Zhang X, Li W (2021) A two-stream neural network for pose-based hand gesture recognition. IEEE Trans Cognit Dev Syst 14(4):1594–1603

    Article  Google Scholar 

  56. Huang Z, Wu J, Van Gool L (2018) Building deep networks on grassmann manifolds. In: Proceedings of the AAAI Conference on Artificial Intelligence

  57. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. In: 2016 IEEE conference on computer vision and pattern recognition, pp 1933–1941

  58. Vemulapalli R, Arrate F, Chellappa R (2014) Human action recognition by representing 3d skeletons as points in a lie group. In: 2014 IEEE conference on computer vision and pattern recognition, pp 588–595

  59. Garcia-Hernando G, Yuan S, Baek S, Kim TK (2018) First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In: 2018 IEEE conference on computer vision and pattern recognition, pp 409–419

  60. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. In: IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 1110–1118

  61. Tekin B, Bogo F, Pollefeys M (2019) H+ o: Unified egocentric recognition of 3d hand-object poses and interactions. In: IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 4511–4520

  62. Celik Y, Talo M, Yildirim O, Karabatak M, Acharya UR (2020) Automated invasive ductal carcinoma detection based using deep transfer learning with whole-slide images. Pattern Recognit Lett 133:232–239

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Electronics and Optical Engineering and College of Flexible Electronics, Nanjing University of Posts and Telecommunications, Jiangsu Nanjing, 210023, China

    Hao Wu, Weigang Wang, Zishan Xia, Yonghao Chen, Yuanjian Liu & Jianfei Chen

Authors
  1. Hao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weigang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zishan Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yonghao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuanjian Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jianfei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weigang Wang.

Ethics declarations

Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Wang, W., Xia, Z. et al. A discriminative multiple-manifold network for image set classification. Appl Intell 53, 25119–25134 (2023). https://doi.org/10.1007/s10489-023-04900-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04900-1

Keywords

Search

Navigation