Multi-view representation learning in multi-task scene

  • Run-kun Lu
  • Jian-wei LiuEmail author
  • Si-ming Lian
  • Xin Zuo
Original Article


Over recent decades have witnessed considerable progress in whether multi-task learning or multi-view learning, but the situation that considers both learning scenes simultaneously has received not too much attention. How to utilize multiple views’ latent representation of each single task to improve each learning task’s performance is a challenge problem. Based on this, we proposed a novel semi-supervised algorithm, termed as multi-task multi-view learning based on common and special features (MTMVCSF). In general, multi-views are the different aspects of an object and every view includes the underlying common or special information of this object. As a consequence, we will mine multiple views’ jointly latent factor of each learning task, jointly latent factor is consisted of each view’s special feature and the common feature of all views. By this way, the original multi-task multi-view data have degenerated into multi-task data, and exploring the correlations among multiple tasks enables to make an improvement on the performance of learning algorithm. Another obvious advantage of this approach is that we get latent representation of the set of unlabeled instances by the constraint of regression task with labeled instances. In classification and semi-supervised clustering tasks, using implicit representation as input peforms much better than using raw data. Furthermore, an anti-noise multi-task multi-view algorithm called AN-MTMVCSF is proposed, which has a strong adaptability to noise labels. The effectiveness of these algorithms is proved by a series of well-designed experiments on both real-world and synthetic data.


Multi-view Multi-task Latent representation Special feature Common feature 



This work was supported by the National Key R&D Program of China (No. 2016YFC030370303). Earlier version of this paper was presented at 2018 37th Chinese Control Conference (CCC), and the doi is 10.23919/ChiCC.2018.8482783. We have benefited from comments at this conference, and based on the early work, we expand the content of this paper by explaining the algorithm more detail and adding more experiments. Furthermore, we proposed an anti-noise version based on the original algorithm, which increases the ability of anti-noise of the algorithm when facing noise labels.


  1. 1.
    Zhang X, Zhang X, Liu H, Liu X (2016) Multi-task multi-view clustering. IEEE Trans Knowl Data Eng 28(12):3324–3338CrossRefGoogle Scholar
  2. 2.
    Chaudhuri K, Kakade SM, Livescu K, Sridharan K (2009) Multi-view clustering via canonical correlation analysis. In: Proceedings of the 26th annual international conference on machine learning. ACM, pp 129–136Google Scholar
  3. 3.
    Kursun O, Alpaydin E (2010) Canonical correlation analysis for multiview semisupervised feature extraction. In: International conference on artificial intelligence and soft computing. Springer, pp 430–436Google Scholar
  4. 4.
    Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings of the eleventh annual conference on computational learning theory. ACM, pp 92–100Google Scholar
  5. 5.
    Liu J, Jiang Y, Li Z, Zhou Z-H, Lu H (2015) Partially shared latent factor learning with multiview data. IEEE Trans Neural Netw Learn Syst 26(6):1233–1246MathSciNetCrossRefGoogle Scholar
  6. 6.
    Brefeld U, Büscher C, Scheffer T (2005) Multi-view hidden Markov perceptrons. In: LWA, pp 134–138Google Scholar
  7. 7.
    Zhou Z-H, Li MJK, Systems I (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439CrossRefGoogle Scholar
  8. 8.
    Wang W, Zhou Z-H (2007) Analyzing co-training style algorithms. In: European conference on machine learning. Springer, pp 454–465Google Scholar
  9. 9.
    He J, Lawrence R (2011) A graphbased framework for multi-task multi-view learning. In: ICML, pp 25–32Google Scholar
  10. 10.
    Zhang J, Huan J (2012) Inductive multi-task learning with multiple view data. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 543–551Google Scholar
  11. 11.
    Qian B, Wang X, Davidson I (2012) A reconstruction error formulation for semi-supervised multi-task and multi-view learning. CoRR abs/1202.0Google Scholar
  12. 12.
    Jin X, Zhuang F, Wang S, He Q, Shi Z (2013) Shared structure learning for multiple tasks with multiple views. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 353–368Google Scholar
  13. 13.
    Jin X, Zhuang F, Xiong H, Du C, Luo P, He Q (2014) Multi-task multi-view learning for heterogeneous tasks. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management. ACM, pp 441–450Google Scholar
  14. 14.
    Hong Z, Mei X, Prokhorov D, Tao D (2013) Tracking via robust multi-task multi-view joint sparse representation. In: Proceedings of the IEEE international conference on computer vision, pp 649–656Google Scholar
  15. 15.
    Mei X, Hong Z, Prokhorov D, Tao D (2015) Robust multitask multiview tracking in videos. IEEE Trans Neural Netw Learn Syst 26(11):2874–2890MathSciNetCrossRefGoogle Scholar
  16. 16.
    Gong P, Ye J, Zhang C (2012) Robust multi-task feature learning. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 895–903Google Scholar
  17. 17.
    Gonçalves AR, Von Zuben FJ, Banerjee A (2016) Multi-task sparse structure learning with gaussian copula models. J Mach Learn Res 17(1):1205–1234MathSciNetzbMATHGoogle Scholar
  18. 18.
    Kong Y, Shao M, Li K, Fu Y (2018) Probabilistic low-rank multitask learning. IEEE Trans Neural Netw Learn Syst 29(3):670–680CrossRefGoogle Scholar
  19. 19.
    Xiao L, Stephen JM, Wilson TW et al (2019) A manifold regularized multi-task learning model for IQ prediction from two fMRI paradigms. IEEE Trans Biomed Eng. CrossRefGoogle Scholar
  20. 20.
    Ranjan R, Patel VM, Chellappa R (2019) HyperFace: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41(1):121–135. CrossRefGoogle Scholar
  21. 21.
    He Y, Zhang J, Shan H, Wang L (2019) Multi-task GANs for view-specific feature learning in gait recognition. IEEE Trans Inf Forensics Secur 14(1):102–113. CrossRefGoogle Scholar
  22. 22.
    Zhang C, Fu H, Hu Q et al (2019) Generalized latent multi-view subspace clustering. IEEE Trans Pattern Anal Mach Intell. CrossRefGoogle Scholar
  23. 23.
    Cheng M, Jing L, Ng MK (2019) Tensor-based low-dimensional representation learning for multi-view clustering. IEEE Trans Image Process 28(5):2399–2414MathSciNetCrossRefGoogle Scholar
  24. 24.
    Yin Q, Wu S, Wang L (2018) Multiview clustering via unified and view-specific embeddings learning. IEEE Trans Neural Netw Learn Syst 99:1–13MathSciNetGoogle Scholar
  25. 25.
    Zhang Z, Qin Z, Li P, Yang Q, Shao J (2018) Multi-view discriminative learning via joint non-negative matrix factorization. In: International conference on database systems for advanced applications. Springer, pp 542–557Google Scholar
  26. 26.
    Sun S, Chao G (2013) Multi-view maximum entropy discrimination. In: Twenty-third international joint conference on artificial intelligenceGoogle Scholar
  27. 27.
    Chao G, Sun S (2016) Alternative multiview maximum entropy discrimination. IEEE Trans Neural Netw Learn Syst 27(7):1445–1456MathSciNetCrossRefGoogle Scholar
  28. 28.
    Fang Y, Zhang H, Ye Y, Li X (2014) Detecting hot topics from Twitter: a multiview approach. J Inf Sci 40(5):578–593CrossRefGoogle Scholar
  29. 29.
    Cui Q, Wu S, Liu Q et al (2018) MV-RNN: a multi-view recurrent neural network for sequential recommendation. IEEE Trans Knowl Data Eng 30:1CrossRefGoogle Scholar
  30. 30.
    Wang Q, Chen M, Nie F, Li X (2019) Detecting coherent groups in crowd scenes by multiview clustering. IEEE Trans Pattern Anal Mach Intell. CrossRefGoogle Scholar
  31. 31.
    Ma C, Guo Y, Yang J, An W (2019) Learning multi-view representation with LSTM for 3-D shape recognition and retrieval. IEEE Trans Multimedia 21(5):1169–1182CrossRefGoogle Scholar
  32. 32.
    Huang F, Zhang X, Zhao Z, Li Z, He Y (2018) Deep multi-view representation learning for social images. Appl Soft Comput 73:106–118CrossRefGoogle Scholar
  33. 33.
    Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127MathSciNetCrossRefGoogle Scholar
  34. 34.
    Lu R, Zuo X, Liu J, Lian S (2018) Multi-view learning based on common and special features in multi-task scenarios. In: 2018 37th Chinese control conference (CCC), pp 9410–9415Google Scholar
  35. 35.
    Wu Y, Zhan D, Jiang Y (2018) DMTMV: a unified learning framework for deep multi-task multi-view learning. In: 2018 IEEE international conference on big knowledge (ICBK), pp 49–56Google Scholar
  36. 36.
    Sun G, Cong Y, Li J, Fu Y (2018) Robust lifelong multi-task multi-view representation learning. In: 2018 IEEE international conference on big knowledge (ICBK), pp 91–98Google Scholar
  37. 37.
    Zhang Q, Zhang L, Du B, Zheng W, Bian W, Tao D (2015) MMFE: multitask multiview feature embedding. In: 2015 IEEE international conference on data mining. IEEE, pp 1105–1110Google Scholar
  38. 38.
    Yan Y, Ricci E, Subramanian R, Lanz O, Sebe N (2013) No matter where you are: flexible graph-guided multi-task learning for multi-view head pose classification under target motion. In: Proceedings of the IEEE international conference on computer vision, pp 1177–1184Google Scholar
  39. 39.
    Yan Y, Subramanian R, Ricci E, Lanz O, Sebe N (2014) Evaluating multi-task learning for multi-view head-pose classification in interactive environments. In: 2014 22nd international conference on pattern recognition, pp 4182–4187Google Scholar
  40. 40.
    Kandemir M, Vetek A, Goenen M, Klami A, Kaski SJN (2014) Multi-task and multi-view learning of user state. Neurocomputing 139:97–106CrossRefGoogle Scholar
  41. 41.
    Javanmardi M, Qi X (2018) Robust structured multi-task multi-view sparse tracking. In: 2018 IEEE international conference on multimedia and expo (ICME), pp 1–6Google Scholar
  42. 42.
    Han H, Kang S, Yoo CD (2017) Multi-view visual speech recognition based on multi task learning. In: 2017 IEEE international conference on image processing (ICIP), pp 3983–3987Google Scholar
  43. 43.
    Li Q, Sun Z, He R (2017) Fast multi-view face alignment via multi-task auto-encoders. In: 2017 IEEE international joint conference on biometrics (IJCB), pp 538–545Google Scholar
  44. 44.
    Argyriou A, Evgeniou T, Pontil M (2007) Multi-task feature learning. In: Advances in neural information processing systems, pp 41–48Google Scholar
  45. 45.
    Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemo Intell Lab Syst 2(1–3):37–52CrossRefGoogle Scholar
  46. 46.
    Kingma DP, Welling M (2013) Auto-encoding variational bayes. CoRR abs/1312.6Google Scholar
  47. 47.
    Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680Google Scholar
  48. 48.
    Li X, Chandrasekaran SN, Huan J (2017) Lifelong multi-task multi-view learning using latent spaces. In: 2017 IEEE international conference on big data (Big Data), pp 37–46Google Scholar
  49. 49.
    Lu C-T, He L, Shao W, Cao B, Yu PS (2017) Multilinear factorization machines for multi-task multi-view learning. In: Proceedings of the tenth ACM international conference on web search and data mining. ACM, pp 701–709Google Scholar
  50. 50.
    Xu C, Tao D, Xu C (2013) A survey on multi-view learning. CoRR abs/1304.5Google Scholar
  51. 51.
    Chowdhury GG (2010) Introduction to modern information retrieval. Facet publishing, LondonGoogle Scholar
  52. 52.
    Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: Workshop track proceedings of 1st international conference on learning representations, {ICLR} 2013, Scottsdale, Arizona, USA, May 2–4, 2013Google Scholar
  53. 53.
    Zhang H, Ho JK, Wu QJ, Ye Y (2013) Multidimensional latent semantic analysis using term spatial information. IEEE Trans Cybern 43(6):1625–1640CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  • Run-kun Lu
    • 1
  • Jian-wei Liu
    • 1
    Email author
  • Si-ming Lian
    • 1
  • Xin Zuo
    • 1
  1. 1.Department of Automation, College of Information Science and EngineeringChina University of PetroleumBeijingChina

Personalised recommendations