TCGM: An Information-Theoretic Framework for Semi-supervised Multi-modality Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12348)


Fusing data from multiple modalities provides more information to train machine learning systems. However, it is prohibitively expensive and time-consuming to label each modality with a large amount of data, which leads to a crucial problem of semi-supervised multi-modal learning. Existing methods suffer from either ineffective fusion across modalities or lack of theoretical guarantees under proper assumptions. In this paper, we propose a novel information-theoretic approach - namely, Total Correlation Gain Maximization (TCGM) – for semi-supervised multi-modal learning, which is endowed with promising properties: (i) it can utilize effectively the information across different modalities of unlabeled data points to facilitate training classifiers of each modality (ii) it has theoretical guarantee to identify Bayesian classifiers, i.e., the ground truth posteriors of all modalities. Specifically, by maximizing TC-induced loss (namely TC gain) over classifiers of all modalities, these classifiers can cooperatively discover the equivalent class of ground-truth classifiers; and identify the unique ones by leveraging limited percentage of labeled data. We apply our method to various tasks and achieve state-of-the-art results, including the news classification (Newsgroup dataset), emotion recognition (IEMOCAP and MOSI datasets), and disease prediction (Alzheimer’s Disease Neuroimaging Initiative dataset).


Total Correlation Semi-supervised Multi-modality Conditional independence Information intersection 



Yizhou Wang’s work is supported by MOST-2018AAA0102004, NSFC-61625201, DFG TRR169/NSFC Major International Collaboration Project “Crossmodal Learning”. Yuqing Kong’s work is supported by Science and Technology Innovation 2030 “The New Generation of Artificial Intelligence” Major Project No. 2018AAA0100901, China. Thanks to Xinwei Sun’s girlfriend Yue Cao, for all her love and support.

Supplementary material (2.2 mb)
Supplementary material 1 (zip 2211 KB)


  1. 1.
    Achille, A., Soatto, S.: Emergence of invariance and disentanglement in deep representations. J. Mach. Learn. Res. 19(1), 1947–1980 (2018)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Ashburner, J.: A fast diffeomorphic image registration algorithm. Neuroimage 38(1), 95–113 (2007)CrossRefGoogle Scholar
  3. 3.
    Balcan, M.F., Blum, A., Yang, K.: Co-training and expansion: towards bridging theory and practice. In: Advances in Neural Information Processing Systems, pp. 89–96 (2005)Google Scholar
  4. 4.
    Belghazi, M.I., et al.: MINE: mutual information neural estimation. arXiv preprint arXiv:1801.04062 (2018)
  5. 5.
    Bisson, G., Grimal, C.: Co-clustering of multi-view datasets: a parallelizable approach. In: 2012 IEEE 12th International Conference on Data Mining, pp. 828–833. IEEE (2012)Google Scholar
  6. 6.
    Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pp. 92–100. ACM (1998)Google Scholar
  7. 7.
    Busso, C., et al.: IEMOCAP: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42, 335–359 (2008)CrossRefGoogle Scholar
  8. 8.
    Cao, P., Xu, Y., Kong, Y., Wang, Y.: Max-MIG: an information theoretic approach for joint learning from crowds. In: ICLR 2019 (2018)Google Scholar
  9. 9.
    Chandar, S., Khapra, M.M., Larochelle, H., Ravindran, B.: Correlational neural networks. Neural Comput. 28(2), 257–285 (2016)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Chang, X., Xiang, T., Hospedales, T.M.: Scalable and effective deep CCA via soft decorrelation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1488–1497 (2018)Google Scholar
  11. 11.
    Cheng, Y., Zhao, X., Huang, K., Tan, T.: Semi-supervised learning and feature evaluation for RGB-D object recognition. Comput. Vis. Image Underst. 139, 149–160 (2015)CrossRefGoogle Scholar
  12. 12.
    Christoudias, C.M., Saenko, K., Morency, L.P., Darrell, T.: Co-adaptation of audio-visual speech and gesture classifiers. In: Proceedings of the 8th International Conference on Multimodal Interfaces, pp. 84–91. ACM (2006)Google Scholar
  13. 13.
    Dasgupta, S., Littman, M.L., McAllester, D.A.: PAC generalization bounds for co-training. In: Advances in Neural Information Processing Systems, pp. 375–382 (2002)Google Scholar
  14. 14.
    Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2018)
  15. 15.
    Huang, S.L., Suh, C., Zheng, L.: Euclidean information theory of networks. IEEE Trans. Inf. Theory 61(12), 6795–6814 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Hussain, S.F., Grimal, C., Bisson, G.: An improved co-similarity measure for document clustering. In: International Conference on Machine Learning and Applications (2010)Google Scholar
  17. 17.
    Jones, R.: Learning to extract entities from labeled and unlabeled text. Ph.D. thesis, Citeseer (2005)Google Scholar
  18. 18.
    Kim, D.H., Lee, M.K., Choi, D.Y., Song, B.C.: Multi-modal emotion recognition using semi-supervised learning and multiple neural networks in the wild. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 529–535. ACM (2017)Google Scholar
  19. 19.
    Kong, Y., Schoenebeck, G.: Water from two rocks: maximizing the mutual information. In: Proceedings of the 2018 ACM Conference on Economics and Computation, pp. 177–194. ACM (2018)Google Scholar
  20. 20.
    Leskes, B.: The value of agreement, a new boosting algorithm. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS (LNAI), vol. 3559, pp. 95–110. Springer, Heidelberg (2005). Scholar
  21. 21.
    Levin, A., Viola, P.A., Freund, Y.: Unsupervised improvement of visual detectors using co-training. In: ICCV, vol. 1, p. 2 (2003)Google Scholar
  22. 22.
    Lewis, D.D.: Naive (Bayes) at forty: the independence assumption in information retrieval. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 4–15. Springer, Heidelberg (1998). Scholar
  23. 23.
    Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., Morency, L.P.: Efficient low-rank multimodal fusion with modality-specific factors. In: ACL (2018)Google Scholar
  24. 24.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: NIPS (2001)Google Scholar
  25. 25.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 689–696 (2011)Google Scholar
  26. 26.
    Nguyen, X., Wainwright, M.J., Jordan, M.I.: Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 56(11), 5847–5861 (2010)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Sohn, K., Shang, W., Lee, H.: Improved multimodal deep learning with variation of information. In: Advances in Neural Information Processing Systems, pp. 2141–2149 (2014)Google Scholar
  28. 28.
    Studenỳ, M., Vejnarová, J.: The multiinformation function as a tool for measuring stochastic dependence. In: Jordan, M.I. (ed.) Learning in Graphical Models. ASID, vol. 89, pp. 261–297. Springer, Dordrecht (1998). Scholar
  29. 29.
    Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1979–1993 (2018)CrossRefGoogle Scholar
  30. 30.
    Veličković, P., Fedus, W., Hamilton, W.L., Liò, P., Bengio, Y., Hjelm, R.D.: Deep graph infomax. arXiv preprint arXiv:1809.10341 (2018)
  31. 31.
    Wang, L., et al.: An efficient approach to informative feature extraction from multimodal data. arXiv preprint arXiv:1811.08979 (2018)
  32. 32.
    Xu, Y., Zhao, S., Song, J., Stewart, R.J., Ermon, S.: A theory of usable information under computational constraints. ArXiv abs/2002.10689 (2020)Google Scholar
  33. 33.
    Yang, Y., Zhan, D.C., Sheng, X.R., Jiang, Y.: Semi-supervised multi-modal learning with incomplete modalities. In: IJCAI, pp. 2998–3004 (2018)Google Scholar
  34. 34.
    Zadeh, A., Zellers, R., Pincus, E., Morency, L.P.: MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. ArXiv abs/1606.06259 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Microsoft Research-AsiaBeijingChina
  2. 2.Center on Frontiers of Computing Studies, Advanced Institute of Information Technology, Department of Computer SciencePeking UniversityBeijingChina
  3. 3.Yanjing Medical CollegeCapital Medical UniversityBeijingChina
  4. 4.UC BerkeleyBerkeleyUSA
  5. 5.Deepwise AI LabBeijingChina

Personalised recommendations