Advertisement

Multiview Deep Learning

  • Shiliang SunEmail author
  • Liang Mao
  • Ziang Dong
  • Lidan Wu
Chapter

Abstract

The multiview deep learning described in this chapter deals with multiview data or simulates constructing its intrinsic structure by using deep learning methods. We highlight three major categories of multiview deep learning methods through three different thoughts. The first category of approaches focuses on obtaining a shared joint representation from different views by building a hierarchical structure. The second category of approaches focuses on constructing structured spaces with different representations of multiple views which gives some constraints between representations on a different view. The third major category approaches focuses on explicitly constructing connections or relationships between different views or representations, which allows different views to be translated or mapped to each other.

References

  1. Andrew G, Arora R, Bilmes J, Livescu K (2013) Deep canonical correlation analysis. In: Proceedings of the 30th international conference on machine learning, pp 1247–1255Google Scholar
  2. Andrews S, Hofmann T, Tsochantaridis I (2002) Multiple instance learning with generalized support vector machines. In: Proceedings of the 8th international association for the advancement of artificial intelligence, pp 943–944Google Scholar
  3. Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick CL, Parikh D (2015) Vqa: Visual question answering. In: Proceedings of the international conference on computer vision, pp 2425–2433Google Scholar
  4. AP SC, Lauly S, Larochelle H, Khapra M, Ravindran B, Raykar VC, Saha A (2014) An autoencoder approach to learning bilingual word representations. In: Advances in neural information processing systems, pp 1853–1861Google Scholar
  5. Ba J, Mnih V, Kavukcuoglu K (2015) Multiple object recognition with visual attention. In: Proceedings of the 3rd international conference on learning representationsGoogle Scholar
  6. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd international conference on learning representationsGoogle Scholar
  7. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166CrossRefGoogle Scholar
  8. Bruni E, Tran NK, Baroni M (2014) Multimodal distributional semantics. J Artif Intell Res 49(1):1–47MathSciNetCrossRefGoogle Scholar
  9. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:14061078
  10. Denton EL, Chintala S, Fergus R, et al (2015) Deep generative image models using a laplacian pyramid of adversarial networks. In: Advances in neural information processing systems, pp 1486–1494Google Scholar
  11. Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Darrell T, Saenko K (2015) Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the 28th IEEE conference on computer vision and pattern recognition, pp 2625–2634Google Scholar
  12. Dong J, Li X, Snoek CGM (2016) Word2visualvec: Cross-media retrieval by visual feature predictionGoogle Scholar
  13. Ehrlich M, Shields TJ, Almaev T, Amer MR (2016) Facial attributes classification using multi-task representation learning. In: Proceedings of the 29th IEEE conference on computer vision and pattern recognition workshops, pp 47–55Google Scholar
  14. Fang H, Gupta S, Iandola FN, Srivastava RK, Deng L, Dollar P, Gao J, He X, Mitchell M, Platt J, et al (2015) From captions to visual concepts and back. In: Proceedings of the 28th IEEE conference on computer vision and pattern recognition pp 1473–1482Google Scholar
  15. Feng F, Wang X, Li R (2014) Cross-modal retrieval with correspondence autoencoder. In: Proceedings of the 22nd ACM international conference on multimedia, ACM, pp 7–16Google Scholar
  16. Feng F, Wang X, Li R, Ahmad I (2015) Correspondence autoencoders for cross-modal retrieval. ACM Trans Multimed Comput Commun Appl (TOMM) 12(1s):26CrossRefGoogle Scholar
  17. Feng Y, Lapata M (2010) Visual information in semantic representation. In: The annual conference of the North American chapter of the association for computational linguistics, association for computational linguistics, pp 91–99Google Scholar
  18. Frome A, Corrado GS, Shlens J, Bengio S, Dean J, Mikolov T, et al (2013) DeViSE: a deep visual-semantic embedding model. In: Advances in neural information processing systems, pp 2121–2129Google Scholar
  19. Ge L, Gao J, Li X, Zhang A (2013) Multi-source deep learning for information trustworthiness estimation. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, ACM, pp 766–774Google Scholar
  20. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680Google Scholar
  21. Gregor K, Danihelka I, Graves A, Rezende DJ, Wierstra D (2015) Draw: a recurrent neural network for image generation. In: Proceedings of the 32nd international conference on machine learning, pp 1462–1471Google Scholar
  22. Hardoon DR, Szedmak S, Shawe-Taylor J (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16(12):2639–2664CrossRefGoogle Scholar
  23. Hinton GE (2009) Deep belief networks. Scholarpedia 4(5):5947CrossRefGoogle Scholar
  24. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507MathSciNetCrossRefGoogle Scholar
  25. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554MathSciNetCrossRefGoogle Scholar
  26. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRefGoogle Scholar
  27. Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3/4):321–377CrossRefGoogle Scholar
  28. Hu H, Liu B, Wang B, Liu M, Wang X (2013) Multimodal DBN for predicting high-quality answers in CQA portals. In: Proceedings of the 51st annual meeting of the association for computational linguistics, vol 2, pp 843–847Google Scholar
  29. Huang J, Kingsbury B (2013) Audio-visual deep learning for noise robust speech recognition. In: Proceedings of the 38th international conference on acoustics, speech, and signal processing, IEEE, pp 7596–7599Google Scholar
  30. Jiang YG, Wu Z, Wang J, Xue X, Chang SF (2018) Exploiting feature and class relationships in video categorization with regularized deep neural networks. IEEE Trans Pattern Anal Mach Intell 40(2):352–364CrossRefGoogle Scholar
  31. Karpathy A, Fei-Fei L (2015) Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the 28th IEEE conference on computer vision and pattern recognition, pp 3128–3137Google Scholar
  32. Karpathy A, Joulin A, Fei-Fei LF (2014) Deep fragment embeddings for bidirectional image sentence mapping. In: Advances in neural information processing systems, pp 1889–1897Google Scholar
  33. Kim Y, Lee H, Provost EM (2013) Deep learning for robust feature generation in audiovisual emotion recognition. In: Proceedings of the 38th international conference on acoustics, speech and signal processing, IEEE, pp 3687–3691Google Scholar
  34. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: Proceedings of the 2nd international conference on learning representationsGoogle Scholar
  35. Kiros R, Salakhutdinov R, Zemel R (2014a) Multimodal neural language models. In: Proceedings of the 31st international conference on machine learning, pp 595–603Google Scholar
  36. Kiros R, Salakhutdinov R, Zemel RS (2014b) Unifying visual-semantic embeddings with multimodal neural language models. arXiv:Learning
  37. Larochelle H, Bengio Y (2008) Classification using discriminative restricted Boltzmann machines. In: Proceedings of the 25th international conference on machine learning, ACM, pp 536–543Google Scholar
  38. Lazaridou A, Baroni M (2015) Combining language and vision with a multimodal skip-gram model. In: The annual conference of the north american chapter of the association for computational linguistics, pp 153–163Google Scholar
  39. Lu A, Wang W, Bansal M, Gimpel K, Livescu K (2015) Deep multilingual correlation for improved word embeddings. In: Proceedings of the North American chapter of the association for computational linguistics: human language technologies, pp 250–256Google Scholar
  40. Mansimov E, Parisotto E, Ba J, Salakhutdinov R (2016) Generating images from captions with attention. In: Proceedings of the 4th international conference on learning representationsGoogle Scholar
  41. Mao J, Xu W, Yang Y, Wang J, Huang Z, Yuille AL (2015) Deep captioning with multimodal recurrent neural networks (m-rnn). In: Proceedings of the 3rd international conference on learning representationsGoogle Scholar
  42. Mikolov T, Karafiát M, Burget L, Černockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: Proceedings of the 11th Annual conference of the international speech communication associationGoogle Scholar
  43. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:13013781
  44. Mnih V, Heess N, Graves A, et al (2014) Recurrent models of visual attention. In: Advances in neural information processing systems, pp 2204–2212Google Scholar
  45. Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY (2011) Multimodal deep learning. In: Proceedings of the 28th international conference on machine learning, pp 689–696Google Scholar
  46. Nojavanasghari B, Gopinath D, Koushik J, Baltrušaitis T, Morency LP (2016) Deep multimodal fusion for persuasiveness prediction. In: Proceedings of the 18th ACM international conference on multimodal interaction, ACM, pp 284–288Google Scholar
  47. Norouzi M, Mikolov T, Bengio S, Singer Y, Shlens J, Frome A, Corrado G, Dean J (2014) Zero-shot learning by convex combination of semantic embeddings. In: Proceedings of the 2nd international conference on learning representationsGoogle Scholar
  48. Ouyang W, Chu X, Wang X (2014) Multi-source deep learning for human pose estimation. In: Proceedings of the 27th IEEE conference on computer vision and pattern recognition, pp 2329–2336Google Scholar
  49. Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, Song X, Ward RK (2016) Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE/ACM Trans Audio Speech Lang Process 24(4):694–707CrossRefGoogle Scholar
  50. Pang L, Ngo CW (2015) Mutlimodal learning with deep boltzmann machine for emotion prediction in user generated videos. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, ACM, pp 619–622Google Scholar
  51. Reed S, Akata Z, Lee H, Schiele B (2016a) Learning deep representations of fine-grained visual descriptions. In: Proceedings of the 29th IEEE conference on computer vision and pattern recognition, pp 49–58Google Scholar
  52. Reed SE, Akata Z, Mohan S, Tenka S, Schiele B, Lee H (2016b) Learning what and where to draw. In: Advances in neural information processing systems, pp 217–225Google Scholar
  53. Reed SE, Akata Z, Yan X, Logeswaran L, Schiele B, Lee H (2016c) Generative adversarial text to image synthesis. In: Proceedings of the 33rd international conference on machine learning, pp 1060–1069Google Scholar
  54. Salakhutdinov R, Larochelle H (2010) Efficient learning of deep boltzmann machines. In: Proceedings of the 13th international conference on artificial intelligence and statistics, pp 693–700Google Scholar
  55. Silberer C, Lapata M (2014) Learning grounded meaning representations with autoencoders. In: Proceedings of the 52nd annual meeting of the association for computational linguistics, vol 1, pp 721–732Google Scholar
  56. Socher R, Karpathy A, Le QV, Manning CD, Ng AY (2014) Grounded compositional semantics for finding and describing images with sentences. Trans Assoc Comput Linguist 2(1):207–218CrossRefGoogle Scholar
  57. Sohn K, Shang W, Lee H (2014) Improved multimodal deep learning with variation of information. In: Advances in neural information processing systems, pp 2141–2149Google Scholar
  58. Song Y, Morency LP, Davis R (2012) Multi-view latent variable discriminative models for action recognition. In: Proceedings of the 25th IEEE conference on computer vision and pattern recognition, IEEE, pp 2120–2127Google Scholar
  59. Srivastava N, Salakhutdinov R (2012a) Learning representations for multimodal data with deep belief nets. In: Proceedings of the 25th IEEE conference on computer vision and pattern recognition workshops, vol 79Google Scholar
  60. Srivastava N, Salakhutdinov R (2012b) Multimodal learning with deep boltzmann machines. In: Advances in neural information processing systems, pp 2222–2230Google Scholar
  61. Suk HI, Lee SW, Shen D, Initiative ADN et al (2014) Hierarchical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. NeuroImage 101:569–582CrossRefGoogle Scholar
  62. Sun S, Liu Q (2018) Multi-view deep gaussian processes. In: Proceedings of the international conference on neural information processingGoogle Scholar
  63. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112Google Scholar
  64. Usunier N, Buffoni D, Gallinari P (2009) Ranking with ordered weighted pairwise classification. In: Proceedings of the 26th international conference on machine learning, ACM, pp 1057–1064Google Scholar
  65. Venugopalan S, Xu H, Donahue J, Rohrbach M, Mooney R, Saenko K (2014) Translating videos to natural language using deep recurrent neural networks. arXiv preprint arXiv:14124729
  66. Venugopalan S, Rohrbach M, Donahue J, Mooney R, Darrell T, Saenko K (2015) Sequence to sequence-video to text. In: Proceedings of the international conference on computer vision, pp 4534–4542Google Scholar
  67. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning, ACM, pp 1096–1103Google Scholar
  68. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11(Dec):3371–3408Google Scholar
  69. Wang D, Cui P, Ou M, Zhu W (2015a) Deep multimodal hashing with orthogonal regularization. In: Proceedings of the 24th international joint conference on artificial intelligence, vol 367, pp 2291–2297Google Scholar
  70. Wang W, Arora R, Livescu K, Bilmes J (2015b) On deep multi-view representation learning. In: Proceedings of the 32nd international conference on machine learning, pp 1083–1092Google Scholar
  71. Weston J, Bengio S, Usunier N (2010) Large scale image annotation: learning to rank with joint word-image embeddings. Mach Learn 81(1):21–35MathSciNetCrossRefGoogle Scholar
  72. Weston J, Bengio S, Usunier N (2011) WSABIE: Scaling up to large vocabulary image annotation. In: Proceedings of the 20th international joint conference on artificial intelligence, vol 11, pp 2764–2770Google Scholar
  73. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015a) Show, attend and tell: Neural image caption generation with visual attention. In: Proceedings of the 32nd international conference on machine learning, pp 2048–2057Google Scholar
  74. Xu R, Xiong C, Chen W, Corso JJ (2015b) Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In: Proceedings of the 29th international association for the advancement of artificial intelligence, vol 5, p 6Google Scholar
  75. Yan F, Mikolajczyk K (2015) Deep correlation for matching images and text. In: Proceedings of the 28th IEEE conference on computer vision and pattern recognition, pp 3441–3450Google Scholar
  76. Yih Wt, He X, Meek C (2014) Semantic parsing for single-relation question answering. In: Proceedings of the 52nd annual meeting of the association for computational linguistics, vol 2, pp 643–648Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyEast China Normal UniversityShanghaiChina

Personalised recommendations