A Deep Approach for Multi-modal User Attribute Modeling

  • Xiu Huang
  • Zihao Yang
  • Yang YangEmail author
  • Fumin Shen
  • Ning Xie
  • Heng Tao Shen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10538)


With the explosive growth of user-generated contents (e.g., texts, images and videos) on social networks, it is of great significance to analyze and extract people’s interests from the massive social media data, thus providing more accurate personalized recommendations and services. In this paper, we propose a novel multimodal deep learning algorithm for user profiling, dubbed multi-modal User Attribute Model (mmUAM), which explores the intrinsic semantic correlations across different modalities. Our proposed model is based on Poisson Gamma Belief Network (PGBN), which is a deep learning topic model for count data in documents. By improving PGBN, we succeed in addressing the problem of learning a shared representation between texts and images in order to obtain textual and visual attributes for users. To evaluate the effectiveness of our proposed method, we collect a real dataset from Sina Weibo. Experimental results demonstrate that the proposed algorithm achieves encouraging performance compared with several state-of-the-art methods.


User profiling Deep learning Multi-model Social media 



This work was supported in part by the National Natural Science Foundation of China under Project 61572108 and Project 61502081, the National Thousand-Young-Talents Program of China, and the Fundamental Research Funds for the Central Universities under Project ZYGX2014Z007 and Project ZYGX2015J055.


  1. 1.
    Abel, F., Gao, Q., Houben, G.-J., Tao, K.: Semantic enrichment of Twitter posts for user profile construction on the social web. In: Antoniou, G., Grobelnik, M., Simperl, E., Parsia, B., Plexousakis, D., Leenheer, P., Pan, J. (eds.) ESWC 2011. LNCS, vol. 6644, pp. 375–389. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21064-8_26 CrossRefGoogle Scholar
  2. 2.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. JMLR 3, 1107–1135 (2003)zbMATHGoogle Scholar
  3. 3.
    Bian, J., Yang, Y., Chua, T.S.: Multimedia summarization for trending topics in microblogs. In: ACM CIKM, pp. 1807–1812 (2013)Google Scholar
  4. 4.
    Bian, J., Yang, Y., Chua, T.S.: Predicting trending messages and diffusion participants in microblogging network. In: ACM SIGIR, pp. 537–546 (2014)Google Scholar
  5. 5.
    Bian, J., Yang, Y., Zhang, H., Chua, T.S.: Multimedia summarization for social events in microblog stream. IEEE Trans. Multimedia 17(2), 216–228 (2015)CrossRefGoogle Scholar
  6. 6.
    Blei, D.M., Jordan, M.I.: Modeling annotated data. In: ACM SIGIR, pp. 127–134 (2003)Google Scholar
  7. 7.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. JMLR 3, 993–1022 (2003)zbMATHGoogle Scholar
  8. 8.
    Cheng, X., Yan, X., Lan, Y., Guo, J.: BTM: topic modeling over short texts. IEEE TKDE 26(12), 2928–2941 (2014)Google Scholar
  9. 9.
    Geng, X., Zhang, H., Song, Z., Yang, Y., Luan, H., Chua, T.S.: One of a kind: User profiling by social curation. In: ACM MM, pp. 567–576 (2014)Google Scholar
  10. 10.
    He, W., Liu, H., He, J., Tang, S., Du, X.: Extracting interest tags for non-famous users in social network. In: ACM CIKM, pp. 861–870 (2015)Google Scholar
  11. 11.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Huang, X., Yang, Y., Hu, Y., Shen, F., Shao, J.: Dynamic user attribute discovery on social media. In: Li, F., Shim, K., Zheng, K., Liu, G. (eds.) APWeb 2016. LNCS, vol. 9931, pp. 256–267. Springer, Cham (2016). doi: 10.1007/978-3-319-45814-4_21 CrossRefGoogle Scholar
  13. 13.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  14. 14.
    Li, X., Cheung, M., She, J.: Connection discovery using shared images by gaussian relational topic model. arxiv:1612.03639 (2016)
  15. 15.
    Liu, S., Cui, P., Zhu, W., Yang, S.: Learning socially embedded visual representation from scratch. In: ACM MM, pp. 109–118 (2015)Google Scholar
  16. 16.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: ICCV, vol. 2, pp. 1150–1157 (1999)Google Scholar
  17. 17.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: ICML, pp. 689–696 (2011)Google Scholar
  18. 18.
    Pang, L., Ngo, C.W.: Mutlimodal learning with deep Boltzmann machine for emotion prediction in user generated videos. In: ICMR, pp. 619–622 (2015)Google Scholar
  19. 19.
    Rosen-Zvi, M., Griffiths, T., Steyvers, M., Smyth, P.: The author-topic model for authors and documents. In: UAI, pp. 487–494 (2004)Google Scholar
  20. 20.
    Salakhutdinov, R., Hinton, G.E.: Deep boltzmann machines. In: AISTATS, vol. 1, p. 3 (2009)Google Scholar
  21. 21.
    Schapire, R.E., Singer, Y.: Boostexter: a boosting-based system for text categorization. Mach. Learn. 39(2–3), 135–168 (2000)CrossRefzbMATHGoogle Scholar
  22. 22.
    Srivastava, N., Salakhutdinov, R.: Learning representations for multimodal data with deep belief nets. In: ICML Workshop (2012)Google Scholar
  23. 23.
    Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep Boltzmann machines. In: NIPS, pp. 2222–2230 (2012)Google Scholar
  24. 24.
    Xu, Z., Ru, L., Xiang, L., Yang, Q.: Discovering user interest on twitter with a modified author-topic model. In: IEEE/WIC/ACM WI-IAT, vol. 1, pp. 422–429 (2011)Google Scholar
  25. 25.
    Yang, X., Li, Y., Luo, J.: Pinterest board recommendation for twitter users. In: ACM MM, pp. 963–966 (2015)Google Scholar
  26. 26.
    Yang, Y., Zha, Z.J., Gao, Y., Zhu, X., Chua, T.S.: Exploiting web images for semantic video indexing via robust sample-specific loss. IEEE Trans. Multimedia 16(6), 1677–1689 (2014)CrossRefGoogle Scholar
  27. 27.
    Zhao, Y., Liang, S., Ren, Z., Ma, J., Yilmaz, E., de Rijke, M.: Explainable user clustering in short text streams. In: ACM SIGIR, pp. 155–164 (2016)Google Scholar
  28. 28.
    Zhou, M., Carin, L.: Negative binomial process count and mixture modeling. IEEE T PAMI 37(2), 307–320 (2015)CrossRefGoogle Scholar
  29. 29.
    Zhou, M., Cong, Y., Chen, B.: The poisson gamma belief network. In: NIPS, pp. 3043–3051 (2015)Google Scholar
  30. 30.
    Zhou, M., Cong, Y., Chen, B.: Augmentable gamma belief networks. JMLR 17(163), 1–44 (2016)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Zhou, M., Hannah, L., Dunson, D.B., Carin, L.: Beta-negative binomial process and poisson factor analysis. In: AISTATS, vol. 22, pp. 1462–1471 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Xiu Huang
    • 1
  • Zihao Yang
    • 1
  • Yang Yang
    • 1
    Email author
  • Fumin Shen
    • 1
  • Ning Xie
    • 1
  • Heng Tao Shen
    • 1
  1. 1.School of Computer Science and Technology and Center for Future MediaUniversity of Electronic Science and Technology of ChinaChengduChina

Personalised recommendations