Probabilistic Topic Modeling, Reinforcement Learning, and Crowdsourcing for Personalized Recommendations

  • Evangelos Tripolitakis
  • Georgios ChalkiadakisEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10207)


We put forward an innovative use of probabilistic topic modeling (PTM) intertwined with reinforcement learning (RL), to provide personalized recommendations. Specifically, we model items under recommendation as mixtures of latent topics following a distribution with Dirichlet priors; this can be achieved via the exploitation of crowdsourced information for each item. Similarly, we model the user herself as an “evolving” document represented by its respective mixture of latent topics. The user’s topic distribution is appropriately updated each time she consumes an item. Recommendations are subsequently based on the divergence between the topic distributions of the user and available items. However, to tackle the exploration versus exploitation dilemma, we apply RL to vary the user’s topic distribution update rate. Our method is immune to the notorious “cold start” problem, and it can effectively cope with changing user preferences. Moreover, it is shown to be competitive against state-of-the-art algorithms, outperforming them in terms of sequential performance.


Recommender systems Applications of reinforcement learning Graphical models Crowdsourcing 


  1. 1.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  2. 2.
    Wang, C., Blei, D.M.: Collaborative topic modeling for recommending scientific articles. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 448–456. ACM (2011)Google Scholar
  3. 3.
    Babas, K., Chalkiadakis, G., Tripolitakis, E.: You are what you consume: a Bayesian method for personalized recommendations. In: Proceedings of the 7th ACM Conference on Recommender Systems, pp. 221–228. ACM (2013)Google Scholar
  4. 4.
    Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009). IEEECrossRefGoogle Scholar
  5. 5.
    Melville, P., Mooney, R.J., Nagarajan, R.: Content-boosted collaborative filtering for improved recommendations. In: AAAI/IAAI, pp. 187–192 (2002)Google Scholar
  6. 6.
    Mooney, R.J., Roy, L.: Content-based book recommending using learning for text categorization. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 195–204. ACM (2000)Google Scholar
  7. 7.
    Bowling, M., Veloso, M.: Rational and convergent learning in stochastic games. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence, vol. 2, pp. 1021–1026 (2001)Google Scholar
  8. 8.
    Bowling, M., Veloso, M.: Multiagent learning using a variable learning rate. Artif. Intell. 136(2), 215–250 (2002). ElsevierMathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)Google Scholar
  10. 10.
    Koren, Y.: The bellkor solution to the netflix grand prize. Netflix Prize Doc. 81, 1–10 (2009)Google Scholar
  11. 11.
    Piotte, M., Chabbert, M.: The pragmatic theory solution to the netflix grand prize. Netflix Prize Doc. (2009).
  12. 12.
    Toscher, A., Jahrer, M., Bell, R.M.: The bigchaos solution to the netflix grand prize. Netflix Prize Doc. (2009).
  13. 13.
    Langseth, H., Nielsen, T.D.: A latent model for collaborative filtering. Int. J. Approx. Reason. 53(4), 447–466 (2012)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Bresler, G., Chen, G.H., Shah, D.: A latent source model for online collaborative filtering. In: Advances in Neural Information Processing Systems, pp. 3347–3355 (2014)Google Scholar
  15. 15.
    Hofmann, T.: Latent semantic models for collaborative filtering. ACM Trans. Inf. Syst. (TOIS) 22(1), 89–115 (2004). ACMCrossRefGoogle Scholar
  16. 16.
    Hu, Y., Koren, Y., Volinsky, C.: Collaborative filtering for implicit feedback datasets. In: Eighth IEEE International Conference on Data Mining, ICDM 2008, pp. 263–272. IEEE (2008)Google Scholar
  17. 17.
    Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012). ACMCrossRefGoogle Scholar
  18. 18.
    Koren, Y., Bell, R.: Advances in collaborative filtering. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 145–186. Springer, Heidelberg (2011). doi: 10.1007/978-0-387-85820-3_5 CrossRefGoogle Scholar
  19. 19.
    Agarwal, D., Chen, B.-C.: fLDA: matrix factorization through latent dirichlet allocation. In: Proceedings of the Third ACM International Conference on Web Search and Data Mining, pp. 91–100. ACM (2010)Google Scholar
  20. 20.
    Ling, G., Lyu, M.R., King, I.: Ratings meet reviews, a combined approach to recommend. In: Proceedings of the 8th ACM Conference on Recommender Systems, pp. 105–112. ACM (2014)Google Scholar
  21. 21.
    Kurimo, M.: Indexing audio documents by using latent semantic analysis and SOM. Elsevier (1999)Google Scholar
  22. 22.
    Wallach, H.M., Murray, I., Salakhutdinov, R., Mimno, D.: Evaluation methods for topic models. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1105–1112. ACM (2009)Google Scholar
  23. 23.
    Hoffman, M., Bach, F.R., Blei, D.M.: Online learning for latent dirichlet allocation. In: Advances in Neural Information Processing Systems, pp. 856–864 (2010)Google Scholar
  24. 24.
    McCallum, A.K.: MALLET: A Machine Learning for Language Toolkit (2002).

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of Electrical and Computer EngineeringTechnical University of CreteChaniaGreece

Personalised recommendations