Cross Domain User Engagement Evaluation

  • Ali Montazeralghaem
  • Hamed Zamani
  • Azadeh ShakeryEmail author
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9626)


Due to the applications of user engagements in recommender systems, predicting user engagement has recently attracted considerable attention. In this task which is firstly proposed in ACM Recommender Systems Challenge 2014, the posts containing users’ opinions about items (e.g., the tweets containing the users’ ratings about movies in the IMDb website) are studied. In this paper, we focus on user engagement evaluation for cold-start web applications in the extreme case, when there is no training data available for the target web application. We propose an adaptive model based on transfer learning (TL) technique to train on the data from a web application and test on another one. We study the problem of detecting tweets with positive engagement, which is a highly imbalanced classification problem. Therefore, we modify the loss function of the employed transfer learning method to cope with imbalanced data. We evaluate our method using a dataset including the tweets of four popular and diverse data sources, i.e., IMDb, YouTube, Goodreads, and Pandora. The experimental results show that in some cases transfer learning can transfer knowledge among domains to improve the user engagement evaluation performance. We further analyze the results to figure out when transfer learning can help to improve the performance.


User engagement Transfer learning Adaptive model Cold-start 



This work was supported in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.


  1. 1.
    Akbani, R., Kwek, S.S., Japkowicz, N.: Applying support vector machines to imbalanced datasets. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 39–50. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Loiacono, D., Lommatzsch, A., Turrin, R.: An Analysis of the 2014 RecSys challenge. In: RecSysChallenge, pp. 1–6 (2014)Google Scholar
  3. 3.
    Long, M., Wang, J., Ding, G., Pan, S.J., Yu, P.S.: Adaptation regularization: a general framework for transfer learning. IEEE Trans. Knowl. Data Eng. 26(5), 1076–1089 (2014)CrossRefGoogle Scholar
  4. 4.
    Petrovic, S., Osborne, M., Lavrenko, V.: RT to win! predicting message propagation in twitter. In: ICWSM, pp. 586–589 (2011)Google Scholar
  5. 5.
    Powers, D.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J. Mach. Learn. Tech. 2(1), 37–63 (2011)MathSciNetGoogle Scholar
  6. 6.
    Said, A., Dooms, S., Loni, B., Tikk, D.: Recommender systems challenge 2014. In: RecSys, pp. 387–388 (2014)Google Scholar
  7. 7.
    de Souza, J.G.C., Zamani, H., Negri, M., Turchi, M., Falavigna, D.: Multitask learning for adaptive quality estimation of automatically transcribed utterances. In: NAACL-HLT, pp. 714–724 (2015)Google Scholar
  8. 8.
    Uysal, I., Croft, W.B.: User oriented tweet ranking: a filtering approach to microblogs. In: CIKM, pp. 2261–2264 (2011)Google Scholar
  9. 9.
    Zamani, H., Moradi, P., Shakery, A.: Adaptive user engagement evaluation via multi-task learning. In: SIGIR, pp. 1011–1014 (2015)Google Scholar
  10. 10.
    Zamani, H., Shakery, A., Moradi, P.: Regression and learning to rank aggregation for user engagement evaluation. In: RecSysChallenge, pp. 29–34 (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Ali Montazeralghaem
    • 1
  • Hamed Zamani
    • 2
  • Azadeh Shakery
    • 1
    Email author
  1. 1.School of Electrical and Computer Engineering, College of EngineeringUniversity of TehranTehranIran
  2. 2.Center for Intelligent Information RetrievalUniversity of Massachusetts AmherstAmherstUSA

Personalised recommendations