Cross Domain User Engagement Evaluation
Due to the applications of user engagements in recommender systems, predicting user engagement has recently attracted considerable attention. In this task which is firstly proposed in ACM Recommender Systems Challenge 2014, the posts containing users’ opinions about items (e.g., the tweets containing the users’ ratings about movies in the IMDb website) are studied. In this paper, we focus on user engagement evaluation for cold-start web applications in the extreme case, when there is no training data available for the target web application. We propose an adaptive model based on transfer learning (TL) technique to train on the data from a web application and test on another one. We study the problem of detecting tweets with positive engagement, which is a highly imbalanced classification problem. Therefore, we modify the loss function of the employed transfer learning method to cope with imbalanced data. We evaluate our method using a dataset including the tweets of four popular and diverse data sources, i.e., IMDb, YouTube, Goodreads, and Pandora. The experimental results show that in some cases transfer learning can transfer knowledge among domains to improve the user engagement evaluation performance. We further analyze the results to figure out when transfer learning can help to improve the performance.
KeywordsUser engagement Transfer learning Adaptive model Cold-start
This work was supported in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.
- 2.Loiacono, D., Lommatzsch, A., Turrin, R.: An Analysis of the 2014 RecSys challenge. In: RecSysChallenge, pp. 1–6 (2014)Google Scholar
- 4.Petrovic, S., Osborne, M., Lavrenko, V.: RT to win! predicting message propagation in twitter. In: ICWSM, pp. 586–589 (2011)Google Scholar
- 6.Said, A., Dooms, S., Loni, B., Tikk, D.: Recommender systems challenge 2014. In: RecSys, pp. 387–388 (2014)Google Scholar
- 7.de Souza, J.G.C., Zamani, H., Negri, M., Turchi, M., Falavigna, D.: Multitask learning for adaptive quality estimation of automatically transcribed utterances. In: NAACL-HLT, pp. 714–724 (2015)Google Scholar
- 8.Uysal, I., Croft, W.B.: User oriented tweet ranking: a filtering approach to microblogs. In: CIKM, pp. 2261–2264 (2011)Google Scholar
- 9.Zamani, H., Moradi, P., Shakery, A.: Adaptive user engagement evaluation via multi-task learning. In: SIGIR, pp. 1011–1014 (2015)Google Scholar
- 10.Zamani, H., Shakery, A., Moradi, P.: Regression and learning to rank aggregation for user engagement evaluation. In: RecSysChallenge, pp. 29–34 (2014)Google Scholar