How to Combine Visual Features with Tags to Improve Movie Recommendation Accuracy?

  • Yashar DeldjooEmail author
  • Mehdi Elahi
  • Paolo Cremonesi
  • Farshad Bakhshandegan Moghaddam
  • Andrea Luigi Edoardo Caielli
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 278)


Previous works have shown the effectiveness of using stylistic visual features, indicative of the movie style, in content-based movie recommendation. However, they have mainly focused on a particular recommendation scenario, i.e., when a new movie is added to the catalogue and no information is available for that movie (New Item scenario). However, the stylistic visual features can be also used when other sources of information is available (Existing Item scenario).

In this work, we address the second scenario and propose a hybrid technique that exploits not only the typical content available for the movies (e.g., tags), but also the stylistic visual content extracted form the movie files and fuse them by applying a fusion method called Canonical Correlation Analysis (CCA). Our experiments on a large catalogue of 13 K movies have shown very promising results which indicates a considerable improvement of the recommendation quality by using a proper fusion of the stylistic visual features with other type of features.


Visual Feature Canonical Correlation Analysis Fusion Method Mean Average Precision Multimedia Document 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Aggarwal, C.C.: Content-based recommender systems. In: Recommender Systems, pp. 139–166. Springer, Heidelberg (2016)Google Scholar
  2. 2.
    Aggarwal, C.C.: Recommender Systems: The Textbook. Springer, Heidelberg (2016)CrossRefGoogle Scholar
  3. 3.
    Bogdanov, D., Herrera, P.: How much metadata do we need in music recommendation? a subjective evaluation using preference sets. In: ISMIR, pp. 97–102 (2011)Google Scholar
  4. 4.
    Bogdanov, D., Serrà, J., Wack, N., Herrera, P., Serra, X.: Unifying low-level and high-level music similarity measures. IEEE Trans. Multimedia 13(4), 687–701 (2011)CrossRefGoogle Scholar
  5. 5.
    Brezeale, D., Cook, D.J.: Automatic video classification: A survey of the literature. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 38(3), 416–430 (2008)CrossRefGoogle Scholar
  6. 6.
    Cantador, I., Szomszor, M., Alani, H., Fernández, M., Castells, P.: Enriching ontological user profiles with tagging history for multi-domain recommendations (2008)Google Scholar
  7. 7.
    Celma, O.: Music recommendation. In: Music Recommendation and Discovery, pp. 43–85. Springer, Heidelberg (2010)Google Scholar
  8. 8.
    Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A.V., Turrin, R.: Looking for “Good” recommendations: a comparative evaluation of recommender systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6948, pp. 152–168. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23765-2_11 CrossRefGoogle Scholar
  9. 9.
    Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, pp. 39–46, 26–30 September 2010Google Scholar
  10. 10.
    de Gemmis, M., Lops, P., Musto, C., Narducci, F., Semeraro, G.: Semantics-aware content-based recommender systems. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 119–159. Springer, Heidelberg (2015)Google Scholar
  11. 11.
    Degemmis, M., Lops, P., Semeraro, G.: A content-collaborative recommender that exploits wordnet-based user profiles for neighborhood formation. User Model. User-Adap. Inter. 17(3), 217–255 (2007)CrossRefGoogle Scholar
  12. 12.
    Deldjoo, Y., Elahi, M., Cremonesi, P.: Using visual features and latent factors for movie recommendation. In: Workshop on New Trends in Content-Based Recommender Systems (CBRecSys), in Conjugation with ACM Recommender Systems Conference (RecSys) (2016)Google Scholar
  13. 13.
    Deldjoo, Y., Elahi, M., Cremonesi, P., Garzotto, F., Piazzolla, P.: Recommending movies based on mise-en-scene design. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1540–1547. ACM (2016)Google Scholar
  14. 14.
    Deldjoo, Y., Elahi, M., Cremonesi, P., Garzotto, F., Piazzolla, P., Quadrana, M.: Content-based video recommendation system based on stylistic visual features. J. Data Semant. 5(2), 99–113 (2016)CrossRefGoogle Scholar
  15. 15.
    Deldjoo, Y., Elahi, M., Quadrana, M., Cremonesi, P.: Toward building a content-based video recommendation system based on low-level features. In: Stuckenschmidt, H., Jannach, D. (eds.) EC-Web 2015. LNBIP, vol. 239, pp. 45–56. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-27729-5_4 CrossRefGoogle Scholar
  16. 16.
    Deldjoo, Y., Elahi, M., Quadrana, M., Cremonesi, P., Garzotto, F.: Toward effective movie recommendations based on mise-en-scène film styles. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter, pp. 162–165. ACM (2015)Google Scholar
  17. 17.
    Elahi, M., Ricci, F., Rubens, N.: A survey of active learning in collaborative filtering recommender systems. Comput. Sci. Rev. 20, 29–50 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Fernández-Tobías, I., Braunhofer, M., Elahi, M., Ricci, F., Cantador, I.: Alleviating the new user problem in collaborative filtering by exploiting personality information. User Model. User-Adap. Inter. 26(2), 221–255 (2016)CrossRefGoogle Scholar
  19. 19.
    Guy, I., Zwerdling, N., Ronen, I., Carmel, D., Uziel, E.: Social media recommendation based on people and tags. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 194–201. ACM (2010)Google Scholar
  20. 20.
    Haghighat, M., Abdel-Mottaleb, M., Alhalabi, W.: Fully automatic face normalization and single sample face recognition in unconstrained environments. Expert Syst. Appl. 47, 23–34 (2016)CrossRefGoogle Scholar
  21. 21.
    Hardoon, D.R., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16(12), 2639–2664 (2004)CrossRefzbMATHGoogle Scholar
  22. 22.
    Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. (TiiS) 5(4), 19 (2015)Google Scholar
  23. 23.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)CrossRefzbMATHGoogle Scholar
  24. 24.
    Hu, W., Xie, N., Li, L., Zeng, X., Maybank, S.: A survey on visual content-based video indexing and retrieval. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 41(6), 797–819 (2011)CrossRefGoogle Scholar
  25. 25.
    Li, X., Guo, L., Zhao, Y.E.: Tag-based social interest discovery. In: Proceedings of the 17th International Conference on World Wide Web, pp. 675–684. ACM (2008)Google Scholar
  26. 26.
    Mei, T., Yang, B., Hua, X.-S., Li, S.: Contextual video recommendation by multimodal relevance and user feedback. ACM Trans. Inf. Syst. (TOIS) 29(2), 10 (2011)CrossRefGoogle Scholar
  27. 27.
    Mooney, R.J., Roy, L.: Content-based book recommending using learning for text categorization. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 195–204. ACM (2000)Google Scholar
  28. 28.
    Nasery, M., Braunhofer, M., Ricci, F.: Recommendations with optimal combination of feature-based and item-based preferences. In: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, pp. 269–273. ACM (2016)Google Scholar
  29. 29.
    Nasery, M., Elahi, M., Cremonesi, P.: Polimovie: a feature-based dataset for recommender systems. In: ACM RecSys Workshop on Crowdsourcing and Human Computation for Recommender Systems (CrawdRec), vol. 3, pp. 25–30. ACM (2015)Google Scholar
  30. 30.
    Pereira, J.C., Coviello, E., Doyle, G., Rasiwasia, N., Lanckriet, G.R., Levy, R., Vasconcelos, N.: On the role of correlation and abstraction in cross-modal multimedia retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 521–535 (2014)CrossRefGoogle Scholar
  31. 31.
    Rasheed, Z., Shah, M.: Video categorization using semantics and semiotics. In: Rosenfeld, A., Doermann, D., DeMenthon, D. (eds.) Video Mining. The Springer International Series in Video Computing, vol. 6, pp. 185–217. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  32. 32.
    Rasheed, Z., Sheikh, Y., Shah, M.: On the use of computable features for film classification. IEEE Trans. Circuits Syst. Video Technol. 15(1), 52–64 (2005)CrossRefGoogle Scholar
  33. 33.
    Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G.R., Levy, R., Vasconcelos, N.: A new approach to cross-modal multimedia retrieval. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 251–260. ACM (2010)Google Scholar
  34. 34.
    Rubens, N., Elahi, M., Sugiyama, M., Kaplan, D.: Active learning in recommender systems. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 809–846. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  35. 35.
    Szomszor, M., Cattuto, C., Alani, H., O’Hara, K., Baldassarri, A., Loreto, V., Servedio, V.D.: Folksonomies, the semantic web, and movie recommendation (2007)Google Scholar
  36. 36.
    Yang, B., Mei, T., Hua, X.-S., Yang, L., Yang, S.-Q., Li, M.: Online video recommendation based on multimodal fusion and relevance feedback. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, pp. 73–80. ACM, (2007)Google Scholar
  37. 37.
    Zhou, H., Hermans, T., Karandikar, A.V., Rehg, J.M.: Movie genre classification via scene categorization. In: Proceedings of the International Conference on Multimedia, pp. 747–750. ACM (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yashar Deldjoo
    • 1
    Email author
  • Mehdi Elahi
    • 1
  • Paolo Cremonesi
    • 1
  • Farshad Bakhshandegan Moghaddam
    • 1
  • Andrea Luigi Edoardo Caielli
    • 1
  1. 1.Politecnico di MilanoMilanItaly

Personalised recommendations