Journal on Data Semantics

, Volume 5, Issue 2, pp 99–113 | Cite as

Content-Based Video Recommendation System Based on Stylistic Visual Features

  • Yashar DeldjooEmail author
  • Mehdi Elahi
  • Paolo Cremonesi
  • Franca Garzotto
  • Pietro Piazzolla
  • Massimo Quadrana
Original Article


This paper investigates the use of automatically extracted visual features of videos in the context of recommender systems and brings some novel contributions in the domain of video recommendations. We propose a new content-based recommender system that encompasses a technique to automatically analyze video contents and to extract a set of representative stylistic features (lighting, color, and motion) grounded on existing approaches of Applied Media Theory. The evaluation of the proposed recommendations, assessed w.r.t. relevance metrics (e.g., recall) and compared with existing content-based recommender systems that exploit explicit features such as movie genre, shows that our technique leads to more accurate recommendations. Our proposed technique achieves better results not only when visual features are extracted from full-length videos, but also when the feature extraction technique operates on movie trailers, pinpointing that our approach is effective also when full-length videos are not available or when there are performance requirements. Our recommender can be used in combination with more traditional content-based recommendation techniques that exploit explicit content features associated to video files, to improve the accuracy of recommendations. Our recommender can also be used alone, to address the problem originated from video files that have no meta-data, a typical situation of popular movie-sharing websites (e.g., YouTube) where every day hundred millions of hours of videos are uploaded by users and may contain no associated information. As they lack explicit content, these items cannot be considered for recommendation purposes by conventional content-based techniques even when they could be relevant for the user.


Recommender System Visual Feature Video Content Cosine Similarity Recommendation Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is supported by Telecom Italia S.p.A., Open Innovation Department, Joint Open Lab S-Cube, Milan.


  1. 1.
    Ricci F, Rokach L, Shapira B (2011) Introduction to recommender systems handbook. In: Ricci F, Rokach L, Shapira B, Kantor PB (eds) Recommender Systems Handbook. Springer, pp 1–35Google Scholar
  2. 2.
    Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749CrossRefGoogle Scholar
  3. 3.
    Burke R (2002) Hybrid recommender systems: Survey and experiments. User Model User Adapt Interact 12(4):331–370. /papers/burke-umuai-ip-2002.pdfGoogle Scholar
  4. 4.
    Su X, Khoshgoftaar TM (2009) A survey of collaborative filtering techniques. Adv Artif Intell 2009:4CrossRefGoogle Scholar
  5. 5.
    Balabanović M, Shoham Y (1997) Fab: Content-based, collaborative recommendation. Commun ACM 40(3):66–72CrossRefGoogle Scholar
  6. 6.
    Lops P, De Gemmis M, Semeraro G (2011) Content-based recommender systems: State of the art and trends. In: Recommender systems handbook. Springer, pp 73–105Google Scholar
  7. 7.
    Pazzani MJ, Billsus D (2007) The adaptive web. chap. Content-based Recommendation Systems, pp 325–341. Springer-Verlag, Berlin, Heidelberg.
  8. 8.
    Zettl H (2002) Essentials of applied media aesthetics. In: Dorai C, Venkatesh S (eds) Media Computing, The Springer International Series in Video Computing, vol 4. Springer, New York, pp 11–38Google Scholar
  9. 9.
    Cremonesi P, Koren Y, Turrin R (2010) Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26–30, 2010, pp 39–46Google Scholar
  10. 10.
    Deshpande M, Karypis G (2004) Item-based top-n recommendation algorithms. ACM Trans Inf Syst (TOIS) 22(1):143–177CrossRefGoogle Scholar
  11. 11.
    Yang B, Mei T, Hua XS, Yang L, Yang SQ, Li M (2007) Online video recommendation based on multimodal fusion and relevance feedback. In: Proceedings of the 6th ACM international conference on Image and video retrieval ACM, pp 73–80Google Scholar
  12. 12.
    Zhao X, Li G, Wang M, Yuan J, Zha ZJ, Li Z, Chua TS (2011) Integrating rich information for video recommendation with multi-task rank aggregation. In: Proceedings of the 19th ACM international conference on Multimedia ACM, pp. 1521–1524Google Scholar
  13. 13.
    Elahi M, Ricci F, Rubens N (2013) Active learning strategies for rating elicitation in collaborative filtering: a system-wide perspective. ACM Trans Intell Syst Technol (TIST) 5(1):13Google Scholar
  14. 14.
    Billsus D, Pazzani MJ (1999) A hybrid user model for news story classification. Springer, New YorkCrossRefGoogle Scholar
  15. 15.
    Kelly D, Teevan J (2003) Implicit feedback for inferring user preference: a bibliography. In: ACM SIGIR Forum ACM, vol 37, pp 18–28Google Scholar
  16. 16.
    Musto C, Narducci F, Lops P, Semeraro G, de Gemmis M, Barbieri M, Korst J, Pronk V, Clout R (2012) Enhanced semantic tv-show representation for personalized electronic program guides. In: User Modeling, Adaptation, and Personalization. Springer, pp 188–199Google Scholar
  17. 17.
    Degemmis M, Lops P, Semeraro G (2007) A content-collaborative recommender that exploits wordnet-based user profiles for neighborhood formation. User Model User Adapt Interact 17(3):217–255CrossRefGoogle Scholar
  18. 18.
    Eirinaki M, Vazirgiannis M, Varlamis I (2003) Sewep: using site semantics and a taxonomy to enhance the web personalization process. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining ACM, pp 99–108Google Scholar
  19. 19.
    Magnini B, Strapparava C (2001) Improving user modelling with content-based techniques. In: User Modeling 2001. Springer, pp 74–83Google Scholar
  20. 20.
    Mooney RJ, Roy L (2000) Content-based book recommending using learning for text categorization. In: Proceedings of the fifth ACM conference on Digital libraries ACM, pp 195–204Google Scholar
  21. 21.
    Ahn JW, Brusilovsky P, Grady J, He D, Syn SY (2007) Open user profiles for adaptive news systems: help or harm? In: Proceedings of the 16th international conference on World Wide Web ACM, pp 11–20Google Scholar
  22. 22.
    Billsus D, Pazzani MJ (2000) User modeling for adaptive news access. User Model User Adapt Interact 10(2–3):147–180CrossRefGoogle Scholar
  23. 23.
    Cantador I, Szomszor M, Alani H, Fernández M, Castells P (2008) Enriching ontological user profiles with tagging history for multi-domain recommendationsGoogle Scholar
  24. 24.
    Middleton SE, Shadbolt NR, De Roure DC (2004) Ontological user profiling in recommender systems. ACM Trans Inf Syst (TOIS) 22(1):54–88CrossRefGoogle Scholar
  25. 25.
    Deldjoo Y, Elahi M, Quadrana M, Cremonesi P (2015) Toward building a content-based video recommendation system based on low-level features. In: E-Commerce and Web Technologies. SpringerGoogle Scholar
  26. 26.
    Deldjoo Y, Elahi M, Quadrana M, Cremonesi P, Garzotto F (2015) Toward effective movie recommendations based on mise-en-scène film styles. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter ACM, pp 162–165Google Scholar
  27. 27.
    Bogdanov D, Herrera P (2011) How much metadata do we need in music recommendation? A subjective evaluation using preference sets. In: International Society for Music Information Retrieval Conference (ISMIR), Miami, Florida, USAGoogle Scholar
  28. 28.
    Bogdanov D, Serrà J, Wack N, Herrera P, Serra X (2011) Unifying low-level and high-level music similarity measures. IEEE Trans Multimed 13(4):687–701CrossRefGoogle Scholar
  29. 29.
    Knees P, Pohle T, Schedl M, Widmer G (2007) A music search engine built upon audio-based and web-based similarity measures. In: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval ACM, pp 447–454Google Scholar
  30. 30.
    Seyerlehner K, Schedl M, Pohle T, Knees P (2010) Using block-level features for genre classification, tag classification and music similarity estimation. Submission to Audio Music Similarity and Retrieval Task of MIREX 2010Google Scholar
  31. 31.
    Canini L, Benini S, Leonardi R (2013) Affective recommendation of movies based on selected connotative features. IEEE Trans Circuits Syst Video Technol 23(4):636–647CrossRefGoogle Scholar
  32. 32.
    Lehinevych T, Kokkinis-Ntrenis N, Siantikos G, Dogruöz AS, Giannakopoulos T, Konstantopoulos S (2014) Discovering similarities for content-based recommendation and browsing in multimedia collections. In: IEEE 2014 Tenth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), pp 237–243Google Scholar
  33. 33.
    Rubens N, Elahi M, Sugiyama M, Kaplan D (2015) Active learning in recommender systems. In: Recommender Systems Handbook. Springer, pp 809–846Google Scholar
  34. 34.
    Davidson J, Liebald B, Liu J, Nandy P, Van Vleet T, Gargi U, Gupta S, He Y, Lambert M, Livingston B et al (2010) The youtube video recommendation system. In: Proceedings of the fourth ACM conference on Recommender systems ACM, pp 293–296Google Scholar
  35. 35.
    Wang Y, Xing C, Zhou L (2006) Video semantic models: survey and evaluation. Int J Comput Sci Netw Secur 6:10–20Google Scholar
  36. 36.
    Hu W, Xie N, Li L, Zeng X, Maybank S (2011) A survey on visual content-based video indexing and retrieval. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):797–819CrossRefGoogle Scholar
  37. 37.
    Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: State of the art and challenges. ACM Trans Multimed Comput Commun Appl (TOMM) 2(1):1–19CrossRefGoogle Scholar
  38. 38.
    Rasheed Z, Sheikh Y, Shah M (2005) On the use of computable features for film classification. IEEE Trans Circuits Syst Video Technol 15(1):52–64CrossRefGoogle Scholar
  39. 39.
    Brezeale D, Cook DJ (2008) Automatic video classification: a survey of the literature. IEEE Trans Syst Man Cybern Part C Appl Rev 38(3):416–430CrossRefGoogle Scholar
  40. 40.
    Rasheed Z, Shah M (2003) Video categorization using semantics and semiotics. In: Video mining. Springer, pp 185–217Google Scholar
  41. 41.
    Zhou H, Hermans T, Karandikar AV, Rehg JM (2010) Movie genre classification via scene categorization. In: Proceedings of the international conference on Multimedia ACM, pp 747–750Google Scholar
  42. 42.
    Dorai C, Venkatesh S (2001) Computational media aesthetics: Finding meaning beautiful. IEEE Multimed 8(4):10–12CrossRefGoogle Scholar
  43. 43.
    Buckland W (2008) What does the statistical style analysis of film involve? A review of moving into pictures. more on film history, style, and analysis. Lit Linguist Comput 23(2):219–230MathSciNetCrossRefGoogle Scholar
  44. 44.
    Valdez P, Mehrabian A (1994) Effects of color on emotions. Journal of Experimental Psychology: General 123(4):394CrossRefGoogle Scholar
  45. 45.
    Wang HL, Cheong LF (2006) Affective understanding in film. IEEE Trans Circuits Syst Video Technol 16(6):689–704CrossRefGoogle Scholar
  46. 46.
    Choroś K (2009) Video shot selection and content-based scene detection for automatic classification of tv sports news. In: Tkacz E, Kapczynski A (eds) Internet Technical Development and Applications, Advances in Intelligent and Soft Computing, vol 64. Springer, Berlin Heidelberg, pp 73–80CrossRefGoogle Scholar
  47. 47.
    Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vis 12(1):43–77CrossRefGoogle Scholar
  48. 48.
    Horn BK, Schunck BG (1981) Determining optical flow. In: 1981 Technical Symposium East. International Society for Optics and Photonics, pp 319–331Google Scholar
  49. 49.
    Tkalcic M, Tasic JF (2003) Colour spaces: perceptual, historical and applicational background. In: EUROCON 2003. Computer as a Tool. The IEEE Region 8, vol 1, pp 304–308Google Scholar
  50. 50.
    Datasets–grouplens. Accessed: 2015-05-01
  51. 51.
    Youtube. Accessed: 2015-04-01
  52. 52.
    Kohavi R (1995) The power of decision tables. In: 8th European Conference on Machine Learning. Springer, pp 174–189Google Scholar
  53. 53.
    Kohavi R, Sommerfield D (1998) Targeting business users with decision table classifiers. In: KDD, pp 249–253Google Scholar
  54. 54.
    Freitas AA (2014) Comprehensible classification models: a position paper. ACM SIGKDD Explor Newsl 15(1):1–10CrossRefGoogle Scholar
  55. 55.
    Guyon I, Matic N, Vapnik V et al (1996) Discovering informative patterns and data cleaningGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Yashar Deldjoo
    • 1
    Email author
  • Mehdi Elahi
    • 1
  • Paolo Cremonesi
    • 1
  • Franca Garzotto
    • 1
  • Pietro Piazzolla
    • 1
  • Massimo Quadrana
    • 1
  1. 1.Politecnico di MilanoMilanItaly

Personalised recommendations