Skip to main content
SpringerLink
Log in

Content-Based Video Recommendation System Based on Stylistic Visual Features

  • Original Article
  • Published:
Journal on Data Semantics

Abstract

This paper investigates the use of automatically extracted visual features of videos in the context of recommender systems and brings some novel contributions in the domain of video recommendations. We propose a new content-based recommender system that encompasses a technique to automatically analyze video contents and to extract a set of representative stylistic features (lighting, color, and motion) grounded on existing approaches of Applied Media Theory. The evaluation of the proposed recommendations, assessed w.r.t. relevance metrics (e.g., recall) and compared with existing content-based recommender systems that exploit explicit features such as movie genre, shows that our technique leads to more accurate recommendations. Our proposed technique achieves better results not only when visual features are extracted from full-length videos, but also when the feature extraction technique operates on movie trailers, pinpointing that our approach is effective also when full-length videos are not available or when there are performance requirements. Our recommender can be used in combination with more traditional content-based recommendation techniques that exploit explicit content features associated to video files, to improve the accuracy of recommendations. Our recommender can also be used alone, to address the problem originated from video files that have no meta-data, a typical situation of popular movie-sharing websites (e.g., YouTube) where every day hundred millions of hours of videos are uploaded by users and may contain no associated information. As they lack explicit content, these items cannot be considered for recommendation purposes by conventional content-based techniques even when they could be relevant for the user.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://recsys.deib.polimi.it/.

  2. http://www.mathworks.com/products/matlab.

  3. https://www.r-project.org.

  4. http://www.cs.waikato.ac.nz/ml/weka.

References

  1. Ricci F, Rokach L, Shapira B (2011) Introduction to recommender systems handbook. In: Ricci F, Rokach L, Shapira B, Kantor PB (eds) Recommender Systems Handbook. Springer, pp 1–35

  2. Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749

    Article  Google Scholar 

  3. Burke R (2002) Hybrid recommender systems: Survey and experiments. User Model User Adapt Interact 12(4):331–370. /papers/burke-umuai-ip-2002.pdf

  4. Su X, Khoshgoftaar TM (2009) A survey of collaborative filtering techniques. Adv Artif Intell 2009:4

    Article  Google Scholar 

  5. Balabanović M, Shoham Y (1997) Fab: Content-based, collaborative recommendation. Commun ACM 40(3):66–72

    Article  Google Scholar 

  6. Lops P, De Gemmis M, Semeraro G (2011) Content-based recommender systems: State of the art and trends. In: Recommender systems handbook. Springer, pp 73–105

  7. Pazzani MJ, Billsus D (2007) The adaptive web. chap. Content-based Recommendation Systems, pp 325–341. Springer-Verlag, Berlin, Heidelberg. http://dl.acm.org/citation.cfm?id=1768197.1768209

  8. Zettl H (2002) Essentials of applied media aesthetics. In: Dorai C, Venkatesh S (eds) Media Computing, The Springer International Series in Video Computing, vol 4. Springer, New York, pp 11–38

    Google Scholar 

  9. Cremonesi P, Koren Y, Turrin R (2010) Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26–30, 2010, pp 39–46

  10. Deshpande M, Karypis G (2004) Item-based top-n recommendation algorithms. ACM Trans Inf Syst (TOIS) 22(1):143–177

    Article  Google Scholar 

  11. Yang B, Mei T, Hua XS, Yang L, Yang SQ, Li M (2007) Online video recommendation based on multimodal fusion and relevance feedback. In: Proceedings of the 6th ACM international conference on Image and video retrieval ACM, pp 73–80

  12. Zhao X, Li G, Wang M, Yuan J, Zha ZJ, Li Z, Chua TS (2011) Integrating rich information for video recommendation with multi-task rank aggregation. In: Proceedings of the 19th ACM international conference on Multimedia ACM, pp. 1521–1524

  13. Elahi M, Ricci F, Rubens N (2013) Active learning strategies for rating elicitation in collaborative filtering: a system-wide perspective. ACM Trans Intell Syst Technol (TIST) 5(1):13

    Google Scholar 

  14. Billsus D, Pazzani MJ (1999) A hybrid user model for news story classification. Springer, New York

    Book  Google Scholar 

  15. Kelly D, Teevan J (2003) Implicit feedback for inferring user preference: a bibliography. In: ACM SIGIR Forum ACM, vol 37, pp 18–28

  16. Musto C, Narducci F, Lops P, Semeraro G, de Gemmis M, Barbieri M, Korst J, Pronk V, Clout R (2012) Enhanced semantic tv-show representation for personalized electronic program guides. In: User Modeling, Adaptation, and Personalization. Springer, pp 188–199

  17. Degemmis M, Lops P, Semeraro G (2007) A content-collaborative recommender that exploits wordnet-based user profiles for neighborhood formation. User Model User Adapt Interact 17(3):217–255

    Article  Google Scholar 

  18. Eirinaki M, Vazirgiannis M, Varlamis I (2003) Sewep: using site semantics and a taxonomy to enhance the web personalization process. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining ACM, pp 99–108

  19. Magnini B, Strapparava C (2001) Improving user modelling with content-based techniques. In: User Modeling 2001. Springer, pp 74–83

  20. Mooney RJ, Roy L (2000) Content-based book recommending using learning for text categorization. In: Proceedings of the fifth ACM conference on Digital libraries ACM, pp 195–204

  21. Ahn JW, Brusilovsky P, Grady J, He D, Syn SY (2007) Open user profiles for adaptive news systems: help or harm? In: Proceedings of the 16th international conference on World Wide Web ACM, pp 11–20

  22. Billsus D, Pazzani MJ (2000) User modeling for adaptive news access. User Model User Adapt Interact 10(2–3):147–180

    Article  Google Scholar 

  23. Cantador I, Szomszor M, Alani H, Fernández M, Castells P (2008) Enriching ontological user profiles with tagging history for multi-domain recommendations

  24. Middleton SE, Shadbolt NR, De Roure DC (2004) Ontological user profiling in recommender systems. ACM Trans Inf Syst (TOIS) 22(1):54–88

    Article  Google Scholar 

  25. Deldjoo Y, Elahi M, Quadrana M, Cremonesi P (2015) Toward building a content-based video recommendation system based on low-level features. In: E-Commerce and Web Technologies. Springer

  26. Deldjoo Y, Elahi M, Quadrana M, Cremonesi P, Garzotto F (2015) Toward effective movie recommendations based on mise-en-scène film styles. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter ACM, pp 162–165

  27. Bogdanov D, Herrera P (2011) How much metadata do we need in music recommendation? A subjective evaluation using preference sets. In: International Society for Music Information Retrieval Conference (ISMIR), Miami, Florida, USA

  28. Bogdanov D, Serrà J, Wack N, Herrera P, Serra X (2011) Unifying low-level and high-level music similarity measures. IEEE Trans Multimed 13(4):687–701

    Article  Google Scholar 

  29. Knees P, Pohle T, Schedl M, Widmer G (2007) A music search engine built upon audio-based and web-based similarity measures. In: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval ACM, pp 447–454

  30. Seyerlehner K, Schedl M, Pohle T, Knees P (2010) Using block-level features for genre classification, tag classification and music similarity estimation. Submission to Audio Music Similarity and Retrieval Task of MIREX 2010

  31. Canini L, Benini S, Leonardi R (2013) Affective recommendation of movies based on selected connotative features. IEEE Trans Circuits Syst Video Technol 23(4):636–647

    Article  Google Scholar 

  32. Lehinevych T, Kokkinis-Ntrenis N, Siantikos G, Dogruöz AS, Giannakopoulos T, Konstantopoulos S (2014) Discovering similarities for content-based recommendation and browsing in multimedia collections. In: IEEE 2014 Tenth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), pp 237–243

  33. Rubens N, Elahi M, Sugiyama M, Kaplan D (2015) Active learning in recommender systems. In: Recommender Systems Handbook. Springer, pp 809–846

  34. Davidson J, Liebald B, Liu J, Nandy P, Van Vleet T, Gargi U, Gupta S, He Y, Lambert M, Livingston B et al (2010) The youtube video recommendation system. In: Proceedings of the fourth ACM conference on Recommender systems ACM, pp 293–296

  35. Wang Y, Xing C, Zhou L (2006) Video semantic models: survey and evaluation. Int J Comput Sci Netw Secur 6:10–20

    Google Scholar 

  36. Hu W, Xie N, Li L, Zeng X, Maybank S (2011) A survey on visual content-based video indexing and retrieval. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):797–819

    Article  Google Scholar 

  37. Lew MS, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: State of the art and challenges. ACM Trans Multimed Comput Commun Appl (TOMM) 2(1):1–19

    Article  Google Scholar 

  38. Rasheed Z, Sheikh Y, Shah M (2005) On the use of computable features for film classification. IEEE Trans Circuits Syst Video Technol 15(1):52–64

    Article  Google Scholar 

  39. Brezeale D, Cook DJ (2008) Automatic video classification: a survey of the literature. IEEE Trans Syst Man Cybern Part C Appl Rev 38(3):416–430

    Article  Google Scholar 

  40. Rasheed Z, Shah M (2003) Video categorization using semantics and semiotics. In: Video mining. Springer, pp 185–217

  41. Zhou H, Hermans T, Karandikar AV, Rehg JM (2010) Movie genre classification via scene categorization. In: Proceedings of the international conference on Multimedia ACM, pp 747–750

  42. Dorai C, Venkatesh S (2001) Computational media aesthetics: Finding meaning beautiful. IEEE Multimed 8(4):10–12

    Article  Google Scholar 

  43. Buckland W (2008) What does the statistical style analysis of film involve? A review of moving into pictures. more on film history, style, and analysis. Lit Linguist Comput 23(2):219–230

    Article  MathSciNet  Google Scholar 

  44. Valdez P, Mehrabian A (1994) Effects of color on emotions. Journal of Experimental Psychology: General 123(4):394

    Article  Google Scholar 

  45. Wang HL, Cheong LF (2006) Affective understanding in film. IEEE Trans Circuits Syst Video Technol 16(6):689–704

    Article  Google Scholar 

  46. Choroś K (2009) Video shot selection and content-based scene detection for automatic classification of tv sports news. In: Tkacz E, Kapczynski A (eds) Internet Technical Development and Applications, Advances in Intelligent and Soft Computing, vol 64. Springer, Berlin Heidelberg, pp 73–80

    Chapter  Google Scholar 

  47. Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vis 12(1):43–77

    Article  Google Scholar 

  48. Horn BK, Schunck BG (1981) Determining optical flow. In: 1981 Technical Symposium East. International Society for Optics and Photonics, pp 319–331

  49. Tkalcic M, Tasic JF (2003) Colour spaces: perceptual, historical and applicational background. In: EUROCON 2003. Computer as a Tool. The IEEE Region 8, vol 1, pp 304–308

  50. Datasets–grouplens. http://grouplens.org/datasets/. Accessed: 2015-05-01

  51. Youtube. http://www.youtube.com. Accessed: 2015-04-01

  52. Kohavi R (1995) The power of decision tables. In: 8th European Conference on Machine Learning. Springer, pp 174–189

  53. Kohavi R, Sommerfield D (1998) Targeting business users with decision table classifiers. In: KDD, pp 249–253

  54. Freitas AA (2014) Comprehensible classification models: a position paper. ACM SIGKDD Explor Newsl 15(1):1–10

    Article  Google Scholar 

  55. Guyon I, Matic N, Vapnik V et al (1996) Discovering informative patterns and data cleaning

Download references

Acknowledgments

This work is supported by Telecom Italia S.p.A., Open Innovation Department, Joint Open Lab S-Cube, Milan.

Author information

Authors and Affiliations

  1. Politecnico di Milano, Milan, Italy

    Yashar Deldjoo, Mehdi Elahi, Paolo Cremonesi, Franca Garzotto, Pietro Piazzolla & Massimo Quadrana

Authors
  1. Yashar Deldjoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehdi Elahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paolo Cremonesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Franca Garzotto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pietro Piazzolla
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Massimo Quadrana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yashar Deldjoo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deldjoo, Y., Elahi, M., Cremonesi, P. et al. Content-Based Video Recommendation System Based on Stylistic Visual Features. J Data Semant 5, 99–113 (2016). https://doi.org/10.1007/s13740-016-0060-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13740-016-0060-9

Keywords

Search

Navigation