An intelligent recommendation system using gaze and emotion detection

Multimedia Tools and Applications volume 78, pages14231–14250(2019)Cite this article

Abstract

Recently, recommendation system has become popular in many e-commerce websites. It helps users by suggesting products which they could buy. Existing work till now uses past feedback of user, similarity of other users’ buying pattern, or a hybrid approach in which both type of information is used. But the pitfall of these approaches is that there is a need to collect and process huge amount of data for good recommendation. This paper is aimed at developing an efficient recommendation system by incorporating user’s emotion and interest to provide good recommendations. The proposed system does not require any of aforementioned data and works without the continuous and interminable attention of the user. In this framework, we capture user’s eye-gaze and facial expression while exploring websites through inexpensive, visible light “webcam”. The eye-gaze detection method uses pupil-center extraction of both eyes and calculates the reference point through a joint probability. The facial expression uses landmark points of face and analyzes the emotion of the user. Both methods work in approximate real time and the proposed framework thus provides intelligent recommendations on-the-fly without requirement of feedback and buying patterns of users.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. 1.

    Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749

    Article  Google Scholar 

  2. 2.

    Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. Comput Vis–ECCV 2006:404–417

    Google Scholar 

  3. 3.

    Beel J, Gipp B, Langer S, Breitinger C (2016) Paper recommender systems: a literature survey. Int J Digit Libr 17(4):305–338

    Article  Google Scholar 

  4. 4.

    Beilin L, Yi S (2013) Survey of personalized recommendation based on society networks analysis. In: 2013 6th International conference on information management, innovation management and industrial engineering (ICIII), Vol. 3, IEEE, pp 337–340

  5. 5.

    Breese JS, Heckerman D, Kadie C (1998) Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, Morgan Kaufmann Publishers Inc., pp 43–52

  6. 6.

    Chen H-H, Gou L, Zhang X, Giles CL (2011) Collabseer: a search engine for collaboration discovery. In: Proceedings of the 11th annual international ACM/IEEE joint conference on digital libraries. ACM, pp 231–240

  7. 7.

    Chen H-H, Ororbia I, Alexander G, Giles CL (2015) Expertseer: a keyphrase based expert recommender for digital libraries. arXiv:1511.02058

  8. 8.

    Cohen I, Garg A, Huang TS et al (2000) Emotion recognition from facial expressions using multilevel hmm. In: Neural information processing systems, vol 2

  9. 9.

    Felfernig A, Isak K, Szabo K, Zachar P (2007) The vita financial services sales support environment. In: Proceedings of the national conference on artificial intelligence, vol 22, p 1692

  10. 10.

    Guo Y, Tao D, Yu J, Xiong H, Li Y, Tao D et al (2016) Deep neural networks with relativity learning for facial expression recognition. In: 2016 IEEE international conference on multimedia & expo workshops, ICMEW. IEEE, pp 1–6

  11. 11.

    Gupta P, Goel A, Lin J, Sharma A, Wang D, Zadeh R (2013) Wtf: the who to follow service at twitter. In: Proceedings of the 22nd international conference on World Wide Web. ACM, pp 505–514

  12. 12.

    Hansen DW, Ji Q (2010) In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell 32(3):478–500

    Article  Google Scholar 

  13. 13.

    Ishwar D, Singh BK (2015) Emotion detection using facial expression. Int J Emerg Res Manage Technol 4(6):144–151

    Google Scholar 

  14. 14.

    Jafarkarimi H, Sim ATH, Saadatdoost R (2012) A naive recommendation model for large databases. Intern J Inform Educ Technol 2(3):216

    Google Scholar 

  15. 15.

    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53

  16. 16.

    Karray F, Alemzadeh M, Saleh JA, Arab MN Human-computer interaction: overview on state of the art. International Journal on Smart Sensing and Intelligent Systems

  17. 17.

    Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874

  18. 18.

    Kermany NR, Alizadeh SH (2017) A hybrid multi-criteria recommender system using ontology and neuro-fuzzy techniques. Electron Commer Res Appl 21:50–64

    Article  Google Scholar 

  19. 19.

    Liao C-L, Lee S-J (2016) A clustering based approach to improving the efficiency of collaborative filtering recommendation. Electron Commer Res Appl 18:1–9

    Article  Google Scholar 

  20. 20.

    Liu W, Zhang L, Tao D, Cheng J (2018) Reinforcement online learning for emotion prediction by using physiological signals. Pattern Recogn Lett 107:123–130

    Article  Google Scholar 

  21. 21.

    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  22. 22.

    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), 2010. IEEE, pp 94–101

  23. 23.

    Ma X, Lu H, Gan Z, Zeng J (2017) An explicit trust and distrust clustering based collaborative filtering recommendation approach. Electron Commer Res Appl 25:29–39

    Article  Google Scholar 

  24. 24.

    Mase K (1991) Recognition of facial expression from optical flow. IEICE Trans 74(E):3474–3483

    Google Scholar 

  25. 25.

    Melville P, Sindhwani V (2011) Recommender systems. In: Encyclopedia of machine learning, Springer, pp 829–838

  26. 26.

    Mooney RJ, Roy L (2000) Content-based book recommending using learning for text categorization. In: Proceedings of the fifth ACM conference on digital libraries. ACM, pp 195–204

  27. 27.

    Pourgholamali F, Kahani M, Bagheri E, Noorian Z (2017) Embedding unstructured side information in product recommendation. Electron Commer Res Appl 25:70–85

    Article  Google Scholar 

  28. 28.

    Timm F, Barth E (2011) Accurate eye centre localisation by means of gradients. Visapp 11:125–130

    Google Scholar 

  29. 29.

    Valenti R, Gevers T (2008) Accurate eye center location and tracking using isophote curvature. In: 2008 IEEE conference on computer vision and pattern recognition, CVPR 2008. IEEE, pp 1–8

  30. 30.

    Venkateswarlu R (2003) Eye gaze estimation from a single image of one eye. In: Proceedings of the 9th IEEE international conference on computer vision, 2003. IEEE, pp 136–143

  31. 31.

    Viola P, Jones M (2001) Robust real-time object detection. Intern J Comput Vis 57(2):137–154

    Article  Google Scholar 

  32. 32.

    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, 2001. CVPR 2001, vol 1. IEEE, pp I–I

  33. 33.

    Yacoob Y, Davis LS (1996) Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans Pattern Anal Mach Intell 18(6):636–642

    Article  Google Scholar 

  34. 34.

    Yadava M, Kumar P, Saini R, Roy PP, Dogra DP (2017) Analysis of eeg signals and its application to neuromarketing. Multimed Tool Appl 76(18):19087–19111

    Article  Google Scholar 

  35. 35.

    Zhao WX, Li S, He Y, Chang EY, Wen J-R, Li X (2016) Connecting social media to e-commerce: Cold-start product recommendation using microblogging information. IEEE Trans Knowl Data Eng 28(5):1147–1159

    Article  Google Scholar 

  36. 36.

    Zhao Z-L, Wang C-D, Wan Y-Y, Lai J-H Recommendation in feature space sphere. Electronic Commerce Research and Applications

Download references

Author information

Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, 247667, India

    Saurabh Jaiswal, Shubham Virmani, Vishal Sethi, Kanjar De & Partha Pratim Roy

Authors
  1. Saurabh Jaiswal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shubham Virmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vishal Sethi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kanjar De
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Partha Pratim Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Partha Pratim Roy.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jaiswal, S., Virmani, S., Sethi, V. et al. An intelligent recommendation system using gaze and emotion detection. Multimed Tools Appl 78, 14231–14250 (2019). https://doi.org/10.1007/s11042-018-6755-1

Download citation

Keywords

  • E-commerce recommendation system
  • Eye gaze
  • Emotion detection