Skip to main content
SpringerLink
Log in

Predicting consumer choice from raw eye-movement data using the RETINA deep learning architecture

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

We propose the use of a deep learning architecture, called RETINA, to predict multi-alternative, multi-attribute consumer choice from eye movement data. RETINA directly uses the complete time series of raw eye-tracking data from both eyes as input to state-of-the art Transformer and Metric Learning Deep Learning methods. Using the raw data input eliminates the information loss that may result from first calculating fixations, deriving metrics from the fixations data and analysing those metrics, as has been often done in eye movement research, and allows us to apply Deep Learning to eye tracking data sets of the size commonly encountered in academic and applied research. Using a data set with 112 respondents who made choices among four laptops, we show that the proposed architecture outperforms other state-of-the-art machine learning methods (standard BERT, LSTM, AutoML, logistic regression) calibrated on raw data or fixation data. The analysis of partial time and partial data segments reveals the ability of RETINA to predict choice outcomes well before participants reach a decision. Specifically, we find that using a mere 5 s of data, the RETINA architecture achieves a predictive validation accuracy of over 0.7. We provide an assessment of which features of the eye movement data contribute to RETINA’s prediction accuracy. We make recommendations on how the proposed deep learning architecture can be used as a basis for future academic research, in particular its application to eye movements collected from front-facing video cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://github.com/tensorflow/tensor2tensor.

  2. http://www.Tobii.com.

  3. https://pypi.org/project/lightautoml/.

  4. https://lightautoml.readthedocs.io/en/latest/pages/modules/ml_algo.html#boosted-trees.

  5. http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.

  6. Note that in contrast to our procedure, Sims and Conati (2020) use the last 5 s of the raw data.

References

  • Akinyelu AA, Blignaut P (2020) Convolutional neural network-based methods for eye gaze estimation: A survey. IEEE Access 8:142581–142605

    Article  Google Scholar 

  • Alaparthi S, Mishra M (2021) Bert: a sentiment analysis odyssey. J Mark Anal 9(2):118–126

    Article  Google Scholar 

  • Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  • Barkley-Levenson E, Galvan A (2017) Eye blink rate predicts reward decisions in adolescents. Dev Sci 20(3):e12412

    Article  Google Scholar 

  • Bednarik R, Vrzakova H, Hradis M (2012) What do you want to do next: a novel approach for intent prediction in gaze-based interaction. In: Proceedings of the symposium on eye tracking research and applications, pp 83–90

  • Bento J, Saleiro P, Cruz AF, Figueiredo MA, Bizarro P (2021) Timeshap: explaining recurrent models through sequence perturbations. In:Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 2565–2573

  • Bhatnagar R, Orquin JL (2022) A meta-analysis on the effect of visual attention on choice. J Exp Psychol Gen 151:2265

    Article  Google Scholar 

  • Bhattacharya N, Rakshit S, Gwizdka J, Kogut P (2020) Relevance prediction from eye-movements using semi-interpretable convolutional neural networks. In: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, pp 223–233

  • Bulling A, Roggen D (2011) Recognition of visual memory recall processes using eye movement analysis. In: Proceedings of the 13th international conference on Ubiquitous computing, pp 455–464

  • Bulling A, Wedel M (2019) Pervasive eye-tracking for real-world consumer behavior analysis. In: A handbook of process tracing methods, pp 27–44 Routledge

  • Bulling A, Zander TO (2014) Cognition-aware computing. IEEE Pervasive Comput 13(3):80–83

    Article  Google Scholar 

  • Bulling A, Ward JA, Gellersen H (2012) Multimodal recognition of reading activity in transit using body-worn sensors. ACM Trans Appl Percept 9(1):1–21

    Article  Google Scholar 

  • Bulling A, Weichel C, Gellersen H (2013) Eyecontext: Recognition of high-level contextual cues from human visual behaviour. In: Proceedings of the sigchi conference on human factors in computing systems, pp 305–308

  • Byrne SA, Reynolds APF, Biliotti C, Bargagli-Stoffi FJ, Polonio L, Riccaboni M (2023) Predicting choice behaviour in economic games using gaze data encoded as scanpath images. Sci Rep 13(1):4722

    Article  Google Scholar 

  • Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  Google Scholar 

  • Collewijn H, Erkelens CJ, Steinman RM (1995) Voluntary binocular gaze-shifts in the plane of regard: dynamics of version and vergence. Vis Res 35(23–24):3335–3358

    Article  Google Scholar 

  • Dalrymple KA, Jiang M, Zhao Q, Elison JT (2019) Machine learning accurately classifies age of toddlers based on eye tracking. Sci Rep 9(1):1–10

    Article  Google Scholar 

  • David-John B, Peacock C, Zhang T, Murdison TS, Benko H, Jonker TR (2021) Towards gaze-based prediction of the intent to interact in virtual reality. In: ACM symposium on eye tracking research and applications, pp 1–7

  • Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  • Duchowski AT, Cournia N, Murphy H (2004) Gaze-contingent displays: a review. Cyberpsychol Behav 7(6):621–634

    Article  Google Scholar 

  • Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  • Feit AM, Vordemann L, Park S, Bérubé C, Hilliges O (2020) Detecting relevance during decision-making from eye movements for ui adaptation. In: ACM symposium on eye tracking research and applications. pp 1–11

  • Ferwerda B, Schedl M, Tkalcic M (2016) Personality traits and the relationship with (non-) disclosure behavior on facebook. In: Proceedings of the 25th international conference companion on World Wide Web. pp 565–568

  • Gabel S, Timoshenko A (2022) Product choice with large assortments: a scalable deep-learning model. Manage Sci 68(3):1808–1827

    Article  Google Scholar 

  • Gao T, Harari D, Tenenbaum J, Ullman S (2014) When computer vision gazes at cognition. arXiv preprint arXiv:1412.2672

  • Gilchrist ID, Harvey M (2006) Evidence for a systematic component within scan paths in visual search. Vis Cogn 14(4–8):704–715

    Article  Google Scholar 

  • Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE

  • Han K, Xiao A, Wu E, Guo J, Xu C, Wang Y (2021) Transformer in transformer. Adv Neural Inf Process Syst 34:15908–15919

    Google Scholar 

  • Heaton JB, Polson NG, Witte JH (2017) Deep learning for finance: deep portfolios. Appl Stoch Model Bus Ind 33(1):3–12

    Article  MathSciNet  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, Van de Weijer J (2011) Eye tracking: a comprehensive guide to methods and measures. OUP Oxford, Oxford

    Google Scholar 

  • Hoppe S, Loetscher T, Morey S, Bulling A (2015) Recognition of curiosity using eye movement analysis. In: Adjunct proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2015 ACM international symposium on wearable computers. pp 185–188

  • Illahi GK, Siekkinen M, Kämäräinen T, Ylä-Jääski A (2022) Real-time gaze prediction in virtual reality. In: Proceedings of the 14th international workshop on immersive mixed and virtual environment systems. pp 12–18

  • Kaya M, Bilge HŞ (2019) Deep metric learning: a survey. Symmetry 11(9):1066

    Article  Google Scholar 

  • Khosravan CH, Naji Turkbey B, Jones EC, Wood B, Bagci U (2019) A collaborative computer aided diagnosis (c-cad) system with eye-tracking, sparse attentional model, and deep learning. Med Image Anal 51:101–115

    Article  Google Scholar 

  • King AJ, Cooper GF, Clermont G, Hochheiser H, Hauskrecht M, Sittig DF, Visweswaran S (2020) Leveraging eye tracking to prioritize relevant medical record data: comparative machine learning study. J Med Internet Res 22(4):e15876

    Article  Google Scholar 

  • Kollias K-F, Syriopoulou-Delli CK, Sarigiannidis P, Fragulis GF (2021) The contribution of machine learning and eye-tracking technology in autism spectrum disorder research: a systematic review. Electronics 10(23):2982

    Article  Google Scholar 

  • Krafka K, Khosla A, Kellnhofer P, Kannan H, Bhandarkar S, Matusik W, Torralba A (2016) Eye tracking for everyone. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2176–2184

  • Krajbich I, Rangel A (2011) Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions. Proc Natl Acad Sci 108:13852–13857

    Article  Google Scholar 

  • Krajbich I, Armel C, Rangel A (2010) Visual fixations and the computation and comparison of value in simple choice. Nat Neurosci 13:1292–1298

    Article  Google Scholar 

  • Krajbich I, Lu D, Camerer C, Rangel A (2012) The attentional drift-diffusion model extends to simple purchasing decisions. Front Psychol 3:193

    Article  Google Scholar 

  • Krejtz K, Żurawska J, Duchowski AT, Wichary S (2020) Pupillary and microsaccadic responses to cognitive effort and emotional arousal during complex decision making. J Eye Mov Res. https://doi.org/10.16910/jemr.13.5.2

    Article  Google Scholar 

  • Krol M, Krol M (2017) A novel approach to studying strategic decisions with eye-tracking and machine learning. Judgm Decis Mak 12(6):596

    Article  Google Scholar 

  • Król M, Król M (2019) Learning from peers’ eye movements in the absence of expert guidance: a proof of concept using laboratory stock trading, eye tracking, and machine learning. Cogn Sci 43(2):e12716

    Article  Google Scholar 

  • Król ME, Król M (2019) A novel machine learning analysis of eye-tracking data reveals suboptimal visual information extraction from facial stimuli in individuals with autism. Neuropsychologia 129:397–406

    Article  Google Scholar 

  • Kübler TC, Rothe C, Schiefer U, Rosenstiel W, Kasneci E (2017) Subsmatch 2.0: Scanpath comparison and classification based on subsequence frequencies. Behav Res Methods 49:1048–1064

    Article  Google Scholar 

  • Kulis B et al (2013) Metric learning: a survey. Found Trends® Mach Learn 5(4):287–364

    Article  MathSciNet  Google Scholar 

  • Kunze K, Utsumi Y, Shiga Y, Kise K, Bulling A (2013) I know what you are reading: recognition of document types using mobile eye tracking. In: Proceedings of the 2013 international symposium on wearable computers. pp 113–116

  • LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  • Lee D, Derrible S, Pereira FC (2018) Comparison of four types of artificial neural network and a multinomial logit model for travel mode choice modeling. Transp Res Rec 2672(49):101–112

    Article  Google Scholar 

  • Lim JZ, Mountstephens J, Teo J (2020) Emotion recognition using eye-tracking: taxonomy, review and current challenges. Sensors 20(8):2384

    Article  Google Scholar 

  • Liu X (2023) Deep learning in marketing: a review and research agenda. Artif Intell Mark 20:239–271

    Google Scholar 

  • Marshall SP (2007) Identifying cognitive state from eye metrics. Aviat Space Environ Med 78:B165–B175

    Google Scholar 

  • Martinovici A, Pieters R, Erdem T (2023) Express: attention trajectories capture utility accumulation and predict brand choice. J Mark Res 60(4):625–645

    Article  Google Scholar 

  • Meißner M, Musalem A, Huber J (2016) Eye tracking reveals processes that enable conjoint choices to become increasingly efficient with practice. J Mark Res 53:1–17

    Article  Google Scholar 

  • Noton D, Stark L (1971) Scanpaths in eye movements during pattern perception. Science 171(3968):308–311

    Article  Google Scholar 

  • Orquin JL, Loose SM (2013) Attention and choice: a review on eye movements in decision making. Acta Physiol (Oxf) 1:190–205

    Google Scholar 

  • Patney A, Salvi M, Kim J, Kaplanyan A, Wyman C, Benty N, Luebke D, Lefohn A (2016) Towards foveated rendering for gaze-tracked virtual reality. ACM Trans Gr 35(6):1–12

    Article  Google Scholar 

  • Pfeiffer J, Pfeiffer T, Meißner M, Weiß E (2020) Eye-tracking-based classification of information search behavior using machine learning: evidence from experiments in physical shops and virtual reality shopping environments. Inf Syst Res 31(3):675–691

    Article  Google Scholar 

  • Pfeiffer UJ, Vogeley K, Schilbach L (2013) From gaze cueing to dual eye-tracking: novel approaches to investigate the neural correlates of gaze in social interaction. Neurosci Biobehav Rev 37(10):2516–2528

    Article  Google Scholar 

  • Pieters R, Warlop L (1999) Visual attention during brand choice: the impact of time pressure and task motivation. Int J Res Mark 16:1–16

    Article  Google Scholar 

  • Pieters R, Wedel M (2020) Heads up: head movements during ad exposure respond to consumer goals and predict brand memory. J Bus Res 111:281–289

    Article  Google Scholar 

  • Polonio L, Di Guida S, Coricelli G (2015) Strategic sophistication and attention in games: an eye-tracking study. Games Econom Behav 94:80–96

    Article  MathSciNet  Google Scholar 

  • Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85:59–108

    Article  Google Scholar 

  • Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124:372–422

    Article  Google Scholar 

  • Rello L, Ballesteros M (2015) Detecting readers with dyslexia using machine learning with eye tracking measures. In: Proceedings of the 12th international web for all conference. pp 1–8

  • Reutskaja E, Nagel R, Camerer CF, Rangel A (2011) Search dynamics in consumer choice under time pressure: an eye-tracking study. Am Econ Rev 101:900–926

    Article  Google Scholar 

  • Ribeiro MT, Singh S, Guestrin C (2016) “ Why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp 1135–1144

  • Russo JE, Leclerc F (1994) An eye-fixation analysis of choice processes for consumer nondurables. J Consum Res 21(2):274–290

    Article  Google Scholar 

  • Russo JE, Rosen LD (1975) An eye fixation analysis of multialternative choice. Mem Cognit 3:267–276

    Article  Google Scholar 

  • Salminen J, Nagpal M, Kwak H, An J, Jung S-g, Jansen BJ (2019) Confusion prediction from eye-tracking data: experiments with machine learning. In: Proceedings of the 9th international conference on information systems and technologies. pp 1–9

  • Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 815–823

  • Sharma P, Joshi S, Gautam S, Maharjan S, Filipe V, Reis MJ (2019) Student engagement detection using emotion analysis, eye tracking and head movement with machine learning. arXiv preprint arXiv:1909.12913

  • Shen C, Huang X, Zhao Q (2015) Predicting eye fixations on webpage with an ensemble of early features and high-level representations from deep network. IEEE Trans Multimed 17(11):2084–2093

    Article  Google Scholar 

  • Shi SW, Wedel M, Pieters R (2013) Information acquisition during online decision making: a model-based exploration using eye-tracking data. Manage Sci 59(5):1009–1026

    Article  Google Scholar 

  • Shimojo S, Simion C, Shimojo E, Scheier C (2003) Gaze bias both reflects and influences preference. Nat Neurosci 6:1317–1322

    Article  Google Scholar 

  • Shojaeizadeh M, Djamasbi S, Paffenroth RC, Trapp AC (2019) Detecting task demand via an eye tracking machine learning system. Decis Support Syst 116:91–101

    Article  Google Scholar 

  • Sims SD, Conati C (2020) A neural architecture for detecting user confusion in eye-tracking data. In: Proceedings of the 2020 international conference on multimodal interaction. pp 15–23

  • Singh A, Bevilacqua A, Nguyen TL, Hu F, McGuinness K, O’Reilly M, Whelan D, Caulfield B, Ifrim G (2023) Fast and robust video-based exercise classification via body pose tracking and scalable multivariate time series classifiers. Data Min Knowl Disc 37(2):873–912

    Article  MathSciNet  Google Scholar 

  • Singh AD, Mehta P, Husain S, Rajkumar R (2016) Quantifying sentence complexity based on eye-tracking measures. In: Proceedings of the workshop on computational linguistics for linguistic complexity (cl4lc). pp 202–212

  • Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427–437

    Article  Google Scholar 

  • Speicher M, Cucerca S, Krüger A (2017) Vrshop: a mobile interactive virtual reality shopping environment combining the benefits of on-and offline shopping. Proc ACM Interact Mobile Wearable Ubiquitous Technol 1(3):1–31

    Article  Google Scholar 

  • Steil J, Bulling A (2015) Discovery of everyday human activities from long-term visual behaviour using topic models. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. pp 75–85

  • Stember JN, Haydar C, Krupinski CPD, Elizabeth MS, Wood BJ, Aea Lignelli (2019) Eye tracking for deep learning segmentation using convolutional neural networks. J Digi Imag 32:597–604

    Article  Google Scholar 

  • Štrumbelj E, Kononenko I (2014) Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst 41(3):647–665

    Article  Google Scholar 

  • Stüttgen P, Boatwright P, Monroe RT (2012) A satisficing choice model. Mark Sci 31:878–899

    Article  Google Scholar 

  • Sugano Y, Zhang X, Bulling A (2016) Aggregaze: Collective estimation of audience attention on public displays. In: Proceedings of the 29th annual symposium on user interface software and technology. pp 821–831

  • Telpaz A, Webb R, Levy DJ (2015) Using EEG to predict consumers’ future choices. J Mark Res 52(4):511–529

    Article  Google Scholar 

  • Tessendorf B, Bulling A, Roggen D, Stiefmeier T, Feilner M, Derleth P, Tröster G (2011) Recognition of hearing needs from body and eye movements to improve hearing instruments. In: International Conference on Pervasive Computing, pp 314–331. Springer

  • Toubia O, Jong MGD, Stieger D, Füller J (2012) Measuring consumer preferences using conjoint poker. Mark Sci 31:138–156

    Article  Google Scholar 

  • Ursu R, Erdem T, Wang Q, Zhang QP (2022) Prior information and consumer search: Evidence from eye-tracking. Available at SSRN

  • van der Lans R, Wedel M (2017) Eye movements during search and choice. In: Wierenga B, van der Lans R (eds) Handbook of marketing decision models. Springer, New York, pp 331–359

    Chapter  Google Scholar 

  • Van der Lans R, Wedel M, Pieters R (2011) Defining eye-fixation sequences across individuals and tasks: the binocular-individual threshold (bit) algorithm. Behav Res Methods 43:239–257

    Article  Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst. Vol 30

  • Vitu F, McConkie GW, Zola D (1998) About regressive saccades in reading and their relation to word identification. In: Eye guidance in reading and scene perception, pp 101–124. Elsevier

  • Wang F, Casalino LP, Khullar D (2019) Deep learning in medicine-promise, progress, and challenges. JAMA Intern Med 179(3):293–294

    Article  Google Scholar 

  • Wedel M, Pieters R (2007) A review of eye-tracking applications in marketing. Rev Mark Res 4:123–147

    Article  Google Scholar 

  • Wedel M, Pieters R, van der Lans R (2023) Modeling eye movements during decision making: a review. Psychometrika 88(2):697–729

    Article  MathSciNet  Google Scholar 

  • Weitzman ML (1979) Optimal search for the best alternative. Econometrica 47:641–654

    Article  MathSciNet  Google Scholar 

  • Willemsen MC, Böckenholt U, Johnson EJ (2011) Choice by value encoding and value construction: processes of loss aversion. J Exp Psychol Gen 140(3):303

    Article  Google Scholar 

  • Wolf J, Hess S, Bachmann D, Lohmeyer Q, Meboldt M (2018) Automating areas of interest analysis in mobile eye tracking experiments based on machine learning. J Eye Mov Res. https://doi.org/10.16910/jemr.11.6.6

    Article  Google Scholar 

  • Wood E, Baltrusaitis T, Zhang X, Sugano Y, Robinson P, Bulling A (2015) Rendering of eyes for eye-shape registration and gaze estimation. In: Proceedings of the IEEE international conference on computer vision. pp 3756–3764

  • Wood E, Baltrušaitis T, Morency L-P, Robinson P, Bulling A (2016) A 3d morphable eye region model for gaze estimation. In: European conference on computer vision. pp 297–313. Springer

  • Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning. pp 2048–2057. PMLR

  • Yaneva V, Eraslan S, Yesilada Y, Mitkov R et al (2020) Detecting high-functioning autism in adults using eye tracking and machine learning. IEEE Trans Neural Syst Rehabil Eng 28(6):1254–1261

    Article  Google Scholar 

  • Yang LC, Toubia O, de Jong MGD (2015) A bounded rationality model of information search and choice in preference measurement. J Mark Res 52:166–183

    Article  Google Scholar 

  • Yarkoni T, Westfall J (2017) Choosing prediction over explanation in psychology: lessons from machine learning. Perspect Psychol Sci 12(6):1100–1122

    Article  Google Scholar 

  • Yu G, Xu B, Zhao Y, Zhang B, Yang M, Kan JYY, Milstein DM, Thevarajah D, Dorris MC (2016) Microsaccade direction reflects the economic value of potential saccade goals and predicts saccade choice. J Neurophysiol 115(2):741–751

    Article  Google Scholar 

  • Zemblys R, Niehorster DC, Holmqvist K (2019) gazenet: end-to-end eye-movement event detection with deep neural networks. Behav Res Methods 51(2):840–864

    Article  Google Scholar 

  • Zhang X, Sugano Y, Bulling A (2017) Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th annual ACM symposium on user interface software and technology. pp 193–203

Download references

Funding

The authors did not receive funds, grants, or other support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Coller School of Management, Tel Aviv University, Tel Aviv, Israel

    Moshe Unger

  2. Robert H. Smith School of Business, University of Maryland, College Park, USA

    Michel Wedel

  3. Stern School of Business, New York University, New York, USA

    Alexander Tuzhilin

Authors
  1. Moshe Unger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michel Wedel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexander Tuzhilin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Moshe Unger.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: extracted features and their Shapely values

Appendix: extracted features and their Shapely values

For the application of autoML, we calculated 82 features, which include (1) features of the raw data, (2) features of the scanpath of fixations of the eyes, (3) features of eye-fixations on AOIs, and (4) features extracted from image data. These features are shown in Table 3.

Table 3 Features extracted from the raw eye-tracking data, fixation data, and image data
Full size table

We computed feature importance (Ribeiro et al. 2016) for the LightAutoML-GBMs method presented in Sect. 4.2 by using Shapley additive explanations values for XGBoost tree models (Štrumbelj and Kononenko 2014). The higher the importance value, the more strongly that feature impacted model performance.

The top 11 most important features for the choice prediction task and their importance values are presented in Fig. 6. The features were extracted from various data sources (see Table 3) and fall into the following classes: (1) features from the raw gaze data points (x and y axis), (2) features of the fixation-based scanpaths of the eyes, (3) features of eye fixations on AOIs, and (4) features extracted from image data. Importance values for these classes, aggregated across the features in each class, are presented in Table 4.

Figure 6 shows that the (top three) features extracted from the raw eye-gaze points, the "sample entropy (x)", "sum of reoccurring data points (x)" and "standard deviation of distance between points (x)" computed from the x-coordinate of the raw eye-tracking data, have the largest impact on the choice prediction. Table 4 reveals that the raw data features have the highest importance overall, 0.553. This again supports our claim that the raw eye-movement data has the largest predictive impact on consumer choices.

Fig. 6
figure 6

LightAutoML-GBM feature importance comparison (importance is measured by Shapley values)

Full size image

Second, we note that the information on the variation of the x-coordinate of the eye gaze points (x,y) is more important to the predictions, compared to the features extracted from the y-coordinate. An explanation is that (as shown in Fig. 3) the horizontal layout of the display and the textual description of the product attributes require horizontal (left-right) eye movements. Movements of consumers’ eyes along the x-axis reflect the dominant layout of the stimulus and are therefore more predictive of choices.

Third, features of the fixation-based scanpath of the eyes are the second-best class of features for the prediction of consumers’ product choices, with an overall importance of 0.248. This again supports our claim that the raw eye-gaze data has a richer representation than the fixation data and therefore contributes more to the prediction results.

Fourth, image features were not included among the top-11 features in Fig. 6, as their overall importance was only 0.087. We note that the images of the personal computers used as stimuli in the experiment may not have had sufficient variation across products to contribute strongly to choice prediction. It remains to be seen, however, if this finding generalizes to other contexts.

Table 4 Feature importance for LightAutoML-GBM by data class
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Unger, M., Wedel, M. & Tuzhilin, A. Predicting consumer choice from raw eye-movement data using the RETINA deep learning architecture. Data Min Knowl Disc (2023). https://doi.org/10.1007/s10618-023-00989-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10618-023-00989-7

Keywords

Search

Navigation