Mind the Gap! Comparing Retrospective and Concurrent Ratings of Emotion in User Experience Evaluation

  • Anders Bruun
  • Simon Ahm
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9296)


User experience (UX) is typically measured retrospectively through subjective questionnaire ratings, yet we know little of how well these retrospective ratings reflect concurrent experiences of an entire event. UX entails a broad range of dimensions of which human emotion is considered to be crucial. This paper presents an empirical study of the discrepancy between concurrent and retrospective ratings of emotions. We induced two experimental conditions of varying pleasantness. Findings show the existence of a significant discrepancy between retrospective and concurrent ratings of emotions. In the most unpleasant condition we found retrospective ratings to be significantly overestimated compared to concurrent ratings. In the most pleasant condition we found retrospective ratings to correlate with the highest and final peaks of emotional arousal. This indicates that we cannot always rely on typical retrospective UX assessments to reflect concurrent experiences. Consequently, we discuss alternative methods of assessing UX, which have considerable implications for practice.


User experience Emotion Memory-experience gap Peak-end rule 


  1. 1.
    Bargas-Avila, J.A., Hornbæk, K.: Old wine in new bottles or novel challenges. In: Procedings CHI, p. 2689. ACM, New York (2011)Google Scholar
  2. 2.
    Thüring, M., Mahlke, S.: Usability, aesthetics and emotions in human–technology interaction. Int. J. Psychol. 42, 253–264 (2007)CrossRefGoogle Scholar
  3. 3.
    Forlizzi, J., Battarbee, K.: Understanding experience in interactive systems. In: Proceedings DIS, pp. 261–268. ACM, New York (2004)Google Scholar
  4. 4.
    Scherer, K.R.: What are emotions? and how can they be measured? Soc. Sci. Inf. 44, 695–729 (2005)CrossRefGoogle Scholar
  5. 5.
    Miron-Shatz, T., Stone, A., Kahneman, D.: Memories of yesterday’s emotions: does the valence of experience affect the memory-experience gap? Emotion 9, 885–891 (2009)CrossRefGoogle Scholar
  6. 6.
    Redelmeier, D.A., Kahneman, D.: Patients’ memories of painful medical treatments: real-time and retrospective evaluations of two minimally invasive procedures. Pain 66, 3–8 (1996)CrossRefGoogle Scholar
  7. 7.
    Hassenzahl, M., Ullrich, D.: To do or not to do: differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interact. Comput. 19, 429–437 (2007)CrossRefGoogle Scholar
  8. 8.
    Mahlke, S., Lindgaard, G.: Emotional experiences and quality perceptions of interactive products. In: Jacko, J. (ed.) Human-Computer Interaction. Interaction Design and Usability SE – 19, pp. 164–173. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  9. 9.
    Mahlke, S., Thüring, M.: Studying antecedents of emotional experiences in interactive contexts. In: Proceedings CHI, pp. 915–918. ACM, New York (2007)Google Scholar
  10. 10.
    Kujala, S., Miron-Shatz, T.: Emotions, experiences and usability in real-life mobile phone use. In: Proceedings CHI, pp. 1061–1070. ACM, New York (2013)Google Scholar
  11. 11.
    Diener, E., Tay, L.: Review of the day reconstruction method (DRM). Soc. Indic. Res. 116, 255–267 (2014)CrossRefGoogle Scholar
  12. 12.
    Bylsma, L.M., Taylor-Clift, A., Rottenberg, J.: Emotional reactivity to daily events in major and minor depression. J. Abnorm. Psychol. 120, 155–167 (2011)CrossRefGoogle Scholar
  13. 13.
    James, W.: What is an emotion? Mind os-IX 9, 188–205 (1884)Google Scholar
  14. 14.
    Lang, P.J.: Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski, J.B., Johnson, J.H., Williams, T.H. (eds.) Technology in Mental Health Care Delivery Systems, pp. 119–137. Ablex, Norwood (1980)Google Scholar
  15. 15.
    Lang, P.J.: The emotion probe: Studies of motivation and attention. Am. Psychol. 50, 372–385 (1995)CrossRefGoogle Scholar
  16. 16.
    Andreassi, J.L.: Psychophysiology: Human Behavior and Physiological Response. Lawrence Erlbaum, Mahwah (2000)Google Scholar
  17. 17.
    Ward, R., Marsden, P.: Physiological responses to different WEB page designs. Int. J. Human-Computer Stud. 59, 199–212 (2003)CrossRefGoogle Scholar
  18. 18.
    Baumeister, R.F., Bratslavsky, E., Finkenauer, C., Vohs, K.D.: Bad is stronger than good. Rev. Gen. Psychol. 5, 323–370 (2001)CrossRefGoogle Scholar
  19. 19.
    Kahneman, D., Fredrickson, B.L., Schreiber, C.A., Redelmeier, D.A.: When more pain is preferred to less: adding a better end. Psychol. Sci. 4, 401–405 (1993)CrossRefGoogle Scholar
  20. 20.
    Mahlke, S., Minge, M., Thüring, M.: Measuring multiple components of emotions in interactive contexts. In: CHI EA, pp. 1061–1066. ACM, New York (2006)Google Scholar
  21. 21.
    Hassenzahl, M., Sandweg, N.: From mental effort to perceived usability: transforming experiences into summary assessments. In: CHI EA, pp. 1283–1286. ACM, New York (2004)Google Scholar
  22. 22.
    Dockray, S., Grant, N., Stone, A.A., Kahneman, D., Wardle, J., Steptoe, A.: A comparison of affect ratings obtained with ecological momentary assessment and the day reconstruction method. Soc. Indic. Res. 99, 269–283 (2010)CrossRefGoogle Scholar
  23. 23.
    Omodei, M.M., McLennan, J.: Studying complex decision making in natural settings: using a head-mounted video camera to study competitive orienteering. Percept. Mot. Skills 79, 1411–1425 (1994)CrossRefzbMATHGoogle Scholar
  24. 24.
    Bentley, T., Johnston, L., von Baggo, K.: Evaluation using cued-recall debrief to elicit information about a user’s affective experiences. In: Proceedings OzCHI, pp. 1–10. CHISIG Australia, Narrabundah (2005)Google Scholar
  25. 25.
    Gao, Y., Bianchi-Berthouze, N., Meng, H.: What does touch tell us about emotions in touchscreen-based gameplay? ACM Trans. Comput. Interact. 19, 31:1–31:30 (2012)zbMATHGoogle Scholar
  26. 26.
    Belford, Z., Neher, C., Pernsteiner, T., Stoffregen, J., Tariq, Z.: Music and physical performance: the effects of different music genres on physical performance as measured by the heart rate, electrodermal arousal, and maximum grip strength. In: JASS. 3 (2013)Google Scholar
  27. 27.
    Karapanos, E., Zimmerman, J., Forlizzi, J., Martens, J.-B.: Measuring the dynamics of remembered experience over time. Interact. Comput. 22, 328–335 (2010)CrossRefGoogle Scholar
  28. 28.
    Karapanos, E., Martens, J.-B., Hassenzahl, M.: On the retrospective assessment of users’ experiences over time: memory or actuality? In: CHI EA, pp. 4075–4080. ACM, New York (2010)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  1. 1.Department of Computer ScienceAalborg UniversityAalborgDenmark

Personalised recommendations