Combined User Physical, Physiological and Subjective Measures for Assessing User Cost

  • Tao Lin
  • Atsumi Imamiya
  • Wanhua Hu
  • Masaki Omata
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4397)


New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multimodal information streams in evaluating user cost. In the study, easy, medium and difficult versions of a game task were used to vary the levels of the cost to user. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Three findings indicate the potential value of multimodal information in evaluating usability: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different user cost levels appeared to correlate with eye movement patterns, especially with a combined eye–hand measure. Third, HRV showed correlations with saccade speed. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.


Heart Rate Variability Pupil Size User Cost Pupillary Response Mental Workload 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hollnagel, E.: From human factors to cognitive systems engineering: Human–machine interactions in the 21st century. In: Kitamura, M., Kimura, I. (eds.) Anzen-no-Tankyu (Researches on Safety), ERC Publishing, Tokyo (2003)Google Scholar
  2. 2.
    Marshall, C., Rossman, G.B.: Designing qualitative research, 3rd edn. Sage, Thousand Oaks (1999)Google Scholar
  3. 3.
    Wilson, G.M., Sasse, M.A.: Investigating the impact of audio degradations on users: Subjective vs. objective assessment methods. In: The Proceedings of OZCHI 2000: Interfacing Reality in the New Millennium, Sydney, Australia, pp. 135–142 (2000)Google Scholar
  4. 4.
    Sweeney, M., Maguire, M., Shackel, B.: Evaluating user-computer interaction: a framework. International Journal of Man-Machine Studies 38, 689–711 (1993)CrossRefGoogle Scholar
  5. 5.
    Wilson, G.M.: Psychophysiological indicators of the impact of media quality on users. In: The Proceedings of CHI 2001 Doctoral Consortium, pp. 95–96. ACM Press, New York (2001)Google Scholar
  6. 6.
    Andreassi, J.L.: Psychophysiology: Human Behavior and Physiological Response, 4th edn. Lawrence Erlbaum, Mahwah (2000)Google Scholar
  7. 7.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)Google Scholar
  8. 8.
    Vicente, K.J., Thornton, C., Moray, N.: Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors 29, 171–182 (1987)Google Scholar
  9. 9.
    Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes among emotions. Science 221(4616), 1208–1210 (1983)CrossRefGoogle Scholar
  10. 10.
    Lin, T., Hu, W.H., Omata, M., Imamiya, A.: Do physiological data relate to traditional usability indexes? In: The Proceedings of the Australian Conference on Computer Human Interaction OZCHI’05, Canberra, Australia, November 15–21 (2005)Google Scholar
  11. 11.
    Meehan, M., Insko, B., Whitton, M., Brooks, F.: Physiological measures of presence in stressful virtual environments. In: The Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (2002)Google Scholar
  12. 12.
    Rowe, D.W., Sibert, J., Irwin, D.: Heart rate variability: Indicator of user state as an aid to human–computer interaction. In: Proceedings of the CHI, pp. 480–487 (1998)Google Scholar
  13. 13.
    Hopman, J.C.W., Kollee, L.A.A., Stoelinga, G.B.A., Van Geijn, H.P., van Ravenswaaij-Arts, C.M.A.: Heart rate variability. Annals of Internal Medicine 118, 436–447 (1993)Google Scholar
  14. 14.
    Malik, M.: Heart rate variability. Circulation 93, 1043–1065 (1996)Google Scholar
  15. 15.
    Pomeranz, B., Macaulay, R.J.B., Caudill, M.A., et al.: Assessment of autonomic function in humans by heart rate spectral analysis. Am. J. Physiol. Heart Circ. Physiol. 248, 151–H153 (1985)Google Scholar
  16. 16.
    Pagani, M., Lombardi, F., Guzzetti, S.: Power spectral analysis of heart rate and arterial pressure variabilities as a maker of sympatho-vagal interaction in man and conscious dog. Cir. Res. 59, 178–193 (1986)Google Scholar
  17. 17.
    Ettema, J., Ziclhuis, R.L.: Physiological parameters of mental load. Ergonomics 14, 137–144 (1971)CrossRefGoogle Scholar
  18. 18.
    Luczak, I.I., Lauring, W.J.: An analysis of heart rate variability. Ergonomics 16, 85–97 (1973)CrossRefGoogle Scholar
  19. 19.
    Hyndman, B.W., Gregory, J.R.: Spectral analysis of sinus arrhythmia during mental loading. Ergonomics 18, 255–270 (1975)CrossRefGoogle Scholar
  20. 20.
    Boutcher, S.H., Naugent, F.W., Mclaren, P.F.: Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology 35, 16–22 (1998)CrossRefGoogle Scholar
  21. 21.
    Hyde, C., Izard, C.E.: Cardiac rhythmicities and attention in young children. Psychophysiology 34, 547–552 (1997)CrossRefGoogle Scholar
  22. 22.
    Tattersall, A.J., Hockey, G.R.J.: Level of operator control and changes in heart rate variability during simulated flight maintenance. Human Factors 37, 682–698 (1995)CrossRefGoogle Scholar
  23. 23.
    Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem solving. Science 132, 1191–1192 (1964)Google Scholar
  24. 24.
    Hoecks, B., Levelt, W.: Pupillary dilation as a measure of attention: A quantitative system analysis. Behavior Research Methods, Instruments, & Computers 25, 16–26 (1993)Google Scholar
  25. 25.
    Juris, M., Velden, M.: The pupillary response to mental overload. Physiological Psychology 5(4), 421–424 (1977)Google Scholar
  26. 26.
    Kahneman, D.: Pupillary responses in a pitch-discrimination task. Perception & Psychophysics 2, 101–105 (1967)Google Scholar
  27. 27.
    Nakayama, M., Takahashi, K.: The act of task difficulty and eye-movement frequency for the oculo-motor indices. In: The Proceedings of the Eye Tracking Research and Application, pp. 37–42 (2002)Google Scholar
  28. 28.
    Beatty, J.: Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin 91(2), 276–292 (1982)CrossRefGoogle Scholar
  29. 29.
    Iqbal, S.T., Zheng, X.S., Bailey, B.P.: Task-evoked pupillary response to mental workload in human–computer interaction. In: The proceeding of CHI, pp. 1477–1480 (2004)Google Scholar
  30. 30.
    Hess, E.H., Petrovich, S.B.: Pupillary behavior in communication. In: Siegman, A.W., Feldstein, S. (eds.) Nonverbal Behavior and Communication, pp. 327–348. Lawrence Erlbaum, Hillsdale (1987)Google Scholar
  31. 31.
    Rayner, K.: Eye movements and information processing: 20 years of research. Psychological Bulletin 124(3), 372–422 (1998)CrossRefGoogle Scholar
  32. 32.
    Noton, D., Stark, L.W.: Scanpath in saccadic eye movements while viewing and recognizing patterns. Vision Research 11, 929–942 (1971)CrossRefGoogle Scholar
  33. 33.
    Goldberg, H., Kotval, X.P.: Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 24, 631–645 (1999)CrossRefGoogle Scholar
  34. 34.
    Hébert, S., Beland, R., Dionne-Fournelle, O., Crête, M., Lupien, S.J.: Physiological stress response to video game playing: The contribution of built-in music. Life Sciences 76, 2371–2380 (2005)CrossRefGoogle Scholar
  35. 35.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): Results of experimental and theoretical research. In: Hancock, P.A., Meshakati, N. (eds.) Human Mental Workload, pp. 39–183. North-Holland, Amsterdam (1988)Google Scholar
  36. 36.
    Hair, J.F., Anderson, R.E., Tatham, R.L., Black, W.C.: Multivariate Data Analysis, 5th edn. Prentice-Hall, Englewood Cliffs (1998)Google Scholar
  37. 37.
    Lin, Y., Zhang, W.J., Koubek, R.J.: Effective Attention Allocation Behavior and its Measurement: A Preliminary Study. Interacting with Computers 16(Issue 6), 1195–1210 (2004)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Tao Lin
    • 1
  • Atsumi Imamiya
    • 1
  • Wanhua Hu
    • 1
  • Masaki Omata
    • 1
  1. 1.Department of Computer Science and Media Engineering, University of Yamanashi, Takeda 4-3-11, Kofu, Yamanashi Prefecture, 400-8511Japan

Personalised recommendations