Personal and Ubiquitous Computing

, Volume 19, Issue 5–6, pp 781–801 | Cite as

Evaluating visual attention for multi-screen television: measures, toolkit, and experimental findings

Original Article

Abstract

New and emerging multi-screen television scenarios and applications need new evaluation procedures, methodologies, and tools to support multi-screen data analysis. In this work, we introduce a set of 12 measures to characterize viewers’ visual attention patterns for multi-screen TV. Out of these, eight measures are computed directly from eye-tracking data, while the other four are evaluated using questionnaires. We applied our measures during a controlled experiment involving nine distinct screen layouts with two, three, and four TV screens, for which we report new findings about viewers’ distributions of visual attention. For example, we found that viewers need an average discovery time up to 4.5 s to visually fixate four screens, and their subjective perceptions of what they watched and for how long they watched each screen are substantially accurate, i.e., we report Pearson’s correlation coefficients up to r = .892 with ground truth measured with eye-tracking equipment. We also analyze and discuss the evolution of our participants’ distributions of visual attention over time from the perspective of our new set of measures. For example, we found that people perform significantly more transitions between screens during the first seconds of watching television, after which their level of visual attention converges to a stable value. We complement the findings revealed by our objective eye gaze measurements with subjective data about participants’ perceived cognitive work load and comfortability while watching more than one TV screen, and we measure viewers’ capacities to understand and recall content delivered simultaneously on multiple screens. To foster new studies and explorations of viewers’ visual attention patterns during multi-screen television watching, we release in the community an update for an existing software toolkit (i.e., VATic-TV, the Visual Attention Toolkit for TV, now at version v2) that automatically computes our measures from data delivered by standard eye-tracking equipment. We hope that our new set of measures and the companion software will benefit the community as a first step toward understanding visual attention for emerging multi-screen TV applications and, consequently, will help researchers and practitioners to design new TV applications that will better exploit viewers’ visual attention patterns toward new, richer television experiences.

Keywords

Visual attention Interactive TV Multi-screen TV  Multi-display Eye gaze Eye tracking Measures  Experiment Evaluation TLX 

References

  1. 1.
    Anderson J (2004) Cognitive psychology and its implications. Worth Publishers, New YorkGoogle Scholar
  2. 2.
    Basapur S, Harboe G, Mandalia H, Novak A, Vuong V, Metcalf C (2011) Field trial of a dual device user experience for iTV. In: Proceedings of the 9th international interactive conference on interactive television, ACM, New York, NY, USA, EuroITV ’11, pp 127–136. doi:10.1145/2000119.2000145
  3. 3.
    Basapur S, Mandalia H, Chaysinh S, Lee Y, Venkitaraman N, Metcalf C (2012) FANFEEDS: evaluation of socially generated information feed on second screen as a TV show companion. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 87–96. doi:10.1145/2325616.2325636
  4. 4.
    Bi X, Bae SH, Balakrishnan R (2010) Effects of interior bezels of tiled-monitor large displays on visual search, tunnel steering, and target selection. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’10, pp 65–74. doi:10.1145/1753326.1753337
  5. 5.
    Borji A, Itti L (2013) State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 35(1):185–207MathSciNetCrossRefGoogle Scholar
  6. 6.
    Cesar P, Bulterman DC, Jansen AJ (2008) Usages of the secondary screen in an interactive television environment: control, enrich, share, and transfer television content. In: Proceedings of the 6th European conference on changing television environments, Springer, Berlin, EuroiTV ’08, pp 168–177. doi:10.1007/978-3-540-69478-6_22
  7. 7.
    Chorianopoulos K, Fernández FJB, Salcines EG, de Castro Lozano C (2010) Delegating the visual interface between a tablet and a TV. In: Proceedings of the international conference on advanced visual interfaces, ACM, New York, NY, USA, AVI ’10, pp 418–418. doi:10.1145/1842993.1843096
  8. 8.
    Courtois C, D’heer E (2012) Second screen applications and tablet users: constellation, awareness, experience, and interest. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 153–156. doi:10.1145/2325616.2325646
  9. 9.
    Cummins R, Tirumala L, Lellis J (2011) Viewer attention to espns mosaic screen: an eye-tracking investigation. J Sports Media 6(1):23–54CrossRefGoogle Scholar
  10. 10.
    Dezfuli N, Khalilbeigi M, Huber J, Müller F, Mühlhäuser M (2012) Palmrc: imaginary palm-based remote control for eyes-free television interaction. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 27–34. doi:10.1145/2325616.2325623
  11. 11.
    Eriksen CW, Hoffman JE (1972) Temporal and spatial characteristics of selective encoding from visual displays. Percept Psychophys 12(2):201–204. doi:10.3758/BF03212870 CrossRefMATHGoogle Scholar
  12. 12.
    Eriksen CW, St James JD (1986) Visual attention within and around the field of focal attention: a zoom lens model. Percept Psychophys 40(4):225–240. doi:10.3758/BF03211502 CrossRefGoogle Scholar
  13. 13.
    Fitts PM (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exper Psychol 47(6):381–391. doi:10.1037/h0055392 CrossRefGoogle Scholar
  14. 14.
    Forlines C, Shen C, Wigdor D, Balakrishnan R (2006) Exploring the effects of group size and display configuration on visual search. In: Proceedings of the 2006 20th anniversary conference on computer supported cooperative work, ACM, New York, NY, USA, CSCW ’06, pp 11–20. doi:10.1145/1180875.1180878
  15. 15.
    Frintrop S, Rome E, Christensen HI (2010) Computational visual attention systems and their cognitive foundations: a survey. ACM Trans Appl Percept 7(1):6:1–6:39. doi:10.1145/1658349.1658355 CrossRefGoogle Scholar
  16. 16.
    Geerts D, Cesar P, Bulterman D (2008) The implications of program genres for the design of social television systems. In: Proceedings of the 1st international conference on designing interactive user experiences for TV and video, ACM, New York, NY, USA, UXTV ’08, pp 71–80. doi:10.1145/1453805.1453822
  17. 17.
    Geerts D, Leenheer R, De Grooff D, Negenman J, Heijstraten S (2014) In front of and behind the second screen: viewer and producer perspectives on a companion app. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 95–102. doi:10.1145/2602299.2602312
  18. 18.
    Grüninger J, Krüger J (2013) The impact of display bezels on stereoscopic vision for tiled displays. In: Proceedings of the 19th ACM symposium on virtual reality software and technology, ACM, New York, NY, USA, VRST ’13, pp 241–250. doi:10.1145/2503713.2503717
  19. 19.
    Hawkins R, Pingree S, Hitchon J, Radler B, Gorham B, Kahlor L, Gilligan E, Serlin R, Schmidt T, Kannaovakun P, Kolbeins G (2005) What produces television attention and attention style? Genre, situation, and individual differences as predictors. Human Commun Res 31(1):162–187Google Scholar
  20. 20.
    Holmes ME, Josephson S, Carney RE (2012) Visual attention to television programs with a second-screen application. In: Proceedings of the symposium on eye tracking research and applications, ACM, New York, NY, USA, ETRA ’12, pp 397–400. doi:10.1145/2168556.2168646
  21. 21.
    Hutchings D (2012) An investigation of Fitts’ law in a multiple-display environment. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’12, pp 3181–3184. doi:10.1145/2207676.2208736
  22. 22.
    Jones B, Sodhi R, Murdock M, Mehra R, Benko H, Wilson A, Ofek E, MacIntyre B, Raghuvanshi N, Shapira L (2014) Roomalive: magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, ACM, New York, NY, USA, UIST ’14, pp 637–644. doi:10.1145/2642918.2647383
  23. 23.
    Jones BR, Benko H, Ofek E, Wilson AD (2013) Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’13, pp 869–878. doi:10.1145/2470654.2466112
  24. 24.
    Kallenbach J, Narhi S, Oittinen P (2007) Effects of extra information on TV viewers’ visual attention, message processing ability, and cognitive workload. Comput Entertain 5(2). doi:10.1145/1279540.1279548
  25. 25.
    Miller G (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 63:81–97CrossRefGoogle Scholar
  26. 26.
  27. 27.
    Posner M (1980) Orienting of attention. Q J Exper Psychol 32:3–25CrossRefGoogle Scholar
  28. 28.
    Posner M, Cohen Y (1984) Components of visual orienting. Atten Perform X Control Lang Process 32:531–556Google Scholar
  29. 29.
    Potter M (1976) Short-term conceptual memory for pictures. J Exper Psychol 2:509–522Google Scholar
  30. 30.
    Rashid U, Nacenta MA, Quigley A (2012) The cost of display switching: a comparison of mobile, large display and hybrid UI configurations. In: Proceedings of the international working conference on advanced visual interfaces, ACM, New York, NY, USA, AVI ’12, pp 99–106. doi:10.1145/2254556.2254577
  31. 31.
    Rashid U, Nacenta MA, Quigley A (2012) Factors influencing visual attention switch in multi-display user interfaces: a survey. In: Proceedings of the 2012 international symposium on pervasive displays, ACM, New York, NY, USA, PerDis ’12, pp 1:1–1:6. doi:10.1145/2307798.2307799
  32. 32.
    Reichle E, Rayner K, Pollatsek A (2004) The e-z reader model of eye-movement control in reading: comparisons to other models. J Behav Brain Sci 26:445–476Google Scholar
  33. 33.
    Riche N, Mancas M, Culibrk D, Crnojevic V, Gosselin B, Dutoit T (2013) Dynamic saliency models and human attention: a comparative study on videos. In: Proceedings of the 11th Asian conference on computer vision—volume part III, Springer, Berlin, ACCV’12, pp 586–598. doi:10.1007/978-3-642-37431-9_45
  34. 34.
    Schwarz J, Klionsky D, Harrison C, Dietz P, Wilson A (2012) Phone as a pixel: enabling ad-hoc, large-scale displays using mobile devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’12, pp 2235–2238. doi:10.1145/2207676.2208378
  35. 35.
    Sohlberg M, Mateer C (1989) Introduction to cognitive rehabilitation: theory and practice. Guilford Press, New YorkGoogle Scholar
  36. 36.
    Tan DS, Czerwinski M (2003) Effects of visual separation and physical discontinuities when distributing information across multiple displays. In: Proceedings of the IFIP TC.13 international conference on human–computer interaction, INTERACT ’03, pp 252–255Google Scholar
  37. 37.
    Theeuwes J (1991) Exogenous and endogenous control of attention: the effect of visual onsets and offsets. Percept Psychophys 49(1):83–90. doi:10.3758/BF03211619 CrossRefGoogle Scholar
  38. 38.
    Thomas BH (2012) A survey of visual, mixed, and augmented reality gaming. Comput Entertain 10(3):3:1–3:33. doi:10.1145/2381876.2381879 CrossRefGoogle Scholar
  39. 39.
    Toet A (2011) Computational versus psychophysical bottom–up image saliency: a comparative evaluation study. IEEE Trans Pattern Anal Mach Intell 33(11):2131–2146CrossRefGoogle Scholar
  40. 40.
    Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12(1):97–136CrossRefGoogle Scholar
  41. 41.
    Vatavu RD (2012) Point & click mediated interactions for large home entertainment displays. Multimed Tools Appl 59(1):113–128. doi:10.1007/s11042-010-0698-5 CrossRefGoogle Scholar
  42. 42.
    Vatavu RD (2012) User-defined gestures for free-hand TV control. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 45–48. doi:10.1145/2325616.2325626
  43. 43.
    Vatavu RD (2013) A comparative study of user-defined handheld vs. freehand gestures for home entertainment environments. J Ambient Intell Smart Environ 5(2):187–211. doi:10.3233/AIS-130200 Google Scholar
  44. 44.
    Vatavu RD (2013) There’s a world outside your TV: exploring interactions beyond the physical TV screen. In: Proceedings of the 11th European conference on interactive TV and video, ACM, New York, NY, USA, EuroITV ’13, pp 143–152. doi:10.1145/2465958.2465972
  45. 45.
    Vatavu RD, Mancas M (2013) Interactive TV potpourris: an overview of designing multi-screen TV installations for home entertainment. In: Mancas M, d’Alessandro N, Siebert X, Gosselin B, Valderrama C, Dutoit T (eds) Intelligent technologies for interactive entertainment. Lecture notes of the institute for computer sciences, social informatics and telecommunications engineering, vol 124. Springer International Publishing, pp 49–54. doi:10.1007/978-3-319-03892-6_6
  46. 46.
    Vatavu RD, Mancas M (2014) Visual attention measures for multi-screen TV. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 111–118. doi:10.1145/2602299.2602305
  47. 47.
    Vatavu RD, Pentiuc SG (2008) Interactive coffee tables: interfacing TV within an intuitive, fun and shared experience. In: Proceedings of the 6th European interactive TV conference, Springer, EuroITV ’08, pp 183–187Google Scholar
  48. 48.
    Vatavu RD, Wobbrock JO (2015) Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’15, pp 1325–1334. doi:10.1145/2702123.2702223
  49. 49.
    Vatavu RD, Zaiti IA (2014) Leap gestures for TV: insights from an elicitation study. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 131–138. doi10.1145/2602299.2602316
  50. 50.
    Wagner J, Nancel M, Gustafson SG, Huot S, Mackay WE (2013) Body-centric design space for multi-surface interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’13, pp 1299–1308. doi:10.1145/2470654.2466170
  51. 51.
    Wallace JR, Vogel D, Lank E (2014) Effect of bezel presence and width on visual search. In: Proceedings of the international symposium on pervasive displays, ACM, New York, NY, USA, PerDis ’14, pp 118:118–118:123. doi:10.1145/2611009.2611019
  52. 52.
    Wallace JR, Vogel D, Lank E (2014) The effect of interior bezel presence and width on magnitude judgement. In: Proceedings of the 2014 graphics interface conference, Canadian Information Processing Society, Toronto, Ont., Canada, Canada, GI ’14, pp 175–182. http://dl.acm.org/citation.cfm?id=2619648.2619678
  53. 53.
    Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input. In: CHI ’05 extended abstracts on human factors in computing systems, ACM, New York, NY, USA, CHI EA ’05, pp 1869–1872. doi:10.1145/1056808.1057043

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  1. 1.Department of Computer Science, Electronics, and Automation and Integrated Center for Research, Development, and Innovation in Advanced Materials, Nanotechnologies, and Distributed Systems for Fabrication and Control (MANSiD)University Stefan cel Mare of SuceavaSuceavaRomania
  2. 2.University of MonsMonsBelgium

Personalised recommendations