Skip to main content
SpringerLink
Log in

Evaluating visual attention for multi-screen television: measures, toolkit, and experimental findings

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

New and emerging multi-screen television scenarios and applications need new evaluation procedures, methodologies, and tools to support multi-screen data analysis. In this work, we introduce a set of 12 measures to characterize viewers’ visual attention patterns for multi-screen TV. Out of these, eight measures are computed directly from eye-tracking data, while the other four are evaluated using questionnaires. We applied our measures during a controlled experiment involving nine distinct screen layouts with two, three, and four TV screens, for which we report new findings about viewers’ distributions of visual attention. For example, we found that viewers need an average discovery time up to 4.5 s to visually fixate four screens, and their subjective perceptions of what they watched and for how long they watched each screen are substantially accurate, i.e., we report Pearson’s correlation coefficients up to r = .892 with ground truth measured with eye-tracking equipment. We also analyze and discuss the evolution of our participants’ distributions of visual attention over time from the perspective of our new set of measures. For example, we found that people perform significantly more transitions between screens during the first seconds of watching television, after which their level of visual attention converges to a stable value. We complement the findings revealed by our objective eye gaze measurements with subjective data about participants’ perceived cognitive work load and comfortability while watching more than one TV screen, and we measure viewers’ capacities to understand and recall content delivered simultaneously on multiple screens. To foster new studies and explorations of viewers’ visual attention patterns during multi-screen television watching, we release in the community an update for an existing software toolkit (i.e., VATic-TV, the Visual Attention Toolkit for TV, now at version v2) that automatically computes our measures from data delivered by standard eye-tracking equipment. We hope that our new set of measures and the companion software will benefit the community as a first step toward understanding visual attention for emerging multi-screen TV applications and, consequently, will help researchers and practitioners to design new TV applications that will better exploit viewers’ visual attention patterns toward new, richer television experiences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. The FaceLab system is a head-free eye tracker, http://www.seeingmachines.com/product/facelab/. We calibrated the tracker with a 9-dot grid and used it to record eye gaze at a rate of 60 Hz.

  2. http://humansystems.arc.nasa.gov/groups/tlx/.

  3. http://www.keithv.com/software/nasatlx/.

  4. Except for the screen watching time and discovery sequence measures, which return more than one value and, consequently, cannot be included in this analysis.

References

  1. Anderson J (2004) Cognitive psychology and its implications. Worth Publishers, New York

    Google Scholar 

  2. Basapur S, Harboe G, Mandalia H, Novak A, Vuong V, Metcalf C (2011) Field trial of a dual device user experience for iTV. In: Proceedings of the 9th international interactive conference on interactive television, ACM, New York, NY, USA, EuroITV ’11, pp 127–136. doi:10.1145/2000119.2000145

  3. Basapur S, Mandalia H, Chaysinh S, Lee Y, Venkitaraman N, Metcalf C (2012) FANFEEDS: evaluation of socially generated information feed on second screen as a TV show companion. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 87–96. doi:10.1145/2325616.2325636

  4. Bi X, Bae SH, Balakrishnan R (2010) Effects of interior bezels of tiled-monitor large displays on visual search, tunnel steering, and target selection. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’10, pp 65–74. doi:10.1145/1753326.1753337

  5. Borji A, Itti L (2013) State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 35(1):185–207

    Article  MathSciNet  Google Scholar 

  6. Cesar P, Bulterman DC, Jansen AJ (2008) Usages of the secondary screen in an interactive television environment: control, enrich, share, and transfer television content. In: Proceedings of the 6th European conference on changing television environments, Springer, Berlin, EuroiTV ’08, pp 168–177. doi:10.1007/978-3-540-69478-6_22

  7. Chorianopoulos K, Fernández FJB, Salcines EG, de Castro Lozano C (2010) Delegating the visual interface between a tablet and a TV. In: Proceedings of the international conference on advanced visual interfaces, ACM, New York, NY, USA, AVI ’10, pp 418–418. doi:10.1145/1842993.1843096

  8. Courtois C, D’heer E (2012) Second screen applications and tablet users: constellation, awareness, experience, and interest. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 153–156. doi:10.1145/2325616.2325646

  9. Cummins R, Tirumala L, Lellis J (2011) Viewer attention to espns mosaic screen: an eye-tracking investigation. J Sports Media 6(1):23–54

    Article  Google Scholar 

  10. Dezfuli N, Khalilbeigi M, Huber J, Müller F, Mühlhäuser M (2012) Palmrc: imaginary palm-based remote control for eyes-free television interaction. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 27–34. doi:10.1145/2325616.2325623

  11. Eriksen CW, Hoffman JE (1972) Temporal and spatial characteristics of selective encoding from visual displays. Percept Psychophys 12(2):201–204. doi:10.3758/BF03212870

    Article  MATH  Google Scholar 

  12. Eriksen CW, St James JD (1986) Visual attention within and around the field of focal attention: a zoom lens model. Percept Psychophys 40(4):225–240. doi:10.3758/BF03211502

    Article  Google Scholar 

  13. Fitts PM (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exper Psychol 47(6):381–391. doi:10.1037/h0055392

    Article  Google Scholar 

  14. Forlines C, Shen C, Wigdor D, Balakrishnan R (2006) Exploring the effects of group size and display configuration on visual search. In: Proceedings of the 2006 20th anniversary conference on computer supported cooperative work, ACM, New York, NY, USA, CSCW ’06, pp 11–20. doi:10.1145/1180875.1180878

  15. Frintrop S, Rome E, Christensen HI (2010) Computational visual attention systems and their cognitive foundations: a survey. ACM Trans Appl Percept 7(1):6:1–6:39. doi:10.1145/1658349.1658355

    Article  Google Scholar 

  16. Geerts D, Cesar P, Bulterman D (2008) The implications of program genres for the design of social television systems. In: Proceedings of the 1st international conference on designing interactive user experiences for TV and video, ACM, New York, NY, USA, UXTV ’08, pp 71–80. doi:10.1145/1453805.1453822

  17. Geerts D, Leenheer R, De Grooff D, Negenman J, Heijstraten S (2014) In front of and behind the second screen: viewer and producer perspectives on a companion app. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 95–102. doi:10.1145/2602299.2602312

  18. Grüninger J, Krüger J (2013) The impact of display bezels on stereoscopic vision for tiled displays. In: Proceedings of the 19th ACM symposium on virtual reality software and technology, ACM, New York, NY, USA, VRST ’13, pp 241–250. doi:10.1145/2503713.2503717

  19. Hawkins R, Pingree S, Hitchon J, Radler B, Gorham B, Kahlor L, Gilligan E, Serlin R, Schmidt T, Kannaovakun P, Kolbeins G (2005) What produces television attention and attention style? Genre, situation, and individual differences as predictors. Human Commun Res 31(1):162–187

    Google Scholar 

  20. Holmes ME, Josephson S, Carney RE (2012) Visual attention to television programs with a second-screen application. In: Proceedings of the symposium on eye tracking research and applications, ACM, New York, NY, USA, ETRA ’12, pp 397–400. doi:10.1145/2168556.2168646

  21. Hutchings D (2012) An investigation of Fitts’ law in a multiple-display environment. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’12, pp 3181–3184. doi:10.1145/2207676.2208736

  22. Jones B, Sodhi R, Murdock M, Mehra R, Benko H, Wilson A, Ofek E, MacIntyre B, Raghuvanshi N, Shapira L (2014) Roomalive: magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, ACM, New York, NY, USA, UIST ’14, pp 637–644. doi:10.1145/2642918.2647383

  23. Jones BR, Benko H, Ofek E, Wilson AD (2013) Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’13, pp 869–878. doi:10.1145/2470654.2466112

  24. Kallenbach J, Narhi S, Oittinen P (2007) Effects of extra information on TV viewers’ visual attention, message processing ability, and cognitive workload. Comput Entertain 5(2). doi:10.1145/1279540.1279548

  25. Miller G (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 63:81–97

    Article  Google Scholar 

  26. Nielsen (2014) http://www.nielsen.com/us/en/insights/reports/2014/the-us-digital-consumer-report.html

  27. Posner M (1980) Orienting of attention. Q J Exper Psychol 32:3–25

    Article  Google Scholar 

  28. Posner M, Cohen Y (1984) Components of visual orienting. Atten Perform X Control Lang Process 32:531–556

    Google Scholar 

  29. Potter M (1976) Short-term conceptual memory for pictures. J Exper Psychol 2:509–522

    Google Scholar 

  30. Rashid U, Nacenta MA, Quigley A (2012) The cost of display switching: a comparison of mobile, large display and hybrid UI configurations. In: Proceedings of the international working conference on advanced visual interfaces, ACM, New York, NY, USA, AVI ’12, pp 99–106. doi:10.1145/2254556.2254577

  31. Rashid U, Nacenta MA, Quigley A (2012) Factors influencing visual attention switch in multi-display user interfaces: a survey. In: Proceedings of the 2012 international symposium on pervasive displays, ACM, New York, NY, USA, PerDis ’12, pp 1:1–1:6. doi:10.1145/2307798.2307799

  32. Reichle E, Rayner K, Pollatsek A (2004) The e-z reader model of eye-movement control in reading: comparisons to other models. J Behav Brain Sci 26:445–476

    Google Scholar 

  33. Riche N, Mancas M, Culibrk D, Crnojevic V, Gosselin B, Dutoit T (2013) Dynamic saliency models and human attention: a comparative study on videos. In: Proceedings of the 11th Asian conference on computer vision—volume part III, Springer, Berlin, ACCV’12, pp 586–598. doi:10.1007/978-3-642-37431-9_45

  34. Schwarz J, Klionsky D, Harrison C, Dietz P, Wilson A (2012) Phone as a pixel: enabling ad-hoc, large-scale displays using mobile devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’12, pp 2235–2238. doi:10.1145/2207676.2208378

  35. Sohlberg M, Mateer C (1989) Introduction to cognitive rehabilitation: theory and practice. Guilford Press, New York

    Google Scholar 

  36. Tan DS, Czerwinski M (2003) Effects of visual separation and physical discontinuities when distributing information across multiple displays. In: Proceedings of the IFIP TC.13 international conference on human–computer interaction, INTERACT ’03, pp 252–255

  37. Theeuwes J (1991) Exogenous and endogenous control of attention: the effect of visual onsets and offsets. Percept Psychophys 49(1):83–90. doi:10.3758/BF03211619

    Article  Google Scholar 

  38. Thomas BH (2012) A survey of visual, mixed, and augmented reality gaming. Comput Entertain 10(3):3:1–3:33. doi:10.1145/2381876.2381879

    Article  Google Scholar 

  39. Toet A (2011) Computational versus psychophysical bottom–up image saliency: a comparative evaluation study. IEEE Trans Pattern Anal Mach Intell 33(11):2131–2146

    Article  Google Scholar 

  40. Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12(1):97–136

    Article  Google Scholar 

  41. Vatavu RD (2012) Point & click mediated interactions for large home entertainment displays. Multimed Tools Appl 59(1):113–128. doi:10.1007/s11042-010-0698-5

    Article  Google Scholar 

  42. Vatavu RD (2012) User-defined gestures for free-hand TV control. In: Proceedings of the 10th European conference on interactive TV and video, ACM, New York, NY, USA, EuroiTV ’12, pp 45–48. doi:10.1145/2325616.2325626

  43. Vatavu RD (2013) A comparative study of user-defined handheld vs. freehand gestures for home entertainment environments. J Ambient Intell Smart Environ 5(2):187–211. doi:10.3233/AIS-130200

    Google Scholar 

  44. Vatavu RD (2013) There’s a world outside your TV: exploring interactions beyond the physical TV screen. In: Proceedings of the 11th European conference on interactive TV and video, ACM, New York, NY, USA, EuroITV ’13, pp 143–152. doi:10.1145/2465958.2465972

  45. Vatavu RD, Mancas M (2013) Interactive TV potpourris: an overview of designing multi-screen TV installations for home entertainment. In: Mancas M, d’Alessandro N, Siebert X, Gosselin B, Valderrama C, Dutoit T (eds) Intelligent technologies for interactive entertainment. Lecture notes of the institute for computer sciences, social informatics and telecommunications engineering, vol 124. Springer International Publishing, pp 49–54. doi:10.1007/978-3-319-03892-6_6

  46. Vatavu RD, Mancas M (2014) Visual attention measures for multi-screen TV. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 111–118. doi:10.1145/2602299.2602305

  47. Vatavu RD, Pentiuc SG (2008) Interactive coffee tables: interfacing TV within an intuitive, fun and shared experience. In: Proceedings of the 6th European interactive TV conference, Springer, EuroITV ’08, pp 183–187

  48. Vatavu RD, Wobbrock JO (2015) Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’15, pp 1325–1334. doi:10.1145/2702123.2702223

  49. Vatavu RD, Zaiti IA (2014) Leap gestures for TV: insights from an elicitation study. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video, ACM, New York, NY, USA, TVX ’14, pp 131–138. doi10.1145/2602299.2602316

  50. Wagner J, Nancel M, Gustafson SG, Huot S, Mackay WE (2013) Body-centric design space for multi-surface interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’13, pp 1299–1308. doi:10.1145/2470654.2466170

  51. Wallace JR, Vogel D, Lank E (2014) Effect of bezel presence and width on visual search. In: Proceedings of the international symposium on pervasive displays, ACM, New York, NY, USA, PerDis ’14, pp 118:118–118:123. doi:10.1145/2611009.2611019

  52. Wallace JR, Vogel D, Lank E (2014) The effect of interior bezel presence and width on magnitude judgement. In: Proceedings of the 2014 graphics interface conference, Canadian Information Processing Society, Toronto, Ont., Canada, Canada, GI ’14, pp 175–182. http://dl.acm.org/citation.cfm?id=2619648.2619678

  53. Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input. In: CHI ’05 extended abstracts on human factors in computing systems, ACM, New York, NY, USA, CHI EA ’05, pp 1869–1872. doi:10.1145/1056808.1057043

Download references

Acknowledgments

This work was partially supported from the project “Integrated Center for research, development and innovation in Advanced Materials, Nanotechnologies, and Distributed Systems for fabrication and control”, Contract No. 671/09.04.2015, Sectoral Operational Program for Increase of the Economic Competitiveness co-funded from the European Regional Development Fund.

Author information

Authors and Affiliations

  1. Department of Computer Science, Electronics, and Automation and Integrated Center for Research, Development, and Innovation in Advanced Materials, Nanotechnologies, and Distributed Systems for Fabrication and Control (MANSiD), University Stefan cel Mare of Suceava, 720229, Suceava, Romania

    Radu-Daniel Vatavu

  2. University of Mons, 20, Place du Parc, 7000, Mons, Belgium

    Matei Mancas

Authors
  1. Radu-Daniel Vatavu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matei Mancas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Radu-Daniel Vatavu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vatavu, RD., Mancas, M. Evaluating visual attention for multi-screen television: measures, toolkit, and experimental findings. Pers Ubiquit Comput 19, 781–801 (2015). https://doi.org/10.1007/s00779-015-0862-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-015-0862-z

Keywords

Search

Navigation