Exploitation of Gaze Data for Photo Region Labeling in an Immersive Environment

  • Tina Walber
  • Ansgar Scherp
  • Steffen Staab
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8325)


Metadata describing the content of photos are of high importance for applications like image search or as part of training sets for object detection algorithms. In this work, we apply tags to image regions for a more detailed description of the photo semantics. This region labeling is performed without additional effort from the user, just from analyzing eye tracking data, recorded while users are playing a gaze-controlled game. In the game EyeGrab, users classify and rate photos falling down the screen. The photos are classified according to a given category under time pressure. The game has been evaluated in a study with 54 subjects. The results show that it is possible to assign the given categories to image regions with a precision of up to 61%. This shows that we can perform an almost equally good region labeling using an immersive environment like in EyeGrab compared to a previous classification experiment that was much more controlled.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arbeláez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE TPAMI 33(5), 898–916 (2011)CrossRefGoogle Scholar
  2. 2.
    Goldberg, J.H., Stimson, M.J., Lewenstein, M., Scott, N., Wichansky, A.M.: Eye tracking in web search tasks: design implications. In: Symposium on Eye Tracking Research & Applications, pp. 51–58. ACM (2002)Google Scholar
  3. 3.
    Klami, A.: Inferring task-relevant image regions from gaze data. In: Workshop on Machine Learning for Signal Processing. IEEE (2010)Google Scholar
  4. 4.
    Kompatsiaris, I., Triantafyllou, E., Strintzis, M.G.: A World Wide Web region-based image search engine. In: Conference on Image Analysis and Processing (2001)Google Scholar
  5. 5.
    Kozma, L., Klami, A., Kaski, S.: Gazir: Gaze-based zooming interface for image retrieval. In: Multimodal Interfaces, pp. 305–312. ACM (2009)Google Scholar
  6. 6.
    Lin, Y.-T., Lin, R.-Y., Lin, Y.-C., Lee, G.C.: Real-time eye-gaze estimation using a low-resolution webcam. In: Multimedia Tools a. Applications (2013)Google Scholar
  7. 7.
    Ni, Y., Dong, J., Feng, J., Yan, S.: Purposive Hidden-Object-Game: Embedding Human Computation in Popular Game. IEEE Transactions on Multimedia 14(5), 1496–1507 (2012)CrossRefGoogle Scholar
  8. 8.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. International Journal of Computer Vision 77(1-3), 157–173 (2008)CrossRefGoogle Scholar
  9. 9.
    Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., Cohen, M.: Gaze-based interaction for semi-automatic photo cropping. In: CHI, p. 780. ACM (2006)Google Scholar
  10. 10.
    Smith, J.D., Graham, T.C.: Use of eye movements for video game control. In: Advances in Computer Entertainment Technology. ACM (2006)Google Scholar
  11. 11.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition. IEEE (2001)Google Scholar
  12. 12.
    Von Ahn, L., Liu, R., Blum, M.: Peekaboom: a game for locating objects in images. In: Human Factors in Computing Systems, pp. 55–64. ACM (2006)Google Scholar
  13. 13.
    Walber, T., Neuhaus, C., Scherp, A.: Eyegrab: A gaze-based game with a purpose to enrich image context information. In: EuroHCIR (2012)Google Scholar
  14. 14.
    Walber, T., Scherp, A., Staab, S.: Identifying objects in images from analyzing the users’s gaze movements for provided tags, pp. 138–148. Springer (2012)Google Scholar
  15. 15.
    Walber, T., Scherp, A., Staab, S.: Can you see it? two novel eye-tracking-based measures for assigning tags to image regions. In: Li, S., El Saddik, A., Wang, M., Mei, T., Sebe, N., Yan, S., Hong, R., Gurrin, C. (eds.) MMM 2013, Part I. LNCS, vol. 7732, pp. 36–46. Springer, Heidelberg (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Tina Walber
    • 1
  • Ansgar Scherp
    • 2
  • Steffen Staab
    • 1
  1. 1.University of Koblenz-LandauGermany
  2. 2.University of MannheimGermany

Personalised recommendations