Skip to main content

Focusing on the face or getting distracted by social signals? The effect of distracting gestures on attentional focus in natural interaction

Abstract

Attentional orienting towards others’ gaze direction or pointing has been well investigated in laboratory conditions. However, less is known about the operation of attentional mechanisms in online naturalistic social interaction scenarios. It is equally plausible that following social directional cues (gaze, pointing) occurs reflexively, and/or that it is influenced by top-down cognitive factors. In a mobile eye-tracking experiment, we show that under natural interaction conditions, overt attentional orienting is not necessarily reflexively triggered by pointing gestures or a combination of gaze shifts and pointing gestures. We found that participants conversing with an experimenter, who, during the interaction, would play out pointing gestures as well as directional gaze movements, continued to mostly focus their gaze on the face of the experimenter, demonstrating the significance of attending to the face of the interaction partner—in line with effective top-down control over reflexive orienting of attention in the direction of social cues.

This is a preview of subscription content, access via your institution.

Fig. 1

Reproduced with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 2

Reproduced with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 3

Adapted with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 4

Adapted with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 5

Adapted with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 6

Adapted with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Fig. 7

Adapted with permission from “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences], by J. Kajopoulos, 2019, University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/

Data availability statement

The data are not publicly available due to them containing information that could compromise research participant privacy/consent, but upon reasonable cause a request may be made to the author [JK].

References

  1. Awh, E., Belopolsky, A. V., & Theeuwes, J. (2012). Top-down versus bottom-up attentional control: a failed theoretical dichotomy. Trends in Cognitive Sciences, 16(8), 437–443.

    PubMed  PubMed Central  Google Scholar 

  2. Bacon, W. F., & Egeth, H. E. (1994). Overriding stimulus-driven attentional capture. Perception & Psychophysics, 55(5), 485–496.

    Google Scholar 

  3. Baron-Cohen, S. (1995). Mindblindness: An essay on autism and the theory of mind. Boston: MIT Press/Bradford Books.

    Google Scholar 

  4. Birmingham, E., Bischof, W. F., & Kingstone, A. (2009). Get real! Resolving the debate about equivalent social stimuli. Visual Cognition, 17(6–7), 904–924.

    Google Scholar 

  5. Broekens, J., Heerink, M., & Rosendal, H. (2009). Assistive social robots in elderly care: A review. Gerontechnology, 8(2), 94–103.

    Google Scholar 

  6. Cabibihan, J.-J., Javed, H., Ang, M., Jr., & Aljunied, S. (2013). Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism. International Journal of Social Robotics, 5(4), 593–618.

    Google Scholar 

  7. Chapman, C. D., Benedict, C., & Schiöth, H. B. (2018). Experimenter gender and replicability in science. Science Advances, 4(1), e1701427.

    PubMed  PubMed Central  Google Scholar 

  8. Cole, G. G., Smith, D. T., & Atkinson, M. A. (2015). Mental state attribution and the gaze cueing effect. Attention, Perception, & Psychophysics, 77(4), 1105–1115.

    Google Scholar 

  9. Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., & Baron-Cohen, S. (1999). Gaze perception triggers reflexive visuospatial orienting. Visual Cognition, 6(5), 509–540.

    Google Scholar 

  10. Ellsworth, P. C., & Ludwig, L. M. (1972). Visual Behavior in social interaction. Journal of Communication, 22(4), 375–403.

    Google Scholar 

  11. Foulsham, T., Walker, E., & Kingstone, A. (2011). The where, what and when of gaze allocation in the lab and the natural environment. Vision Research, 51(17), 1920–1931.

    PubMed  Google Scholar 

  12. Freeth, M., Foulsham, T., & Kingstone, A. (2013). What affects social attention? Social presence, eye contact and autistic traits. PLoS One, 8(1), e53286.

    PubMed  PubMed Central  Google Scholar 

  13. Friesen, C. K., & Kingstone, A. (1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin & Review, 5(3), 490–495.

    Google Scholar 

  14. Friesen, C. K., Moore, C., & Kingstone, A. (2005). Does gaze direction really trigger a reflexive shift of spatial attention? Brain and Cognition, 57(1), 66–69.

    PubMed  Google Scholar 

  15. Gluckman, M., & Johnson, S. P. (2013). Attentional capture by social stimuli in young infants. Frontiers in Psychology, 4, 527.

    PubMed  PubMed Central  Google Scholar 

  16. Hamilton, S. J. (2017). The effects of pointing gestures on visual attention. University Honors Program Theses, 243.

  17. Hayward, D. A., Voorhies, W., Morris, J. L., Capozzi, F., & Ristic, J. (2017). Staring reality in the face: A comparison of social attention across laboratory and real world measures suggests little common ground. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 71(3), 212.

    PubMed  Google Scholar 

  18. Kajopoulos, J. (2019) Social cognition and robotics. (Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences). University library repository of the Ludwig-Maximilians-Universität München. https://edoc.ub.uni-muenchen.de/25034/.

  19. Kajopoulos, J., Wong, A. H. Y., Yuen, A. W. C., Dung, T. A., Kee, T. Y., & Wykowska, A. (2015). Robot-assisted training of joint attention skills in children diagnosed with autism. Social Robotics: 7th International Conference, ICSR 2015 (pp. 296–305). Paris, France.

  20. Kerzel, D., Zarian, L., & Souto, D. (2009). Involuntary cueing effects on accuracy measures: Stimulus and task dependence. Journal of Vision, 9(11), 1–16.

    PubMed  Google Scholar 

  21. Kleinke, C. L. (1986). Gaze and eye contact: a research review. Psychological Bulletin, 100(1), 78.

    PubMed  Google Scholar 

  22. Kompatsiari, K., Ciardo, F., Tikhanoff, V., Metta, A., & Wykowska, A. (2018a). On the role of eye contact in gaze cueing. Scientific Reports, 8, 17842.

    PubMed  PubMed Central  Google Scholar 

  23. Kompatsiari, K., Pérez-Osorio, J., De Tommaso, D., Metta, G., Wykowska, A. (2018b). Neuroscientifically-grounded research for improved human-robot interaction. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3403–3408). Madrid, Spain.

  24. Kuhn, G., & Kingstone, A. (2009). Look away! Eyes and arrows engage oculomotor responses automatically. Attention, Perception, & Psychophysics, 71(2), 314–327.

    Google Scholar 

  25. Langton, S. R., & Bruce, V. (2000). You must see the point: automatic processing of cues to the direction of social attention. Journal of Experimental Psychology: Human Perception and Performance, 26(2), 747.

    PubMed  Google Scholar 

  26. Macdonald, R. G., & Tatler, B. W. (2018). Gaze in a real-world social interaction: A dual eye-tracking study. Quarterly Journal of Experimental Psychology, 71(10), 2162–2173.

    Google Scholar 

  27. Moore, C., & Dunham, P. J. (2014). Joint attention: Its origins and role in development. New York: Psychology Press.

    Google Scholar 

  28. Müller, H. J., Geyer, T., Zehetleitner, M., & Krummenacher, J. (2009). Attentional capture by salient color singleton distractors is modulated by top-down dimensional set. Journal of Experimental Psychology: Human Perception and Performance, 35(1), 1–16.

    PubMed  Google Scholar 

  29. Mundy, P. (2017). A review of joint attention and social-cognitive brain systems in typical development and autism spectrum disorder. European Journal of Neuroscience, 47(6), 497–514.

    Google Scholar 

  30. Perez-Osorio, J., Müller, H. J., Wiese, E., & Wykowska, A. (2015). Gaze following is modulated by expectations regarding others’ action goals. PLoS One, 10(11), e0143614.

    PubMed  PubMed Central  Google Scholar 

  31. Prinzmetal, W., McCool, C., & Park, S. (2005). Attention: reaction time and accuracy reveal different mechanisms. Journal of Experimental Psychology: General, 134(1), 73.

    Google Scholar 

  32. Ricciardelli, P., Bricolo, E., Aglioti, S. M., & Chelazzi, L. (2002). My eyes want to look where your eyes are looking: exploring the tendency to imitate another individual’s gaze. NeuroReport, 13(17), 2259–2264.

    PubMed  Google Scholar 

  33. Ristic, J., & Kingstone, A. (2005). Taking control of reflexive social attention. Cognition International Journal of Cognitive Science, 94(3), B55–B65.

    Google Scholar 

  34. Schellen, E., & Wykowska, A. (2019). Intentional mindset toward robots—open questions and methodological challenges. Frontiers in Robotics and AI, 5, 139.

    PubMed  PubMed Central  Google Scholar 

  35. Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley, K. (2013). Toward a second-person neuroscience. Behavioral and Brain Sciences, 36(4), 393–414.

    Google Scholar 

  36. Skarratt, P. A., Cole, G. G., & Kingstone, A. (2010). Social inhibition of return. Acta Psychologica, 134(1), 48–54.

    PubMed  Google Scholar 

  37. Teufel, C., Fletcher, P. C., & Davis, G. (2010). Seeing other minds: attributed mental states influence perception. Trends in Cognitive Sciences, 14(8), 376–382.

    PubMed  Google Scholar 

  38. Theeuwes, J. (1994). Stimulus-driven capture and attentional set: Selective search for color and visual abrupt onsets. Journal of Experimental Psychology: Human Perception and Performance, 20(4), 799–806.

    PubMed  Google Scholar 

  39. Theeuwes, J. (2004). Top-down search strategies cannot override attentional capture. Psychonomic Bulletin & Review, 11(1), 65–70.

    Google Scholar 

  40. Theeuwes, J. (2010). Top-down and bottom-up control of visual selection. Acta Psychologica, 135(2), 77–99.

    PubMed  Google Scholar 

  41. Theeuwes, J., Atchley, P., & Kramer, A. F. (2000). On the time course of top-down and bottom-up control of visual attention. In S. M. J. Driver (Ed.), Attention and Performance, 18 (pp. 105–125). Cambridge: MIT Press.

    Google Scholar 

  42. Tomasello, M. (1995). Joint attention as social cognition. In C. Moore & P. J. Dunham (Eds.), Joint attention: Its origins and role in development (pp. 103–130). USA: Lawrence Erlbaum Associates Inc.

    Google Scholar 

  43. Wiese, E., Metta, G., & Wykowska, A. (2017). Robots as intentional agents: Using neuroscientific methods to make robots appear more social. Frontiers in Psychology, 8, 1663.

    PubMed  PubMed Central  Google Scholar 

  44. Wiese, E., Wykowska, A., & Müller, H. J. (2014). What we observe is biased by what other people tell us: Beliefs about the reliability of gaze behavior modulate attentional orienting to gaze cues. PLoS One, 9(4), e94529.

    PubMed  PubMed Central  Google Scholar 

  45. Wiese, E., Wykowska, A., Zwickel, J., & Müller, H. J. (2012). I see what you mean: how attentional selection is shaped by ascribing intentions to others. PLoS One, 7(9), e45391.

    PubMed  PubMed Central  Google Scholar 

  46. Wiese, E., Zwickel, J., & Müller, H. J. (2013). The importance of context information for the spatial specificity of gaze cueing. Attention, Perception, & Psychophysics, 75(5), 967–982.

    Google Scholar 

  47. Willemse, C., & Wykowska, A. (2019). In natural interaction with embodied robots we prefer it when they follow our gaze: a gaze-contingent mobile eyetracking study. Philosophical Transactions of the Royal Society B, 374, 20180036.

    Google Scholar 

  48. Wykowska, A., Chaminade, T., & Cheng, G. (2016). Embodied artificial agents for understanding human social cognition. Philosophical Transactions of the Royal Society London: B. Biological Sciences, 371, 20150375.

    Google Scholar 

  49. Wykowska, A., Kajopoulos, J., Obando-Leitón, M., Chauhan, S., Cabibihan, J.-J., & Cheng, G. (2015). Humans are well tuned to detecting agents among non-agents: Examining the sensitivity of human perception to behavioral characteristics of intentional systems. International Journal of Social Robotics, 7(5), 767–781.

    Google Scholar 

  50. Wykowska, A., & Schubö, A. (2010). On the temporal relation of top-down and bottom-up mechanisms during guidance of attention. Journal of Cognitive Neuroscience, 22(4), 640–654.

    PubMed  Google Scholar 

  51. Wykowska, A., Wiese, E., Prosser, A., & Müller, H. J. (2014). Beliefs about the minds of others influence how we process sensory information. PLoS One, 9(4), e94339.

    PubMed  PubMed Central  Google Scholar 

  52. Zehetleitner, M., Hegenloh, M., & Müller, H. J. (2011). Visually guided pointing movements are driven by the salience map. Journal of Vision, 11(1):1–18.

    Google Scholar 

  53. Zehetleitner, M., Krummenacher, J., & Müller, H. J. (2009). The detection of feature singletons defined in two dimensions is based on salience summation, rather than on serial exhaustive or interactive race architectures. Attention, Perception, & Psychophysics, 71(8), 1739–1759.

    Google Scholar 

Download references

Acknowledgements

We thank the Japan Society for the Promotion of Science (JSPS) for supporting Jasmin Kajopoulos as an International Research Fellow. We are grateful towards Ayano Okoso and several students from the OPU, who supported us with data collection and participant recruitment. The content of this article and the figures of this article have been extracted, reproduced and partly adapted (see figure captions in particular) with permission from the doctoral thesis (mainly from Chapter 1) of Kajopoulos, J. (2019), “Social cognition and robotics” [Doctoral thesis, Ludwig-Maximilians-Universität München: Graduate School of Systemic Neurosciences] published at the university library repository of the Ludwig-Maximilians-Universität München, accessible at https://edoc.ub.uni-muenchen.de/25034/.

Funding

This research was supported by the Deutsche Forschungsgemeinschaft (WY-122/1-1, awarded to AW; MU 773/16-2, awarded to HJM ), the Japan Society for the Promotion of Science (JSPS Summer Program, awarded to JK), the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program: Grant ERC-2016-StG-715058 awarded to AW.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jasmin Kajopoulos.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 23 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kajopoulos, J., Cheng, G., Kise, K. et al. Focusing on the face or getting distracted by social signals? The effect of distracting gestures on attentional focus in natural interaction. Psychological Research 85, 491–502 (2021). https://doi.org/10.1007/s00426-020-01383-4

Download citation