Cognitive Therapy and Research

, Volume 41, Issue 4, pp 588–599 | Cite as

How Well Can We Measure Visual Attention? Psychometric Properties of Manual Response Times and First Fixation Latencies in a Visual Search Paradigm

  • Richard WermesEmail author
  • Tania M. Lincoln
  • Sylvia Helbig-Lang
Original Article


Attentional biases are considered to be important for the development and maintenance of anxiety disorders, but despite widespread research in this area, the psychometric properties of measures targeting attentional allocation processes have rarely been evaluated. The current study assessed the reliability and validity of manual response times and first fixation latencies within a visual search paradigm, including data from one hundred twenty-two participants (n = sixty-two participants with social anxiety disorder). We found raw manual response times to be highly reliable, while raw first fixation latencies, in most cases, were not. Bias scores were neither reliable for manual responses nor for first fixation latencies. However, we found indicators of convergent validity, as raw values and also bias scores of both measures were significantly correlated. These results raise doubts about our ability to accurately measure visual attention, especially via eye-tracking procedures.


Attentional biases Psychometric properties Visual search Eye movements Social anxiety disorder 



Financial support for this study was provided by a grant from the German Research Foundation provided to the last author.

Compliance with Ethical Standards

Conflict of interest

Richard Wermes, Tania M. Lincoln and Sylvia Helbig-Lang declare that they have no conflicts of interest.

Research involving Human and Animal Rights

No animal studies were carried out for this article.

Informed Consent

All study procedures were in accordance with ethical standards of the responsible ethics committee. All participants provided a written informed consent before they were enrolled in the study.

Supplementary material

10608_2016_9830_MOESM1_ESM.pdf (195 kb)
Supplementary material 1 (PDF 195 KB)


  1. Amir, N., Beard, C., Taylor, C. T., Klumpp, H., Elias, J., Burns, M., & Chen, X. (2009). Attention training in individuals with generalized social phobia: a randomized controlled trial. Journal of Consulting and Clinical Psychology, 77(5), 961–973. doi:  10.1037/a0016685.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Amir, N., Elias, J., Klumpp, H., & Przeworski, A. (2003). Attentional bias to threat in social phobia: Facilitated processing of threat or difficulty disengaging attention from threat? Behaviour Research and Therapy, 41(11), 1325–1335. doi:  10.1016/S0005-7967(03)00039-1.CrossRefPubMedGoogle Scholar
  3. Baker, S. L., Heinrichs, N., Kim, H. J., & Hofmann, S. G. (2002). The Liebowitz social anxiety scale as a self-report instrument: A preliminary psychometric analysis. Behaviour Research and Therapy, 40(6), 701–715. doi:  10.1016/S0005-7967(01)00060-2.CrossRefPubMedGoogle Scholar
  4. Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007). Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study. Psychological Bulletin, 133(1), 1–24. doi:  10.1037/0033-2909.133.1.1.CrossRefPubMedGoogle Scholar
  5. Boettcher, J., Leek, L., Matson, L., Holmes, E. A., Browning, M., MacLeod, C., Andersson, G., & Carlbring, P. (2013). Internet-based attention bias modification for social anxiety: a randomised controlled comparison of training towards negative and training towards positive cues. PLoS ONE, 8(9), 1–14. doi:  10.1371/journal.pone.0071760.CrossRefGoogle Scholar
  6. Bradley, B. P., Mogg, K., White, J., Groom, C., & de Bono, J. (1999). Attentional bias for emotional faces in generalized anxiety disorder. The British Journal of Clinical Psychology/The British Psychological Society, 38(3), 267–278. doi:  10.1348/014466599162845.CrossRefGoogle Scholar
  7. Brosschot, J. F., De Ruiter, C., & Kindt, M. (1999). Processing bias in anxious subjects and repressors, measured by emotional Stroop interference and attentional allocation. Personality and Individual Differences, 26(5), 777–793. doi:  10.1016/S0191-8869(98)00173-1.CrossRefGoogle Scholar
  8. Brown, H. M., Eley, T. C., Broeren, S., MacLeod, C., Rinck, M., Hadwin, J. A., & Lester, K. J. (2014). Psychometric properties of reaction time based experimental paradigms measuring anxiety-related information-processing biases in children. Journal of Anxiety Disorders, 28(1), 97–107. doi:  10.1016/j.janxdis.2013.11.004.CrossRefPubMedGoogle Scholar
  9. Byrne, A., & Eysenck, M. W. (1995). Trait anxiety, anxious mood, and threat detection. Cognition and Emotion, 9(6), 549–562. doi:  10.1080/02699939508408982.CrossRefGoogle Scholar
  10. Chen, N. T. M., Clarke, P. J. F., MacLeod, C., & Guastella, A. J. (2012). Biased attentional processing of positive stimuli in social anxiety disorder: an eye movement study. Cognitive Behaviour Therapy, 41(2), 96–107. doi:  10.1080/16506073.2012.666562.CrossRefPubMedGoogle Scholar
  11. Chen, N. T. M., Clarke, P. J. F., Watson, T. L., MacLeod, C., & Guastella, A. J. (2014). Biased saccadic responses to emotional stimuli in anxiety: an antisaccade study. PLoS ONE, 9(2), 1–7. doi: 10.1371/journal.pone.0086474.
  12. Cisler, J. M., & Koster, E. H. W. (2010). Mechanisms of attentional biases towards threat in anxiety disorders: an integrative review. Clinical Psychology Review, 30(2), 203–216. doi:  10.1016/j.cpr.2009.11.003.CrossRefPubMedGoogle Scholar
  13. Cooper, R. M., Bailey, J. E., Diaper, A., Stirland, R., Renton, L. E., Benton, C. P., … & Munafo, M. R. (2011). Effects of 7.5% CO2 inhalation on allocation of spatial attention to facial cues of emotional expression. Cognition and Emotion, 25(4), 626–638. doi:  10.1177/0269881108096981.CrossRefPubMedGoogle Scholar
  14. Cristea, I. A., Kok, R. N., & Cuijpers, P. (2015). Efficacy of cognitive bias modification interventions in anxiety and depression: Meta-analysis. British Journal of Psychiatry, 206(1), 7–16. doi:  10.1192/bjp.bp.114.146761.CrossRefPubMedGoogle Scholar
  15. Dalgleish, T., Taghavi, R., Neshat-Doost, H., Moradi, A., Canterbury, R., & Yule, W. (2003). Patterns of processing bias for emotional information across clinical disorders: a comparison of attention, memory, and prospective cognition in children and adolescents with depression, generalized anxiety, and posttraumatic stress disorder. Journal of Clinical Child and Adolescent Psychology 32(1), 10–21. doi:  10.1207/S15374424JCCP3201_02.CrossRefPubMedGoogle Scholar
  16. Derakshan, N., & Koster, E. H. W. (2010). Processing efficiency in anxiety: Evidence from eye-movements during visual search. Behaviour Research and Therapy, 48(12), 1180–1185. doi:  10.1016/j.brat.2010.08.009.CrossRefPubMedGoogle Scholar
  17. Eastwood, J., Smile, D., Oakman, J., Farvolden, P., van Ameringen, M., Mancini, C., & Merikle, P. (2005). Individuals with social phobia are biased to become aware of negative faces. Visual Cognition, 12(1), 159–179. doi:  10.1080/13506280444000175.CrossRefGoogle Scholar
  18. Egloff, B., & Hock, M. (2003). Assessing attention allocation toward threat-related stimuli: A comparison of the emotional Stroop task and the attentional probe task. Personality and Individual Differences, 35(2), 475–483. doi:  10.1016/S0191-8869(02)00265-9.CrossRefGoogle Scholar
  19. Eide, P., Kemp, A., Silberstein, R. B., Nathan, P. J., & Stough, C. (2002). Test–retest reliability of the emotional stroop task: examining the paradox of measurement change. The Journal of Psychology, 136(5), 514–520. doi:  10.1080/00223980209605547.CrossRefPubMedGoogle Scholar
  20. Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: attentional control theory. Emotion, 7(2), 336–353. doi:  10.1037/1528-3542.7.2.336.CrossRefPubMedGoogle Scholar
  21. Fox, E., Russo, R., Bowles, R., & Dutton, K. (2001). Do threatening stimuli draw or hold visual attention in subclinical anxiety?. Journal of Experimental Psychology, 130, 681–700. doi:  10.1037/0096-3445.130.4.681.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Fresco, D. M., Coles, M. E., Heimberg, R. G., Liebowitz, M. R., Hami, S., Stein, M. B., & Goetz, D. (2001). The Liebowitz social anxiety scale: a comparison of the psychometric properties of self-report and clinician-administered formats. Psychological Medicine, 31(6), 1025–1035. doi:  10.1017/S0033291701004056.CrossRefPubMedGoogle Scholar
  23. Gamble, A. L., & Rapee, R. M. (2010). The time-course of attention to emotional faces in social phobia. Journal of Behavior Therapy and Experimental Psychiatry, 41(1), 39–44. doi:  10.1016/j.jbtep.2009.08.008.CrossRefPubMedGoogle Scholar
  24. Garner, M., Mogg, K., & Bradley, B. P. (2006). Orienting and maintenance of gaze to facial expressions in social anxiety. Journal of Abnormal Psychology, 115(4), 760–770. doi:  10.1037/0021-843X.115.4.760.CrossRefPubMedGoogle Scholar
  25. Gilboa-Schechtman, E., Foa, E. B., & Amir, N. (1999). Attentional biases for facial expressions in social phobia: The face-in-the-crowd paradigm. Cognition and Emotion, 13(3), 305–318. doi:  10.1080/026999399379294.CrossRefGoogle Scholar
  26. Gotlib, I. H., Kasch, K. L., Traill, S., Joormann, J., Arnow, B. A., & Johnson, S. L. (2004). Coherence and specificity of information-processing biases in depression and social phobia. Journal of Abnormal Psychology, 113(3), 386–398. doi:  10.1037/0021-843X.113.3.386.CrossRefPubMedGoogle Scholar
  27. Heeren, A., Mogoase, C., Philippot, P., & McNally, R. J. (2015). Attention bias modification for social anxiety: A systematic review and meta-analysis. Clinical Psychology Review, 40, 76–90. doi:  10.1016/j.cpr.2015.06.001.CrossRefPubMedGoogle Scholar
  28. Hoffman, J. E. (1998). Visual attention and eye movements. Attention, 384, 119–153.Google Scholar
  29. Juth, P., Lundqvist, D., Karlsson, A., Ohman, A., & Öhman, A. (2005). Looking for foes and friends: perceptual and emotional factors when finding a face in the crowd. Emotion, 5(4), 379–395. doi:  10.1037/1528-3542.5.4.379.CrossRefPubMedGoogle Scholar
  30. Kendall, P. C., Finch, A. J. Jr., Auerbach, S. M., Hooke, J. F., Mikulka, P. J. (1976). The state-trait anxiety inventory: a systematic evaluation. Journal of Consulting and Clinical Psychology, 44(3), 406–412. doi:  10.1037/0022-006X.44.3.406.CrossRefPubMedGoogle Scholar
  31. Koster, E. H. W., Crombez, G., Verschuere, B., & De Houwer, J. (2006). Attention to threat in anxiety-prone individuals: Mechanisms underlying attentional bias. Cognitive Therapy and Research, 30(5), 635–643. doi:  10.1007/s10608-006-9042-9.CrossRefGoogle Scholar
  32. Kowler, E., Anderson, E., Dosher, B., & Blaser, E. (1995). The role of attention in the programming of saccades. Vision Research, 35(13), 1897–1916. doi:  10.1016/0042-6989(94)00279-U.CrossRefPubMedGoogle Scholar
  33. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377–1388. doi:  10.1080/02699930903485076.CrossRefGoogle Scholar
  34. Laux, L., Glanzmann, P., Schaffner, P., & Spielberger, C. D. (1981). State-Trait-Anxiety-Inventory (STAI). Theoretical Foundations and Instructions. Weinheim: Beltz.Google Scholar
  35. MacLeod, J. W., Lawrence, M. A., McConnell, M. M., Eskes, G. A., Klein, R. M., & Shore, D. I. (2010). Appraising the ANT: Psychometric and theoretical considerations of the attention network test. Neuropsychology, 24(5), 637–651. doi:  10.1037/a0019803.CrossRefPubMedGoogle Scholar
  36. Mansell, W., Clark, D. M., Ehlers, A., & Chen, Y.-P. (1999). Social anxiety and attention away from emotional faces. Cognition and Emotion, 13(6), 673–690. doi:  10.1080/026999399379032.CrossRefGoogle Scholar
  37. Miskovic, V., & Schmidt, L. A. (2012). Social fearfulness in the human brain. Neuroscience and Biobehavioral Reviews, 36(1), 459–478. doi:  10.1016/j.neubiorev.2011.08.002.CrossRefPubMedGoogle Scholar
  38. Mogg, K., Bradley, B., Miles, F., & Dixon, R. (2004). Time course of attentional bias for threat scenes: testing the vigilance-avoidance hypothesis. Cognition and Emotion, 18(5), 689–700. doi:  10.1080/02699930341000158.CrossRefGoogle Scholar
  39. Mogg, K., Bradley, B. P., Dixon, C., Fisher, S., Twelftree, H., & McWilliams, A. (2000). Trait anxiety, defensiveness and selective processing of threat: an investigation using two measures of attentional bias. Personality and Individual Differences, 28(1), 1063–1077. doi:  10.1016/S0191-8869(99)00157-9.CrossRefGoogle Scholar
  40. Mogg, K., Millar, N., & Bradley, B. P. (2000). Biases in eye movements to threatening facial expressions in generalized anxiety disorder and depressive disorder. Journal of Abnormal Psychology, 109(4), 695–704. doi:  10.1037//0021-843X.CrossRefPubMedGoogle Scholar
  41. Murphy, K. R., & Davidshofer, C. O. (2005). Psychological testing: principles and applications. New Jersey: Upper Saddle River.Google Scholar
  42. Neubauer, K., von Auer, M., Murray, E., Petermann, F., Helbig-Lang, S., & Gerlach, A. L. (2013). Internet-delivered attention modification training as a treatment for social phobia: A randomized controlled trial. Behaviour Research and Therapy, 51(2), 87–97. doi:  10.1016/j.brat.2012.10.006.CrossRefPubMedGoogle Scholar
  43. Nothdurft, H. C. (1999). Focal attention in visual search. Vision Research, 39(14), 2305–2310. doi:  10.1016/S0042-6989(99)00006-1.CrossRefPubMedGoogle Scholar
  44. Ohman, A., Flykt, A., Esteves, F., Öhman, A., Flykt, A., & Esteves, F. (2001). Emotion drives attention: detecting the snake in the grass. Journal of Experimental Psychology, 130(3), 466–478. doi:  10.1037/0096-3445.130.3.466.CrossRefPubMedGoogle Scholar
  45. Pflugshaupt, T., Mosimann, U. P., Von Wartburg, R., Schmitt, W., Nyffeler, T., & Müri, R. M. (2005). Hypervigilance-avoidance pattern in spider phobia. Journal of Anxiety Disorders, 19(1), 105–116. doi:  10.1016/j.janxdis.2003.12.002.CrossRefPubMedGoogle Scholar
  46. Pishyar, R., Harris, L. M., & Menzies, R. G. (2004). Attentional bias for words and faces in social anxiety. Anxiety, Stress and Coping, 17(1), 23–36. doi:  10.1080/10615800310001601458.CrossRefGoogle Scholar
  47. Quigley, L., Nelson, A. L., Carriere, J., Smilek, D., & Purdon, C. (2012). The effects of trait and state anxiety on attention to emotional images: An eye-tracking study. Cognition and Emotion, 26, 1–22. doi:  10.1080/02699931.2012.662892.CrossRefGoogle Scholar
  48. Rapee, R. M., & Heimberg, R. G. (1997). A cognitive-behavioral model of anxiety in social phobia. Behaviour Research and Therapy, 35(8), 741–756. doi:  10.1016/S0005-7967(97)00022-3.CrossRefPubMedGoogle Scholar
  49. Rinck, M., Becker, E. S., Kellermann, J., & Roth, W. T. (2003). Selective attention in anxiety: distraction and enhancement in visual search. Depression and Anxiety, 18(1), 18–28. doi:  10.1002/da.10105.CrossRefPubMedGoogle Scholar
  50. Rinck, M., Reinecke, A., Ellwart, T., Heuer, K., & Becker, E. S. (2005). Speeded detection and increased distraction in fear of spiders: evidence from eye movements. Journal of Abnormal Psychology, 114(2), 235–248. doi:  10.1037/0021-843X.114.2.235.CrossRefPubMedGoogle Scholar
  51. Schmidt, N. B., Richey, J. A., Buckner, J. D., & Timpano, K. R. (2009). Attention training for generalized social anxiety disorder. Journal of Abnormal Psychology, 118(1), 5–14. doi:  10.1037/a0013643.CrossRefPubMedGoogle Scholar
  52. Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19(7), 595–605. doi:  10.1002/per.554.CrossRefGoogle Scholar
  53. Skre, I., Onstad, S., Torgensen, S., & Kringlen, E. (1991). High interrater reliability for the structured clinical interview for DSM-III-R axis I (SCID-I). Acta Psychiatrica Scandinavica, 84(2), 167–173. doi:  10.1111/j.1600-0447.1991.tb03123.x.CrossRefPubMedGoogle Scholar
  54. Sposari, J. A., & Rapee, R. M. (2007). Attentional bias toward facial stimuli under conditions of social threat in socially phobic and nonclinical participants. Cognitive Therapy and Research, 31(1), 23–37. doi:  10.1007/s10608-006-9073-2.CrossRefGoogle Scholar
  55. Stangier, U., & Heidenreich, T. (2005). Liebowitz Soziale Angst-Skala (LSAS) [Liebowitz Social Anxiety Scale]. In Collegium Internationale Psychiatriae Scalarum (Ed.), Internationale Skalen für Psychiatrie (pp. 299–305). Göttingen: Hogrefe.Google Scholar
  56. Staugaard, S. R. (2009). Reliability of two versions of the dot-probe task using photographic faces. Psychology Science Quarterly, 51(3), 339–350.
  57. Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014). Measuring attentional bias to threat: Reliability of dot probe and eye movement indices. Cognitive Therapy and Research, 38(3), 313–333. doi:  10.1007/s10608-013-9588-2.CrossRefGoogle Scholar
  58. Waechter, S., & Stolz, J. A. (2015). Trait anxiety, state anxiety, and attentional bias to threat: assessing the psychometric properties of response time measures. Cognitive Therapy and Research, 39(4), 441–458. doi:  10.1007/s10608-015-9670-z.CrossRefGoogle Scholar
  59. Williams, J. M., Mathews, A., & MacLeod, C. (1996). The emotional Stroop task and psychopathology. Psychological Bulletin, 120(1), 3–24. doi:  10.1037/0033-2909.120.1.3.CrossRefPubMedGoogle Scholar
  60. Wittchen, H.-U., Zaudig, M., & Fydrich, T. H. (1997). SKID - Strukturiertes Klinisches Interview für DSM-IV. Achse I und II Handanweisungen [Structured Clinical Interview for DSM-IV]. Göttingen: Hogrefe. doi:  10.1026//0084-5345.28.1.68.Google Scholar
  61. Zanarini, M. C., Skodol, A. E., Bender, D., Dolan, R., Sanislow, C., Schaefer, E., … Gundersono, J. G. (2000). The collaborative longitudinal personality disorders study: Reliability of axis I and II diagnoses. Journal of Personality Disorders, 14(4), 291–299. doi:  10.1521/pedi.2000.14.4.291.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Clinical Psychology and PsychotherapyUniversity of HamburgHamburgGermany

Personalised recommendations