Fleeting reliability in the dot-probe task

  • Angus Chapman
  • Christel Devue
  • Gina M. GrimshawEmail author
Original Article


In a dot-probe task, two cues—one emotional and one neutral—are followed by a probe in one of their locations. Faster responses to probes co-located with the emotional stimulus are taken as evidence of attentional bias. Several studies indicate that such attentional bias measures have poor reliability, even though ERP studies show that people reliably attend to the emotional stimulus. This inconsistency might arise because the emotional stimulus captures attention briefly (as indicated by ERP), but cues appear for long enough that attention can be redistributed before the probe onset, causing RT measures of bias to vary across trials. We tested this hypothesis by manipulating SOA (stimulus onset asynchrony between onset of the cues and onset of the probe) in a dot-probe task using angry and neutral faces. Across three experiments, the internal reliability of behavioural biases was significantly greater than zero when probes followed faces by 100 ms, but not when the SOA was 300, 500, or 900 ms. Thus, the initial capture of attention shows some level of consistency, but this diminishes quickly. Even at the shortest SOA internal reliability estimates were poor, and not sufficient to justify the use of the task as an index of individual differences in attentional bias.


Attention Dot probe Emotion Attentional bias Threat bias Reliability 



We thank Laura Kranz and Emma O’Brien for assistance with data collection. This research was supported by a Grant from the Royal Society of New Zealand Marsden Fund (VUW1307) to GG. Development of the MacBrain Face Stimulus Set was overseen by Nim Tottenham and supported by the John D. and Catherine T. MacArthur Foundation Research Network on Early Experience and Brain Development. Please contact Nim Tottenham at for more information concerning the stimulus set.

Compliance with ethical standards

Ethical standards

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Victoria University of Wellington Human Ethics Committee. Informed consent was obtained from all individual participants included in the study.

Data availability

The data sets generated and analysed for this study are not publicly available due to ethical constraints, but are available from the corresponding author on reasonable request.

Supplementary material

426_2017_947_MOESM1_ESM.docx (19 kb)
Supplementary material 1 (DOCX 18 KB)


  1. Amir, I., Zvielli, A., & Bernstein, A. (2016). De(coupling) of our eyes and our mind’s eye: A dynamic process perspective on attentional bias. Emotion, 16(7), 978–986. Scholar
  2. Bar-Haim, Y., Holoshitz, Y., Eldar, S., Frenkel, T. I., Muller, D., Charney, D. S., Pine, D. S., Fox, N. A., & Wald, I. (2010). Life-threatening danger and suppression of attention bias to threat. American Journal of Psychiatry, 167(6), 694–698. Scholar
  3. Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007). Threat-related attentional bias in anxious and non-anxious individuals: A meta-analytic study. Psychological Bulletin, 133(1), 1–24. Scholar
  4. Bishop, S. J. (2008). Neural mechanisms underlying selective attention to threat. Annals of the New York Academy of Sciences, 1129, 141–152. Scholar
  5. Booth, R. W. (2014). Uncontrolled avoidance of threat: Vigilance-avoidance, executive control, inhibition and shifting. Cognition and Emotion, 28(8), 1465–1473. Scholar
  6. Bradley, B. P., Mogg, K., Falla, S. J., & Hamilton, L. R. (1998). Attentional bias for threatening facial expressions in anxiety: Manipulation of stimulus duration. Cognition and Emotion, 12(6), 737–753. Scholar
  7. Bradley, B. P., Mogg, K., Millar, N., Bonham-Carter, C., Fergusson, E., Jenkins, J., & Parr, M. (1997). Attentional biases for emotional faces. Cognition and Emotion, 11(1), 25–42. Scholar
  8. Buschman, T. J., & Miller, E. K. (2009). Serial, covert shifts of attention during visual search are reflected by the frontal eye fields and correlated with population oscillations. Neuron, 63(3), 386–396. Scholar
  9. Cisler, J. M., Bacon, A. K., & Williams, N. L. (2009). Phenomenological characteristics of attentional biases towards threat: A critical review. Cognitive Therapy and Research, 33, 221–234.CrossRefPubMedPubMedCentralGoogle Scholar
  10. Cooper, R. M., Bailey, J. E., Diaper, A., Stirland, R., Renton, L. E., Benton, C. P., Penton-Voak, I. S., Nutt, D. J., & Munafò, M. R. (2011). Effects of 7.5% CO2 inhalation on allocation of spatial attention to facial cues of emotional expression. Cognition and Emotion, 25(4), 626–638. Scholar
  11. Cooper, R. M. & Langton, S. R. H. (2006). Attentional bias to angry face using the dot-probe task? It depends when you look for it. Behavior Research and Therapy, 44, 1321–1329.
  12. Dear, B. F., Sharpe, L., Nicholas, M. K., & Refshauge, K. (2011). The psychometric properties of the dot-probe paradigm when used in pain-related attentional bias research. The Journal of Pain, 12(12), 1247–1254. Scholar
  13. Derryberry, D., & Reed, M. A. (2002). Anxiety-related attentional biases and their regulation by attentional control. Journal of Abnormal Psychology, 111(2), 225–236. Scholar
  14. DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park: Sage.Google Scholar
  15. Eimer, M., & Kiss, M. (2007). Attentional capture by task-irrelevant fearful faces is revealed by the N2pc component. Biological Psychology, 74(1), 108–112. Scholar
  16. Eysenck, M. W., & Derakshan, N. (2011). New perspectives in attentional control theory. Personality and Individual Differences, 50(7), 955–960. Scholar
  17. Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7(2), 336–353. Scholar
  18. Grimshaw, G. M., Foster, J. J., & Corballis, P. M. (2014). Frontal and parietal EEG asymmetries interact to predict attentional bias to threat. Brain and Cognition, 90, 78–86. Scholar
  19. Hedge, C., Powell, G., & Sumner, P. (2017). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods. Scholar
  20. Hilimire, M. R., Hickey, C., & Corballis, P. M. (2013). Target resolution in visual search involves the direct suppression of distractors: Evidence from electrophysiology. Psychophysiology, 49, 504–509. Scholar
  21. Holmes, A., Bradley, B. P., Nielsen, K., M., & Mogg, K. (2009). Attentional selectivity for emotional faces: Evidence from human electrophysiology. Psychophysiology, 46(1), 62–68. Scholar
  22. Holmes, A., Green, S., & Vuilleumier, P. (2005). The involvement of distinct visual channels in rapid attention towards fearful facial expressions. Cognition and Emotion, 19(6), 899–922.
  23. Kappenman, E. S., Farrens, J. L., Luck, S. J., & Proudfit, G. H. (2014). Behavioral and ERP measures of attentional bias to threat in the dot-probe task: Poor reliability and lack of correlation with anxiety. Frontiers in Psychology. Scholar
  24. Kappenman, E. S., MacNamara, A., & Proudift, G. H. (2015). Electrocortical evidence for rapid allocation of attention to threat in the dot-probe task. Social Cognitive and Affective Neuroscience, 10(4), 577–583. Scholar
  25. Koster, E. H. W., Crombez, G., Verschuere, B., & De Houwer, J. (2004). Selective attention to threat in the dot probe paradigm: Differentiating vigilance and difficulty to disengage. Behaviour Research and Therapy, 42, 1183–1192. Scholar
  26. Koster, E. H. W., Crombez, G., Verschuere, B., Van Damme, S., & Wiersema, J. R. (2006). Components of attentional bias to threat in high trait anxiety: Facilitated engagement, impaired disengagement, and attentional avoidance. Behaviour Research and Therapy, 44, 1757–1771. Scholar
  27. Koster, E. H. W., Verschuere, B., Crombez, G., & Van Damme, S. (2005). Time-course of attentional bias for threatening pictures in high and low trait anxiety. Behaviour Research and Therapy, 45, 1087–1098. Scholar
  28. MacLeod, C., Mathews, A., & Tata, P. (1986). Attentional bias in emotional disorders. Journal of Abnormal Psychology, 95(1), 15–20. Scholar
  29. MacLeod, J. W., Lawrence, M. A., McConnell, M. M., Eskes, G. A., Klein, R. M., & Shore, D. L. (2010). Appraising the ANT: Psychometric and theoretical considerations of the Attention Network Task. Neuropsychology, 24(5), 637–651. Scholar
  30. Mathôt, S., Schreij, D., & Theeuwes, J. (2012). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314–324. Scholar
  31. Miller, J., & Ulrich, R. (2013). Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect size. Psychonomic Bulletin and Review, 20, 819–858. Scholar
  32. Mogg, K., & Bradley, B. P. (1999). Some methodological issues in assessing attentional biases for threatening faces in anxiety: A replication study using a modified version of the probe detection task. Behavior Therapy and Research, 37, 595–604. Scholar
  33. Mogg, K., Bradley, B. P., De Bono, J., & Painter, M. (1997). Time course of attentional bias for threat information in non-clinical anxiety. Behaviour Research and Therapy, 35(4), 297–303. Scholar
  34. Müller, H. J., & Rabbitt, P. M. A. (1989). Reflexive and voluntary orienting of visual attention: Time course of activation and resistance to interruption. Journal of Experimental Psychology Human Perception and Performance, 15(2), 315–330. Scholar
  35. Öhman, A., Lundqvist, D., & Esteves, F. (2001). The face in the crowd revisited: A threat advantage with schematic stimuli. Journal of Personality and Social Psychology, 80(3), 381–396. Scholar
  36. Okon-Singer, H., Lichtenstein-Vidne, L., & Cohen, N. (2013). Dynamic modulation of emotional processing. Biological Psychology, 92, 480–491. Scholar
  37. Onnis, R., Dadds, M. R., & Bryant, R. A. (2011). Is there a mutual relationship between opposite attentional biases underlying anxiety? Emotion, 11(3), 582–594. Scholar
  38. Posner, M. I., Cohen, Y., & Rafal, R. D. (1982). Neural systems control of spatial attention. Philosophical Transactions of the Royal Society B: Biological Sciences, 298(1089), 187–198.
  39. Pourtois, G., Schettino, A., & Vuilleumier, P. (2013). Brain mechanisms for emotional influences on perception and attention: What is magic and what is not. Biological Psychology, 92, 492–512. Scholar
  40. R Core Team (2015). R: A Language and Environment for Statistical Computing [Computer Software]. Retrieved from
  41. Ratcliff, R. (1993). Methods for dealing with response time outliers. Psychological Bulletin, 114(3), 510–532. Scholar
  42. Reutter, M., Hewig, J., Wieser, M. J., & Osinsky, R. (2017). The N2pc component reliably captures attentional bias in social anxiety. Psychophysiology, 54, 519–527. Scholar
  43. Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., Zvielli, A., & Lenze, E. J. (2016). Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of Abnormal Psychology, 125(6), 840–851. Scholar
  44. Ross, D. A., Richler, J. J., & Gauthier, I. (2015). Reliability of composite-task measurements of holistic face processing. Behavior Research Methods, 47, 736–743.CrossRefPubMedPubMedCentralGoogle Scholar
  45. Sawaki, R., & Luck, S. J. (2010). Capture versus suppression of attention by salient singletons: Electrophysiological evidence for an automatic attend-to-me signal. Attention Perception and Psychophysics, 72(6), 1455–1470. Scholar
  46. Schäfer, J., Bernstein, A., Zvielli, A., Hölfer, M., Wittchen, H., & Schönfeld, S. (2016). Attentional bias temporal dynamics predict posttraumatic stress symptoms: A prospective-longitudinal study among soldiers. Depression and Anxiety, 33, 630–639. Scholar
  47. Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19, 595–605. Scholar
  48. Spearman, C. C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3(3), 271–295. Scholar
  49. Spielberger, C. D., & Sydeman, S. J. (1994). State-trait anxiety inventory and state-trait anger expression inventory. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (pp. 292–321). Hillsdale: Lawrence Erlbaum Associates.Google Scholar
  50. Staugaard, S. R. (2009). Reliability of two versions of the dot-probe task using photographic faces. Psychology Science Quarterly, 51(3), 339–350.Google Scholar
  51. Strauss, G. P., Allen, D. N., Jorgensen, M. L., & Cramer, S. L. (2005). Test-retest reliability of standard and emotional Stroop tasks. Assessment, 12(3), 330–337. Scholar
  52. Szymanski, J., & O’Donohue, W. (1995). Fear of Spiders Questionnaire. Journal of Behavioral Therapy and Experimental Psychology, 26(1), 31–34. Scholar
  53. Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249. Scholar
  54. Van Bockstaele, B., Vershuere, B., Koster, E. H. W., Tibboel, H., De Houwer, J., & Crombez, G. (2011). Differential predictive power of self report and implicit measures on behavioural and physiological fear responses to spiders. International Journal of Psychophysiology, 79, 166–174. Scholar
  55. Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014). Measuring attention bias to threat: Reliability of dot probe and eye movement indices. Cognitive Therapy and Research, 38(3), 313–333. Scholar
  56. Waechter, S., & Stolz, J. A. (2015). Trait anxiety, state anxiety, and attentional bias to threat: Assessing the psychometric properties of response time measures. Cognitive Therapy and Research, 39(4), 441–458. Scholar
  57. White, L. K., Britton, J. C., Sequiera, S., Ronkin, E. G., Chen, G., Bar-Haim, Y., Pine, D. S. (2016). Behavioral and neural stability of attentional bias to threat in healthy adolescents. NeuroImage, 136, 84–93. Scholar
  58. Willenbockel, V., Sadr, J., Fiset, D., Horne, G. O., Gosselin, F., & Tanaka, J. W. (2010). Controlling low-level image properties: The SHINE toolbox. Behavior Research Methods, 42, 671–684. Scholar
  59. Williams, B. J., & Kaufmann, L. M. (2012). Reliability of the go/no-go association task. Journal of Experimental Social Psychology, 48, 879–891. Scholar
  60. Yiend, J. (2010). The effects of emotion on attention: A review of attentional processing of emotional information. Cognition and Emotion, 24(1), 3–47. Scholar
  61. Zvielli, A., Bernstein, A., & Koster, E. H. W. (2015). Temporal dynamics of attentional bias. Clinical Psychological Science, 3(5), 772–788. Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  1. 1.School of PsychologyVictoria University of WellingtonWellingtonNew Zealand

Personalised recommendations