Abstract
In a dot-probe task, two cues—one emotional and one neutral—are followed by a probe in one of their locations. Faster responses to probes co-located with the emotional stimulus are taken as evidence of attentional bias. Several studies indicate that such attentional bias measures have poor reliability, even though ERP studies show that people reliably attend to the emotional stimulus. This inconsistency might arise because the emotional stimulus captures attention briefly (as indicated by ERP), but cues appear for long enough that attention can be redistributed before the probe onset, causing RT measures of bias to vary across trials. We tested this hypothesis by manipulating SOA (stimulus onset asynchrony between onset of the cues and onset of the probe) in a dot-probe task using angry and neutral faces. Across three experiments, the internal reliability of behavioural biases was significantly greater than zero when probes followed faces by 100 ms, but not when the SOA was 300, 500, or 900 ms. Thus, the initial capture of attention shows some level of consistency, but this diminishes quickly. Even at the shortest SOA internal reliability estimates were poor, and not sufficient to justify the use of the task as an index of individual differences in attentional bias.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Power was calculated assuming a one-tailed test, as negative reliabilities are meaningless. This method underestimates the true power of our Experiments, as it is based on a single correlation coefficient and not the distribution of reliabilities that our computational method produces.
References
Amir, I., Zvielli, A., & Bernstein, A. (2016). De(coupling) of our eyes and our mind’s eye: A dynamic process perspective on attentional bias. Emotion, 16(7), 978–986. https://doi.org/10.1037/emo0000172.
Bar-Haim, Y., Holoshitz, Y., Eldar, S., Frenkel, T. I., Muller, D., Charney, D. S., Pine, D. S., Fox, N. A., & Wald, I. (2010). Life-threatening danger and suppression of attention bias to threat. American Journal of Psychiatry, 167(6), 694–698. https://doi.org/10.1176/appi.ajp.2009.09070956.
Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007). Threat-related attentional bias in anxious and non-anxious individuals: A meta-analytic study. Psychological Bulletin, 133(1), 1–24. https://doi.org/10.1037/0033-2909.133.1.1.
Bishop, S. J. (2008). Neural mechanisms underlying selective attention to threat. Annals of the New York Academy of Sciences, 1129, 141–152. https://doi.org/10.1196/annals.1417.016.
Booth, R. W. (2014). Uncontrolled avoidance of threat: Vigilance-avoidance, executive control, inhibition and shifting. Cognition and Emotion, 28(8), 1465–1473. https://doi.org/10.1080/02699931.2014.882294.
Bradley, B. P., Mogg, K., Falla, S. J., & Hamilton, L. R. (1998). Attentional bias for threatening facial expressions in anxiety: Manipulation of stimulus duration. Cognition and Emotion, 12(6), 737–753. https://doi.org/10.1080/026999398379411.
Bradley, B. P., Mogg, K., Millar, N., Bonham-Carter, C., Fergusson, E., Jenkins, J., & Parr, M. (1997). Attentional biases for emotional faces. Cognition and Emotion, 11(1), 25–42. https://doi.org/10.1080/026999397380014.
Buschman, T. J., & Miller, E. K. (2009). Serial, covert shifts of attention during visual search are reflected by the frontal eye fields and correlated with population oscillations. Neuron, 63(3), 386–396. https://doi.org/10.1016/j.neuron.2009.06.020.
Cisler, J. M., Bacon, A. K., & Williams, N. L. (2009). Phenomenological characteristics of attentional biases towards threat: A critical review. Cognitive Therapy and Research, 33, 221–234.
Cooper, R. M., Bailey, J. E., Diaper, A., Stirland, R., Renton, L. E., Benton, C. P., Penton-Voak, I. S., Nutt, D. J., & Munafò, M. R. (2011). Effects of 7.5% CO2 inhalation on allocation of spatial attention to facial cues of emotional expression. Cognition and Emotion, 25(4), 626–638. https://doi.org/10.1080/02699931.2010.508887.
Cooper, R. M. & Langton, S. R. H. (2006). Attentional bias to angry face using the dot-probe task? It depends when you look for it. Behavior Research and Therapy, 44, 1321–1329. https://doi.org/10.1016/j.brat.2005.10.004.
Dear, B. F., Sharpe, L., Nicholas, M. K., & Refshauge, K. (2011). The psychometric properties of the dot-probe paradigm when used in pain-related attentional bias research. The Journal of Pain, 12(12), 1247–1254. https://doi.org/10.1016/j.jpain.2011.07.003.
Derryberry, D., & Reed, M. A. (2002). Anxiety-related attentional biases and their regulation by attentional control. Journal of Abnormal Psychology, 111(2), 225–236. https://doi.org/10.1037/0021-843X.111.2.225.
DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park: Sage.
Eimer, M., & Kiss, M. (2007). Attentional capture by task-irrelevant fearful faces is revealed by the N2pc component. Biological Psychology, 74(1), 108–112. https://doi.org/10.1016/j.biopsycho.2006.06.008.
Eysenck, M. W., & Derakshan, N. (2011). New perspectives in attentional control theory. Personality and Individual Differences, 50(7), 955–960. https://doi.org/10.1016/j.paid.2010.08.019.
Eysenck, M. W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7(2), 336–353. https://doi.org/10.1037/1528-3542.7.2.336.
Grimshaw, G. M., Foster, J. J., & Corballis, P. M. (2014). Frontal and parietal EEG asymmetries interact to predict attentional bias to threat. Brain and Cognition, 90, 78–86. https://doi.org/10.1016/j.bandc.2014.06.008.
Hedge, C., Powell, G., & Sumner, P. (2017). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods. https://doi.org/10.3758/s13428-017-0935-1.
Hilimire, M. R., Hickey, C., & Corballis, P. M. (2013). Target resolution in visual search involves the direct suppression of distractors: Evidence from electrophysiology. Psychophysiology, 49, 504–509. https://doi.org/10.1111/j.1469-8986.2011.01326.x.
Holmes, A., Bradley, B. P., Nielsen, K., M., & Mogg, K. (2009). Attentional selectivity for emotional faces: Evidence from human electrophysiology. Psychophysiology, 46(1), 62–68. https://doi.org/10.1111/j.1469-8986.2008.00750.x.
Holmes, A., Green, S., & Vuilleumier, P. (2005). The involvement of distinct visual channels in rapid attention towards fearful facial expressions. Cognition and Emotion, 19(6), 899–922. https://doi.org/10.1080/02699930441000454.
Kappenman, E. S., Farrens, J. L., Luck, S. J., & Proudfit, G. H. (2014). Behavioral and ERP measures of attentional bias to threat in the dot-probe task: Poor reliability and lack of correlation with anxiety. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2014.01368.
Kappenman, E. S., MacNamara, A., & Proudift, G. H. (2015). Electrocortical evidence for rapid allocation of attention to threat in the dot-probe task. Social Cognitive and Affective Neuroscience, 10(4), 577–583. https://doi.org/10.1093/scan/nsu098.
Koster, E. H. W., Crombez, G., Verschuere, B., & De Houwer, J. (2004). Selective attention to threat in the dot probe paradigm: Differentiating vigilance and difficulty to disengage. Behaviour Research and Therapy, 42, 1183–1192. https://doi.org/10.1016/j.brat.2003.08.001.
Koster, E. H. W., Crombez, G., Verschuere, B., Van Damme, S., & Wiersema, J. R. (2006). Components of attentional bias to threat in high trait anxiety: Facilitated engagement, impaired disengagement, and attentional avoidance. Behaviour Research and Therapy, 44, 1757–1771. https://doi.org/10.1016/j.brat.2005.12.011.
Koster, E. H. W., Verschuere, B., Crombez, G., & Van Damme, S. (2005). Time-course of attentional bias for threatening pictures in high and low trait anxiety. Behaviour Research and Therapy, 45, 1087–1098. https://doi.org/10.1016/j.brat.2004.08.004.
MacLeod, C., Mathews, A., & Tata, P. (1986). Attentional bias in emotional disorders. Journal of Abnormal Psychology, 95(1), 15–20. https://doi.org/10.1037/0021-843X.95.1.15.
MacLeod, J. W., Lawrence, M. A., McConnell, M. M., Eskes, G. A., Klein, R. M., & Shore, D. L. (2010). Appraising the ANT: Psychometric and theoretical considerations of the Attention Network Task. Neuropsychology, 24(5), 637–651. https://doi.org/10.1037/a0019803.
Mathôt, S., Schreij, D., & Theeuwes, J. (2012). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314–324. https://doi.org/10.3758/s13428-011-0168-7.
Miller, J., & Ulrich, R. (2013). Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect size. Psychonomic Bulletin and Review, 20, 819–858. https://doi.org/10.3758/s13423-013-0404-5.
Mogg, K., & Bradley, B. P. (1999). Some methodological issues in assessing attentional biases for threatening faces in anxiety: A replication study using a modified version of the probe detection task. Behavior Therapy and Research, 37, 595–604. https://doi.org/10.1016/S0005-7967(98)00158-2.
Mogg, K., Bradley, B. P., De Bono, J., & Painter, M. (1997). Time course of attentional bias for threat information in non-clinical anxiety. Behaviour Research and Therapy, 35(4), 297–303. https://doi.org/10.1016/S0005-7967(96)00109-X.
Müller, H. J., & Rabbitt, P. M. A. (1989). Reflexive and voluntary orienting of visual attention: Time course of activation and resistance to interruption. Journal of Experimental Psychology Human Perception and Performance, 15(2), 315–330. https://doi.org/10.1037/0096-1523.15.2.315.
Öhman, A., Lundqvist, D., & Esteves, F. (2001). The face in the crowd revisited: A threat advantage with schematic stimuli. Journal of Personality and Social Psychology, 80(3), 381–396. https://doi.org/10.1037/0022-3514.80.3.381.
Okon-Singer, H., Lichtenstein-Vidne, L., & Cohen, N. (2013). Dynamic modulation of emotional processing. Biological Psychology, 92, 480–491. https://doi.org/10.1016/j.biopsycho.2012.05.010.
Onnis, R., Dadds, M. R., & Bryant, R. A. (2011). Is there a mutual relationship between opposite attentional biases underlying anxiety? Emotion, 11(3), 582–594. https://doi.org/10.1037/a0022019.
Posner, M. I., Cohen, Y., & Rafal, R. D. (1982). Neural systems control of spatial attention. Philosophical Transactions of the Royal Society B: Biological Sciences, 298(1089), 187–198. https://doi.org/10.1098/rstb.1982.0081.
Pourtois, G., Schettino, A., & Vuilleumier, P. (2013). Brain mechanisms for emotional influences on perception and attention: What is magic and what is not. Biological Psychology, 92, 492–512. https://doi.org/10.1016/j.biopsycho.2012.02.007.
R Core Team (2015). R: A Language and Environment for Statistical Computing [Computer Software]. Retrieved from http://www.R-project.org.
Ratcliff, R. (1993). Methods for dealing with response time outliers. Psychological Bulletin, 114(3), 510–532. https://doi.org/10.1037/0033-2909.114.3.510.
Reutter, M., Hewig, J., Wieser, M. J., & Osinsky, R. (2017). The N2pc component reliably captures attentional bias in social anxiety. Psychophysiology, 54, 519–527. https://doi.org/10.1111/psyp.12809.
Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., Zvielli, A., & Lenze, E. J. (2016). Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of Abnormal Psychology, 125(6), 840–851. https://doi.org/10.1037/abn0000184.
Ross, D. A., Richler, J. J., & Gauthier, I. (2015). Reliability of composite-task measurements of holistic face processing. Behavior Research Methods, 47, 736–743.
Sawaki, R., & Luck, S. J. (2010). Capture versus suppression of attention by salient singletons: Electrophysiological evidence for an automatic attend-to-me signal. Attention Perception and Psychophysics, 72(6), 1455–1470. https://doi.org/10.3758/APP.72.6.1455.
Schäfer, J., Bernstein, A., Zvielli, A., Hölfer, M., Wittchen, H., & Schönfeld, S. (2016). Attentional bias temporal dynamics predict posttraumatic stress symptoms: A prospective-longitudinal study among soldiers. Depression and Anxiety, 33, 630–639. https://doi.org/10.1002/da.22526.
Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19, 595–605. https://doi.org/10.1002/per.554.
Spearman, C. C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3(3), 271–295. https://doi.org/10.1111/j.2044-8295.1910.tb00206.x.
Spielberger, C. D., & Sydeman, S. J. (1994). State-trait anxiety inventory and state-trait anger expression inventory. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (pp. 292–321). Hillsdale: Lawrence Erlbaum Associates.
Staugaard, S. R. (2009). Reliability of two versions of the dot-probe task using photographic faces. Psychology Science Quarterly, 51(3), 339–350.
Strauss, G. P., Allen, D. N., Jorgensen, M. L., & Cramer, S. L. (2005). Test-retest reliability of standard and emotional Stroop tasks. Assessment, 12(3), 330–337. https://doi.org/10.1177/1073191105276375.
Szymanski, J., & O’Donohue, W. (1995). Fear of Spiders Questionnaire. Journal of Behavioral Therapy and Experimental Psychology, 26(1), 31–34. https://doi.org/10.1016/0005-7916(94)00072-T.
Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249. https://doi.org/10.1016/j.psychres.2008.05.006.
Van Bockstaele, B., Vershuere, B., Koster, E. H. W., Tibboel, H., De Houwer, J., & Crombez, G. (2011). Differential predictive power of self report and implicit measures on behavioural and physiological fear responses to spiders. International Journal of Psychophysiology, 79, 166–174. https://doi.org/10.1016/j.ijpsycho.2010.10.003.
Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014). Measuring attention bias to threat: Reliability of dot probe and eye movement indices. Cognitive Therapy and Research, 38(3), 313–333. https://doi.org/10.1007/s10608-013-9588-2.
Waechter, S., & Stolz, J. A. (2015). Trait anxiety, state anxiety, and attentional bias to threat: Assessing the psychometric properties of response time measures. Cognitive Therapy and Research, 39(4), 441–458. https://doi.org/10.1007/s10608-015-9670-z.
White, L. K., Britton, J. C., Sequiera, S., Ronkin, E. G., Chen, G., Bar-Haim, Y., Pine, D. S. (2016). Behavioral and neural stability of attentional bias to threat in healthy adolescents. NeuroImage, 136, 84–93. https://doi.org/10.1016/j.neuroimage.2016.04.058.
Willenbockel, V., Sadr, J., Fiset, D., Horne, G. O., Gosselin, F., & Tanaka, J. W. (2010). Controlling low-level image properties: The SHINE toolbox. Behavior Research Methods, 42, 671–684. https://doi.org/10.3758/BRM.42.3.671.
Williams, B. J., & Kaufmann, L. M. (2012). Reliability of the go/no-go association task. Journal of Experimental Social Psychology, 48, 879–891. https://doi.org/10.1016/j.jesp.2012.03.001.
Yiend, J. (2010). The effects of emotion on attention: A review of attentional processing of emotional information. Cognition and Emotion, 24(1), 3–47. https://doi.org/10.1080/02699930903205698.
Zvielli, A., Bernstein, A., & Koster, E. H. W. (2015). Temporal dynamics of attentional bias. Clinical Psychological Science, 3(5), 772–788. https://doi.org/10.1177/2167702614551572.
Acknowledgements
We thank Laura Kranz and Emma O’Brien for assistance with data collection. This research was supported by a Grant from the Royal Society of New Zealand Marsden Fund (VUW1307) to GG. Development of the MacBrain Face Stimulus Set was overseen by Nim Tottenham and supported by the John D. and Catherine T. MacArthur Foundation Research Network on Early Experience and Brain Development. Please contact Nim Tottenham at tott0006@tc.umn.edu for more information concerning the stimulus set.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical standards
All procedures performed in studies involving human participants were in accordance with the ethical standards of the Victoria University of Wellington Human Ethics Committee. Informed consent was obtained from all individual participants included in the study.
Data availability
The data sets generated and analysed for this study are not publicly available due to ethical constraints, but are available from the corresponding author on reasonable request.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Chapman, A., Devue, C. & Grimshaw, G.M. Fleeting reliability in the dot-probe task. Psychological Research 83, 308–320 (2019). https://doi.org/10.1007/s00426-017-0947-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00426-017-0947-6