Skip to main content
SpringerLink
Log in

Beyond Face Value: Assessing the Factor Structure of an Eye-Tracking Based Attention Bias Task

  • Original Article
  • Published:
Cognitive Therapy and Research Aims and scope Submit manuscript

Abstract

Background

Behavioral measurement of attention bias for emotional stimuli has traditionally ignored whether trial-level task data have a strong enough general factor to justify a unidimensional measurement model. This is surprising, as unidimensionality across trials is an important assumption for computing bias scores.

Methods

In the present study, we assess the psychometric properties of a free-viewing, eye-tracking task measuring attention for emotional stimuli. Undergraduate students (N = 130) viewed two counterbalanced blocks of 4 × 4 matrices of sad/neutral and happy/neutral facial expressions for 10 seconds each across 60 trials. We applied a bifactor measurement model across ten attention bias metrics (e.g., total dwell time for neutral and emotional stimuli, ratio of emotional to total dwell time, difference in dwell time for emotional and neutral stimuli, a variable indicating whether dwell time on emotional stimuli exceeded dwell time on neutral stimuli) to assess whether trial-level data load on to a single, general factor. Unidimensionality was evaluated using omega hierarchical, explained common variance, and percentage of uncontaminated correlations.

Results

Total dwell time had excellent internal consistency for sad (ɑ = .95, ɷ = .96) and neutral stimuli (ɑ = .95, ɷ = .95), and met criteria for unidimensionality, suggesting the trial-level data within each task reflect a single underlying construct. However, the remaining bias metrics fell short of the unidimensionality thresholds, suggesting not all metrics are good candidates for creating bias scores.

Conclusion

Total dwell time by valence had the best psychometrics in terms of internal consistency and unidimensionality. This study demonstrates the importance of assessing whether trial-level data load onto a general factor, as not all metrics are equivalent, even when derived from the same task data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available in our Supplementary Materials within the Texas Data Repository at https://dataverse.tdl.org/dataverse/factor_structure_attention.

Notes

  1. While 18–45 was the inclusion criteria for age of the sample, average age was 19.4 years (SD = 1.4), and maximum age was 28 years. Only 4 participants were over the age of 22 (23, 23, 24, and 28). We confirmed the inclusion of these participants in our sample had no meaningful impact on the results through supplementary analyses that can be found at https://dataverse.tdl.org/dataverse/factor_structure_attention.

  2. As a reminder, the stimuli selection processes for each task were independent, so different pools of neutral facial expressions are used in the happy version and sad version (some of the same neutral faces may be in both tasks).

References

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub.

    Book  Google Scholar 

  • Armstrong, T., & Olatunji, B. O. (2012). Eye tracking of attention in the affective disorders: A meta-analytic review and synthesis. Clinical Psychology Review, 32(8), 704–723.

    Article  PubMed  PubMed Central  Google Scholar 

  • Armstrong, T., Wilbanks, D., Leong, D., & Hsu, K. (2021). Beyond vernacular: Measurement solutions to the lexical fallacy in disgust research. Journal of Anxiety Disorders, 82, 102408.

    Article  PubMed  Google Scholar 

  • Beck, A. T. (2008). The evolution of the cognitive model of depression and its neurobiological correlates. The American Journal of Psychiatry, 165(8), 969–977.

    Article  PubMed  Google Scholar 

  • Beck, A. T., Steer, R. A., & Brown, G. (1996). Beck Depression Inventory–II. Psychological Assessment. https://doi.org/10.1037/t00742-000

    Article  Google Scholar 

  • Beevers, C. G., Lee, H.-J., Wells, T. T., Ellis, A. J., & Telch, M. J. (2011). Association of predeployment gaze bias for emotion stimuli with later symptoms of PTSD and depression in soldiers deployed in Iraq. The American Journal of Psychiatry, 168(7), 735–741.

    Article  PubMed  Google Scholar 

  • Beevers, C. G., Mullarkey, M. C., Dainer-Best, J., Stewart, R. A., Labrada, J., Allen, J. J. B., McGeary, J. E., & Shumake, J. (2019). Association between negative cognitive bias and depression: A symptom-level approach. Journal of Abnormal Psychology, 128(3), 212–227.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. In Educational and Psychological Measurement (Vol. 65, pp. 676–696).

  • Bodenschatz, C. M., Kersting, A., & Suslow, T. (2019). Effects of briefly presented masked emotional facial expressions on gaze behavior: An eye-tracking study. Psychological Reports, 122(4), 1432–1448.

    Article  PubMed  Google Scholar 

  • Canty, A., & Ripley, B. D. (2019). boot: Bootstrap R (S-Plus) Functions.

  • Chong, L. J., & Meyer, A. (2021). Psychometric properties of threat-related attentional bias in young children using eye-tracking. Developmental Psychobiology, 63(5), 1120–1131.

    Article  PubMed  Google Scholar 

  • Disner, S. G., Beevers, C. G., Haigh, E. A. P., & Beck, A. T. (2011). Neural mechanisms of the cognitive model of depression. Nature Reviews. Neuroscience, 12(8), 467–477.

    Article  CAS  PubMed  Google Scholar 

  • Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412.

    Article  PubMed  Google Scholar 

  • Duque, A., & Vázquez, C. (2015). Double attention bias for positive and negative emotional faces in clinical depression: Evidence from an eye-tracking study. Journal of Behavior Therapy and Experimental Psychiatry, 46, 107–114.

    Article  PubMed  Google Scholar 

  • Ebner, N. C., Riediger, M., & Lindenberger, U. (2010). FACES—A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior Research Methods, 42(1), 351–362.

    Article  PubMed  Google Scholar 

  • Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using r to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484–501.

    Article  Google Scholar 

  • Fried, E. I., van Borkulo, C. D., Epskamp, S., Schoevers, R. A., Tuerlinckx, F., & Borsboom, D. (2016). Measuring depression over time … Or not? Lack of unidimensionality and longitudinal measurement invariance in four common rating scales of depression. Psychological Assessment, 28(11), 1354–1367.

    Article  PubMed  Google Scholar 

  • Gibb, B. E., Alloy, L. B., Abramson, L. Y., Beevers, C. G., & Miller, I. W. (2004). Cognitive vulnerability to depression: A taxometric analysis. Journal of Abnormal Psychology, 113(1), 81–89.

    Article  PubMed  Google Scholar 

  • Green, S. B., & Yang, Y. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement Issues and Practice, 34(4), 14–20.

    Article  Google Scholar 

  • Hankin, B. L., Fraley, R. C., Lahey, B. B., & Waldman, I. D. (2005). Is depression best viewed as a continuum or discrete category? A taxometric analysis of childhood and adolescent depression in a population-based sample. Journal of Abnormal Psychology, 114(1), 96–110.

    Article  PubMed  Google Scholar 

  • Harrell, F. E., Jr, from Charles Dupont, W. C., & others., M. (2020). Hmisc: Harrell Miscellaneous. Retrieved from https://CRAN.R-project.org/package=Hmisc

  • Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4), 188–194.

    Article  PubMed  Google Scholar 

  • Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186.

    Article  PubMed  Google Scholar 

  • Hsu, K. J., McNamara, M. E., Shumake, J., Stewart, R. A., Labrada, J., Alario, A., Gonzalez, G. D. S., Schnyer, D. M., & Beevers, C. G. (2020). Neurocognitive predictors of self-reported reward responsivity and approach motivation in depression: A data-driven approach. Depression and Anxiety, 37(7), 682–697.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hsu, K. J., Shumake, J., Caffey, K., Risom, S., Labrada, J., Smits, J. A. J., Schnyer, D. M., & Beevers, C. G. (2021). Efficacy of attention bias modification training for depressed adults: A randomized clinical trial. Psychological Medicine, 1–9.

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

    Article  Google Scholar 

  • Iannone, R., Cheng, J., & Schloerke, B. (2020). gt: Easily Create Presentation-Ready Display Tables. Retrieved from https://CRAN.R-project.org/package=gt

  • Kappenman, E. S., Farrens, J. L., Luck, S. J., & Proudfit, G. H. (2014). Behavioral and ERP measures of attentional bias to threat in the dot-probe task: Poor reliability and lack of correlation with anxiety. Frontiers in Psychology, 5, 1368.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kellough, J. L., Beevers, C. G., Ellis, A. J., & Wells, T. T. (2008). Time course of selective attention in clinically depressed young adults: An eye tracking study. Behaviour Research and Therapy, 46(11), 1238–1243.

    Article  PubMed  PubMed Central  Google Scholar 

  • Klawohn, J., Bruchnak, A., Burani, K., Meyer, A., Lazarov, A., Bar-Haim, Y., & Hajcak, G. (2020). Aberrant attentional bias to sad faces in depression and the role of stressful life events: Evidence from an eye-tracking paradigm. Behaviour Research and Therapy, 135, 103762.

    Article  PubMed  Google Scholar 

  • Lanza, C., Müller, C., & Riepe, M. W. (2018). Positive mood on negative self-statements: Paradoxical intervention in geriatric patients with major depressive disorder. Aging & Mental Health, 22(6), 748–754.

    Article  Google Scholar 

  • Lazarov, A., Abend, R., & Bar-Haim, Y. (2016). Social anxiety is related to increased dwell time on socially threatening faces. Journal of Affective Disorders, 193, 282–288.

    Article  PubMed  Google Scholar 

  • Lazarov, A., Ben-Zion, Z., Shamai, D., Pine, D. S., & Bar-Haim, Y. (2018). Free viewing of sad and happy faces in depression: A potential target for attention bias modification. Journal of Affective Disorders, 238, 94–100.

    Article  PubMed  PubMed Central  Google Scholar 

  • Liu, Y., Ding, Y., Lu, L., & Chen, X. (2017). Attention bias of avoidant individuals to attachment emotion pictures. Scientific Reports, 7, 41631.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Machulska, A., Kleinke, K., & Klucken, T. (2022). Same same, but different: A psychometric examination of three frequently used experimental tasks for cognitive bias assessment in a sample of healthy young adults. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01804-9

    Article  PubMed  PubMed Central  Google Scholar 

  • Mathôt, S., Schreij, D., & Theeuwes, J. (2012). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314–324.

    Article  PubMed  Google Scholar 

  • McNamara, M. E., Shumake, J., Stewart, R. A., Labrada, J., Alario, A., Allen, J. J. B., Palmer, R., Schnyer, D. M., McGeary, J. E., & Beevers, C. G. (2021). Multifactorial prediction of depression diagnosis and symptom dimensions. Psychiatry Research, 298, 113805.

    Article  PubMed  PubMed Central  Google Scholar 

  • Miles, W. R. (1930). Ocular dominance in human adults. The Journal of General Psychology, 3(3), 412–430.

    Article  Google Scholar 

  • Owens, M., & Gibb, B. E. (2017). Brooding rumination and attentional biases in currently non-depressed individuals: An eye-tracking study. Cognition & Emotion, 31(5), 1062–1069.

    Article  Google Scholar 

  • Parsons, S. (2020). splithalf; robust estimates of split half reliability. Retrieved from https://doi.org/10.6084/m9.figshare.5559175.v5

  • Parsons, S., Kruijt, A.-W., & Fox, E. (2019). Psychological science needs a standard practice of reporting the reliability of cognitive-behavioral measurements. Advances in Methods and Practices in Psychological Science, 2(4), 378–395.

    Article  Google Scholar 

  • Price, R. B., Kuckertz, J. M., Siegle, G. J., Ladouceur, C. D., Silk, J. S., Ryan, N. D., ... & Amir, N. (2015). Empirical recommendations for improving the stability of the dot-probe task in clinical research. Psychological assessment, 27(2), 365.

  • Price, R. B., Rosen, D., Siegle, G. J., Ladouceur, C. D., Tang, K., Allen, K. B., Ryan, N. D., Dahl, R. E., Forbes, E. E., & Silk, J. S. (2016). From anxious youth to depressed adolescents: Prospective prediction of 2-year depression symptoms via attentional bias measures. Journal of Abnormal Psychology, 125(2), 267–278.

    Article  PubMed  Google Scholar 

  • R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

    Google Scholar 

  • Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26.

    Article  Google Scholar 

  • Revelle, W. (2019). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University. Retrieved from https://CRAN.R-project.org/package=psych

  • Robinson, D., Hayes, A., & Couch, S. (2020). broom: Convert Statistical Objects into Tidy Tibbles. Retrieved from https://CRAN.R-project.org/package=broom

  • Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., Zvielli, A., & Lenze, E. J. (2016). Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of Abnormal Psychology, 125(6), 840–851.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016a). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237.

    Article  PubMed  Google Scholar 

  • Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016b). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150.

    Article  PubMed  Google Scholar 

  • Sanchez, A., Vazquez, C., Marker, C., LeMoult, J., & Joormann, J. (2013). Attentional disengagement predicts stress recovery in depression: An eye-tracking study. Journal of Abnormal Psychology, 122(2), 303–313.

    Article  PubMed  Google Scholar 

  • Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19(7), 595–605.

    Article  Google Scholar 

  • Sears, C. R., Quigley, L., Fernandez, A., Newman, K., & Dobson, K. (2019). The reliability of attentional biases for emotional images measured using a free-viewing eye-tracking paradigm. Behavior Research Methods, 51(6), 2748–2760.

    Article  PubMed  Google Scholar 

  • Shi, D., Lee, T., & Maydeu-Olivares, A. (2019). Understanding the Model Size Effect on SEM Fit Indices. Educational and Psychological Measurement, 79(2), 310–334.

    Article  PubMed  Google Scholar 

  • Skinner, I. W., Hübscher, M., Moseley, G. L., Lee, H., Wand, B. M., Traeger, A. C., Gustin, S. M., & McAuley, J. H. (2018). The reliability of eyetracking to assess attentional bias to threatening words in healthy individuals. Behavior Research Methods, 50(5), 1778–1792.

    Article  PubMed  Google Scholar 

  • Soltani, S., Newman, K., Quigley, L., Fernandez, A., Dobson, K., & Sears, C. (2015). Temporal changes in attention to sad and happy faces distinguish currently and remitted depressed individuals from never depressed individuals. Psychiatry Research, 230(2), 454–463.

  • Staugaard, S. R. (2009). Reliability of two versions of the dot-probe task using photographic faces. Citeseer. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.6498&rep=rep1&type=pdf

  • Steiner, M. D., & Grieder, S. (2020). EFAtools: An R package with fast and flexible implementations of exploratory factor analysis tools. Journal of Open Source Software, 5(53), 2521. https://doi.org/10.21105/joss.02521

    Article  Google Scholar 

  • Stochl, J., Fried, E. I., Fritz, J., Croudace, T. J., Russo, D. A., Knight, C., Jones, P. B., & Perez, J. (2020). On Dimensionality, measurement invariance, and suitability of sum scores for the PHQ-9 and the GAD-7. Assessment, 1073191120976863.

  • Suslow, T., Hußlack, A., Kersting, A., & Bodenschatz, C. M. (2020). Attentional biases to emotional information in clinical depression: A systematic and meta-analytic review of eye tracking findings. Journal of Affective Disorders, 274, 632–642.

    Article  PubMed  Google Scholar 

  • Tang, W., Cui, Y., & Babenko, O. (2014). Internal consistency: Do we really know what it is and how to assess it. Journal of Psychology & Clinical Psychiatry., 2(2), 205–220.

    Google Scholar 

  • Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55.

    Article  PubMed  PubMed Central  Google Scholar 

  • Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wang, Y.-P., & Gorenstein, C. (2013). Psychometric properties of the Beck Depression Inventory-II: A comprehensive review. Revista Brasileira de Psiquiatria (Sao Paulo, Brazil : 1999), 35(4), 416–431.

    Article  PubMed  Google Scholar 

  • Wells, T. T., Clerkin, E. M., Ellis, A. J., & Beevers, C. G. (2014). Effect of antidepressant medication use on emotional information processing in major depression. The American Journal of Psychiatry, 171(2), 195–200.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., & Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686

    Article  Google Scholar 

  • Wickham, H., & Bryan, J. (2019). readxl: Read Excel Files. Retrieved from https://CRAN.R-project.org/package=readxl

  • Wickham, H., & Seidel, D. (2020). scales: Scale Functions for Visualization. Retrieved from https://CRAN.R-project.org/package=scales

  • Xie, Y. (2014). knitr: a comprehensive tool for reproducible research in R. In V. Stodden, F. Leisch, & R. D. Peng (Eds.), Implementing reproducible computational research. Chapman and Hall/CRC.

    Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Psychology and Institute for Mental Health Research, University of Texas at Austin, 108 E Dean Keeton St, Austin, TX, 78712, USA

    Mary E. McNamara, Bryan A. McSpadden, Semeon Risom, Jason Shumake & Christopher G. Beevers

  2. Department of Psychology, National University of Singapore, Singapore, Singapore

    Kean J. Hsu

  3. Department of Psychiatry, Georgetown University Medical Center, Washington, DC, USA

    Kean J. Hsu

Authors
  1. Mary E. McNamara
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kean J. Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bryan A. McSpadden
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Semeon Risom
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jason Shumake
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christopher G. Beevers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mary E. McNamara.

Ethics declarations

Conflict of Interest

Mary McNamara, Kean Hsu, Bryan McSpadden, and Semeon Risom report no conflicts of interest. Christopher Beevers has received grant funding from the National Institutes of Health, Brain and Behavior Foundation, and other not-for profit foundations. He has received income from the Association for Psychological Science for his editorial work and from Orexo, Inc. for serving on a Scientific Advisory Board related to digital therapeutics. Dr. Beevers’ financial disclosures have been reviewed and approved by the University of Texas at Austin in accordance with its conflict-of-interest policies. Jason Shumake has received grant funding from the National Institutes of Health as well as salary and stock options from Aiberry, Inc. Drs. Beevers’ and Shumake’s financial disclosures have been reviewed and approved by the University of Texas at Austin in accordance with its conflict of interest policies.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McNamara, M.E., Hsu, K.J., McSpadden, B.A. et al. Beyond Face Value: Assessing the Factor Structure of an Eye-Tracking Based Attention Bias Task. Cogn Ther Res 47, 772–787 (2023). https://doi.org/10.1007/s10608-023-10395-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10608-023-10395-4

Keywords

Search

Navigation