A comparative investigation of seven indirect attitude measures

Behavior Research Methods volume 46, pages668–688(2014)Cite this article

Abstract

We compared the psychometric qualities of seven indirect attitude measures across three attitude domains (race, politics, and self-esteem) with a large sample (N = 23,413). We compared the measures on internal consistency, sensitivity to known effects, relationships with indirect and direct measures of the same topic, the reliability and validity of single-category attitude measurement, their ability to detect meaningful variance among people with nonextreme attitudes, and their robustness to the exclusion of misbehaving or well-behaving participants. All seven indirect measures correlated with each other and with direct measures of the same topic. These relations were always weak for self-esteem, moderate for race, and strong for politics. This pattern suggests that some of the sources of variation in the reliability and predictive validity of the indirect measures is a function of the concepts rather than the methods. The Implicit Association Test (IAT) and Brief IAT (BIAT) showed the best overall psychometric quality, followed by the Go–No-Go association task, Single-Target IAT (ST-IAT), Affective Misattribution Procedure (AMP), Sorting Paired Features task, and Evaluative Priming. The AMP showed a steep decline in its psychometric qualities when people with extreme attitude scores were removed. Single-category attitude scores computed for the IAT and BIAT showed good relationships with other attitude measures but no evidence of discriminant validity between paired categories. The other measures, especially the AMP and ST-IAT, showed better evidence for discriminant validity. These results inform us on the validity of the measures as attitude assessments, but do not speak to the implicitness of the measured constructs.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1

References

  1. Altemeyer, B. (1981). Right-wing authoritarianism. Winnipeg, MB: University of Manitoba Press.

    Google Scholar 

  2. Altemeyer, B. (1996). The authoritarian specter. Cambridge, MA: Harvard University Press.

    Google Scholar 

  3. Bar-Anan, Y., & Nosek, B. A. (2012). Reporting intentional rating of the primes predicts priming effects in the affective misattribution procedure. Personality and Social Psychology Bulletin, 38, 1193–1207.

    Article  Google Scholar 

  4. Bar-Anan, Y., & Nosek, B. A. (2013). The effect of number of trials and stimulus set on the psychometric qualities of the affective misattribution procedure. Open Science Framework, bHNd2. http://openscienceframework.org/project/bHNd2/

  5. Bar-Anan, Y., Nosek, B. A., & Vianello, M. (2009). The sorting paired features task: A measure of association strengths. Experimental Psychology, 56, 329–343.

    PubMed  Article  Google Scholar 

  6. Bar-Anan, Y., Shahar, G., & Nosek, B. A. (2013). A multitrait-multimethod structural analysis of direct and indirect attitude measures. Unpublished raw data.

  7. Bosson, J. K., Swann, W. B., & Pennebaker, J. W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631–643.

    PubMed  Article  Google Scholar 

  8. Cameron, C. D., Brown-Iannuzzi, J. L., & Payne, B. K. (2012). Sequential priming measures of implicit social cognition: A meta-analysis of associations with behavior and explicit attitudes. Personality and Social Psychology Review, 16, 330–350.

    PubMed  Article  Google Scholar 

  9. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

    Article  Google Scholar 

  10. Cunningham, W. A., Preacher, K. J., & Banaji, M. R. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163–170.

    PubMed  Article  Google Scholar 

  11. Fazio, R. H. (2007). Attitudes as object–evaluation associations of varying strength. Social Cognition, 25, 603–637.

    PubMed Central  PubMed  Article  Google Scholar 

  12. Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013–1027.

    PubMed  Article  Google Scholar 

  13. Fazio, R. H., & Olson, M. A. (2003). Indirect measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297–327.

    PubMed  Article  Google Scholar 

  14. Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 50, 229–238.

    PubMed  Article  Google Scholar 

  15. Feldt, L. S. (1969). A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests. Psychometrika, 34(3), 363–373.

    Google Scholar 

  16. Gawronski, B., & Bodenhausen, G. V. (2006). Associative and prepositional processes in evaluation: An integrative review of implicit and explicit attitude change. Psychological Bulletin, 132, 692–731. doi:10.1037/0033-2909.132.5.692

    PubMed  Article  Google Scholar 

  17. Gawronski, B., & De Houwer, J. (in press). Indirect measures in social and personality psychology. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (2nd ed.). New York, NY: Cambridge University Press.

  18. Gawronski, B., & Payne, B. K. (2010). Handbook of implicit social cognition: Measurement, theory, and applications. New York, NY: Guilford Press.

    Google Scholar 

  19. Greenwald, A. G., McGhee, D. E., & Schwartz, J. K. L. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480.

    PubMed  Article  Google Scholar 

  20. Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.

    PubMed  Article  Google Scholar 

  21. John, O. P., & Benet-Martinez, V. (2000). Measurement: Reliability, construct validation, and scale construction. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 339–369). New York, NY: Cambridge University Press.

    Google Scholar 

  22. Jost, J. T., Glaser, J., Kruglanski, A. W., & Sulloway, F. J. (2003). Political conservatism as motivated social cognition. Psychological Bulletin, 129, 339–375. doi:10.1037/0033-2909.129.3.339

    PubMed  Article  Google Scholar 

  23. Karpinski, A., & Steinman, R. B. (2006). The Single Category Implicit Association Test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91, 16–32.

    PubMed  Article  Google Scholar 

  24. McConahay, J. B. (1983). Modern racism and modern discrimination. Personality and Social Psychology Bulletin, 9, 551–558.

    Article  Google Scholar 

  25. McConahay, J. B. (1986). Modern racism, ambivalence, and the Modern Racism Scale. In J. F. Dovidio & S. L. Gaertner (Eds.), Prejudice, discrimination, and racism (pp. 91–125). San Diego, CA: Academic Press.

    Google Scholar 

  26. Mierke, J., & Klauer, K. C. (2003). Method-specific variance in the Implicit Association Test. Journal of Personality and Social Psychology, 85, 1180–1192.

    PubMed  Article  Google Scholar 

  27. Nosek, B. A. (2005). Moderators of the relationship between implicit and explicit evaluation. Journal of Experimental Psychology: General, 134, 565–584. doi:10.1037/0096-3445.134.4.565

    Article  Google Scholar 

  28. Nosek, B. A. (2007). Implicit–explicit relations. Current Directions in Psychological Science, 16, 65–69.

    Article  Google Scholar 

  29. Nosek, B. A., & Banaji, M. R. (2001). The Go/No-Go association task. Social Cognition, 19, 625–666.

    Google Scholar 

  30. Nosek, B. A., Bar-Anan, Y., Sriram, N., & Greenwald, A. G. (2013). Understanding and using the brief Implicit Association Test: I. Recommended scoring procedures. Unpublished manuscript.

  31. Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the Implicit Association Test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31, 166–180.

    PubMed  Article  Google Scholar 

  32. Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265–292). New York, NY: Psychology Press.

    Google Scholar 

  33. Nosek, B. A., Hawkins, C. B., & Frazier, R. S. (2011). Implicit social cognition: From measures to mechanisms. Trends in Cognitive Sciences, 15, 152–159.

    PubMed Central  PubMed  Article  Google Scholar 

  34. Nosek, B. A., & Smyth, F. L. (2007). A multitrait-multimethod validation of the Implicit Association Test: Implicit and explicit attitudes are related but distinct constructs. Experimental Psychology, 54, 14–29.

    PubMed  Article  Google Scholar 

  35. Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., . . . Banaji, M. R. (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18, 36–88.

    Google Scholar 

  36. Olson, M. A., & Fazio, R. H. (2003). Relations between indirect measures of prejudice. Psychological Science, 14, 636–639.

    PubMed  Article  Google Scholar 

  37. Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. (2005). An inkblot for attitudes: Affect misattribution as indirect measurement. Journal of Personality and Social Psychology, 89, 277–293.

    PubMed  Article  Google Scholar 

  38. Payne, B. K., Govorun, O., & Arbukle, N. L. (2008). Automatic attitudes and alcohol: Does implicit liking predict drinking? Cognition and Emotion, 22, 238–271.

    Article  Google Scholar 

  39. Perugini, M., Richetin, J., & Zogmaister, C. (2010). Prediction of behavior. In B. Gawronski & B. K. Payne (Eds.), Handbook of implicit social cognition: Measurement, theory, and applications (pp. 255–277). New York, NY: Guilford Press.

    Google Scholar 

  40. Ranganath, K. A., Smith, C. T., & Nosek, B. A. (2008). Distinguishing automatic and controlled components of attitudes from direct and indirect measurement methods. Journal of Experimental Social Psychology, 44, 386–396.

    PubMed Central  PubMed  Article  Google Scholar 

  41. Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.

    Google Scholar 

  42. Scherer, L. D., & Lambert, A. J. (2009). Contrast effects in priming paradigms: Implications for theory and research on implicit attitudes. Journal of Personality and Social Psychology, 97, 383–403.

    PubMed  Article  Google Scholar 

  43. Sriram, N., & Greenwald, A. G. (2009). The brief implicit association test. Experimental Psychology, 56, 283–294.

    PubMed  Article  Google Scholar 

  44. Stieger, R. S., Göritz, A. S., Hergovich, A., & Voracek, M. (2011). Intentional faking of the single category implicit association test and the implicit association test. Psychological Reports, 109, 219–230.

    PubMed  Article  Google Scholar 

  45. Wigboldus, D. H. J., Holland, R. W., & van Knippenberg, A. (2004). Single target implicit associations. Unpublished manuscript.

Download references

Author note

This project was supported by grants from the European Union (PIRG06‐GA‐2009‐256467), and the Israeli Science Foundation (1012/10) to Y. B.‐A, and from Project Implicit Inc. to Y.B.‐A and B.A.N.. B.A.N. is an officer of Project Implicit Inc., a nonprofit organization that provided financial and technical support to this project, and includes in its mission “To develop and deliver methods for investigating and applying phenomena of implicit social cognition, including especially phenomena of implicit bias based on age, race, gender or other factors.”

Author information

Affiliations

  1. Psychology Department, Ben Gurion University of the Negev, Be’er Sheva, Israel

    Yoav Bar-Anan

  2. University of Virginia, Charlottesville, VA, USA

    Brian A. Nosek

Authors
  1. Yoav Bar-Anan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian A. Nosek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoav Bar-Anan.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 686 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bar-Anan, Y., Nosek, B.A. A comparative investigation of seven indirect attitude measures. Behav Res 46, 668–688 (2014). https://doi.org/10.3758/s13428-013-0410-6

Download citation

Keywords

  • Implicit social cognition
  • Indirect measures
  • Implicit attitudes
  • The Brief Implicit Association Test