Behavior Research Methods

, Volume 46, Issue 3, pp 668–688 | Cite as

A comparative investigation of seven indirect attitude measures

  • Yoav Bar-Anan
  • Brian A. Nosek


We compared the psychometric qualities of seven indirect attitude measures across three attitude domains (race, politics, and self-esteem) with a large sample (N = 23,413). We compared the measures on internal consistency, sensitivity to known effects, relationships with indirect and direct measures of the same topic, the reliability and validity of single-category attitude measurement, their ability to detect meaningful variance among people with nonextreme attitudes, and their robustness to the exclusion of misbehaving or well-behaving participants. All seven indirect measures correlated with each other and with direct measures of the same topic. These relations were always weak for self-esteem, moderate for race, and strong for politics. This pattern suggests that some of the sources of variation in the reliability and predictive validity of the indirect measures is a function of the concepts rather than the methods. The Implicit Association Test (IAT) and Brief IAT (BIAT) showed the best overall psychometric quality, followed by the Go–No-Go association task, Single-Target IAT (ST-IAT), Affective Misattribution Procedure (AMP), Sorting Paired Features task, and Evaluative Priming. The AMP showed a steep decline in its psychometric qualities when people with extreme attitude scores were removed. Single-category attitude scores computed for the IAT and BIAT showed good relationships with other attitude measures but no evidence of discriminant validity between paired categories. The other measures, especially the AMP and ST-IAT, showed better evidence for discriminant validity. These results inform us on the validity of the measures as attitude assessments, but do not speak to the implicitness of the measured constructs.


Implicit social cognition Indirect measures Implicit attitudes The Brief Implicit Association Test 


Author note

This project was supported by grants from the European Union (PIRG06‐GA‐2009‐256467), and the Israeli Science Foundation (1012/10) to Y. B.‐A, and from Project Implicit Inc. to Y.B.‐A and B.A.N.. B.A.N. is an officer of Project Implicit Inc., a nonprofit organization that provided financial and technical support to this project, and includes in its mission “To develop and deliver methods for investigating and applying phenomena of implicit social cognition, including especially phenomena of implicit bias based on age, race, gender or other factors.”

Supplementary material

13428_2013_410_MOESM1_ESM.pdf (687 kb)
ESM 1 (PDF 686 kb)


  1. Altemeyer, B. (1981). Right-wing authoritarianism. Winnipeg, MB: University of Manitoba Press.Google Scholar
  2. Altemeyer, B. (1996). The authoritarian specter. Cambridge, MA: Harvard University Press.Google Scholar
  3. Bar-Anan, Y., & Nosek, B. A. (2012). Reporting intentional rating of the primes predicts priming effects in the affective misattribution procedure. Personality and Social Psychology Bulletin, 38, 1193–1207.CrossRefGoogle Scholar
  4. Bar-Anan, Y., & Nosek, B. A. (2013). The effect of number of trials and stimulus set on the psychometric qualities of the affective misattribution procedure. Open Science Framework, bHNd2.
  5. Bar-Anan, Y., Nosek, B. A., & Vianello, M. (2009). The sorting paired features task: A measure of association strengths. Experimental Psychology, 56, 329–343.PubMedCrossRefGoogle Scholar
  6. Bar-Anan, Y., Shahar, G., & Nosek, B. A. (2013). A multitrait-multimethod structural analysis of direct and indirect attitude measures. Unpublished raw data.Google Scholar
  7. Bosson, J. K., Swann, W. B., & Pennebaker, J. W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631–643.PubMedCrossRefGoogle Scholar
  8. Cameron, C. D., Brown-Iannuzzi, J. L., & Payne, B. K. (2012). Sequential priming measures of implicit social cognition: A meta-analysis of associations with behavior and explicit attitudes. Personality and Social Psychology Review, 16, 330–350.PubMedCrossRefGoogle Scholar
  9. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.CrossRefGoogle Scholar
  10. Cunningham, W. A., Preacher, K. J., & Banaji, M. R. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163–170.PubMedCrossRefGoogle Scholar
  11. Fazio, R. H. (2007). Attitudes as object–evaluation associations of varying strength. Social Cognition, 25, 603–637.PubMedCentralPubMedCrossRefGoogle Scholar
  12. Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013–1027.PubMedCrossRefGoogle Scholar
  13. Fazio, R. H., & Olson, M. A. (2003). Indirect measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297–327.PubMedCrossRefGoogle Scholar
  14. Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 50, 229–238.PubMedCrossRefGoogle Scholar
  15. Feldt, L. S. (1969). A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests. Psychometrika, 34(3), 363–373.Google Scholar
  16. Gawronski, B., & Bodenhausen, G. V. (2006). Associative and prepositional processes in evaluation: An integrative review of implicit and explicit attitude change. Psychological Bulletin, 132, 692–731. doi: 10.1037/0033-2909.132.5.692 PubMedCrossRefGoogle Scholar
  17. Gawronski, B., & De Houwer, J. (in press). Indirect measures in social and personality psychology. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (2nd ed.). New York, NY: Cambridge University Press.Google Scholar
  18. Gawronski, B., & Payne, B. K. (2010). Handbook of implicit social cognition: Measurement, theory, and applications. New York, NY: Guilford Press.Google Scholar
  19. Greenwald, A. G., McGhee, D. E., & Schwartz, J. K. L. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480.PubMedCrossRefGoogle Scholar
  20. Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.PubMedCrossRefGoogle Scholar
  21. John, O. P., & Benet-Martinez, V. (2000). Measurement: Reliability, construct validation, and scale construction. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 339–369). New York, NY: Cambridge University Press.Google Scholar
  22. Jost, J. T., Glaser, J., Kruglanski, A. W., & Sulloway, F. J. (2003). Political conservatism as motivated social cognition. Psychological Bulletin, 129, 339–375. doi: 10.1037/0033-2909.129.3.339 PubMedCrossRefGoogle Scholar
  23. Karpinski, A., & Steinman, R. B. (2006). The Single Category Implicit Association Test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91, 16–32.PubMedCrossRefGoogle Scholar
  24. McConahay, J. B. (1983). Modern racism and modern discrimination. Personality and Social Psychology Bulletin, 9, 551–558.CrossRefGoogle Scholar
  25. McConahay, J. B. (1986). Modern racism, ambivalence, and the Modern Racism Scale. In J. F. Dovidio & S. L. Gaertner (Eds.), Prejudice, discrimination, and racism (pp. 91–125). San Diego, CA: Academic Press.Google Scholar
  26. Mierke, J., & Klauer, K. C. (2003). Method-specific variance in the Implicit Association Test. Journal of Personality and Social Psychology, 85, 1180–1192.PubMedCrossRefGoogle Scholar
  27. Nosek, B. A. (2005). Moderators of the relationship between implicit and explicit evaluation. Journal of Experimental Psychology: General, 134, 565–584. doi: 10.1037/0096-3445.134.4.565 CrossRefGoogle Scholar
  28. Nosek, B. A. (2007). Implicit–explicit relations. Current Directions in Psychological Science, 16, 65–69.CrossRefGoogle Scholar
  29. Nosek, B. A., & Banaji, M. R. (2001). The Go/No-Go association task. Social Cognition, 19, 625–666.Google Scholar
  30. Nosek, B. A., Bar-Anan, Y., Sriram, N., & Greenwald, A. G. (2013). Understanding and using the brief Implicit Association Test: I. Recommended scoring procedures. Unpublished manuscript.Google Scholar
  31. Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the Implicit Association Test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31, 166–180.PubMedCrossRefGoogle Scholar
  32. Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265–292). New York, NY: Psychology Press.Google Scholar
  33. Nosek, B. A., Hawkins, C. B., & Frazier, R. S. (2011). Implicit social cognition: From measures to mechanisms. Trends in Cognitive Sciences, 15, 152–159.PubMedCentralPubMedCrossRefGoogle Scholar
  34. Nosek, B. A., & Smyth, F. L. (2007). A multitrait-multimethod validation of the Implicit Association Test: Implicit and explicit attitudes are related but distinct constructs. Experimental Psychology, 54, 14–29.PubMedCrossRefGoogle Scholar
  35. Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., . . . Banaji, M. R. (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18, 36–88.Google Scholar
  36. Olson, M. A., & Fazio, R. H. (2003). Relations between indirect measures of prejudice. Psychological Science, 14, 636–639.PubMedCrossRefGoogle Scholar
  37. Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. (2005). An inkblot for attitudes: Affect misattribution as indirect measurement. Journal of Personality and Social Psychology, 89, 277–293.PubMedCrossRefGoogle Scholar
  38. Payne, B. K., Govorun, O., & Arbukle, N. L. (2008). Automatic attitudes and alcohol: Does implicit liking predict drinking? Cognition and Emotion, 22, 238–271.CrossRefGoogle Scholar
  39. Perugini, M., Richetin, J., & Zogmaister, C. (2010). Prediction of behavior. In B. Gawronski & B. K. Payne (Eds.), Handbook of implicit social cognition: Measurement, theory, and applications (pp. 255–277). New York, NY: Guilford Press.Google Scholar
  40. Ranganath, K. A., Smith, C. T., & Nosek, B. A. (2008). Distinguishing automatic and controlled components of attitudes from direct and indirect measurement methods. Journal of Experimental Social Psychology, 44, 386–396.PubMedCentralPubMedCrossRefGoogle Scholar
  41. Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.Google Scholar
  42. Scherer, L. D., & Lambert, A. J. (2009). Contrast effects in priming paradigms: Implications for theory and research on implicit attitudes. Journal of Personality and Social Psychology, 97, 383–403.PubMedCrossRefGoogle Scholar
  43. Sriram, N., & Greenwald, A. G. (2009). The brief implicit association test. Experimental Psychology, 56, 283–294.PubMedCrossRefGoogle Scholar
  44. Stieger, R. S., Göritz, A. S., Hergovich, A., & Voracek, M. (2011). Intentional faking of the single category implicit association test and the implicit association test. Psychological Reports, 109, 219–230.PubMedCrossRefGoogle Scholar
  45. Wigboldus, D. H. J., Holland, R. W., & van Knippenberg, A. (2004). Single target implicit associations. Unpublished manuscript.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  1. 1.Psychology DepartmentBen Gurion University of the NegevBe’er ShevaIsrael
  2. 2.University of VirginiaCharlottesvilleUSA

Personalised recommendations