Behavior Research Methods

, Volume 50, Issue 6, pp 2345–2365 | Cite as

The failing measurement of attitudes: How semantic determinants of individual survey responses come to replace measures of attitude strength

  • Jan Ketil Arnulf
  • Kai Rune Larsen
  • Øyvind Lund Martinsen
  • Thore Egeland


The traditional understanding of data from Likert scales is that the quantifications involved result from measures of attitude strength. Applying a recently proposed semantic theory of survey response, we claim that survey responses tap two different sources: a mixture of attitudes plus the semantic structure of the survey. Exploring the degree to which individual responses are influenced by semantics, we hypothesized that in many cases, information about attitude strength is actually filtered out as noise in the commonly used correlation matrix. We developed a procedure to separate the semantic influence from attitude strength in individual response patterns, and compared these results to, respectively, the observed sample correlation matrices and the semantic similarity structures arising from text analysis algorithms. This was done with four datasets, comprising a total of 7,787 subjects and 27,461,502 observed item pair responses. As we argued, attitude strength seemed to account for much information about the individual respondents. However, this information did not seem to carry over into the observed sample correlation matrices, which instead converged around the semantic structures offered by the survey items. This is potentially disturbing for the traditional understanding of what survey data represent. We argue that this approach contributes to a better understanding of the cognitive processes involved in survey responses. In turn, this could help us make better use of the data that such methods provide.


Semantic analysis Surveys Survey response Semantic theory of survey response (STSR) Attitude strength 


Author note

We thank the U.S. National Science Foundation for research support under Grant NSF 0965338, and the National Institutes of Health through Colorado Clinical & Translational Sciences Institute for research support under Grant NIH/CTSI 5 UL1 RR025780.


  1. Abdi, H. (2003). Factor rotations in factor analysis. In M. Lewis-Beck, A. Bryman, & T. Futing (Eds.), Encyclopedia of social sciences research methods (pp. 792–795). Thousand Oaks, CA: Sage.Google Scholar
  2. Aiken, L. S., West, S. G., & Reno, R. R. (1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: Sage.Google Scholar
  3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  4. Andrich, D. (1996). A hyperbolic cosine latent trait model for unfolding polytomous responses: Reconciling Thurstone and Likert methodologies. British Journal of Mathematical and Statistical Psychology, 49, 347–365.CrossRefGoogle Scholar
  5. Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120.CrossRefGoogle Scholar
  6. Arnulf, J. K., & Larsen, K. R. (2015). Overlapping semantics of leadership and heroism: Expectations of omnipotence, identification with ideal leaders and disappointment in real managers. Scandinavian Psychologist, 2, 1–28. doi: CrossRefGoogle Scholar
  7. Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2014). Predicting survey responses: How and why semantics shape survey statistics in organizational behavior. PLoS ONE, 9, e106361:1–13. doi: CrossRefGoogle Scholar
  8. Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2015). A new approach to psychological measures in leadership research. Scandinavian Psychologist, 2, e4. doi: CrossRefGoogle Scholar
  9. Avolio, B. J., Bass, B. M., & Jung, D. I. (1995). Multifactor Leadership Questionnaire (Technical Report). Redwood City, CA: Mind Garden.Google Scholar
  10. Bagozzi, R. P. (2011). Measurement and meaning in information systems and organizational research: Methodological and philosophical foundations. MIS Quarterly, 35, 261–292.CrossRefGoogle Scholar
  11. Bernerth, J. B., Armenakis, A. A., Feild, H. S., Giles, W. F., & Walker, H. J. (2007). Leader–member social exchange (LMSX): Development and validation of a scale. Journal of Organizational Behavior, 28, 979–1003. doi: CrossRefGoogle Scholar
  12. Borsboom, D. (2008). Latent variable theory. Measurement, 6, 25–53.Google Scholar
  13. Borsboom, D. (2009). Educational measurement [Book review]. Structural Equation Modeling, 16, 702–711. doi: CrossRefGoogle Scholar
  14. Brennan, R. L. (2010). Generalizability theory. In P. P. B. McGaw (Ed.), International encyclopedia of education (3rd ed., pp. 61–68). Oxford, UK: Elsevier.CrossRefGoogle Scholar
  15. Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan organizational assessment questionnaire. Ann Arbor, MI: University of Michigan.Google Scholar
  16. Cascio, W. F. (2012). Methodological issues in international HR management research. International Journal of Human Resource Management, 23, 2532–2545. doi: CrossRefGoogle Scholar
  17. Conger, J. A., & Kanungo, R. N. (1998). Charismatic leadership in organizations. Thousand Oaks, CA: Sage.Google Scholar
  18. Coombs, C. H., & Kao, R. C. (1960). On a connection between factor analysis and multidimensional unfolding. Psychometrika, 25, 219–231.CrossRefGoogle Scholar
  19. Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality Inventory: Professional manual. Odessa, FL: Psychological Assessment Resources.Google Scholar
  20. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.CrossRefGoogle Scholar
  21. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–340. doi: CrossRefGoogle Scholar
  22. de Vries, J., Byrne, M., & Kehoe, E. (2015). Cognitive dissonance induction in everyday life: An fMRI study. Social Neuroscience, 10, 268–281. doi: CrossRefPubMedGoogle Scholar
  23. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., & Harshman, R. (1990). Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science, 41, 391–407.CrossRefGoogle Scholar
  24. Dennis, S., Landauer, T., Kintsch, W., & Quesada, J. (2003). Introduction to latent semantic analysis. Paper presented at the 25th Annual Meeting of the Cognitive Science Society, Boston, MA.Google Scholar
  25. Dennis, S., Landauer, T., Kintsch, W., & Quesada, J. (2013). Introduction to latent semantic analysis. Boulder, CO: University of Colorado.Google Scholar
  26. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.CrossRefGoogle Scholar
  27. Feldman, J. M., & Lynch, J. G. J. (1988). Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73, 421–435.CrossRefGoogle Scholar
  28. Felfe, J., & Schyns, B. (2014). Romance of leadership and motivation to lead. Journal of Managerial Psychology, 29, 850–865. doi: CrossRefGoogle Scholar
  29. Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.Google Scholar
  30. Fisher, H. E., Island, H. D., Rich, J., Marchalik, D., & Brown, L. L. (2015). Four broad temperament dimensions: Description, convergent validation correlations, and comparison with the Big Five. Frontiers in Psychology, 6, 1098. doi: CrossRefPubMedPubMedCentralGoogle Scholar
  31. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50.CrossRefGoogle Scholar
  32. Gefen, D., & Larsen, K. R. (2017). Controlling for lexical closeness in survey research: A demonstration on the Technology Acceptance Model. Journal of the Association for Information Systems, 18(10), 1.CrossRefGoogle Scholar
  33. Gollwitzer, P., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69–118.Google Scholar
  34. Honey, C. J., Thomson, C. R., Lerner, Y., & Hasson, U. (2012). Not lost in translation: Neural responses shared across languages. Journal of Neuroscience, 32, 15277–15283.CrossRefGoogle Scholar
  35. Ilies, R., Nahrgang, J. D., & Morgeson, F. P. (2007). Leader–member exchange and citizenship behaviors: A meta-analysis. Journal of Applied Psychology, 92, 269–277.CrossRefGoogle Scholar
  36. Jackson, R. L., Hoffman, P., Pobric, G., & Ralph, M. A. L. (2016). The semantic network at work and rest: Differential connectivity of anterior temporal lobe subregions. Journal of Neuroscience, 36, 1490–1501. doi: CrossRefPubMedGoogle Scholar
  37. Jöreskog, K. G. (1993). Testing structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 294–316). Newbury Park, CA: Sage.Google Scholar
  38. Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Chicago, IL: SSI Scientific Software.Google Scholar
  39. Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus & Giroux.Google Scholar
  40. Kern, M. L., Eichstaedt, J. C., Schwartz, H. A., Park, G., Ungar, L. H., Stillwell, D. J., … Seligman, M. E. P. (2014). From “sooo excited!!!” to “so proud”: Using language to study development. Developmental Psychology, 50, 178–188. doi: CrossRefPubMedGoogle Scholar
  41. Kim, S., Hahn, H. J., & Lee, J. (2015). Organizational attitudes as precursors to training performance. Human Resource Development Quarterly, 26, 409–429. doi: CrossRefGoogle Scholar
  42. Kline, P. (2000). Handbook of psychological testing (2nd ed.). London, UK: Routledge.Google Scholar
  43. Kosinski, M., Bachrach, Y., Kasneci, G., Van-Gael, J., & Graepel, T. (2012). Crowd IQ: measuring the intelligence of crowdsourcing platforms. Proceedings of the 4th Annual ACM Web Science Conference, Evanston, Illinois, June 22–24 (pp. 151–160). New York, NY: ACM. doi:
  44. Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110, 5802–5805. doi: CrossRefGoogle Scholar
  45. Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56, 16–26.CrossRefGoogle Scholar
  46. Kuvaas, B. (2006a). Performance appraisal satisfaction and employee outcomes: Mediating and moderating roles of motivation. International Journal of Human Resource Management, 17, 504–522.CrossRefGoogle Scholar
  47. Kuvaas, B. (2006b). Work performance, affective commitment, and work motivation: The roles of pay administration and pay level. Journal of Organizational Behavior, 27, 365–385.CrossRefGoogle Scholar
  48. Kuvaas, B., Arnulf, J. K., & Dysvik, A. (2008). Social and economic exchange perceptions and intrinsic motivation among knowledge workers. Paper presented at the Academy of Management Annual Meeting, Anaheim, CA.Google Scholar
  49. Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31, 65–71. doi: CrossRefGoogle Scholar
  50. Landauer, T. K. (2007). LSA as a theory of meaning. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsh (Eds.), Handbook of latent semantic analysis (pp. 3–34). Mahwah, NJ: Erlbaum.Google Scholar
  51. LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13, 230–237.CrossRefGoogle Scholar
  52. LaPointe, L. L. (2013). Paul Broca and the origins of language in the brain. San Diego, CA, US: Plural Publishing.Google Scholar
  53. Larsen, K. R., & Bong, C. H. (2013). Construct identity detection. Leeds School Working Paper Series (pp. 1–70). Boulder, CO: University of Colorado, Boulder.Google Scholar
  54. Larsen, K. R., & Bong, C. H. (2016). A tool for addressing construct identity in literature reviews and meta-analyses. MIS Quarterly, 40, 529–551.CrossRefGoogle Scholar
  55. Larsen, K. R., & Monarchi, D. E. (2004). A mathematical approach to categorization and labeling of qualitative data: The latent categorization method. Sociological Methodology, 34, 349–392.CrossRefGoogle Scholar
  56. Larsen, K. R., Nevo, D., & Rich, E. (2008). Exploring the semantic validity of questionnaire scales. Paper presented at the Proceedings of the 41st Hawaii International Conference on System Sciences, Waikoloa, Hawaii.Google Scholar
  57. Li, Y., McLean, D., Bandar, Z. A., O’Shea, J. D., & Crockett, K. (2006). Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering, 18, 1138–1150.CrossRefGoogle Scholar
  58. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55.Google Scholar
  59. Lynam, D. R., Hoyle, R. H., & Newman, J. P. (2006). The perils of partialling. Assessment, 13, 328–341. doi: CrossRefPubMedPubMedCentralGoogle Scholar
  60. MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Quarterly, 35, 293–334.CrossRefGoogle Scholar
  61. Markowetz, A., Blaszkiewicz, K., Montag, C., Switala, C., & Schlaepfer, T. E. (2014). Psycho-Informatics: Big data shaping modern psychometrics. Medical Hypotheses, 82, 405–411. doi: CrossRefPubMedGoogle Scholar
  62. Marsh, H. W., Ludtke, O., Muthén, B., Asparouhov, T., Morin, A. J. S., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471–491. doi: CrossRefPubMedGoogle Scholar
  63. Martinsen, Ø. L., Nordvik, H., & Østbø, L. (2003). Norske versjoner av NEO PI- R og NEO FFI (Norwegian versions of the NEO PI-R and NEO FFI). Oslo, Norway: Gyldendal Akademisk forlag.Google Scholar
  64. Maul, A. (2017). Rethinking traditional methods of survey validation. Measurement: Interdisciplinary Research and Perspectives, 15, 51–69. doi: Google Scholar
  65. McClelland, D. C., Koestner, R., & Weinberger, J. (1989). How do self-attributed and implicit motives differ. Psychological Review, 96, 690–702.CrossRefGoogle Scholar
  66. McKenzie, J. (1998). Fundamental flaws in the five factor model: A re-analysis of the seminal correlation matrix from which the “openness-to-experience” factor was extracted. Personality and Individual Differences, 24, 475–480. doi: CrossRefGoogle Scholar
  67. Meyer, J. P., Allen, N. J., & Smith, C. A. (1993). Commitment to organizations and occupations: Extension and test of a three-component conceptualization. Journal of Applied Psychology, 78, 538–551.CrossRefGoogle Scholar
  68. Michell, J. (1994). Measuring dimensions of belief by unidimensional unfolding. Journal of Mathematical Psychology, 38, 244–273.CrossRefGoogle Scholar
  69. Michell, J. (2013). Constructs, inferences, and mental measurement. New Ideas in Psychology, 31, 13–21. doi: CrossRefGoogle Scholar
  70. Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. Paper presented at the 21st National Conference on Artificial Intelligence, Boston, MA.Google Scholar
  71. Miller, G. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41. doi: CrossRefGoogle Scholar
  72. Mohler, M., & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. Paper presented at the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009), Athens, Greece.Google Scholar
  73. Nakagawa, J., Takahashi, M., Okada, R., Matsushima, E., & Matsuda, T. (2015). Women’s preference for a male acquaintance enhances social reward processing of material goods in the anterior cingulate cortex. PLoS ONE, 10, e136168:1–14. doi: CrossRefGoogle Scholar
  74. Nimon, K., Shuck, B., & Zigarmi, D. (2015). Construct overlap between employee engagement and job satisfaction: A function of semantic equivalence? Journal of Happiness Studies, 17, 1149–1171. doi: CrossRefGoogle Scholar
  75. Nunnally, J. C., & Bernstein, I. H. (2010). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.Google Scholar
  76. Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. doi: CrossRefPubMedGoogle Scholar
  77. Podsakoff, P. M., & Organ, D. W. (1986). Self-reports in organizational research—Problems and prospects. Journal of Management, 12, 531–544.CrossRefGoogle Scholar
  78. Poli, R., Healy, M., & Kameas, A. (2010). WordNet. In C. Fellbaum (Ed.), Theory and applications of ontology: Computer applications (pp. 231–243). New York, NY: Springer.CrossRefGoogle Scholar
  79. Prinz, W., Aschersleben, G., & Koch, I. (2009). Cognition and action. In E. Morsella, J. A. Bargh, & P. M. Gollwitzer (Eds.), Oxford handbook of human action (pp. 33–70). New York, NY: Oxford University Press.Google Scholar
  80. Regev, M., Honey, U., & Hasson, U. (2013). Modality-selective and modality-invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33, 15978–15988.CrossRefGoogle Scholar
  81. Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1–12. doi: CrossRefGoogle Scholar
  82. Russell, B. (1922). An introduction to the Tractatus Logico-Philosophicus. In L. Wittgenstein, Tractatus logico-philosophicus. London, UK: Kegan Paul.Google Scholar
  83. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105.CrossRefGoogle Scholar
  84. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.Google Scholar
  85. Shore, L. M., Tetrick, L. E., Lynch, P., & Barksdale, K. (2006). Social and economic exchange: Construct development and validation. Journal of Applied Social Psychology, 36, 837–867.CrossRefGoogle Scholar
  86. Slaney, K. L. (2017). Validating psychological constructs: Historical, philosophical, and practical dimensions. London, UK: Palgrave MacMillan.CrossRefGoogle Scholar
  87. Slaney, K. L., & Racine, T. P. (2013). What’s in a name? Psychology’s ever evasive construct. New Ideas in Psychology, 31, 4–12. doi: CrossRefGoogle Scholar
  88. Smedslund, J. (1994). Nonempirical and empirical components in the hypotheses of 5 social-psychological experiments. Scandinavian Journal of Psychology, 35, 1–15.CrossRefGoogle Scholar
  89. Stogdill, R. M. (1963). Manual for the leader behavior description questionnaire, form XII. Columbus, OH: Bureau of Business Research, Ohio State University.Google Scholar
  90. Triandis, H. C. (1980). Value, attitudes and interpersonal behavior. In M. M. Page (Ed.), Nebraska Symposium on Motivation, Beliefs, Attitudes and Values (pp. 159–295). Lincoln, NE: University of Nebraska.Google Scholar
  91. Tversky, B. (1993). Cognitive maps, cognitive collages, and spatial mental models. In A. U. Frank & I. Campari (Eds.), Spatial information theory: A theoretical basis for GIS (pp. 14–24). Berlin, Germany: Springer.CrossRefGoogle Scholar
  92. Van Berkum, J. J. A., Koornneef, A. W., Otten, M., & Neuwland, M. S. (2007). Establishing reference in language comprehension: An electrophysiological perspective. Brain Research, 1146, 158–171. doi: CrossRefPubMedGoogle Scholar
  93. van der Linden, D., te Nijenhuis, J., & Bakker, A. B. (2010). The general factor of personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study. Journal of Research in Personality, 44, 315–327. doi: Scholar
  94. Van Dyne, L., & LePine, J. A. (1998). Helping and voice extra-role behaviors: Evidence of construct and predictive validity. Academy of Management Journal, 41, 108–119.Google Scholar
  95. Van Knippenberg, D., & Sitkin, S. B. (2013). A critical assessment of charismatic–transformational leadership research: Back to the drawing board? Academy of Management Annals, 7, 1–60. doi: CrossRefGoogle Scholar
  96. van Schuur, W. H. (2003). Mokken scale analysis: Between the Guttman scale and parametric item response theory. Political Analysis, 11, 139–163. doi: CrossRefGoogle Scholar
  97. Wittgenstein, L. (1922). Tractatus logico-philosophicus. London, UK: Kegan Paul.Google Scholar
  98. Zimbardo, P. G., Ebbesen, E. B., & Maslach, C. (1977). Influencing attitudes and changing behavior (2nd ed.). Reading, MA: Addison Wesley.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Jan Ketil Arnulf
    • 1
  • Kai Rune Larsen
    • 2
  • Øyvind Lund Martinsen
    • 1
  • Thore Egeland
    • 3
  1. 1.BI Norwegian Business SchoolOsloNorway
  2. 2.Leeds Business SchoolUniversity of ColoradoBoulderUSA
  3. 3.Norwegian University of Life SciencesAasNorway

Personalised recommendations