Skip to main content

Reliability, Validity and True Values in Surveys

Abstract

In general, it is assumed that distinct true values will be found behind what is actually measured in surveys. By acquiring sufficient knowledge of measurement error, its extent and nature, we are supposed to be able to obtain adequate knowledge of underlying properties. It could be maintained, however, that this idea of a stable and comprehensible underlying reality is often only a theoretical construction. The existence of a clear measurable reality can often be questioned on both theoretical and empirical grounds. This paper provides some arguments and some illustrative results based on method studies. One observation that is often made when examining survey data is that responses to similar questions have a tendency to show poor correspondence with each other. Also, responses to the same questions posed to the same people at different times tend to correspond not as well as might be expected. Data is presented that shows far greater inconsistency in contexts where people have to make judgements than in those where they provide descriptions. Also, much less consistency was found among people whose standpoints are relatively unclear. It seems plausible to interpret lack of consistency as partly an expression of difficulties on the part of respondents in adopting unequivocal stances. Consequently, inconsistency cannot be indiscriminately used to gauge the measuring instrument’s reliability. It is not possible to manage deficiencies by means of any simple methodological technique. If so, the instrument’s reliability will be consistently underestimated. Part of the uncertainty among individuals is then displaced into the measuring instrument. Uncertainty has to be handled also through stricter choices of questions, by using indicators of sufficient clarity, and by differentiating between clear and unclear standpoints.

This is a preview of subscription content, access via your institution.

References

  • F. M. Andrews S. B. Withey (1976) Social Indicators of Well-Being: Americans Perceptions of Life Quality Plenum Press New York

    Google Scholar 

  • G. W. Bohrnstedt (1983) Measurement. Chapter 3 P. H. Rossi J. D. Wright A. B. Andersson (Eds) Handbook and Survey Research Academic Press New York

    Google Scholar 

  • A. Campell P. E. Converse W. L. Rodgers (1976) The Quality of American Life: Perceptions, Evaluations and Satisfactions Russel Sage Foundation New York

    Google Scholar 

  • P. Converse ( 1964) The Nature of Belief Systems D.E. Apter Ideology and Discontent Free Press of Glencoe New York

    Google Scholar 

  • J. Fleishman J. Benson (1987) ArticleTitleUsing Lisrel to evaluate measurement models and scale reliability Educational and Psychological Measurement 47 925–939

    Google Scholar 

  • G. Forsman I. Schreiner (1991) The design and analysis of reinterview: an overview P. P. Biemer R. M. Groves L. Lyberg N. A. Mathiowetz A. Nancy S. Sudman (Eds) Measurement Errors in Surveys John Wiley and Sons New York

    Google Scholar 

  • Goocher, B. E.: 1965, ‘Effects of attitude and experience on the selection of frequency adverbs’, Journal of Verbal Learning and Verbal Behaviour (4)

  • R. M. Groves (1991) Measurement error across the disciplines P. P. Biemer R. M. Groves L. Lyberg N. A. Mathiowetz A. Nancy S. Sudman (Eds) Measurement Errors in Surveys John Wiley and Sons New York

    Google Scholar 

  • Hakel, M. D.: 1969, ‘How often is often?’ American Psychologists (23)

  • K. G. Jöreskog ( 1971) ArticleTitleStatistical analysis of sets of congeneric measures Psychometrica 36 109–134 Occurrence Handle10.1007/BF02291393

    Article  Google Scholar 

  • Jöreskog K. G. and D. Sörbom: 1987, ‘New development in Lisrel’, Data Analyst 4(1)

  • Levnadsförhållanden: 1985, Appendix 8. ‘Återintervjustudier i undersökningarna om levnadsförhållanden’ [Re-interview studies in surveys of living conditions] ULF 1981, 1983 and 1984 (SCB, Stockholm)

  • Levnadsförhållanden: 1991, Appendix 12. ‘Återintervjustudie i undersökningarna om levnadsförhållanden’ [Re-interview studies in surveys of living conditions], ULF 1989 (SCB, Stockholm)

  • R. Luskin (1990) ArticleTitleExplaining political sophistication Political Behaviour 12 4 Occurrence Handle10.1007/BF00992793

    Article  Google Scholar 

  • I. Munck (1979) Model Building in Comparative Education. Applications of the Lisrel Method to Cross-National Survey Data Almqvist och Wiksell International Stockholm

    Google Scholar 

  • Nordiska ministerrådet (Nordic Council of Ministers): 1989, Rapport om Nordiskt projekt nr 42.70.04 [Report on Nordic project no. 42.70.04] (Copenhagen)

  • Österman, T.: 1998, Opinionens mekanismer – om värderingar och verklighet [The mechanisms of opinions – on valuations and reality]. Doctoral dissertation at the Department of Sociology, Stockholm University (Akademitryck, Stockholm)

  • Pace C. R. and J. Friedlander: 1982, ‘The meaning of response categories. How often is ‘occasionally’, ‘often’ and ‘very often’?’ Research in Higher Education 17(3)

  • Pepper, S. and L.S. Prytulak: 1974, ‘Sometimes frequently means seldom: context effects in the interpretation of quantitative expressions’, Journal of Research in Personality (8)

  • J. P. Robinson J. G. Rusk K. B. Head (1968) Measures of Political Attitudes Ann Arbor Michigan

    Google Scholar 

  • Schwarz, N.: 1979, ‘Self Reports’, The American Psychologist February 1999

  • Simpson R. H.: 1944, ‘The specific meanings of certain terms indicating differing degrees of frequency’, The Quarterly Journal of Speech (30)

  • M. Sirken D. Herrmann S. Schechter N. Schwarz J. Tanur R. Tourangeau ( 1999) Cognition and Survey Research Wiley Series in Probability and Statistics New York

    Google Scholar 

  • Statistics Sweden: 1992, The Working Environment 1989/91 (SCB, Stockholm)

  • Suchman E. A.: 1950, ‘The intensity component in attitude and opinion research’, in S. A. Stouffer et al. (eds.), Measurement and Prediction (Princeton University Press, Princeton, NJ)

  • Wärneryd, B.: 1985, Återintervjustudier i undersökningarna av levnadsförhållanden 81, 83 och 84 [Re-interview studies in investigations of living conditions 81, 83 and 84]. Levnadsförhållanden appendix 8 (SCB)

  • Wikman, A.: 1979, ‘Mätning av opinioner – en mätteknisk metodstudie’ [Measurement of opinions – a methodological study]. Beredskapsnämnden för psykologiskt försvar (BN), now called the Swedish Emergency Management Agency

  • Wikman, A.: 1980, ‘Svarsprecisionen i surveyundersökningar om levnadsförhållanden’, [Response precision in surveys of living conditions] Metodproblem i individ- och hushållsstatistiken, no. 14 (SCB, Stockholm)

  • Wikman, A.: 1991, ‘Att utveckla sociala indikatorer – en surveyansats belyst med exemplet arbetsmiljö,’ [Developing social indicators – an “effort” with the survey method, illustrated with the example of working environment. Urval, no. 21 (SCB, Stockholm).

  • Wikman A. and B. Wärneryd: 1990, ‘Measurement errors in survey questions: explaining response variability,’ Social Indicators Research 22, pp. 199–212

  • L. J. Williams (1995) ArticleTitle‘Covariance structure modeling in organizational research: problems with the method versus applications of the method’ Journal of Organizational Behavior 16 225–233

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anders Wikman.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wikman, A. Reliability, Validity and True Values in Surveys. Soc Indic Res 78, 85 (2006). https://doi.org/10.1007/s11205-005-5372-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11205-005-5372-3

Keywords

  • non-attitudes
  • social indicator
  • survey methods
  • reliability
  • re-interviews validity