Reliability, Validity and True Values in Surveys

Abstract

In general, it is assumed that distinct true values will be found behind what is actually measured in surveys. By acquiring sufficient knowledge of measurement error, its extent and nature, we are supposed to be able to obtain adequate knowledge of underlying properties. It could be maintained, however, that this idea of a stable and comprehensible underlying reality is often only a theoretical construction. The existence of a clear measurable reality can often be questioned on both theoretical and empirical grounds. This paper provides some arguments and some illustrative results based on method studies. One observation that is often made when examining survey data is that responses to similar questions have a tendency to show poor correspondence with each other. Also, responses to the same questions posed to the same people at different times tend to correspond not as well as might be expected. Data is presented that shows far greater inconsistency in contexts where people have to make judgements than in those where they provide descriptions. Also, much less consistency was found among people whose standpoints are relatively unclear. It seems plausible to interpret lack of consistency as partly an expression of difficulties on the part of respondents in adopting unequivocal stances. Consequently, inconsistency cannot be indiscriminately used to gauge the measuring instrument’s reliability. It is not possible to manage deficiencies by means of any simple methodological technique. If so, the instrument’s reliability will be consistently underestimated. Part of the uncertainty among individuals is then displaced into the measuring instrument. Uncertainty has to be handled also through stricter choices of questions, by using indicators of sufficient clarity, and by differentiating between clear and unclear standpoints.