Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Correction for Measurement Errors in Survey Research: Necessary and Possible

Abstract

Survey research is the most frequently used data collection method in many disciplines. Nearly, everybody agrees that such data contain serious measurement errors. However, only few researchers try to correct for them. If the measurement errors in the variables vary, the comparison of the sizes of effects of these variables on each other will be wrong. If the sizes of the measurement errors are different across countries, cross-national comparisons of relationships between variables cannot be made. There is ample evidence for these differences in measurement errors across variables, methods and countries (Saris and Gallhofer in Design, evaluation and analysis of questionnaires for survey. Wiley, Hoboken, 2007; Oberski in Measurement errors in comparative surveys. PhD thesis, University of Tilburg, 2011). Therefore, correction for measurement errors is essential for the social sciences. The correction for measurement errors can be made in a simple way, but it requires that the sizes of the error variances are known for all observed variables. Many experiments are carried out to determine the quality of questions. The relationship between the quality and the characteristics of the questions has been studied. Because this relationship is rather strong, one can also predict the quality of new questions. A program SQP has been developed to predict the quality of questions. Using this program, the quality of the questions (complement of error variance) can be obtained for nearly all questions measuring subjective concepts. For objective variables, other research needs to be used (e.g., Alwin in Margins of error: a study of reliability in survey measurement. Wiley, Hoboken, 2007). Using these two sources of information, making correction for measurement error in survey research is possible. We illustrate here that correction for measurement errors can and should be performed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    We are grateful to Wiebke Weber of RECSM for collecting the data in Table 1. Pei-Shan Liao of the Academia Sinica and Zih-Wei Wang of the National Taipei University reported to us that in the Taiwanese Sociological Journal and the Political science review, the situation was very similar: Together these journals published in 2010–2012 in total 67 papers of which 18 used survey research and 9 mentioned the problem of measurement errors but only one makes corrections for measurement errors. We are very grateful for this information.

  2. 2.

    y i  = f i  + e i and cov(f i e i ) = 0 then var(y i ) = var(f i ) + var(e i ) and dividing by var(y i ) gives 1 = [var(f i )/var(y i )] + [var(e i )/var(y i )] or 1 = q i 2  + var(e i ) ,so the total variance of y i , which is equal to 1, can be decomposed into the quality of the question i plus the proportion error in the variance in the observed variable y i . Normally, q i 2 has been called the reliability of a measurement instrument (Lord and Novick 1968) but we prefer the term quality for reason that will become clear in Sect. 4.

  3. 3.

    http://essedunet.nsd.uib.no/cms/topics/measurement/.

  4. 4.

    However, this information is not enough to correct for measurement errors in all ESS questions because at the same time in the ESS more than 60,000 questions were asked with respect to values, opinions, attitudes, preferences, feelings, etc. This illustrates that a different approach is required.

  5. 5.

    SQP is free of change available at sqp.upf.edu.

  6. 6.

    It can be shown that the CMV = r 1j m ij m 2j r 2j .

References

  1. Alwin, D. F. (2007). Margins of error: A study of reliability in survey measurement. Hoboken: Wiley.

  2. Alwin, D. F., & Krosnick, I. A. (1991). The reliability of survey attitude measurement. The influence of question and respondent attributes. Sociological Methods and Research, 20, 139–181.

  3. Andrews, F. M. (1984). Construct validity and error components of survey measures: A structural equation approach. Public Opinion Quarterly, 48, 409–442.

  4. Belson, W. (1981). The design and understanding of survey questions. London: Gower.

  5. Biemer, P. R. (2011). Latent class analysis of survey errors. Hoboken: Wiley.

  6. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

  7. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrices. Psychological Bulletin, 56, 81–105.

  8. De Castellarnau, A., & Saris, W. E. (2014). A simple way to correct for measurement errors. European Social Survey Education Net (ESS EduNet). http://essedunet.nsd.uib.no/cms/topics/measurement/

  9. Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. Hoboken: Wiley.

  10. Goldberger, A. S., & Duncan, O. D. (Eds.). (1973). Structural equation models in the social sciences. New York: Seminar Press.

  11. Hagenaars, J. (1988). Latent structure model with direct effects between indicators; local dependency models. Sociological Methods and Research, 16, 379–405.

  12. Hambleto, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. London: Sage.

  13. Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago: University of Chicago Press.

  14. Heise, D. R. (1969). Separating reliability and stability in test–retest-correlation. American Sociological Review, 34, 93–101.

  15. Jöreskog, K. G. (1973). A general method for estimating a linear structural equation system. In A. S. Goldberger & O. D. Duncan (Eds.), Structural equation models in the social sciences. New York: Academic Press.

  16. Költringer, R. (1993). Gültigkeit von Umfragendaten. Wien: Bohlau.

  17. Költringer, R. (1995). Measurement quality in Austrian personal interview surveys. In W. E. Saris & A. Münnich (Eds.), The multitrait-multimethod approach to evaluate measurement instruments (pp. 207–225). Budapest: Eötvös University Press.

  18. Lawley, D. N., & Maxwell, A. E. (1971). Factor analysis as a statistical method. London: Butterworth.

  19. Lord, F., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesly.

  20. Mokken, R. J. (1971). A theory and procedure of scale analysis: With applications in political research. The Hague: Mouton.

  21. Molenaar, N. I. (1986). Formuleringseffecten in survey-interviews. PhD thesis, Amsterdam: Free University.

  22. Oberski, D.T. (2011). Measurement errors in comparative surveys. PhD thesis, University of Tilburg.

  23. Oberski, D.T., Gruner, T., & Saris, W. E. (2011). The prediction procedure the quality of the questions based on the present data base of questions In W. E Saris, D. Oberski, M. Revilla, D. Zavalla, L. Lilleoja, I. Gallhofer, & T. Grüner. (Eds.), The development of the Program SQP 2.0 for the prediction of the quality of survey questions. RECSM Working paper 24, chapter 6.

  24. Saris, W. E, Oberski, D., Revilla, M., Zavalla, D., Lilleoja, L., Gallhofer, I., & Grüner, T. (2011). The development of the Program SQP 2.0 for the prediction of the quality of survey questions. RECSM Working paper 24.

  25. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Education Research.

  26. Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation and analysis of questionnaires for survey research. Hoboken: Wiley.

  27. Saris, W. E., & Gallhofer, I. N. (2014). Design, evaluation and analysis of questionnaires for survey research (2nd ed.). Hoboken: Wiley.

  28. Saris, W. E., Satorra, A., & Coenders, G. (2004). A new approach for evaluating quality of measurement instruments. Sociological Methodology, 3, 311–347.

  29. Oberski, D. T., & Satorra, A. (2013). Measurement error models with uncertainty about the error variance. Structural Equation Modeling, 20, 409–428.

  30. Scherpenzeel, A. C. (1995). A question of quality. Evaluating survey questions by multitrait-multimethod studies. Leidschendam: KPN Research.

  31. Schuman, H., & Presser, S. (1981). Questions and answers in attitude survey: Experiments on question form, wording and context. New York: Academic Press.

  32. Sudman, S., & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey Bass.

  33. Torgerson, W. S. (1958). Theory and methods of scaling. London: Wiley.

  34. Tourangeau, R., Rips, J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, MA: Cambridge University Press.

  35. Van Schuur, W. H. (1997). Nonparametric IRT models for dominance and proximity data. In M. Wilson, G. Engelhard Jr, & K. Draney (Eds.), Objective measurement: Theory into practice (Vol. 4, pp. 313–331). Greenwich: Ablex Publishing Corporation.

  36. Vermunt, J. K. (2003). Multilevel latent class models. Sociological Methodology, 33, 213–239.

  37. Wiley, D. E., & Wiley, I. A. (1970). The estimation of measurement error in panel data. American Sociological Review, 35, 112–117.

Download references

Author information

Correspondence to Willem E. Saris.

Appendices

Appendix 1

See Table 7.

Table 7 Procedure to correct for measurement error using LISREL

Appendix 2: The different questions

figurea

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Saris, W.E., Revilla, M. Correction for Measurement Errors in Survey Research: Necessary and Possible. Soc Indic Res 127, 1005–1020 (2016). https://doi.org/10.1007/s11205-015-1002-x

Download citation

Keywords

  • Correction for measurement errors
  • Quality
  • SQP 2.0
  • European Social Survey