Skip to main content

Pretesting survey instruments: An overview of cognitive methods

Quality of Life Research volume 12, pages 229–238 (2003)Cite this article

Abstract

This article puts forward the case that survey questionnaires, which are a type of measuring instrument, can and should be tested to ensure they meet their purpose. Traditionally survey researchers have been pre-occupied with 'standardising' data collection instruments and procedures such as question wording and have assumed that experience in questionnaire design, coupled with pilot testing of questionnaires, will then ensure valid and reliable results. However, implicit in the notion of standardisation are the assumptions that respondents are able to understand the questions being asked, that questions are understood in the same way by all respondents, and that respondents are willing and able to answer such questions. The development of cognitive question testing methods has provided social researchers with a number of theories and tools to test these assumptions, and to develop better survey instruments and questionnaires. This paper describes some of these theories and tools, and argues that cognitive testing should be a standard part of the development process of any survey instrument.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Fowler FJ. ImprovingSurvey Questions: Design and Evaluation. Applied Social Research Methods Series: 38. Thousand Oaks: Sage, 1995.

    Google Scholar 

  2. Fowler FJ, Mangione TW. Standardized Survey Interviewing: Minimizing Interviewer-Related Error. Applied Social Research Methods Series: 18. Newbury Park: Sage, 1990.

    Google Scholar 

  3. Oksenberg L, Cannell C, Kalton G. New strategies for pretesting survey questions. J Off Stat 1991; 7(3): 349–365.

    Google Scholar 

  4. Belson WA. The Design and Understanding of Survey Questions. Aldershot: Gower, 1981.

    Google Scholar 

  5. Clark HH, Schober MF. Asking questions and influencing answers. In: Tanur JM (ed.) Questions About Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 15–48.

    Google Scholar 

  6. Tourangeau R, Rips LJ, Rasinski K. The Psychology of Survey Response. Cambridge: Cambridge University Press, 2000.

    Google Scholar 

  7. Krosnick JA. Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 1991; 5: 213–236.

    Google Scholar 

  8. Krosnick JA, Alwin DF. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opin Quart 1987; 51: 201–219.

    Google Scholar 

  9. Cannell CF, Oksenberg L, Converse JM (eds). Experiments in Interviewing Techniques: Field Experiments in Health Reporting, 1971-1977. Survey Research Centre, Institute of Social Research, University of Michigan. Ann Arbor, MI: University of Michigan, 1979.

    Google Scholar 

  10. Cantril H. Gauging Public Opinion. Princeton, NJ: Princeton University Press, 1944.

    Google Scholar 

  11. Payne SL. The Art of Asking Questions. Princeton, NJ: Princeton University Press, 1951.

    Google Scholar 

  12. Rugg D. Experiments in wording questions: II. Public Opin Quart. 1941; 5: 91–92.

    Google Scholar 

  13. Sudman S, Bradburn NM. Response Effects in Surveys: A Review and Synthesis. Chicago: Aldine, 1974.

    Google Scholar 

  14. Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984.

    Google Scholar 

  15. Hippler HJ, Schwarz N, Sudman S (eds). Social Information Processing and Survey Methodology. New York: Springer-Verlag, 1987.

    Google Scholar 

  16. Tanur JM. Looking backwards and forwards at the CASM movement. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 13–19

    Google Scholar 

  17. Aborn M. CASM revisited. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 21–38.

    Google Scholar 

  18. Tourangeau R. Cognitive sciences and survey methods. In: Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984: 73–100.

    Google Scholar 

  19. Esposito J, Jobe J. A General Model of the Survey Interaction Process. Bureau of the Census Seventh Annual Research Conference Proceedings. Washington DC: US Bureau of the Census, 1991: 537–560.

    Google Scholar 

  20. Schober MF. Making sense of questions: an interactional approach. In: Sirken, MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 77–93.

    Google Scholar 

  21. Suchman L, Jordan B. Validity and the collaborative construction of meaning in face-to-face surveys. In: Tanur JM (ed.) Questions about Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 241–267.

    Google Scholar 

  22. Sudman S, Bradburn NM, Schwarz N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Josey-Bass, 1996.

    Google Scholar 

  23. Tulving E, Thomson DM. Encoding specificity and retrieval processes in episodic memory. Psychol Rev 1973; 80: 352–373.

    Google Scholar 

  24. Anderson J. The Architecture of Cognition. Cambridge, MA: Harvard University Press, 1983.

    Google Scholar 

  25. Loftus E, Palmer J. Reconstruction of automobile destruction: An example of the interaction between language and memory. J Verb Learn Verb Behav 1974; 13: 585–589

    Google Scholar 

  26. Jobe JB, Tourangeau R, Smith AF. Contributions of survey research to the understanding of memory. Appl Cogn Psychol 1993; 7: 567–584.

    Google Scholar 

  27. Blair EA, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14: 280–288.

    Google Scholar 

  28. Sudman S, Schwarz N. Contributions of cognitive psychology to advertising research. J Advertising Res 1989; 29: 43–53.

    Google Scholar 

  29. Tversky A, Kahneman D. Availability: A heuristic for judging frequency and probability. Cogn Psychol 1973; 5: 207–232.

    Google Scholar 

  30. Tversky A, Kahneman D. Judgement under uncertainty: Heuristics and biases. Science 1974; 185: 1124–1131.

    Google Scholar 

  31. Schwarz N, Hippler HJ. Response alternatives: The impact of their choice and presentation order. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds), Measurement Errors in Surveys. New York: Wiley, 1991: 41–56.

    Google Scholar 

  32. Czaja, R. Questionnaire pretesting comes of age. Market Bull 1998; 9: 52–66.

    Google Scholar 

  33. Forsyth BH, Lessler JT. Cognitive laboratory methods: A taxonomy. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds) Measurement Errors in Surveys. New York: Wiley, 1991: 393–418.

    Google Scholar 

  34. Jobe JB, Mingay DJ. (1991) Cognition and survey measurement: History and overview. Appl Cogn Psychol 1991; 5: 175–192.

    Google Scholar 

  35. Schwarz N. Questionnaire design: The rocky road from concepts to answers. In: LybergL, Biemer P, Collins M, de Leeuw E, Dippo C, Schwarz N, Trewin D (eds), Survey Measurement and Process Quality. New York: Wiley, 1997: 29–45.

    Google Scholar 

  36. Willis GB. Cognitive Interviewing and Questionnaire Design: A training manual. Hyattsville, MD: National Centre for Health Statistics, 1994.

    Google Scholar 

  37. Wilson TD, LaFleur SJ, Anderson DA. The validity and consequences of verbal reports about attitudes. In: Schwarz N, Sudman S (eds), Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey-Bass, 1995: 91–114.

    Google Scholar 

  38. Willis GB, DeMaio T, Harris-Kojetin B. Is the bandwagon headed to the methodological promised land? Evaluating the validity of cognitive interviewing techniques. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 133–153.

    Google Scholar 

  39. Presser S, Blair J. Survey pretesting: Do different methods produce different results? In: Marsden PV (ed), Sociological Methodology. Beverley Hills, CA: Sage, 1994; 24: 73–104.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Centre for Social Research, London, UK

    Debbie Collins

Authors
  1. Debbie Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Collins, D. Pretesting survey instruments: An overview of cognitive methods. Qual Life Res 12, 229–238 (2003). https://doi.org/10.1023/A:1023254226592

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1023254226592

  • Cognitive aspects of survey methodology
  • Cognitive interviews
  • Data quality