Quality of Life Research

, Volume 12, Issue 3, pp 229–238

Pretesting survey instruments: An overview of cognitive methods

  • Debbie Collins
Article

Abstract

This article puts forward the case that survey questionnaires, which are a type of measuring instrument, can and should be tested to ensure they meet their purpose. Traditionally survey researchers have been pre-occupied with 'standardising' data collection instruments and procedures such as question wording and have assumed that experience in questionnaire design, coupled with pilot testing of questionnaires, will then ensure valid and reliable results. However, implicit in the notion of standardisation are the assumptions that respondents are able to understand the questions being asked, that questions are understood in the same way by all respondents, and that respondents are willing and able to answer such questions. The development of cognitive question testing methods has provided social researchers with a number of theories and tools to test these assumptions, and to develop better survey instruments and questionnaires. This paper describes some of these theories and tools, and argues that cognitive testing should be a standard part of the development process of any survey instrument.

Cognitive aspects of survey methodology Cognitive interviews Data quality 

References

  1. 1.
    Fowler FJ. ImprovingSurvey Questions: Design and Evaluation. Applied Social Research Methods Series: 38. Thousand Oaks: Sage, 1995.Google Scholar
  2. 2.
    Fowler FJ, Mangione TW. Standardized Survey Interviewing: Minimizing Interviewer-Related Error. Applied Social Research Methods Series: 18. Newbury Park: Sage, 1990.Google Scholar
  3. 3.
    Oksenberg L, Cannell C, Kalton G. New strategies for pretesting survey questions. J Off Stat 1991; 7(3): 349–365.Google Scholar
  4. 4.
    Belson WA. The Design and Understanding of Survey Questions. Aldershot: Gower, 1981.Google Scholar
  5. 5.
    Clark HH, Schober MF. Asking questions and influencing answers. In: Tanur JM (ed.) Questions About Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 15–48.Google Scholar
  6. 6.
    Tourangeau R, Rips LJ, Rasinski K. The Psychology of Survey Response. Cambridge: Cambridge University Press, 2000.Google Scholar
  7. 7.
    Krosnick JA. Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 1991; 5: 213–236.Google Scholar
  8. 8.
    Krosnick JA, Alwin DF. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opin Quart 1987; 51: 201–219.Google Scholar
  9. 9.
    Cannell CF, Oksenberg L, Converse JM (eds). Experiments in Interviewing Techniques: Field Experiments in Health Reporting, 1971-1977. Survey Research Centre, Institute of Social Research, University of Michigan. Ann Arbor, MI: University of Michigan, 1979.Google Scholar
  10. 10.
    Cantril H. Gauging Public Opinion. Princeton, NJ: Princeton University Press, 1944.Google Scholar
  11. 11.
    Payne SL. The Art of Asking Questions. Princeton, NJ: Princeton University Press, 1951.Google Scholar
  12. 12.
    Rugg D. Experiments in wording questions: II. Public Opin Quart. 1941; 5: 91–92.Google Scholar
  13. 13.
    Sudman S, Bradburn NM. Response Effects in Surveys: A Review and Synthesis. Chicago: Aldine, 1974.Google Scholar
  14. 14.
    Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984.Google Scholar
  15. 15.
    Hippler HJ, Schwarz N, Sudman S (eds). Social Information Processing and Survey Methodology. New York: Springer-Verlag, 1987.Google Scholar
  16. 16.
    Tanur JM. Looking backwards and forwards at the CASM movement. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 13–19Google Scholar
  17. 17.
    Aborn M. CASM revisited. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 21–38.Google Scholar
  18. 18.
    Tourangeau R. Cognitive sciences and survey methods. In: Jabine T, Straf M, Tanur J, Tourangeau R (eds). Cognitive Aspects of Survey Methodology: Building a Bridge Between the Disciplines. Washington, DC: National Academy Press, 1984: 73–100.Google Scholar
  19. 19.
    Esposito J, Jobe J. A General Model of the Survey Interaction Process. Bureau of the Census Seventh Annual Research Conference Proceedings. Washington DC: US Bureau of the Census, 1991: 537–560.Google Scholar
  20. 20.
    Schober MF. Making sense of questions: an interactional approach. In: Sirken, MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 77–93.Google Scholar
  21. 21.
    Suchman L, Jordan B. Validity and the collaborative construction of meaning in face-to-face surveys. In: Tanur JM (ed.) Questions about Questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation, 1992: 241–267.Google Scholar
  22. 22.
    Sudman S, Bradburn NM, Schwarz N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Josey-Bass, 1996.Google Scholar
  23. 23.
    Tulving E, Thomson DM. Encoding specificity and retrieval processes in episodic memory. Psychol Rev 1973; 80: 352–373.Google Scholar
  24. 24.
    Anderson J. The Architecture of Cognition. Cambridge, MA: Harvard University Press, 1983.Google Scholar
  25. 25.
    Loftus E, Palmer J. Reconstruction of automobile destruction: An example of the interaction between language and memory. J Verb Learn Verb Behav 1974; 13: 585–589Google Scholar
  26. 26.
    Jobe JB, Tourangeau R, Smith AF. Contributions of survey research to the understanding of memory. Appl Cogn Psychol 1993; 7: 567–584.Google Scholar
  27. 27.
    Blair EA, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14: 280–288.Google Scholar
  28. 28.
    Sudman S, Schwarz N. Contributions of cognitive psychology to advertising research. J Advertising Res 1989; 29: 43–53.Google Scholar
  29. 29.
    Tversky A, Kahneman D. Availability: A heuristic for judging frequency and probability. Cogn Psychol 1973; 5: 207–232.Google Scholar
  30. 30.
    Tversky A, Kahneman D. Judgement under uncertainty: Heuristics and biases. Science 1974; 185: 1124–1131.Google Scholar
  31. 31.
    Schwarz N, Hippler HJ. Response alternatives: The impact of their choice and presentation order. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds), Measurement Errors in Surveys. New York: Wiley, 1991: 41–56.Google Scholar
  32. 32.
    Czaja, R. Questionnaire pretesting comes of age. Market Bull 1998; 9: 52–66.Google Scholar
  33. 33.
    Forsyth BH, Lessler JT. Cognitive laboratory methods: A taxonomy. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds) Measurement Errors in Surveys. New York: Wiley, 1991: 393–418.Google Scholar
  34. 34.
    Jobe JB, Mingay DJ. (1991) Cognition and survey measurement: History and overview. Appl Cogn Psychol 1991; 5: 175–192.Google Scholar
  35. 35.
    Schwarz N. Questionnaire design: The rocky road from concepts to answers. In: LybergL, Biemer P, Collins M, de Leeuw E, Dippo C, Schwarz N, Trewin D (eds), Survey Measurement and Process Quality. New York: Wiley, 1997: 29–45.Google Scholar
  36. 36.
    Willis GB. Cognitive Interviewing and Questionnaire Design: A training manual. Hyattsville, MD: National Centre for Health Statistics, 1994.Google Scholar
  37. 37.
    Wilson TD, LaFleur SJ, Anderson DA. The validity and consequences of verbal reports about attitudes. In: Schwarz N, Sudman S (eds), Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey-Bass, 1995: 91–114.Google Scholar
  38. 38.
    Willis GB, DeMaio T, Harris-Kojetin B. Is the bandwagon headed to the methodological promised land? Evaluating the validity of cognitive interviewing techniques. In: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R (eds), Cognition and Survey Research. New York: Wiley, 1999: 133–153.Google Scholar
  39. 39.
    Presser S, Blair J. Survey pretesting: Do different methods produce different results? In: Marsden PV (ed), Sociological Methodology. Beverley Hills, CA: Sage, 1994; 24: 73–104.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Debbie Collins
    • 1
  1. 1.National Centre for Social ResearchLondonUK

Personalised recommendations