Abstract
It has become increasingly apparent that the conventional piloting, or pre-testing, of survey questions does not ensure that questions are understood by respondents as researchers intend them to be understood. This has led survey methodologists to advocate the use of indepth testing procedures. While some of the procedures used to identify comprehension problems appear to be reasonably effective, almost all of the procedures used to identify other kinds of problems have not proved to be very useful.
Two reasons stand out as explanations for this general lack of success. First, the probes used are themselves not likely to be interpreted in the same way by different respondents. Second, the cognitive psychological assumptions that have suggested many of the procedures may not adequately reflect the processes that underlie the way questions are answered.
A number of suggestions are made with the view to making the in-depth testing of survey questions a more profitable exercise.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
BelsonW.A. (1981). The Design and Understanding of Survey Questions. London: Gower.
BelsonW.A. (1986). Validity in Survey Research. London: Gower.
BishopG.F. (1992). Qualitative analysis of question-order and context effects: the use of thinkaloud responses, Chapter 11 in N.Schwarz & S.Sudman (eds.), Context Effects in Social Psychological Research. New York: Springer-Verlag.
CantrilH. & E.Fried (1944). The meaning of questions, Chapter 1 in H.Cantril and ResearchAssociates (eds.), Gauging Public Opinion. Princeton: Princeton University Press.
CrutchfieldR. & D.Gordon (1947). Variations in respondents interpretations of an opinion poll question, International Journal of Opinion and Attitude Research 1: 1–2.
DeMarioT. (ed.) (1983). Approaches to Developing Questionnaires. Statistical Policy Working Paper 10. Washington DC: Office of Management and Budget.
FeeJ.F. (1981). Symbols in survey questions: solving the problems of multiple word meanings, Political Methodology 7: 71–195.
FienbergS.E., E.Loftus, & J.M.Tanur (1985a). Cognitive aspects of health survey methodology: an overview, Milbank Memorial Fund Quarterly/Health and Society 63: 547–564.
FienbergS.E., E.Loftus, & J.M.Tanur (1985b). Cognitive aspects of health surveys for public information policy, Milbank Memorial Fund Quarterly/Health and Society 63: 598–614.
FowlerF.Jr. (1992). How unclear terms affect survey data, Public Opinion Quarterly 56: 218–231.
HuntS.D., R.D.SparkmanJr., & J.B.Wilcox (1982). The pretest in survey research: issues and preliminary findings, Journal of Marketing Research XIX: 269–273.
JobeJ.B. & D.J.Mingay (1989). Cognitive research improves questionnaires, Amer. J. of Public Health 79: 1053–1055.
LazarsfeldP.F. (1944). The controversy over detailed interviews — an offer for negotiation, Public Opinion Quarterly 8: 38–60.
LesslerJ.T. & M.G.Sirken (1965). Laboratory-based research on the cognitive aspects of survey methodology: the goals and methods of the national center for health statistics study, Milbank Memorial Fund Quarterly/Health and Society 63: 565–581.
MeansB., D.J.Mingay, A.Nigam & M.Zarrow (1988). A cognitive approach to enhancing health survey reports of medical visits, in M.Gruneberg et al. (eds.), Practical Aspects of Memory. New York: Wiley.
NuckolsR.C. (1953). A note on pre-testing public opinion questions, Journal of Applied Psychology 2: 119–120.
OksenbergL., C.Cannell, & G.Kalton (1991). New strategies for pretesting survey questions, Journal of Official Statistics 7: 349–365.
PayneS.L. (1951). The Art of Asking Questions. Princeton: Princeton University Press.
SchumanH. (1966). The random probe: a technique for evaluating the validity of closed questions, American Sociological Review 31: 218–222.
SirkenM.G., D.J.Mingay, P.N.Royston, D.H.Bercini, & J.B.Jobe (1988). Interdisciplinary research in cognition and survey measurement, in M.Gruneberg, P.Morris, and R.Sykes (eds.), Practical Aspects of Memory: Current Research and Issues. Vol. 1. New York: Wiley.
SmithT.W. (1989). Random probles of GSS questions, International Journal of Public Opinion Research 1: 305–325.
StrackF. & L.L.Martin (1987). Thinking, judging and communicating: a process account of context effects in surveys, Chapter 7 in H.Hippler, N.Schwarz, and S.Sudman (eds.), Social Information Processing and Survey Methodology, New York: Springer-Verlag.
TourangeauR. & R.Rasinski (1988). Cognitive processes underlying context effects in attitude measurement, Psychological Bulletin 103: 299–314.
Willis, G.B., P. Royston, & D. Bercini (1989). Problems with survey questions revealed by cognitively-based interviewing techniques. U.S. Dept. of Commerce, Bureau of the Census, Proceedings of the Fifth Annual Research Conference.
WillisG.B., P.Royston, & D.Bercini (1991). The use of verbal report methods in the development and testing of survey questionnaires, Applied Cognitive Psychology 5: 251–267.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Foddy, W. The in-depth testing of survey questions: A critical appraisal of methods. Qual Quant 30, 361–370 (1996). https://doi.org/10.1007/BF00170142
Issue Date:
DOI: https://doi.org/10.1007/BF00170142