Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Retrospective questions: data quality, task difficulty, and the use of a checklist

  • 149 Accesses

  • 16 Citations

Abstract

Data from a national panel study with waves in 1987 and 1991 (N=1257), present evidence of the low accuracy of responses to retrospective questions, concerning both attitudes and behaviour. Applying a split ballot design, it is investigated whether using a checklist improves the response accuracy for a retrospective question about one single event: how respondents (N=363) did obtain the job they had four years ago. Furthermore interaction effects of ‘task difficulty’ are examined.

The response accuracy indeed increases by using a checklist; however, this increase is not statistically significant. The expected increase of the checklist effect with higher ‘task difficulty’ appears for longer recall intervals, but not for more frequent changes of jobs.

It turned out that for male respondents all the predicted effects are indeed clearly present. But for female respondents the checklist appeared to have no effect, irrespective of the task difficulty. A tentative explanation for this unexpected gender effect is suggested by pointing at indications that the male and female respondents differed in the way they obtained their job.

This is a preview of subscription content, log in to check access.

References

  1. Baddeley, A. (1979). The limitations of human memory: Implications for the design of retrospective surveys, pp. 13–27 in: L. Moss & H. Goldstein (eds.),The Recall Method in Social Surveys. London: University of London Institute of Education.

  2. Baddeley, A. D., Lewis, V. & Nimmo-Smith, I. (1978). When did you last ...?, pp. 77–83 in: M. M. Gruneberg, P. E. Morris & R. N. Sykes (eds.),Practical Aspects of Memory. New York: Academic Press.

  3. Bahr, H. M. & Houts, K. C. (1971). Can you trust a homeless man? A comparison of official records and interview responses by bowery men,Public Opinion Quart. 35: 374–382.

  4. Belson, W. & Duncan, J. A. (1962). A comparison of the checklist and the open response questioning systems,Appl. Stat. 11: 120–132.

  5. Bradburn, N. & Danis, C. (1984). Potential contributions of cognitive research to survey questionnaire design, pp. 101–129, in: T. B. Jabine, M. L. Straf, J. M. Tanur & R. Tourangeau (eds.),Cognitive Aspects of Survey Methodology. Building a Bridge Between Disciplines. Washington, D. C.: National Academy Press.

  6. Cannell, C. F., Marquis, Kent H. & Laurent, A. (1977). A Summary of Research Studies of Interviewing Methodology, 1959–1970,Vital Health Statistics, Ser. 2, No. 69. Rockville, MD: U.S. Department of Health, Education, and Welfare.

  7. Cannell, C. F., Miller, P. V. & Oksenberg, L. (1981). Research on Interviewing Techniques, pp. 389–437 in: S. Leinhardt (ed.),Sociological Methodology. San Francisco: Jossey-Bass.

  8. Cannell, C. F., Oksenburg, L. & Converse, J. M. (1977). Striving for response accuracy: Experiments in interviewing techniques,J. Marketing Res. 14: 306–315.

  9. Cohen, J. (1960). A coefficient of agreement for nominal scales,Educat. Psycholog. Measurement 20: 37–46.

  10. Conway, M. & Ross, M. (1984). Getting what you want by revising what you had,J. Personality, Social Psychol. 47(4): 738–748.

  11. Dijkstra, W. (ed.) (1989).Het Proces van Sociale Integratie van Jong-Volwassenen: de Gegevens Verzameling voor de Eerste Hoofdmeting (The Process of Social Integration of Young Adults: the Data Collection for the First Main Wave). Amsterdam: VU Boekhandel/Uitgeverij.

  12. Dijkstra, W. (ed.) (1993).Het Proces van Sociale Integratie van Jong-Volwassenen: de Gegevens, Verzameling voor de Tussenmeting en de Tweede Hoofdmeting (The Process of Social Integration of Young Adults: the Data Collection for the In-Between Wave and the Second Main Wave). Amsterdam: VU Boekhandel/Uitgeverij.

  13. Eisenhower, D., Mathiowetz, N. A. & Morganstein, D. (1991). Recall error: Sources and bias reduction techniques, pp. 127–144 in: P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz & S. Sudman (eds.),Measurement Errors in Surveys. New York: Wiley.

  14. Finney, H. C. (1981). Improving the reliability of retrospective survey measures: Results of a longitudinal yield survey,Evaluation Rev. 5(2): 207–229.

  15. Fleiss, J. L. (1981).Statistical Methods for Rates and Proportions (2nd edn.). New York: Wiley.

  16. Gems, B., Ghosh, D. & Hitlin, R. (1982). A recall experiment: impact of time on recall of recreational fishing trips, pp. 372–375 in:Proc. Section on Survey Research Methods. Washington: American Statistical Association.

  17. Haggard, E. A., Brekstad, A. & Skard, A. G. (1960). On the reliability of the anamnestic interview,J. Abnormal, Social Psychol. 61(3): 311–318.

  18. Kalton, G. & Schuman, H. (1982). The effect of the question on survey responses: A review,J. Roy. Stat. Soc. 145: 42–57.

  19. Klein, D. N. & Rubovits, D. R. (1987). The reliability of subject's reports on stressful life events inventories: a longitudinal study,J. Behavioral Med. 10(5): 501–512.

  20. Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data,Biometrics 33: 159–174.

  21. Mathiowetz, N. A. (1985). The problem of omissions and telescoping error: New evidence from a study of unemployment, pp. 482–487 in:Proc. Section on Survey Research Methods. Washington: American Statistical Association.

  22. Mathiowetz, N. A. & Duncan, G. J. (1988). Out of work, out of mind: response errors in retrospective reports of unemployment,J. Business, Economic Stat. 6(2): 221–229.

  23. Means, B., Mingay, D. J., Nigam, A. & Zarrow, M. (1988). A cognitive approach to enhancing health survey reports of medical visits, pp. 537–542 in: M. M. Gruneberg, P. E. Morris & R. N. Sykes (eds.),Practical Aspects of Memory:Current Research and Issues: Vol. 1. Memory in Everyday Life. Chichester: Wiley.

  24. Neter, J. & Waksberg, J. (1964). A study of response errors in expenditures data from household interviews,J. Amer. Stat. Ass. 59: 18–55.

  25. Powers, E. A., Goudy, W. J. & Keith, P. M. (1978). Congruence between panel and recall data in longitudinal research,Public Opinion Quart. 42(3): 380–389.

  26. Roos, E. J. J. & Bartels, C. P. A. (1989).Hollen of Stilstaan, Wisselingen in Arbeidsmarktdiscrepanties in de Jaren '80 (All or Nothing, Variations in Labour Market Discrepancies in the Eighties). OSA-rapport. Den Haag: Staats Drukkerij en Uitgeverij.

  27. Schwarz, N., Hippler, H.-J. & Noelle-Neumann, E. (1991). A cognitive model of response order effects in survey measurement, pp. 187–201 in: N. Schwarz & S. Sudman (eds.),Context Effects in Social and Psychological Research. New York: Springer.

  28. Schwarz, N. & Hippler, H.-J. (1991). Response alternatives: The impact of their choice and presentation order, pp. 41–56 in: P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz & S. Sudman (eds.),Measurement Errors in Surveys. New York: Wiley.

  29. Sudman, S. & Bradbum, N. M. (1973). Effects of time and memory factors on response in surveys.J. Amer. Stat. Ass. 68(344): 805–815.

  30. Sudman, S. & Bradburn, N. M. (1982).Asking Questions: A Practical Guide to Questionnaire Design, London: Jossey-Bass.

  31. Van der Vaart, W. (1990).Response Effects of Retrospective Questions: a Review, Technical Report 1. Department of Social Research Methodology, Vrije Universiteit, Amsterdam. Sipublication wi-90/41.

  32. Van der Vaart, W. (1991).Measures to Reduce Response Effects of Recall Questions, Technical Report 2. Department of Social Research Methodology, Vrije Universiteit, Amsterdam. SI-publication wi-91/44.

  33. Van der Zouwen, J., Dijkstra, W. & Van der Vaart, W. (1993). Effects of measures aimed at increasing the quality of recall data,Bull. Méthodol. Sociol. 39: 3–19.

  34. Zinkhan, G. M. (1982). An empirical investigation of aided recall in advertising, pp. 137–160 in: J. H. Leigh & C. R. Martin (eds.),Current Issues and Research in Advertising, 1982. Ann Arbor, Michigan: Division of Research, Graduate School of Business Administration, University of Michigan.

Download references

Author information

Additional information

This article is produced within the framework of the research program: “The process of social integration of young adults” (SI-project). That research program is directed by the Department of Social Research Methodology and the Department of Work and Organizational Psychology at the Vrije Universiteit Amsterdam. An earlier version of this article was presented at the ‘1993 European Meeting of the Psychometric Society’, Barcelona, Spain.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Van Der Vaart, W., Van Der Zouwen, J. & Dijkstra, W. Retrospective questions: data quality, task difficulty, and the use of a checklist. Qual Quant 29, 299–315 (1995). https://doi.org/10.1007/BF01101976

Download citation

Keywords

  • Interaction Effect
  • Data Quality
  • Frequent Change
  • Single Event
  • Task Difficulty