Detecting and Deterring Insufficient Effort Responding to Surveys
Responses provided by unmotivated survey participants in a careless, haphazard, or random fashion can threaten the quality of data in psychological and organizational research. The purpose of this study was to summarize existing approaches to detect insufficient effort responding (IER) to low-stakes surveys and to comprehensively evaluate these approaches.
In an experiment (Study 1) and a nonexperimental survey (Study 2), 725 undergraduates responded to a personality survey online.
Study 1 examined the presentation of warnings to respondents as a means of deterrence and showed the relative effectiveness of four indices for detecting IE responses: response time, long string, psychometric antonyms, and individual reliability coefficients. Study 2 demonstrated that the detection indices measured the same underlying construct and showed the improvement of psychometric properties (item interrelatedness, facet dimensionality, and factor structure) after removing IE respondents identified by each index. Three approaches (response time, psychometric antonyms, and individual reliability) with high specificity and moderate sensitivity were recommended as candidates for future application in survey research.
The identification of effective IER indices may help researchers ensure the quality of their low-stake survey data.
This study is a first attempt to comprehensively evaluate IER detection methods using both experimental and nonexperimental designs. Results from both studies corroborated each other in suggesting the three more effective approaches. This study also provided convergent validity evidence regarding various indices for IER.
KeywordsCareless responding Random responding Inconsistent responding Online surveys Data screening
We thank Goran Kuljanin for collecting data for the two studies. We are grateful for the constructive comments from Neal Schmitt and Ann Marie Ryan on an earlier draft of this article.
- Babbie, E. (2001). The practice of social research (9th ed.). Belmont, CA: Wadsworth.Google Scholar
- Beach, D. A. (1989). Identifying the random responder. Journal of Psychology: Interdisciplinary and Applied, 123, 101–103.Google Scholar
- Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods. doi:10.3758/s13428-011-0081-0
- Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota Multiphasic Personality Inventory-2 (MMPI-2): manual for administration and scoring. Minneapolis: University of Minnesota Press.Google Scholar
- Butcher, J. N., Williams, C. L., Graham, J. R., Archer, R. P., Tellegen, A., Ben-Porath, Y. S., et al. (1992). MMPI–A: Minnesota Multiphasic Personality Inventory–Adolescent: manual for administration, scoring, and interpretation. Minneapolis: University of Minnesota Press.Google Scholar
- Costa, P. T., Jr., & McCrae, R. R. (1992). NEO PI-R professional manual. Odessa, FL: Psychological Assessment Resources.Google Scholar
- Costa, P. T., Jr., & McCrae, R. R. (2008). The Revised NEO Personality Inventory (NEO-PI-R). In G. J. Boyle, G. Matthews, & D. H. Saklofske (Eds.), The Sage handbook of personality theory and assessment: personality measurement and testing (pp. 179–198). London: Sage.Google Scholar
- Curran, P. G., Kotrba, L., & Denison, D. (2010, April). Careless responding in surveys: applying traditional techniques to organizational settings. Paper presented at the 25th annual conference of Society for Industrial and Organizational Psychology, Atlanta, GA.Google Scholar
- DiLalla, D. L., & Dollinger, S. J. (2006). Cleaning up data and running preliminary analyses. In F. T. L. Leong & J. T. Austin (Eds.), The psychology research handbook: a guide for graduate students and research assistants (2nd ed., pp. 241–253). Thousand Oaks, CA: Sage.Google Scholar
- Gleason, J. M., & Barnum, D. T. (1991). Predictive probabilities in employee drug-testing. Risk, 2, 3–18.Google Scholar
- Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. D. Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg, The Netherlands: Tilburg University Press.Google Scholar
- Goldberg, L. R., & Kilkowski, J. M. (1985). The prediction of semantic consistency in self-descriptions: characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48, 82–98.PubMedCrossRefGoogle Scholar
- Hartwig, F., & Dearing, B. E. (1979). Exploratory data analysis. Thousand Oaks, CA: Sage.Google Scholar
- Jackson, D. N. (1976). The appraisal of personal reliability. Paper presented at the meetings of the Society of Multivariate Experimental Psychology, University Park, PA.Google Scholar
- Kline, R. B. (2009). Becoming a behavioral science researcher: a guide to producing research that matters. New York: Guilford.Google Scholar
- Lucas, R. E., & Baird, B. M. (2005). Global self-assessment. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 29–42). Washington, DC: American Psychological Association.Google Scholar
- Marsh, H. W. (1987). The Self-Description Questionnaire 1: manual and research monograph. San Antonio, TX: Psychological Corporation.Google Scholar
- Martin, S. L., & Terris, W. (1990). The four-cell classification table in personnel selection: a heuristic device gone awry. The Industrial-Organizational Psychologist, 47(3), 49–55.Google Scholar
- Meade, A. W., & Craig, S. B. (2011, April). Identifying careless responses in survey data. Paper presented at the 26th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL.Google Scholar
- O’Rourke, T. (2000). Techniques for screening and cleaning data for analysis. American Journal of Health Studies, 16, 205–207.Google Scholar
- Rosse, J. G., Levin, R. A., & Nowicki, M. D. (1999, April). Assessing the impact of faking on job performance and counter-productive behaviors. Paper presented at the 14th annual meeting of the Society for Industrial and Organizational Psychology, Atlanta.Google Scholar
- Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.Google Scholar