Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions
- 1.7k Downloads
Insufficient effort responding (IER), which occurs due to a lack of motivation to comply with survey instructions and to correctly interpret item content, represents a serious problem for researchers and practitioners who employ survey methodology (Huang et al. 2012). Extending prior research, we examine the validity of the infrequency approach to detecting IER and assess participant reactions to such an approach.
Two online surveys (Studies 1 and 2) completed by employed undergraduates were utilized to assess the validity of the infrequency approach. An on-line survey of paid participants (Study 3) and a paper-and-pencil survey in an organization (Study 4) were conducted to evaluate participant reactions, using random assignment into survey conditions that either did or did not contain infrequency items.
Studies 1 and 2 provided evidence for the reliability, unidimensionality, and criterion-related validity of the infrequency scales. Study 3 and Study 4 showed that surveys that contained infrequency items did not lead to more negative reactions than did surveys that did not contain such items.
The current findings provide evidence of the effectiveness and feasibility of the infrequency approach for detecting IER, supporting its application in low-stakes organizational survey contexts.
The current studies provide a more in-depth examination of the infrequency approach to IER detection than had been done in prior research. In particular, the evaluation of participant reactions to infrequency scales represents a novel contribution to the IER literature.
KeywordsInsufficient effort responding Careless responding Random responding Inconsistent responding Data screening Online surveys
We would like to thank Neal Schmitt, Fred Oswald, and Adam Meade for comments on earlier drafts of this paper, and Jessica Keeney and Paul Curran for suggestions during the early stages of this research. We also thank Travis Walker for assisting with data collection.
- Furr, R. M., & Bacharach, V. R. (2014). Psychometrics: An introduction. Thousand Oaks, CA: Sage.Google Scholar
- Hogan, R., & Hogan, J. (2007). Hogan Personality Inventory manual (3rd ed.). Tulsa, OK: Hogan Assessment Systems.Google Scholar
- Huang, J. L., Bowling, N. A., & Liu, M. (2014). The effects of insufficient effort responding on the convergent and discriminant validity of substantive measures. Unpublished manuscript.Google Scholar
- Huang, J. L., Liu, M., & Bowling, N. A. (2014, May). Insufficient effort responding: Uncovering an insidious threat to data quality. In J. H. Huang & M. Liu (Co-chairs), Insufficient effort responding to surveys: From impact to solutions. Symposium to be presented at the Annual Conference of Society for Industrial and Organizational Psychology, Honolulu, HA.Google Scholar
- Jackson, D. N. (1974). Personality Research Form manual. Goshen, NY: Research Psychologists Press.Google Scholar
- Liu, M., Bowling, N. A., Huang, J. L., & Kent, T. A. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial-Organizational Psychologist, 51(1), 32–38.Google Scholar
- Muthén, B., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Unpublished manuscript.Google Scholar
- Muthén, L. K., & Muthén, B. O. (2011). Mplus User’s Guide. Los Angeles, CA: Muthén & Muthén.Google Scholar