Abstract
Surveys are one of the most popular ways to collect employee information. Because of their widespread use, data quality is an increasingly important concern. The purpose of this paper is to (1) introduce the intra-individual response variability (IRV) index as an easily calculated and flexible way to detect insufficient effort responding (IER); (2) examine the extent to which various IER indices detect the same or different respondents engaging in IER behavior; and (3) investigate relationships between individual differences and commonly used IER indices to better understand systematic and theoretically relevant IER behavior. In a two-part study, 199 undergraduates responded to questionnaires online, and various IER indices were calculated. The IRV index identifies different respondents than other IER indices. Values on the IRV index (as well as other IER indices) are related to scores on theoretically meaningful individual differences in conscientiousness, agreeableness, and boredom proneness. This study provides researchers with a robust, easily calculated, and flexible means for screening questionnaire data for IER behavior. Practical recommendations for finding and making decisions about IER behavior patterns are provided. This study introduces the IRV index, an extension of the long string, used to identify survey research participants who likely engaged in one type of IER behavior. It is also one of the first studies to evaluate the extent to which IER indices identify different respondents as having engaged in IER and provides additional evidence that values on these indices are related to individual differences.
Similar content being viewed by others
References
Allen, N. J., & Meyer, J.P. (1990). The measurement and antecedents of affective, continuance, and normative commitment to the organization. Journal of Occupational Psychology, 63, 1–18.
Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995–2008: a meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25, 335–349.
Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: black-white differences in response style. Public Opinion Quarterly, 48, 491–509.
Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48, 5–37.
Bardo, J. W., Yeager, S. J., & Klingsporn, M. J. (1982). Preliminary assessment of format-specific central tendency and leniency error in summated rating scales. Perceptual and Motor Skills, 54, 227–234.
Barrick, M. A., & Mount, M. K. (1991). The big five personality dimensions and job performance: a meta-analysis. Personnel Psychology, 44, 1–26.
Barrick, M. R., Mitchell, T. R., & Stewart, G. L. (2003). Situational and motivational influences on trait-behavior relationships. In M. R. Barrick & A. M. Ryan (Eds.), Personality and work: reconsidering the role of personality in organizations (pp. 60–82). San Francisco: Jossey-Bass.
Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMP 2 random responding indices: validation using a self-report methodology. Psychological Assessment, 4, 340–345.
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229.
Christian, L.M., Dillman, D.A., & Smyth, J.D. (2008). The effects of mode and format on answers to scalar questions in telephone and web surveys. Advances in telephone survey methodology, 250–275.
Costa Jr., P. T., & McCrae, R. R. (2008). The revised NEO personality inventory (NEO-PI-R). In G. J. Boyle, G. Matthews, & D. H. Saklofske (Eds.), The sage handbook of personality theory and assessment: personality measurement and testing (pp. 179–198). London: Sage.
DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171–181.
Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.
Farmer, R., & Sundberg, N. D. (1986). Boredom proneness: the development and correlates of a new scale. Journal of Personality Assessment, 50, 4–17.
Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg: Tilburg University Press.
Greene, R. L. (1978). Am empirically derived MMPI carelessness scale. Journal of Clinical Psychology, 34, 407–410.
Guay, R. P., Oh, I. S., Choi, D., Mitchell, M. S., Mount, M. K., & Shin, K. (2013). The interactive effect of conscientiousness and agreeableness on job performance dimensions in South Korea. International Journal of Selection and Assessment, 21, 233–238.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.
Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
Little, R. J. A., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: treatment of no-shows using Rubin’s causal model. Psychological Methods, 3, 147–159.
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.
McGonagle, A. K., Huang, J. L., & Walsh, B. M. (2015). Insufficient effort survey responding: an under-appreciated problem in work and Organisational Health Psychology research. Applied Psychology, 65, 287–321. doi:10.1111/apps.12058.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–457.
Merritt, S. M. (2012). The two-factor solution to Allen and Meyer’s (1990) affective commitment scale: effects of negatively worded items. Journal of Business and Psychology, 27, 421–436.
Nichols, D. S., Greene, R. L., & Schmolck, P. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials. Journal of Clinical Psychology, 45, 239–250.
Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behavior. Personnel Psychology, 48, 775–802.
Rogelberg, S. G., & Stanton, J. M. (2007). Introduction: understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195–209.
Skowronski, M. (2012). When the bored behave badly (or exceptionally). Personnel Review, 41, 143–159.
Watt, J. D., & Hargis, M. B. (2010). Boredom proneness: its relationship with subjective underemployment, perceived organizational support, and job performance. Journal of Business and Psychology, 25, 163–174.
Weiner, S.P., & Dalessio, A.T. (2006). Oversurveying: Causes, consequences, and cures. Getting action from organizational surveys: New concepts, methods, and applications, 294–311.
Woods, C. M. (2006). Careless responding to reverse-worded items: implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186–191. doi:10.1007/s10862-005-9004-7.
Author information
Authors and Affiliations
Corresponding author
Appendix A. Catch Question Index Used in the Present Study.
Appendix A. Catch Question Index Used in the Present Study.
Scoring: Scoring recommendations are based on the use of a 5-point Likert-type response scale from 1 = Strongly Disagree to 5 = Strongly Agree). For items without an * designation, responses other than Strongly Disagree or Disagree are given one point for the Catch Question Index. For the items denoted by an *, responses other than Strongly Agree or Agree are given one point for the Catch Question Index.
Note: Item 11 is specific to a college student population.
-
1.
I lie 100 % of the time.
-
2.
I am under three years old.
-
3.
I have traveled around the world twenty three times.
-
4.
I have been to the moon.
-
5.
I own Starbucks.
-
6.
I am ninety-nine years old.
-
7.
I wrote the questions on this survey.
-
8.
I graduated high school at age eight.
-
9.
I am the president of the United States.
-
10.
I have been eaten by a shark.
-
11.
I have never been to a college class before.
-
12.
I can walk on water.
-
13.
I have never laughed before.
-
14.
I am the richest person in the world.
-
15.
I know that the sky is blue.*
-
16.
I have never used a computer before.
-
17.
I eat food.*
-
18.
I have seen a moving car.*
-
19.
I know how to count to ten.*
-
20.
I know how to spell my name.*
-
21.
I know the months of the year.*
-
22.
I have never been mad before.
-
23.
I never smile.
-
24.
I do not know what dancing is.
-
25.
I have never used a pen before.
Rights and permissions
About this article
Cite this article
Dunn, A.M., Heggestad, E.D., Shanock, L.R. et al. Intra-individual Response Variability as an Indicator of Insufficient Effort Responding: Comparison to Other Indicators and Relationships with Individual Differences. J Bus Psychol 33, 105–121 (2018). https://doi.org/10.1007/s10869-016-9479-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10869-016-9479-0