Skip to main content

Stop With the Questions Already! Does Data Quality Suffer for Scales Positioned Near the End of a Lengthy Questionnaire?

Journal of Business and Psychology volume 37, pages 1099–1116 (2022)Cite this article

Abstract

Research questionnaires frequently include dozens—if not hundreds—of self-report items. Lengthy questionnaires, however, are often a necessity. In some cases, they are needed to assess the many variables found in a complex model; in other cases, they are the result of the inclusion of a single lengthy measure. This raises an important question: Do participants provide accurate responses to measures positioned at the end of a lengthy questionnaire? One possibility is that participants experience fatigue during questionnaire completion, leading them to engage in careless responding, and thus compromising the accuracy of their responses. Another possibility is that even the longest research questionnaires are generally too short to evoke participant fatigue. This latter possibility suggests that participants are largely able to maintain their attention while completing most questionnaires. Given the lack of clarity on this issue, we conducted two experiments (Study 1 N = 244; Study 2 N = 461) in which we randomly assigned each participant to complete a block of target scales at either the beginning or the end of a lengthy (> 300-item) questionnaire. Each participant also recruited an informant who provided reports of the participant’s personality, attitudes, and behaviors. These informant data allowed us to examine the effects of the experimental manipulation on the target scales’ convergent and criterion-related validity. The findings of both studies indicated that the target scales performed similarly across the two conditions. Given the ubiquity of lengthy questionnaires, these findings have far-reaching practical implications.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1

Notes

  1. Personality inventories often include hundreds of items. Some noteworthy examples include the Revised NEO Personality Inventory (Costa & McCrae, 1992; 240 items), the Minnesota Multiphasic Personality Inventory-2 Restructured Form (Ben-Porath & Tellegen, 2008; 338 items), and the California Psychological Inventory (Gough & Bradley, 1996; 434 items).

  2. Convergent validity is present when a given measure is strongly related to alternative measures of the same construct (e.g., when a self-reported measure of conscientiousness is strongly related to an informant-reported measure of conscientiousness; see Campbell & Fiske, 1959; Hinkin, 1998). Criterion-related validity, on the other hand, is present when a given predictor measure relates to a criterion measure in the hypothesized manner (e.g., when a self-reported measure of conscientiousness is positively related to informant-reported job performance; see Cronbach & Meehl, 1955; Hinkin, 1998).

  3. Involvement in a romantic relationship, of course, was not inherently required by our research question and the inclusion of this requirement may limit the extent to which Study 2 findings can be generalized to participants who are not involved in a romantic relationship; however, our need to collect dyad data from Prolific made it necessary to impose this requirement. Prolific asks participants of dyad studies to recruit their romantic partners. Closer inspection of our data suggests that most (94%)—but not all—informants were the participants’ romantic partners. The remaining informants were either family members (4%), friends (2%), or coworkers (< 1%) of the participants.

  4. The complete set of five infrequency items is available from the first author.

References

  • Ackerman, P. L., & Kanfer, R. (2009). Test length and cognitive fatigue: An empirical examination of effects on performance and test-taker reactions. Journal of Experimental Psychology: Applied, 15, 163–181.

    PubMed  Google Scholar 

  • Ackerman, P. L., Kanfer, R., Shapiro, S. W., Newton, S., & Beier, M. E. (2010). Cognitive fatigue during testing: An examination of trait, time-on-task, and strategy influences. Human Performance, 23, 381–402.

    Article  Google Scholar 

  • Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94–107.

    Article  PubMed  Google Scholar 

  • Aquino, K., & Thau, S. (2009). Workplace victimization: Aggression from the target’s perspective. Annual Review of Psychology, 60, 717–741.

    Article  PubMed  Google Scholar 

  • Bäckström, M., & Björklund, F. (2019). Is reliability compromised towards the end of long personality inventories? European Journal of Psychological Assessment, 35, 14–21.

    Article  Google Scholar 

  • Barańczuk, U. (2019). The Five Factor Model of personality and social support: A meta-analysis. Journal of Research in Personality, 81, 38–46.

    Article  Google Scholar 

  • Barber, L. K., Barnes, C. M., & Carlson, K. D. (2013). Random and systematic error effects of insomnia on survey behavior. Organizational Research Methods, 16, 616–649.

    Article  Google Scholar 

  • Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection and Assessment, 9, 9–30.

    Article  Google Scholar 

  • Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74, 1252–1265.

    Article  PubMed  Google Scholar 

  • Baumeister, R. F., & Vohs, K. D. (2007). Self-regulation, ego depletion, and motivation. Social and Personality Psychology Compass, 1, 115–128.

    Article  Google Scholar 

  • Bennett, R. J., & Robinson, S. L. (2000). Development of a measure of workplace deviance. Journal of Applied Psychology, 85, 349–360.

    Article  PubMed  Google Scholar 

  • Ben-Porath, Y. S., & Tellegen, A. (2008). MMPI-2-RF, Minnesota Multiphasic Personality Inventory-2 Restructured Form: Manual for Administration. University of Minnesota Press.

    Google Scholar 

  • Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2012). Do other-reports of counterproductive work behavior provide an incremental contribution over self-reports? A meta-analytic comparison. Journal of Applied Psychology, 97, 613–636.

    Article  PubMed  Google Scholar 

  • Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92, 410–424.

    Article  PubMed  Google Scholar 

  • Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI–2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.

    Article  Google Scholar 

  • Bowling, N. A., & Beehr, T. A. (2006). Workplace harassment from the victim’s perspective: A theoretical model and meta-analysis. Journal of Applied Psychology, 91(5), 998–1012.

    Article  PubMed  Google Scholar 

  • Bowling, N. A., Gibson, A. M., Houpt, J. W., & Brower, C. K. (2021). Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organizational Research Methods, 24, 718–738.

    Article  Google Scholar 

  • Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229.

    Article  PubMed  Google Scholar 

  • Bowling, N. A., Huang, J. L., Brower, C. K., & Bragg, C. B. (2021). The quick and the careless: The construct validity of page time as a measure of insufficient effort responding to surveys. Organizational Research Methods. https://doi.org/10.1177/10944281211056520 (in press)

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  PubMed  Google Scholar 

  • Cialdini, R. B. (2001). Influence: Science and practice. Allyn & Bacon.

    Google Scholar 

  • Connelly, B. S., & Ones, D. S. (2010). Another perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136, 1092–1122.

    Article  PubMed  Google Scholar 

  • Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) Professional Manual. Psychological Assessment Resources Inc.

    Google Scholar 

  • Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.

    Article  Google Scholar 

  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.

    Article  PubMed  Google Scholar 

  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.

    Article  Google Scholar 

  • Dalal, D. K., & Zickar, M. J. (2012). Come common myths about centering predictor variables in moderated multiple regression and polynomial regression. Organizational Research Methods, 15, 339–362.

    Article  Google Scholar 

  • DeNeve, K. M., & Cooper, H. (1998). The happy personality: A meta-analysis of 137 personality traits and subjective well-being. Psychological Bulletin, 124, 197–229.

    Article  PubMed  Google Scholar 

  • DeSimone, J. A., & Harms, P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33, 559–577.

    Article  Google Scholar 

  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.

    Article  Google Scholar 

  • DeSimone, J. A, Harms, P. D., & Lowman, G. H. (2019). Are you tired of hearing about fatigue? The effects of survey response fatigue on data quality. Paper presented at the 79th annual meeting of the Academy of Management, Boston, MA.

  • Diener, E. D., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71–75.

    Article  PubMed  Google Scholar 

  • Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18, 192–203.

    Article  PubMed  Google Scholar 

  • Fisher, G. G., Matthews, R. A., & Gibbons, A. M. (2016). Developing and investigating the use of single-item measures in organizational research. Journal of Occupational Health Psychology, 21, 3–23.

    Article  PubMed  Google Scholar 

  • Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670.

    Article  PubMed  Google Scholar 

  • Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.

    Article  Google Scholar 

  • Gibson, A. M., & Bowling, N. A. (2020). The effects of questionnaire length and behavioral consequences on careless responding. European Journal of Psychological Assessment, 36, 410–420.

    Article  Google Scholar 

  • Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4, 26–42.

    Article  Google Scholar 

  • Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96.

    Article  Google Scholar 

  • Gough, H. G., & Bradley, P. (1996). California Psychological Inventory: Administrator’s guide (3rd ed.). Consulting Psychologists Press.

    Google Scholar 

  • Gouldner, A. W. (1960). The norm of reciprocity: A preliminary statement. American Sociological Review, 161-178.

  • Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. (2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136, 495–525.

    Article  PubMed  Google Scholar 

  • Hardy, J. H., III., Gibson, C., Sloan, M., & Carr, A. (2017). Are applicants more likely to quit longer assessments? Examining the effect of assessment length on applicant attrition behavior. Journal of Applied Psychology, 102, 1148–1158.

    Article  PubMed  Google Scholar 

  • Herzog, A. R., & Bachman, J. G. (1981). Effects of questionnaire length on response quality. Public Opinion Quarterly, 45, 549–559.

    Article  Google Scholar 

  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104–121.

    Article  Google Scholar 

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99–114.

    Article  Google Scholar 

  • Huang, J. L., & DeSimone, J. A. (2021). Insufficient effort responding as a potential confound between survey measures and objective tests. Journal of Business and Psychology, 36, 807–828.

    Article  Google Scholar 

  • Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.

    Article  PubMed  Google Scholar 

  • Isaakyan, S., Sherf, E. N., Tangirala, S., & Guenter, H. (2021). Keeping it between us: Managerial endorsement of public versus private voice. Journal of Applied Psychology, 106, 1049–1066.

    Article  PubMed  Google Scholar 

  • Jiang, G., & Mai, Y. (2020). equaltestMI: Examine measurement invariance via equivalence testing and projection method.

  • Judge, T. A., Locke, E. A., Durham, C. C., & Kluger, A. N. (1998). Dispositional effects on job and life satisfaction: The role of core evaluations. Journal of Applied Psychology, 83, 17–34.

    Article  PubMed  Google Scholar 

  • Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

    Article  Google Scholar 

  • Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., ... & Conaway, M. (2002). The impact of" no opinion" response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371-403

  • Landis, R. S., James, L. R., Lance, C. E., Pierce, C. A., & Rogelberg, S. G. (2014). When is nothing something? Journal of Business and Psychology, 29, 163–167.

    Article  Google Scholar 

  • Landis, R. S., & Rogelberg, S. G. (2013). Our scholarly practices are derailing our progress: The importance of “nothing” in the organizational sciences. Industrial and Organizational Psychology, 6, 299–302.

    Article  Google Scholar 

  • Lyubomirsky, S., & Lepper, H. S. (1999). A measure of subjective happiness: Preliminary reliability and construct validation. Social Indicators Research, 46, 137–155.

    Article  Google Scholar 

  • Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.

    Article  Google Scholar 

  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.

    Article  PubMed  Google Scholar 

  • Nagy, M. S. (2002). Using a single-item approach to measure facet job satisfaction. Journal of Occupational and Organizational Psychology, 75, 77–86.

    Article  Google Scholar 

  • Nicolas, S., & Makowski, D. (2016). Can mental fatigue be measured by Weber’s Compass? Alfred Binet’s answer on the value of aesthesiometry (tactile sensitivity) as an objective measure of mental fatigue. In M. Antonelli & H. Gundlach (Eds.), European yearbook of the history of psychology: Sources, theories, and models (Vol. 2, pp. 11–46). Brepols Publishers.

    Google Scholar 

  • Palan, S., & Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.

    Article  Google Scholar 

  • Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.

    Article  PubMed  Google Scholar 

  • Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338.

    Article  PubMed  Google Scholar 

  • Rolstad, S., Adler, J., & Rydén, A. (2011). Response burden and questionnaire length: Is shorter better? A review and meta-analysis. Value in Health, 14, 1101–1108.

    Article  PubMed  Google Scholar 

  • Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010, April). Who are the crowdworkers? Shifting demographics in Mechanical Turk. Paper presented at the CHI’10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, GA.

  • Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Journal of Statistical Software, 48, 1–36.

    Article  Google Scholar 

  • Saucier, G. (1994). Mini-Markers: A brief version of Goldberg’s unipolar Big-Five markers. Journal of Personality Assessment, 63, 506–516.

    Article  PubMed  Google Scholar 

  • Schleifer, S. (1986). Trends in attitudes toward and participation in survey research. Public Opinion Quarterly, 50, 17–26.

    Article  Google Scholar 

  • Sherf, E. N., & Morrison, E. W. (2020). I do not need feedback! Or do I? Self-efficacy, perspective taking, and feedback seeking. Journal of Applied Psychology, 105, 146–165.

    Article  PubMed  Google Scholar 

  • Sitzmann, T., & Wang, M. (2015). The survey effect: Does administering surveys affect trainees’ behavior?. Learning and Individual Differences, 37, 1–12.

    Article  Google Scholar 

  • Spector, P. E., Fox, S., Penney, L. M., Bruursema, K., Goh, A., & Kessler, S. (2006). The dimensionality of counterproductivity: Are all counterproductive behaviors created equal? Journal of Vocational Behavior, 68, 446–460.

    Article  Google Scholar 

  • Spector, P. E., & Jex, S. M. (1998). Development of four self-report measures of job stressors and strain: Interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. Journal of Occupational Health Psychology, 3, 356–367.

    Article  PubMed  Google Scholar 

  • Stocké, V. (2006). Attitudes toward surveys, attitude accessibility and the effect on respondents’ susceptibility to nonresponse. Quality and Quantity, 40, 259–288.

    Article  Google Scholar 

  • Takeuchi, R., Guo, N., Teschner, R. S., & Kautz, J. (2021). Reflecting on death amidst COVID-19 and individual creativity: Cross-lagged panel data analysis using four-wave longitudinal data. Journal of Applied Psychology, 106, 1156–1168.

    Article  PubMed  Google Scholar 

  • Tourangeau, R. (1984). Cognitive science and survey methods. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines, 73-100.

  • Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.

    Article  Google Scholar 

  • Voce, A. C., & Moston, S. (2016). Does monitoring performance eliminate the ego-depletion phenomenon and influence perception of time? Self and Identity, 15, 32–46.

    Article  Google Scholar 

  • Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82, 247–252.

    Article  PubMed  Google Scholar 

  • Yentes, R. D., & Wilhelm, F. (2018). careless: Procedures for computing indices of careless responding. R package version, 1, 2018

  • Jiang, G., & Mai, Y. (2020). equaltestMI: Examine measurement invariance via equivalence testing and projection method.

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Wright State University, Dayton, OH, USA

    Nathan A. Bowling

  2. VHA National Center for Organizational Development, Cincinnati, OH, USA

    Anthony M. Gibson

  3. Department of Management, Culverhouse College of Business, University of Alabama, Tuscaloosa, USA

    Justin A. DeSimone

Authors
  1. Nathan A. Bowling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anthony M. Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Justin A. DeSimone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nathan A. Bowling.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 27 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bowling, N.A., Gibson, A.M. & DeSimone, J.A. Stop With the Questions Already! Does Data Quality Suffer for Scales Positioned Near the End of a Lengthy Questionnaire?. J Bus Psychol 37, 1099–1116 (2022). https://doi.org/10.1007/s10869-021-09787-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-021-09787-8

Keywords

  • Construct validity
  • Questionnaires
  • Careless responding
  • Insufficient effort responding
  • Response effort