Abstract
Three studies experimentally investigated response formats of agreement (A) and frequency (F) to determine their effect on correlations between often used measures of job stressors and strains. Study 1 used a within-subject design where respondents were given two sets of the same 6 stressor measures that varied in A versus F formats, as well as measures of 6 strains. Study 2 replicated Study 1 using a between-subject design where each respondent was randomly assigned to either the A or F stressor scale format. Study 3 utilized a 2 (A vs. F for stressors) by 2 (A vs. F for strains) design to test the assumption that matched response formats would result in higher correlations than unmatched due to common method variance. The first two studies provide little evidence for a systematic difference between A and F formats in predicting strains. Study 3 found that matching formats more often resulted in lower, not higher, correlations between stressor and strain pairs. Taken together, these experimental studies suggest that response format has little effect on correlations, means, standard deviations, and internal consistencies, and is not a method variance concern, at least with these commonly used measures of job stressors and strains.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amstad, F. T., Meier, L. L., Fasel, U., Elfering, A., & Semmer, N. K. (2011). A meta-analysis of work–family conflict and various outcomes with a special emphasis on cross-domain versus matching-domain relations. Journal of Occupational Health Psychology, 16(2), 151–169. https://doi.org/10.1037/a0022170.
Baumgartner, H., & Steenkamp, J.-B. E. (2001). Response styles in marketing research: a cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840.
Blau, G., & Andersson, L. (2005). Testing a measure of instigated workplace incivility. Journal of Occupational and Organizational Psychology, 78(4), 595–614. https://doi.org/10.1348/096317905x26822.
Bowling, N. A., Alarcon, G. M., Bragg, C. B., & Hartman, M. J. (2015). A meta-analytic examination of the potential correlates and consequences of workload. Work and Stress, 29(2), 95–113. https://doi.org/10.1080/02678373.2015.1033037.
Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan organizational assessment questionnaire. Unpublished manuscript, University of Michigan, Ann Arbor. University of Michigan, Ann Arbor.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016.
Caplan, R. D., Cobb, S., French, J. R. P., Van Harrison, R., & Penneau, S. R. (1980). Job demands and worker health. An Arbor: University of Michigan, Institute for Social Research.
Dalal, R. S. (2005). A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90(6), 1241–1255. https://doi.org/10.1037/0021-9010.90.6.1241.
De Beuckelaer, A., Weijters, B., & Rutten, A. (2010). Using ad hoc measures for response styles: a cautionary note. Quality & Quantity: International Journal of Methodology, 44(4), 761–775. https://doi.org/10.1007/s11135-009-9225-z.
Fiske, D. W. (1987). On understanding our methods and their effects. Diagnostica, 33(3), 188–194.
Guidroz, A. M., Burnfield-Geimer, J. L., Clark, O., Schwetschenau, H. M., & Jex, S. M. (2010). The nursing incivility scale: development and validation of an occupation-specific measure. Journal of Nursing Measurement, 18(3), 176–200.
House, R. J., Schuler, R. S., & Levanoni, E. (1983). Role conflict and ambiguity scales: reality or artifacts? Journal of Applied Psychology, 68(2), 334–337. https://doi.org/10.1037/0021-9010.68.2.334.
Judge, T. A., Locke, E. A., & Durham, C. C. (1997). The dispositional causes of job satisfaction: a core evaluations approach. Research in Organizational Behavior, 19, 151–188.
Malhotra, N. K., Schaller, T. K., & Patil, A. (2017). Common method variance in advertising research: when to be concerned and how to control for it. Journal of Advertising, 46(1), 193–212. https://doi.org/10.1080/00913367.2016.1252287.
Moorman, R. H., & Podsakoff, P. M. (1992). A meta-analytic review and empirical test of the potential confounding effects of social desirability response sets in organizational behaviour research. Journal of Occupational and Organizational Psychology, 65(2), 131–149.
Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: a critical analysis. Journal of Applied Psychology, 78(2), 218–225. https://doi.org/10.1037/0021-9010.78.2.218.
Netemeyer, R. G., Boles, J. S., & McMurrian, R. (1996). Development and validation of work–family conflict and family–work conflict scales. Journal of Applied Psychology, 81(4), 400–410. https://doi.org/10.1037/0021-9010.81.4.400.
Nixon, A. E., Mazzola, J. J., Bauer, J., Krueger, J. R., & Spector, P. E. (2011). Can work make you sick? A meta-analysis of the relationships between job stressors and physical symptoms. Work and Stress, 25(1), 1–22. https://doi.org/10.1080/02678373.2011.569175.
Pace, V. L. (2010). Method variance from the perspectives of reviewers: poorly understood problem or overemphasized complaint? Organizational Research Methods, 13(3), 421–434. https://doi.org/10.1177/1094428109351751.
Pindek, S., & Spector, P. E. (2016). Organizational constraints: a meta-analysis of a major stressor. Work and Stress, 30(1), 7–25. https://doi.org/10.1080/02678373.2015.1137376.
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. https://doi.org/10.1146/annurev-psych-120710-100452.
Rorer, L. G. (1965). The great response-style myth. Psychological Bulletin, 63(3), 129–156. https://doi.org/10.1037/h0021888.
Spector, P. E., & Jex, S. M. (1998). Development of four self-report measures of job stressors and strain: interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. Journal of Occupational Health Psychology, 3(4), 356–367. https://doi.org/10.1037/1076-8998.3.4.356.
Spector, P. E., Dwyer, D. J., & Jex, S. M. (1988). Relation of job stressors to affective, health, and performance outcomes: a comparison of multiple data sources. Journal of Applied Psychology, 73(1), 11–19. https://doi.org/10.1037/0021-9010.73.1.11.
Spector, P. E., Bauer, J. A., & Fox, S. (2010). Measurement artifacts in the assessment of counterproductive work behavior and organizational citizenship behavior: do we know what we think we know? Journal of Applied Psychology, 95(4), 781–790. https://doi.org/10.1037/a0019477.
Spector, P. E., Rosen, C. C., Richardson, H. A., Williams, L. J., & Johnson, R. E. (2019). A new perspective on method variance: a measure-centric approach. Journal of Management, 45(3), 855–880. https://doi.org/10.1177/0149206316687295.
Weijters, B., Cabooter, E., & Schillewaert, N. (2010a). The effect of rating scale format on response styles: the number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236–247. https://doi.org/10.1016/j.ijresmar.2010.02.004.
Weijters, B., Geuens, M., & Schillewaert, N. (2010b). The individual consistency of acquiescence and extreme response style in self-report questionnaires. Applied Psychological Measurement, 34(2), 105–121. https://doi.org/10.1177/0146621609338593.
Weijters, B., Geuens, M., & Schillewaert, N. (2010c). The stability of individual response styles. Psychological Methods, 15(1), 96–110. https://doi.org/10.1037/a0018721.
Weitz, J. (1952). A neglected concept in the study of job satisfaction. Personnel Psychology, 5, 201–205. https://doi.org/10.1111/j.1744-6570.1952.tb01012.x.
Williams, E. J. (1959). The comparison of regression variables. Journal of the Royal Statistical Society (Series B), 21, 396–399.
Williams, L. J., Gavin, M. B., & Williams, M. L. (1996). Measurement and nonmeasurement processes with negative affectivity and employee attitudes. Journal of Applied Psychology, 81(1), 88–101. https://doi.org/10.1037/0021-9010.81.1.88.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Spector, P.E., Nixon, A.E. How Often Do I Agree: an Experimental Test of Item Format Method Variance in Stress Measures. Occup Health Sci 3, 125–143 (2019). https://doi.org/10.1007/s41542-019-00039-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41542-019-00039-z