Advertisement

Journal of Business and Psychology

, Volume 25, Issue 3, pp 325–334 | Cite as

What Reviewers Should Expect from Authors Regarding Common Method Bias in Organizational Research

  • James M. Conway
  • Charles E. Lance
Article

Abstract

We believe that journal reviewers (as well as editors and dissertation or thesis committee members) have to some extent perpetuated misconceptions about common method bias in self-report measures, including (a) that relationships between self-reported variables are necessarily and routinely upwardly biased, (b) other-reports (or other methods) are superior to self-reports, and (c) rating sources (e.g., self, other) constitute measurement methods. We argue against these misconceptions and make recommendations for what reviewers (and others) should reasonably expect from authors regarding common method bias. We believe it is reasonable to expect (a) an argument for why self-reports are appropriate, (b) construct validity evidence, (c) lack of overlap in items for different constructs, and (d) evidence that authors took proactive design steps to mitigate threats of method effects. We specifically do not recommend post hoc statistical control strategies; while some statistical strategies are promising, all have significant drawbacks and some have shown poor empirical results.

Keywords

Common method bias Method variance Self-report measures Reviewing 

References

  1. Arora, R. (1982). Validation of an S-O-R model for situation, enduring and response components of involvement. Journal of Marketing Research, 19, 505–516.CrossRefGoogle Scholar
  2. Bagozzi, R. P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547–560.CrossRefGoogle Scholar
  3. Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625–641.CrossRefGoogle Scholar
  4. Bozeman, D., & Perrewé, P. (2001). The effect of item content overlap on Organizational Commitment Questionnaire–turnover cognitions relationships. Journal of Applied Psychology, 86, 161–173.CrossRefPubMedGoogle Scholar
  5. Brannick, M. T., Chan, D., Conway, J. M., Lance, C. E., & Spector, D. (in press). What is method variance and how can we cope with it? A panel discussion. Organizational Research Methods.Google Scholar
  6. Campbell, J. (1982). Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691–700.CrossRefGoogle Scholar
  7. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–106.CrossRefPubMedGoogle Scholar
  8. Chan, D. (2001). Method effects of positive affectivity, negative affectivity, and impression management in self-reports of work attitudes. Human Performance, 14, 77–96.CrossRefGoogle Scholar
  9. Chan, D. (2009). So why ask me? Are self-report data really that bad? In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 311–338). New York: Routledge.Google Scholar
  10. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.CrossRefGoogle Scholar
  11. Conway, J. M. (2002). Method variance and method bias in I/O psychology. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 344–365). Oxford: Blackwell Publishers.Google Scholar
  12. Cote, J. A., & Buckley, M. R. (1987). Estimating trait, method and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 26, 315–318.CrossRefGoogle Scholar
  13. Crampton, S., & Wagner, J. (1994). Percept-percept inflation in microorganizational research: An investigation of prevalence and effect. Journal of Applied Psychology, 79, 67–76.CrossRefGoogle Scholar
  14. Dooley, W. K., Shaffer, D. R., Lance, C. E., & Williamson, G. M. (2007). Informal care can be better than adequate: Development and evaluation of the exemplary care scale. Rehabilitation Psychology, 52, 359–369.CrossRefGoogle Scholar
  15. Doty, D. H., & Glick, W. H. (1998). Common method bias: Does common methods variance really bias results? Organizational Research Methods, 1, 374–406.CrossRefGoogle Scholar
  16. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105–146). Washington, DC: American Council on Education and National Council on Measurement in Education.Google Scholar
  17. Heisenberg, W. (1927). Über den anschaulichen Inhalt der quantentheoretischen Kinematik und Mechanik. [About the graphic content of quantum theoretic kinematics and mechanics]. Zeitschrift für Physik, 43, 172–198.CrossRefGoogle Scholar
  18. Hoffman, B. J., Lance, C. E., Bynum, B. H., & Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119–151.CrossRefGoogle Scholar
  19. Hoffman, B. J., & Woehr, D. J. (2009). Disentangling the meaning of multisource performance rating source and dimension factors. Personnel Psychology, 62, 735–765.CrossRefGoogle Scholar
  20. Hunter, J. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3–7.CrossRefGoogle Scholar
  21. International Personality Item Pool. (2010). International Personality Item Pool: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences (http://ipip.ori.org/). http://ipip.ori.org/newNEOKey.htm#Conscientiousness. February 5, 2010.
  22. James, L. A., & James, L. R. (1989). Integrating work environment perceptions: Explorations into the measurement of meaning. Journal of Applied Psychology, 74, 739–751.CrossRefGoogle Scholar
  23. Judge, T., Bono, J., & Locke, E. (2000). Personality and job satisfaction: The mediating role of job characteristics. Journal of Applied Psychology, 85, 237–249.CrossRefPubMedGoogle Scholar
  24. Lance, C. E., Baranik, L. E., Lau, A. R., & Scharlau, E. A. (2009). If it ain’t trait it must be method: (Mis)application of the multitrait-multimethod design in organizational research. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 339–362). New York: Routledge.Google Scholar
  25. Lance, C. E., Baxter, D., & Mahan, R. P. (2006). Multi-source performance measurement: A reconceptualization. In W. Bennett, C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 49–76). Mahwah, NJ: Erlbaum.Google Scholar
  26. Lance, C. E., Dawson, B., Birklebach, D., & Hoffman, B. J. (in press). Method effects, measurement error, and substantive conclusions. Organizational Research Methods.Google Scholar
  27. Lance, C. E., Hoffman, B. J., Baranik, L. E., & Gentry, W. A. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223–232.CrossRefGoogle Scholar
  28. Lance, C. E., Teachout, M. S., & Donnelly, T. M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437–452.CrossRefGoogle Scholar
  29. Lance, C. E., & Vandenberg, R. J. (2009). Introduction. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 1–4). New York: Routledge.Google Scholar
  30. Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86, 114–121.CrossRefPubMedGoogle Scholar
  31. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley Publishing Company.Google Scholar
  32. MacKenzie, S. B., Podsakoff, P. M., & Ahearne, M. (1998). Some possible antecedents and consequences of in-role and extra-role salesperson performance. Journal of Marketing, 62, 87–98.CrossRefGoogle Scholar
  33. Mallard, A. G. C., & Lance, C. E. (1998). Development and evaluation of a parent–employee interrole conflict scale. Social Indicators Research, 45, 343–370.CrossRefGoogle Scholar
  34. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.Google Scholar
  35. Motowidlo, S., Borman, W., & Schmit, M. (1997). A theory of individual differences in task and contextual performance. Human Performance, 10, 71–83.CrossRefGoogle Scholar
  36. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557–576.CrossRefGoogle Scholar
  37. Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224–247.CrossRefGoogle Scholar
  38. Netemeyer, R. G., Boles, J. S., & McMurrian, R. (1996). Development and validation of work-family conflict and family-work conflict scales. Journal of Applied Psychology, 81, 400–410.CrossRefGoogle Scholar
  39. Ones, D., Viswesvaran, C., & Reiss, A. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660–679.CrossRefGoogle Scholar
  40. Organ, D., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behavior. Personnel Psychology, 48, 775–802.CrossRefGoogle Scholar
  41. Podsakoff, P., MacKenzie, S., Lee, J., & Podsakoff, N. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.CrossRefPubMedGoogle Scholar
  42. Podsakoff, P., & Todor, W. (1985). Relationships between leader reward and punishment behavior and group processes and productivity. Journal of Management, 11, 55–73.CrossRefGoogle Scholar
  43. Putka, D. M., & Sackett, P. R. (2010). Reliability and validity. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (pp. 9–49). New York: Routledge.Google Scholar
  44. Richardson, H., Simmering, M., & Sturman, M. (2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12, 762–800.CrossRefGoogle Scholar
  45. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956–970.CrossRefPubMedGoogle Scholar
  46. Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50–66.CrossRefPubMedGoogle Scholar
  47. Shalley, C., Gilson, L., & Blum, T. (2009). Interactive effects of growth need strength, work context, and job complexity on self-reported creative performance. Academy of Management Journal, 52, 489–505.Google Scholar
  48. Skinner, B. F. (1957). Verbal behavior. New York: Appleton-Century-Crofts.CrossRefGoogle Scholar
  49. Spector, P. (2006). Method variance in organizational research: Truth or urban legend? Organizational Research Methods, 9, 221–232.CrossRefGoogle Scholar
  50. Tornow, W. W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221–229.CrossRefGoogle Scholar
  51. Van Scotter, J. R., & Motowidlo, S. J. (1996). Interpersonal facilitation and job dedication as separate facets of contextual performance. Journal of Applied Psychology, 81, 525–531.CrossRefGoogle Scholar
  52. Vandenberg, R. J. (2006). Statistical and methodological myths and urban legends: Where, pray tell, did they get this idea? Organizational Research Methods, 9, 194–201.CrossRefGoogle Scholar
  53. Westaby, J., & Braithwaite, K. (2003). Specific factors underlying reemployment self-efficacy: Comparing control belief and motivational reason methods for the recently unemployed. Journal of Applied Behavioral Science, 39, 415–437.CrossRefGoogle Scholar
  54. Widaman, K. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.CrossRefGoogle Scholar
  55. Williams, L. J., & Anderson, S. E. (1994). An alternative approach to method effects by using latent-variable models: Applications in organizational behavior research. Journal of Applied Psychology, 79, 323–331.CrossRefGoogle Scholar
  56. Williams, L. J., Cote, J. A., & Buckley, M. R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? Journal of Applied Psychology, 74, 462–468.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Department of PsychologyCentral Connecticut State UniversityNew BritainUSA
  2. 2.University of GeorgiaAthensUSA

Personalised recommendations