Advertisement

Behavior Research Methods

, Volume 46, Issue 3, pp 798–807 | Cite as

Current misuses of multiple regression for investigating bivariate hypotheses: an example from the organizational domain

  • Thomas A. O’Neill
  • Matthew J. W. McLarnon
  • Travis J. Schneider
  • Robert C. Gardner
Article

Abstract

By definition, multiple regression (MR) considers more than one predictor variable, and each variable’s beta will depend on both its correlation with the criterion and its correlation with the other predictor(s). Despite ad nauseam coverage of this characteristic in organizational psychology and statistical texts, researchers’ applications of MR in bivariate hypothesis testing has been the subject of recent and renewed interest. Accordingly, we conducted a targeted survey of the literature by coding articles, covering a five-year span from two top-tier organizational journals, that employed MR for testing bivariate relations. The results suggest that MR coefficients, rather than correlation coefficients, were most common for testing hypotheses of bivariate relations, yet supporting theoretical rationales were rarely offered. Regarding the potential impact on scientific advancement, in almost half of the articles reviewed (44 %), at least one conclusion of each study (i.e., that the hypothesis was or was not supported) would have been different, depending on the author’s use of correlation or beta to test the bivariate hypothesis. It follows that inappropriate decisions to interpret the correlation versus the beta will affect the accumulation of consistent and replicable scientific evidence. We conclude with recommendations for improving bivariate hypothesis testing.

Keywords

Multiple regression Organizational science Research methods Beta Correlation 

References

  1. Atinc, G., Simmering, M. J., & Kroll, M. J. (2012). Control variable use and reporting in macro and micro management research. Organizational Research Methods, 15, 57–75. doi: 10.1177/1094428110397773 CrossRefGoogle Scholar
  2. Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman.Google Scholar
  3. Bauer, T. N., Erdogan, B., Liden, R. C., & Wayne, S. J. (2006). A longitudinal study of the moderating role of extraversion: Leader-member exchange, performance, and turnover during new executive development. Journal of Applied Psychology, 91, 298–310.PubMedCrossRefGoogle Scholar
  4. Becker, T. E. (2005). Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations. Organizational Research Methods, 8, 274–289. doi: 10.1177/1094428105278021 CrossRefGoogle Scholar
  5. Bernstein, J. (2004). Evidence-based medicine. American Academy of Orthopaedic Surgeons, 12, 80–88.Google Scholar
  6. Biemann, T. (2013). What if we were Texas sharpshooters? Predictor reporting bias in regression analysis. Organizational Research Methods, 16, 335–363. doi: 10.1177/1094428113485135 CrossRefGoogle Scholar
  7. Bobko, P. (1990). Multivariate correlational analysis. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 637–686). Palo Alto, CA: Consulting Psychologists Press.Google Scholar
  8. Bobko, P. (2001). Correlation and regression: Applications for industrial and organizational psychology and management. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  9. Braun, M. T., & Oswald, F. L. (2011). Exploratory regression analysis: A tool for selecting models and determining predictor importance. Behavior Research Methods, 43, 331–339. doi: 10.3758/s13428-010-0046-8 PubMedCrossRefGoogle Scholar
  10. Breaugh, J. A. (2006). Rethinking the control of nuisance variables in theory testing. Journal of Business and Psychology, 20, 429–443. doi: 10.1007/s10869-005-9009-y CrossRefGoogle Scholar
  11. Breaugh, J. A. (2008). Important considerations in using statistical procedures to control for nuisance variables in non-experimental studies. Human Resource Management Review, 18, 282–293. doi: 10.1016/j.hrmr.2008.03.001 CrossRefGoogle Scholar
  12. Budescu, D. V. (1993). Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. Psychological Bulletin, 114, 542–551. doi: 10.1037/0033-2909.114.3.542 CrossRefGoogle Scholar
  13. Campion, M. A. (1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 705–718. doi: 10.1111/j.1744-6570.1993.tb00896.x CrossRefGoogle Scholar
  14. Carlson, K. D., & Wu, J. (2012). The illusion of statistical control: Control variable practice in management research. Organizational Research Methods, 15, 413–435. doi: 10.1177/1094428111428817 CrossRefGoogle Scholar
  15. Chan, D. (2011). Advances in analytic strategies. In S. Zedeck (Ed.), APA handbook of industrial and organizational psychology (Vol. 1, pp. 85–113). Washington, DC: American Psychological Association.Google Scholar
  16. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Erlbaum.Google Scholar
  17. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.Google Scholar
  18. Courville, T., & Thompson, B. (2001). Use of structure coefficient in published multiple regression articles: β is not enough. Educational and Psychological Measurement, 61, 229–248. doi: 10.1177/0013164401612006 Google Scholar
  19. Darlington, R. B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69, 161–182. doi: 10.1037/h0025471 PubMedCrossRefGoogle Scholar
  20. Darlington, R. B. (1990). Regression and linear models. New York, NY: McGraw-Hill.Google Scholar
  21. Dickson, J. M., Moberly, N. J., Hannon, E. M., & Bates, G. W. (2009). Are repressors so special after all? Specificity of negative personal events as a function of anxiety and defensiveness. Journal of Research in Personality, 43, 386–391. doi: 10.1016/j.jrp.2009.01.002 CrossRefGoogle Scholar
  22. Dineen, B. R., Lewicki, R. J., & Tomlinson, E. C. (2006). Supervisory guidance and behavioral integrity: Relationships with employee citizenship and deviant behaviors. Journal of Applied Psychology, 91, 622–635. doi: 10.1037/0021-9010.91.3.622 PubMedCrossRefGoogle Scholar
  23. Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi: 10.3758/s13423-012-0227-9 CrossRefGoogle Scholar
  24. Gardner, R. C. (2001). Psychological statistics using SPSS for Windows. Upper Saddle River, NJ: Prentice-Hall.Google Scholar
  25. Gordon, R. A. (1968). Issues in multiple regression. American Journal of Sociology, 73, 592–616.CrossRefGoogle Scholar
  26. Hinkin, T. R., & Schriesheim, C. A. (2008). An examination of “nonleadership”: From laissez-faire leadership to leader reward omission and punishment omission. Journal of Applied Psychology, 93, 1234–1248. doi: 10.1037/a0012875 PubMedCrossRefGoogle Scholar
  27. Hirschfeld, R. R., & Bernerth, J. B. (2008). Mental efficacy and physical efficacy at the team level: Inputs and outcomes among newly formed action teams. Journal of Applied Psychology, 93, 1429–1437. doi: 10.1037/a0012574 PubMedCrossRefGoogle Scholar
  28. Hoffman, B. J., Bynum, B. H., Piccolo, R. F., & Sutton, A. W. (2011). Person–organization value congruence: How transformational leaders influence work. Academy of Management Journal, 54, 779–796. doi: 10.5465/AMJ.2011.64870139 CrossRefGoogle Scholar
  29. Homan, A. C., Hollenbeck, J. R., Humphrey, S. E., van Knippenberg, D., Ilgen, D. R., & Van Kleef, G. (2008). Facing differences with an open mind: Openness to experience, salience of intragroup differences, and performance of diverse work groups. Academy of Management Journal, 51, 1204–1222. doi: 10.5465/AMJ.2008.35732995 CrossRefGoogle Scholar
  30. Horst, P. (1941). The role of the predictor variables which are independent of the criterion. Social Science Research Council, 48, 431–436.Google Scholar
  31. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.Google Scholar
  32. Johnson, J. W. (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, 1–19. doi: 10.1207/S15327906MBR3501_1 CrossRefGoogle Scholar
  33. Johnson, J. W., & LeBreton, J. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, 238–257. doi: 10.1177/1094428104266510 CrossRefGoogle Scholar
  34. Keller, R. T. (2006). Transformational leadership, initiating structure, and substitutes for leadership: A longitudinal study of research and development project team performance. Journal of Applied Psychology, 91, 202–210. doi: 10.1037/0021-9010.91.1.202 PubMedCrossRefGoogle Scholar
  35. Kraha, A., Turner, H., Nimon, K., Zientek, L., & Henson, R. K. (2012). The multifaceted concept of predictor importance: Tools to support interpreting multiple regression. Frontiers in Psychology, 3, 1–16. doi: 10.3389/fpsyg.2012.00044 CrossRefGoogle Scholar
  36. LeBreton, J. M., Hargis, M. B., Griepentrog, B., Oswald, F. L., & Ployhart, R. E. (2007). A multidimensional approach for evaluating variables in organizational research and practice. Personnel Psychology, 60, 475–498. doi: 10.1111/j.1744-6570.2007.00080.x CrossRefGoogle Scholar
  37. Lehman, W. E. K., & Simpson, D. D. (1992). Employee substance use and on-the-job behaviors. Journal of Applied Psychology, 77, 309–321. doi: 10.1037/0021-9010.77.3.309 PubMedCrossRefGoogle Scholar
  38. Lindeman, R., Merenda, P., & Gold, R. (1980). Introduction to bivariate and multivariate analysis. Glenview, IL: Scott, Foresman.Google Scholar
  39. Liu, C., Spector, P. E., & Jex, S. M. (2005). The relation of job control with job strains: A comparison of multiple data sources. Journal of Occupational and Organizational Psychology, 78, 325–336. doi: 10.1348/096317905X26002 CrossRefGoogle Scholar
  40. Lorenzo-Seva, U., & Ferrando, P. J. (2011). FIRE: an SPSS program for variable selection in multiple linear regression via the relative importance of predictors. Behavior Research Methods, 43, 1–7. doi: 10.3758/s13428-010-0043-y PubMedCrossRefGoogle Scholar
  41. Lorenzo-Seva, U., Ferrando, P. J., & Chico, E. (2010). Two SPSS programs for interpreting multiple regression results. Behavior Research Methods, 42, 29–35. doi: 10.3758/BRM.42.1.29 PubMedCrossRefGoogle Scholar
  42. MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation, confounding, and suppression effect. Prevention Science, 4, 173–181. doi: 10.1023/A:1026595011371 CrossRefGoogle Scholar
  43. Meade, A. W., Behrend, T. S., & Lance, C. E. (2009). Dr. StrangeLOVE, or how I learned to stop worrying and love omitted variables. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends (pp. 247–265). New York, NY: Routledge.Google Scholar
  44. Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40–48. doi: 10.1037/0021-843X.110.1.40 PubMedCrossRefGoogle Scholar
  45. Newcombe, N. S. (2003). Some controls control too much. Child Development, 74, 1050–1052. doi: 10.1111/1467-8624.00588 PubMedCrossRefGoogle Scholar
  46. Nimon, K. F., & Oswald, F. L. (2013). Understanding the results of multiple linear regression: Beyond standardized regression coefficients. Organizational Research Methods, 16, 650–674. doi: 10.1177/1094428113493929 CrossRefGoogle Scholar
  47. Park, H., & Antonioni, D. (2007). Personality, reciprocity, and strength of conflict resolution strategy. Journal of Research in Personality, 41, 110–125. doi: 10.1016/j.jrp.2006.03.003 CrossRefGoogle Scholar
  48. Pea, R., Nass, C., Meheula, L., Rance, M., Kumar, A., Bamford, H., & Zhou, M. (2012). Media use, face-to-face communication, media multitasking, and social well-being among 8- to 12-year-old girls. Developmental Psychology, 48, 327–336. doi: 10.1037/a0027030 PubMedCrossRefGoogle Scholar
  49. Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York, NY: Holt, Rinehart, & Winston.Google Scholar
  50. Pedhazur, E. J. (1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). Forth Worth, TX: Harcourt.Google Scholar
  51. Peterson, R. A., & Brown, S. P. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90, 175–181. doi: 10.1037/0021-9010.90.1.175 PubMedCrossRefGoogle Scholar
  52. Porath, C. L., & Bateman, T. S. (2006). Self-regulation: From goal orientation to job performance. Journal of Applied Psychology, 91, 185–192. doi: 10.1037/0021-9010.91.1.185 PubMedCrossRefGoogle Scholar
  53. Ramirez, M., Roth, L., Young, T., & Peek-Asa, C. (2013). Rural roadway safety perceptions among rural teen drivers living in and outside of towns. Journal of Rural Health, 29, 46–54. doi: 10.1111/j.1748-0361.2012.00435.x PubMedCrossRefGoogle Scholar
  54. Rousseau, D. M. (Ed.). (2012). The Oxford handbook of evidence-based management. New York, NY: Oxford University Press.Google Scholar
  55. Rowe, C. L., La Greca, A. M., & Alexandersson, A. (2010). Family and individual factors associated with substance involvement and PTS symptoms among adolescents in Greater New Orleans after Hurricane Katrina. Journal of Consulting and Clinical Psychology, 78, 806–817. doi: 10.1037/a0020808 PubMedCrossRefGoogle Scholar
  56. Schmitt, N., Keeny, J., & Oswald, F. L. (2009). Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. Journal of Applied Psychology, 94, 1479–1497. doi: 10.1037/a0016810 PubMedCrossRefGoogle Scholar
  57. Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470, 437. doi: 10.1038/470437a PubMedCrossRefGoogle Scholar
  58. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi: 10.1177/0956797611417632 PubMedCrossRefGoogle Scholar
  59. Spector, P. E., & Brannick, M. T. (2011). Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods, 14, 287–305. doi: 10.1177/1094428110369842 CrossRefGoogle Scholar
  60. Spector, P. E., Dwyer, D., & Jex, S. M. (1988). Relation of job stressors to affective, health, and performance outcomes: A comparison of multiple data sources. Journal of Applied Psychology, 73, 11–19.PubMedCrossRefGoogle Scholar
  61. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Boston, MA: Pearson/Allyn & Bacon.Google Scholar
  62. Tepper, B. J., Moss, S. E., Lockhart, D. E., & Carr, J. C. (2007). Abusive supervision, upward maintenance communication, and subordinates’ psychological distress. Academy of Management Journal, 50, 1169–1180. doi: 10.2307/20159918 CrossRefGoogle Scholar
  63. Tonidandel, S., LeBreton, J. M., & Johnson, J. W. (2009). Determining the statistical significance of relative weights. Psychological Methods, 14, 387–399. doi: 10.1037/a0017735 PubMedCrossRefGoogle Scholar
  64. Wanberg, C. R., Zhu, J., Kanfer, R., & Zhang, Z. (2012). After the pink slip: Applying dynamic motivation frameworks to the job search experience. Academy of Management Journal, 55, 261–284. doi: 10.5465/amj.2010.0157 CrossRefGoogle Scholar
  65. Wang, S., Tomlinson, E. C., & Noe, R. A. (2010). The role of mentor trust and protégé internal locus of control in formal mentoring relationships. Journal of Applied Psychology, 95, 358–367. doi: 10.1037/a0017663 PubMedCrossRefGoogle Scholar
  66. Watson, D., Pennebaker, J. W., & Folger, R. (1987). Beyond negative affectivity: Measuring stress and satisfaction in the workplace. Journal of Organizational Behavior Management, 8, 141–158. doi: 10.1300/J075v08n02_09 CrossRefGoogle Scholar
  67. Winne, P. H. (1983). Distortions of construct validity in multiple regression analysis. Canadian Journal of Behavioral Science, 15, 187–202. doi: 10.1037/h0080736 CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  • Thomas A. O’Neill
    • 1
  • Matthew J. W. McLarnon
    • 2
  • Travis J. Schneider
    • 2
  • Robert C. Gardner
    • 2
  1. 1.Department of PsychologyUniversity of CalgaryCalgaryCanada
  2. 2.Department of PsychologyUniversity of Western OntarioLondonCanada

Personalised recommendations