Effect sizes and the interpretation of research results in international business

Abstract

Journal editors and academy presidents are increasingly calling on researchers to evaluate the substantive, as opposed to the statistical, significance of their results. To measure the extent to which these calls have been heeded, I aggregated the meta-analytically derived effect size estimates obtained from 965 individual samples. I then surveyed 204 studies published in the Journal of International Business Studies. I found that the average effect size in international business research is small, and that most published studies lack the statistical power to detect such effects reliably. I also found that many authors confuse statistical with substantive significance when interpreting their research results. These practices have likely led to unacceptably high Type II error rates and invalid inferences regarding real-world effects. By emphasizing p values over their effect size estimates, researchers are under-selling their results and settling for contributions that are less than what they really have to offer. In view of this, I offer four recommendations for improving research and reporting practices.

This is a preview of subscription content, log in to check access.

References

  1. Abraham, W. T., & Russell, D. W. 2008. Statistical power analysis in psychological research. Social and Personality Compass, 2 (1): 283–301.

    Article  Google Scholar 

  2. AERA. 2006. Standards for reporting on empirical social science research in AERA publications. http://www.aera.net/opportunities/?id=1850. Accessed 11 September 2008.

  3. APA. 2001. Publication manual of the American Psychological Association, (5th ed.). Washington, DC: American Psychological Association.

  4. APA. 2010. Publication manual of the American Psychological Association, (6th ed.). Washington, DC: American Psychological Association.

  5. Baggs, J., & Brander, J. A. 2006. Trade liberalization, profitability, and financial leverage. Journal of International Business Studies, 37 (2): 196–211.

    Article  Google Scholar 

  6. Barden, J. Q., Steensma, H. K., & Lyles, M. A. 2005. The influence of parent control structure on parent conflict in Vietnamese international joint ventures: An organizational justice-based contingency approach. Journal of International Business Studies, 36 (2): 156–174.

    Article  Google Scholar 

  7. Brock, J. 2003. The “power” of international business research. Journal of International Business Studies, 34 (1): 90–99.

    Article  Google Scholar 

  8. Campbell, J. P. 1982. Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67 (6): 691–700.

    Article  Google Scholar 

  9. Campion, M. A. 1993. Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46 (3): 705–718.

    Article  Google Scholar 

  10. Cano, C. R., Carrillat, F. A., & Jaramillo, F. 2004. A meta-analysis of the relationship between market orientation and business performance: Evidence from five continents. International Journal of Research in Marketing, 21 (2): 179–200.

    Article  Google Scholar 

  11. Carver, R. P. 1978. The case against statistical significance testing. Harvard Educational Review, 48 (3): 378–399.

    Article  Google Scholar 

  12. Cashen, L. H., & Geiger, S. W. 2004. Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7 (2): 151–167.

    Article  Google Scholar 

  13. Child, J., Chung, L., & Davies, H. 2003. The performance of cross-border units in China: A test of natural selection, strategic choice and contingency theories. Journal of International Business Studies, 34 (3): 242–254.

    Article  Google Scholar 

  14. Cohen, J. 1988. Statistical power for the behavioral analysis, (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  15. Combs, J. G. 2010. Big samples and small effects: Let's not trade relevance and rigor for power. Academy of Management Journal, 53 (1): 9–13.

    Article  Google Scholar 

  16. Cortina, J. M., & Dunlap, W. P. 1997. On the logic and purpose of significance testing. Psychological Methods, 2 (2): 161–172.

    Article  Google Scholar 

  17. Cumming, G., & Finch, S. 2005. Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60 (2): 170–180.

    Article  Google Scholar 

  18. Cummings, T. G. 2007. 2006 presidential address: Quest for an engaged academy. Academy of Management Review, 32 (2): 355–360.

    Article  Google Scholar 

  19. Ellis, P. D. 2007. Distance, dependence and diversity of markets: Effects on market orientation. Journal of International Business Studies, 38 (3): 374–386.

    Article  Google Scholar 

  20. Ellis, P. D. 2008. Does psychic distance moderate the market size–entry sequence relationship? Journal of International Business Studies, 39 (3): 351–369.

    Article  Google Scholar 

  21. Ellis, P. D. 2010a. Effect size FAQs. http://www.effectsizefaq.com. Accessed 2 June 2010.

  22. Ellis, P. D. 2010b. The essential guide to effect sizes: An introduction to statistical power, meta-analysis and the interpretation of research results. Cambridge: Cambridge University Press.

    Google Scholar 

  23. Hambrick, D. C. 1994. 1993 Presidential Address: What if the academy actually mattered? Academy of Management Review, 19 (1): 11–16.

    Google Scholar 

  24. Hoenig, J. M., & Heisey, D. M. 2001. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55 (1): 19–24.

    Article  Google Scholar 

  25. Iacobucci, D. 2005. From the editor. Journal of Consumer Research, 32 (1): 1–6.

    Article  Google Scholar 

  26. JARS. 2008. Reporting standards for research in psychology. American Psychologist, 63 (9): 839–851.

  27. JEP. 2003. Instructions to authors. Journal of Educational Psychology, 95 (1): 201.

  28. Luk, C. L., Yau, O., Sin, L., Tse, A., Chow, R., & Lee, J. 2008. The effects of social capital and organizational innovativeness in different institutional contexts. Journal of International Business Studies, 39 (4): 589–612.

    Article  Google Scholar 

  29. Mazen, A. M., Graf, L. A., Kellogg, L. A., & Hemmasi, M. 1987. Statistical power in contemporary management research. Academy of Management Journal, 30 (2): 369–380.

    Article  Google Scholar 

  30. Parboteeah, K. P., Hoegl, M., & Cullen, J. B. 2008. Managers’ gender role attitudes: A country institutional profile approach. Journal of International Business Studies, 39 (5): 795–813.

    Article  Google Scholar 

  31. Qian, G. M., Li, L., Li, J., & Qian, Z. M. 2008. Regional diversification and firm performance. Journal of International Business Studies, 39 (2): 197–214.

    Article  Google Scholar 

  32. Rynes, S. L. 2007. Editors afterword: Let's create a tipping point – What academics and practitioners can do, alone and together. Academy of Management Journal, 50 (5): 1046–1054.

    Article  Google Scholar 

  33. Scherbaum, C. A., & Ferreter, J. M. 2009. Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12 (2): 347–367.

    Article  Google Scholar 

  34. Shaver, J. M. 2006. Interpreting empirical findings. Journal of International Business Studies, 37 (4): 451–452.

    Article  Google Scholar 

  35. Shaver, J. M. 2008. Organizational significance. Strategic Organization, 6 (2): 185–193.

    Article  Google Scholar 

  36. Van de Vliert, E. V. 2003. Thermoclimate, culture, and poverty as country-level roots of workers’ wages. Journal of International Business Studies, 34 (1): 40–52.

    Article  Google Scholar 

  37. Zedeck, S. 2003. Editorial. Journal of Applied Psychology, 88 (1): 3–5.

    Article  Google Scholar 

Download references

Acknowledgements

I am grateful to Juergen Brock, Sylvie Chetty, Flora Gu, Evert Van de Vliert, Robert Wright, Consulting Editor Myles Shaver, and three anonymous JIBS referees for providing feedback and constructive criticism on earlier drafts of this paper. The research reported in this paper was supported by a Internal Research Grant provided by the Hong Kong Polytechnic University (Project 4-ZZ9V).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Paul D Ellis.

Additional information

Accepted by Myles Shaver, Consulting Editor, 25 July 2010. This paper has been with the author for three revisions.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ellis, P. Effect sizes and the interpretation of research results in international business. J Int Bus Stud 41, 1581–1588 (2010). https://doi.org/10.1057/jibs.2010.39

Download citation

Keywords

  • evaluation of current empirical approaches
  • theory–method intersection
  • meta-analysis
  • statistical power
  • effect size