Journal of International Business Studies

, Volume 41, Issue 9, pp 1581–1588 | Cite as

Effect sizes and the interpretation of research results in international business

  • Paul D Ellis
Research Note

Abstract

Journal editors and academy presidents are increasingly calling on researchers to evaluate the substantive, as opposed to the statistical, significance of their results. To measure the extent to which these calls have been heeded, I aggregated the meta-analytically derived effect size estimates obtained from 965 individual samples. I then surveyed 204 studies published in the Journal of International Business Studies. I found that the average effect size in international business research is small, and that most published studies lack the statistical power to detect such effects reliably. I also found that many authors confuse statistical with substantive significance when interpreting their research results. These practices have likely led to unacceptably high Type II error rates and invalid inferences regarding real-world effects. By emphasizing p values over their effect size estimates, researchers are under-selling their results and settling for contributions that are less than what they really have to offer. In view of this, I offer four recommendations for improving research and reporting practices.

Keywords

evaluation of current empirical approaches theory–method intersection meta-analysis statistical power effect size 

References

  1. Abraham, W. T., & Russell, D. W. 2008. Statistical power analysis in psychological research. Social and Personality Compass, 2 (1): 283–301.CrossRefGoogle Scholar
  2. AERA. 2006. Standards for reporting on empirical social science research in AERA publications. http://www.aera.net/opportunities/?id=1850. Accessed 11 September 2008.
  3. APA. 2001. Publication manual of the American Psychological Association, (5th ed.). Washington, DC: American Psychological Association.Google Scholar
  4. APA. 2010. Publication manual of the American Psychological Association, (6th ed.). Washington, DC: American Psychological Association.Google Scholar
  5. Baggs, J., & Brander, J. A. 2006. Trade liberalization, profitability, and financial leverage. Journal of International Business Studies, 37 (2): 196–211.CrossRefGoogle Scholar
  6. Barden, J. Q., Steensma, H. K., & Lyles, M. A. 2005. The influence of parent control structure on parent conflict in Vietnamese international joint ventures: An organizational justice-based contingency approach. Journal of International Business Studies, 36 (2): 156–174.CrossRefGoogle Scholar
  7. Brock, J. 2003. The “power” of international business research. Journal of International Business Studies, 34 (1): 90–99.CrossRefGoogle Scholar
  8. Campbell, J. P. 1982. Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67 (6): 691–700.CrossRefGoogle Scholar
  9. Campion, M. A. 1993. Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46 (3): 705–718.CrossRefGoogle Scholar
  10. Cano, C. R., Carrillat, F. A., & Jaramillo, F. 2004. A meta-analysis of the relationship between market orientation and business performance: Evidence from five continents. International Journal of Research in Marketing, 21 (2): 179–200.CrossRefGoogle Scholar
  11. Carver, R. P. 1978. The case against statistical significance testing. Harvard Educational Review, 48 (3): 378–399.CrossRefGoogle Scholar
  12. Cashen, L. H., & Geiger, S. W. 2004. Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7 (2): 151–167.CrossRefGoogle Scholar
  13. Child, J., Chung, L., & Davies, H. 2003. The performance of cross-border units in China: A test of natural selection, strategic choice and contingency theories. Journal of International Business Studies, 34 (3): 242–254.CrossRefGoogle Scholar
  14. Cohen, J. 1988. Statistical power for the behavioral analysis, (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  15. Combs, J. G. 2010. Big samples and small effects: Let's not trade relevance and rigor for power. Academy of Management Journal, 53 (1): 9–13.CrossRefGoogle Scholar
  16. Cortina, J. M., & Dunlap, W. P. 1997. On the logic and purpose of significance testing. Psychological Methods, 2 (2): 161–172.CrossRefGoogle Scholar
  17. Cumming, G., & Finch, S. 2005. Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60 (2): 170–180.CrossRefGoogle Scholar
  18. Cummings, T. G. 2007. 2006 presidential address: Quest for an engaged academy. Academy of Management Review, 32 (2): 355–360.CrossRefGoogle Scholar
  19. Ellis, P. D. 2007. Distance, dependence and diversity of markets: Effects on market orientation. Journal of International Business Studies, 38 (3): 374–386.CrossRefGoogle Scholar
  20. Ellis, P. D. 2008. Does psychic distance moderate the market size–entry sequence relationship? Journal of International Business Studies, 39 (3): 351–369.CrossRefGoogle Scholar
  21. Ellis, P. D. 2010a. Effect size FAQs. http://www.effectsizefaq.com. Accessed 2 June 2010.
  22. Ellis, P. D. 2010b. The essential guide to effect sizes: An introduction to statistical power, meta-analysis and the interpretation of research results. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  23. Hambrick, D. C. 1994. 1993 Presidential Address: What if the academy actually mattered? Academy of Management Review, 19 (1): 11–16.Google Scholar
  24. Hoenig, J. M., & Heisey, D. M. 2001. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55 (1): 19–24.CrossRefGoogle Scholar
  25. Iacobucci, D. 2005. From the editor. Journal of Consumer Research, 32 (1): 1–6.CrossRefGoogle Scholar
  26. JARS. 2008. Reporting standards for research in psychology. American Psychologist, 63 (9): 839–851.Google Scholar
  27. JEP. 2003. Instructions to authors. Journal of Educational Psychology, 95 (1): 201.Google Scholar
  28. Luk, C. L., Yau, O., Sin, L., Tse, A., Chow, R., & Lee, J. 2008. The effects of social capital and organizational innovativeness in different institutional contexts. Journal of International Business Studies, 39 (4): 589–612.CrossRefGoogle Scholar
  29. Mazen, A. M., Graf, L. A., Kellogg, L. A., & Hemmasi, M. 1987. Statistical power in contemporary management research. Academy of Management Journal, 30 (2): 369–380.CrossRefGoogle Scholar
  30. Parboteeah, K. P., Hoegl, M., & Cullen, J. B. 2008. Managers’ gender role attitudes: A country institutional profile approach. Journal of International Business Studies, 39 (5): 795–813.CrossRefGoogle Scholar
  31. Qian, G. M., Li, L., Li, J., & Qian, Z. M. 2008. Regional diversification and firm performance. Journal of International Business Studies, 39 (2): 197–214.CrossRefGoogle Scholar
  32. Rynes, S. L. 2007. Editors afterword: Let's create a tipping point – What academics and practitioners can do, alone and together. Academy of Management Journal, 50 (5): 1046–1054.CrossRefGoogle Scholar
  33. Scherbaum, C. A., & Ferreter, J. M. 2009. Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12 (2): 347–367.CrossRefGoogle Scholar
  34. Shaver, J. M. 2006. Interpreting empirical findings. Journal of International Business Studies, 37 (4): 451–452.CrossRefGoogle Scholar
  35. Shaver, J. M. 2008. Organizational significance. Strategic Organization, 6 (2): 185–193.CrossRefGoogle Scholar
  36. Van de Vliert, E. V. 2003. Thermoclimate, culture, and poverty as country-level roots of workers’ wages. Journal of International Business Studies, 34 (1): 40–52.CrossRefGoogle Scholar
  37. Zedeck, S. 2003. Editorial. Journal of Applied Psychology, 88 (1): 3–5.CrossRefGoogle Scholar

Copyright information

© Academy of International Business 2010

Authors and Affiliations

  • Paul D Ellis
    • 1
  1. 1.Department of Management & MarketingHong Kong Polytechnic UniversityKowloonHong Kong

Personalised recommendations