Effect sizes and the interpretation of research results in international business
- 304 Downloads
Journal editors and academy presidents are increasingly calling on researchers to evaluate the substantive, as opposed to the statistical, significance of their results. To measure the extent to which these calls have been heeded, I aggregated the meta-analytically derived effect size estimates obtained from 965 individual samples. I then surveyed 204 studies published in the Journal of International Business Studies. I found that the average effect size in international business research is small, and that most published studies lack the statistical power to detect such effects reliably. I also found that many authors confuse statistical with substantive significance when interpreting their research results. These practices have likely led to unacceptably high Type II error rates and invalid inferences regarding real-world effects. By emphasizing p values over their effect size estimates, researchers are under-selling their results and settling for contributions that are less than what they really have to offer. In view of this, I offer four recommendations for improving research and reporting practices.
Keywordsevaluation of current empirical approaches theory–method intersection meta-analysis statistical power effect size
I am grateful to Juergen Brock, Sylvie Chetty, Flora Gu, Evert Van de Vliert, Robert Wright, Consulting Editor Myles Shaver, and three anonymous JIBS referees for providing feedback and constructive criticism on earlier drafts of this paper. The research reported in this paper was supported by a Internal Research Grant provided by the Hong Kong Polytechnic University (Project 4-ZZ9V).
- AERA. 2006. Standards for reporting on empirical social science research in AERA publications. http://www.aera.net/opportunities/?id=1850. Accessed 11 September 2008.
- APA. 2001. Publication manual of the American Psychological Association, (5th ed.). Washington, DC: American Psychological Association.Google Scholar
- APA. 2010. Publication manual of the American Psychological Association, (6th ed.). Washington, DC: American Psychological Association.Google Scholar
- Cohen, J. 1988. Statistical power for the behavioral analysis, (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
- Ellis, P. D. 2010a. Effect size FAQs. http://www.effectsizefaq.com. Accessed 2 June 2010.
- Hambrick, D. C. 1994. 1993 Presidential Address: What if the academy actually mattered? Academy of Management Review, 19 (1): 11–16.Google Scholar
- JARS. 2008. Reporting standards for research in psychology. American Psychologist, 63 (9): 839–851.Google Scholar
- JEP. 2003. Instructions to authors. Journal of Educational Psychology, 95 (1): 201.Google Scholar