Abstract
Although researchers often aim to observe significant results, this chapter will show that the mere significance of a test does not necessarily yield any information about the size of an effect. Statistical significance therefore does not necessarily imply “substantive relevance” or “practical significance”. This chapter introduces the concepts needed to assess these issues, beginning with a systematic consideration of statistical decisions and corresponding decision errors. Subsequently, we introduce Cohen’s d as a measure of effect size for the previously discussed t-tests. Finally, the concept of effect size leads to the concept of power and the question of an optimal sample size.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The procedure described in Formula 7.4 corresponds to the proposal of Cohen (1988). Furthermore, Cohen recommends using a corrected effect \(d_c=d\sqrt {2}\) when calculating power (see Sect. 7.3) for dependent samples, and many computer programs automatically take this correction into account. Some authors go further to suggest reporting dc as effect size instead of d, while other authors describe an adaptation of the conventions for interpreting effect sizes in case of dependent samples (Bortz 2005; Dunlap et al. 1996; Eid et al. 2010).
- 2.
An elegant alternative to this procedure is to take the standard deviation of the difference directly from the output of t.test(). The corresponding syntax is described in the online material. Direct access to effect sizes is also provided by various R packages such as compute.es or MBESS.
References
APA. (2020). Publication manual of the American Psychological Association (7th ed.). APA.
Bortz, J. (2005). Statistik für Human- und Sozialwissenschaftler. Springer.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Erlbaum.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170–177.
Eid, M., Gollwitzer, M., & Schmitt, M. (2010). Statistik und Forschungsmethoden [Statistics and research methods]. Beltz.
Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
Francis, G., Tanzman, J., & Matthews, W. (2014). Excess success for psychology articles in the journal Science. PLoS One, 9, e114255.
Goulet-Pelletier, J. C., & Cousineau, D. (2018). A review of effect sizes and their confidence intervals, Part I: The Cohen’s d family. The Quantitative Methods for Psychology, 14, 242–265.
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.
Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8, 355–362.
Langenberg, B., Janczyk, M., Koob, V., Kliegl, R., & Mayer, A. (2022). A tutorial on using the paired t test for power calculations in repeated measures ANOVA with interactions. Behavior Research Methods.
Rosnow, R. L., & Rosenthal, R. (2003). Effect sizes for experimenting psychologists. Canadian Journal of Experimental Psychology, 57, 221–237.
Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1359–1366.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer-Verlag GmbH Germany, part of Springer Nature
About this chapter
Cite this chapter
Janczyk, M., Pfister, R. (2023). Decision Errors, Effect Sizes, and Power. In: Understanding Inferential Statistics. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-66786-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-662-66786-6_7
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-66785-9
Online ISBN: 978-3-662-66786-6
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)