Abstract
This chapter focuses on the use of monofactorial statistical tests to compare two or more groups of speakers or two or more corpora. It starts with a discussion of the Null-Hypothesis Significance Testing Procedure (NHSTP) and its applications to corpus data. The chapter then offers seven different statistical procedures showing their principles, equations, and underlying assumptions. These procedures include the chi-squared test, the t-test, Mann-Whitney U test, ANOVA, Kruskal-Wallis test, Pearson’s correlation and non-parametric correlations. Effect sizes and confidence intervals are also discussed. A particular attention is paid to the distinction between the parametric and non-parametric tests and their respective assumptions. The ‘Practical Guide with R’ section offers the readers a step-by-step guide on how to run the tests discussed in the chapter in the statistical package R.
The writing of this chapter has been supported by UK Economic and Social Research Council (grants ES/R008906/1 and EP/P001559/1).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Relative frequencies are used because speaker samples are of unequal sizes (number of tokens). These are calculated as absolute frequency/number of tokens in the sample x basis for normalisation (e.g. 1000).
- 2.
An independent variable (also known as an explanatory variable or a predictor variable) is a variable that is used to explain linguistic patterns measured as dependent variables. In corpus research, independent variables are typically related to the context (like the genre in this example) or speaker characteristics (gender, age etc.).
- 3.
If we have two or more samples from each speaker and are interested in the difference in their language between sample 1 and sample 2 etc. (e.g. linguistic change/development), we are dealing with a so-called repeated measures design, which requires a different version of the statistic (Crowder 1990).
- 4.
Generally, the cut-off point depends on how much chance we are comfortable with in our discipline. Without testing the whole population (which is usually impracticable), we will always operate in the realm of probability (p-values) and will never have a 100% certainty. Imagine that you have a jar full of sweets of different flavours, some of which you like and some dislike. Would you be willing to pick one at random? Would your answer to this question change if you knew that one of these sweets is poisoned?
References
Balakrishnan, N., Voinov, V., & Nikulin, M. S. (2013). Chi-squared goodness of fit tests with applications. Oxford: Academic.
Boneau, C. A. (1960). The effects of violations of assumptions underlying the t test. Psychological Bulletin, 57(1), 49–64. https://doi.org/10.1037/h0041412.
Bonett, D. G., & Wright, T. A. (2000). Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika, 65(1), 23–28. https://doi.org/10.1007/BF02294183.
Brezina, V. (2018). Statistics in corpus linguistics: A practical guide. Cambridge: Cambridge University Press.
Brezina, V., & Meyerhoff, M. (2014). Significant or random? A critical review of sociolinguistic generalisations based on large corpora. International Journal of Corpus Linguistics, 19(1), 1–28. https://doi.org/10.1075/ijcl.19.1.01bre.
Brezina, V., McEnery, T., & Wattam, S. (2015). Collocations in context: A new perspective on collocation networks. International Journal of Corpus Linguistics, 20(2), 139–173. https://doi.org/10.1075/ijcl.20.2.01bre.
Cabin, R. J., & Mitchell, R. J. (2000). To Bonferroni or not to Bonferroni: When and how are the questions. Bulletin of the Ecological Society of America, 81(3), 246–248.
Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). Hillsdale: Erlbaum.
Cohen, J. (1995). The earth is round (p < 0.05): Rejoinder. American Psychologist, 50(12), 1103. https://doi.org/10.1037/0003-066X.50.12.1103.
Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (pp. 103–126). New York: Russell Sage Foundation.
Corder, G. W., & Foreman, D. I. (2009). Nonparametric statistics for non-statisticians: A step-by-step approach. New York: Wiley.
Cox, D. R., & Donnelly, C. A. (2011). Principles of applied statistics. Cambridge: Cambridge University Press.
Crowder, M. J. (1990). Analysis of repeated measures. New York: Chapman and Hall.
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016). Comparing the Pearson and Spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological Methods, 21(3), 273–290. https://doi.org/10.1037/met0000079.
Duncan, G. T., & Layard, M. W. J. (1973). A Monte-Carlo study of asymptotically robust tests for correlation coefficients. Biometrika, 60(3), 551–558. https://doi.org/10.2307/2335004.
Everitt, B. (2006). The Cambridge dictionary of statistics (3rd ed.). Cambridge: Cambridge University Press.
Fisher, R. A. (1935). The design of experiments. London: Oliver and Boyd.
Gayen, A. K. (1951). The frequency distribution of the product moment correlation in random samples of any size drawn from non-normal universes. Biometrika, 38, 219–247. https://doi.org/10.2307/2332329.
Greenwood, P. E., & Nikulin, M. S. (1996). A guide to chi-squared testing (Vol. 280). New York: Wiley.
Gries, S. T. (2013). Statistics for linguistics with R: A practical introduction (2nd ed.). Berlin: Mouton de Gruyter.
Kapadia, A. S., Chan, W., & Moyé, L. (2005). Mathematical statistics with applications. Boca Raton: CRC Press.
Kepes, S., Banks, G. C., & Oh, I. S. (2014). Avoiding bias in publication bias research: The value of “null” findings. Journal of Business and Psychology, 29(2), 183–203. https://doi.org/10.1007/s10869-012-9279-0.
Kerby, D. S. (2014). The simple difference formula: An approach to teaching nonparametric correlation. Innovative Teaching, 3, 1–9. https://doi.org/10.2466/11.IT.3.1.
Kilgarriff, A. (2005). Language is never, ever, ever, random. Corpus Linguistics and Linguistic Theory, 1(2), 263–275. https://doi.org/10.1515/cllt.2005.1.2.263.
Kowalski, C. J. (1972). On the effects of non-normality on the distribution of the sample product moment correlation coefficient. Applied Statistics, 21(1), 1–12. https://doi.org/10.2307/2346598.
Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. https://doi.org/10.2307/2280779.
Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002). The importance of the normality assumption in large public health data sets. Annual Review of Public Health, 23(1), 151–169. https://doi.org/10.1146/annurev.publhealth.23.100901.140546.
Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491.
Miller, R. G. (1997). Beyond ANOVA: Basics of applied statistics. New York: Chapman and Hall/CRC.
Plonsky, L. (2015). Statistical power, p values, descriptive statistics, and effect sizes: A “back-to-basics” approach to advancing quantitative methods in L2 research. In Advancing quantitative methods in second language research (pp. 43–65). London: Routledge.
Rasch, D., Kubinger, K. D., & Moder, K. (2011). The two-sample t test: Pre-testing its assumptions does not pay off. Statistical Papers, 52(1), 219–231. https://doi.org/10.1007/s00362-009-0224-x.
Rayson, P., Berridge, D., & Francis, B. (2004). Extending the Cochran rule for the comparison of word frequencies between corpora. In: Proceedings from 7th international conference on statistical analysis of textual data, JADT 2004, pp. 926–936.
Ruxton, G. D., & Neuhäuser, M. (2010). When should we use one-tailed hypothesis testing? Methods in Ecology and Evolution, 1(2), 114–117. https://doi.org/10.1111/j.2041-210X.2010.00014.x.
Schmider, E., Ziegler, M., Danay, E., Beyer, L., & Bühner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6(4), 147–151. https://doi.org/10.1027/1614-2241/a000016.
Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47(5), 609–612. https://doi.org/10.1016/j.jrp.2013.05.009.
Schucany, W. R., & Tony Ng, H. K. (2006). Preliminary goodness-of-fit tests for normality do not validate the one-sample Student t. Communications in Statistics-Theory and Methods, 35(12), 2275–2286. https://doi.org/10.1080/03610920600853308.
Shaffer, J. P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46(1), 561–584.
Sherman, G. R. (1954). The “Student” T test when applied to non-normal populations. Department of Statistics, Stanford University.
Sheskin, D. (2004). Handbook of parametric and nonparametric statistical procedures (3rd ed.). London: Chapman & Hall/CRC.
Shingala, M. C., & Rajyaguru, A. (2015). Comparison of post hoc tests for unequal variance. International Journal of New Technologies in Science and Engineering, 2(5), 22–33.
Sprent, P. (2011). Fisher exact test. In M. Lovric (Ed.), International encyclopedia of statistical science (pp. 524–525). Berlin/Heidelberg: Springer.
Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37, 1–2.
Upton, G. J. G. (1992). Fisher’s exact test. Journal of the Royal Statistical Society. Series A (Statistics in Society), 155(3), 395–402. https://doi.org/10.2307/2982890.
Yates, F. (1934). Contingency table involving small numbers and the χ2 test. Supplement to the Journal of the Royal Statistical Society, 1(2), 217–235.
Zimmerman, D. W. (1998). Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. The Journal of Experimental Education, 67(1), 55–68. https://doi.org/10.1080/00220979809598344.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic Supplementary Materials
Further Reading
Further Reading
-
Brezina, V. 2018. Statistics in corpus linguistics: A practical guide . Cambridge University Press, Cambridge.
Brezina (2018: Chaps. 6 and 8) provides more information and real examples (case studies) of the use of the statistical measures discussed in this chapter. The book is intended for beginner and intermediate users of statistical techniques in corpus linguistics and does not presuppose any knowledge of statistics. It is accompanied by Lancaster Stats Tools online, a free and easy to use (‘click and analyse’) statistical tool (http://corpora.lancs.ac.uk/stats) (accessed 14 June 2019).
-
Gablasova, D., Brezina, V., and McEnery, T. 2017. Exploring learner language through corpora: Comparing and interpreting corpus frequency information. Language Learning 67(S1):130–154. doi:10.1111/lang.12226.
This article offers a critical view on using frequency data in corpus linguistics in the context of language learning; this critique is, however, applicable to other contexts as well. The paper investigates the sources of variation in corpora and shows how these can be dealt with systematically using different statistical and visualisation techniques.
-
Gries, S.T. 2013. Statistics for linguistics with R: A practical introduction , 2nd ed. Mouton de Gruyter, Berlin.
Gries (2013) provides an informative introduction to using R in statistical analysis of language. Chapter 2 will be of interest to anyone new to the statistical package R. The book will appeal to a wide range of users who seek statistical sophistication in their analyses.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Brezina, V. (2020). Classical Monofactorial (Parametric and Non-parametric) Tests. In: Paquot, M., Gries, S.T. (eds) A Practical Handbook of Corpus Linguistics. Springer, Cham. https://doi.org/10.1007/978-3-030-46216-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-46216-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-46215-4
Online ISBN: 978-3-030-46216-1
eBook Packages: Religion and PhilosophyPhilosophy and Religion (R0)