MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment

Abstract

The main purpose of this study was to examine the validity of the approach to lexical diversity assessment known as the measure of textual lexical diversity (MTLD). The index for this approach is calculated as the mean length of word strings that maintain a criterion level of lexical variation. To validate the MTLD approach, we compared it against the performances of the primary competing indices in the field, which include vocd-D, TTR, Maas, Yule’s K, and an HD-D index derived directly from the hypergeometric distribution function. The comparisons involved assessments of convergent validity, divergent validity, internal validity, and incremental validity. The results of our assessments of these indices across two separate corpora suggest three major findings. First, MTLD performs well with respect to all four types of validity and is, in fact, the only index not found to vary as a function of text length. Second, HD-D is a viable alternative to the vocd-D standard. And third, three of the indices—MTLD, vocd-D (or HD-D), and Maas—appear to capture unique lexical information. We conclude by advising researchers to consider using MTLD, vocd-D (or HD-D), and Maas in their studies, rather than any single index, noting that lexical diversity can be assessed in many ways and each approach may be informative as to the construct under investigation.

References

  1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  2. Best, R., Ozuru, Y., Floyd, R., & McNamara, D. S. (2006). Children’s text comprehension: Effects of genre, knowledge, and text cohesion. In S. A. Barab, K. E. Hay, & D. T. Hickey (Eds.), Proceedings of the Seventh International Conference of the Learning Sciences (pp. 37–42). Mahwah, NJ: Erlbaum.

    Google Scholar 

  3. Biber, D. (1988). Variation across speech and writing. Cambridge: Cambridge University Press.

    Google Scholar 

  4. Biber, D. (1989). A typology of English texts. Linguistics, 27, 3–43.

    Article  Google Scholar 

  5. Biggs, A., Daniel, L., Feather, R. M., Ortleb, E., Rillero, P., Snyder, S. L., & Zike, D. (2003). Glencoe science: Science level green. New York: Glencoe/McGraw-Hill.

    Google Scholar 

  6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  7. Crossley, S. A., & McNamara, D. S. (2009). Computationally assessing lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18, 119–135.

    Article  Google Scholar 

  8. Crossley, S. A., & McNamara, D. S. (in press). Predicting second language writing proficiency: The role of cohesion, readability, and lexical difficulty. Journal of Research in Reading.

  9. Crossley, S. A., Salsbury, T., & McNamara, D. S. (2009). Measuring second language lexical growth using hypernymic relationships. Language Learning, 59, 307–334.

    Article  Google Scholar 

  10. Dempsey, K. B., McCarthy, P. M., & McNamara, D. S. (2007). Using phrasal verbs as an index to distinguish text genres. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference (pp. 217–222). Menlo Park, CA: AAAI Press.

    Google Scholar 

  11. Dugast, D. (1978). Sur quoi se fonde la notion d’étendue théoretique du vocabulaire? Le Français Moderne, 46, 25–32.

    Google Scholar 

  12. Ertmer, P. A., Bai, H., Dong, C., Khalil, M., Park, S. H., & Wang, L. (2002). Online professional development: Building administrators’ capacity for technology leadership. Journal in Computing Teacher Education, 19, 5–11.

    Google Scholar 

  13. Glaser, B. G., & Strauss, A. (1967). Discovery of grounded theory: Strategies for qualitative research. New York: Aldine.

    Google Scholar 

  14. Harris Wright, H., Silverman, S. W., & Newhoff, M. (2003). Measures of lexical diversity in aphasia. Aphasiology, 17, 443–452.

    Article  Google Scholar 

  15. Herdan, G. (1964). Quantitative linguistics. London: Butterworths.

    Google Scholar 

  16. Hess, C. W., Sefton, K. M., & Landry, R. G. (1986). Sample size and type-token ratios for oral language of preschool children. Journal of Speech & Hearing Research, 29, 129–134.

    Google Scholar 

  17. Honoré, A. (1979). Some simple measures of richness of vocabulary. Association for Literary & Linguistic Computing Bulletin, 7, 172–177.

    Google Scholar 

  18. Jarvis, S. (2002). Short texts, best fitting curves, and new measures of lexical diversity. Language Testing, 19, 57–84.

    Article  Google Scholar 

  19. Johansson, S., Leech, G., & Goodluck, H. (1978). Manual of information to accompany the Lancaster-Oslo/Bergen Corpus of British English, for use with digital computers. Oslo: University of Oslo, Department of English.

    Google Scholar 

  20. Johnson, W. (1944). Studies in language behavior: I. A program of research. Psychological Monographs, 56, 1–15.

    Article  Google Scholar 

  21. Kučera, H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.

    Google Scholar 

  22. Landauer, T. K., Laham, D., Rehder, B., & Schreiner, M. E. (1997). How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. In M. G. Shafto & P. Langley (Eds.), Proceedings of the 19th Annual Meeting of the Cognitive Science Society (pp. 412–417). Mahwah, NJ: Erlbaum.

    Google Scholar 

  23. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

    Google Scholar 

  24. Louwerse, M. M., McCarthy, P. M., McNamara, D. S., & Graesser, A. C. (2004). Variation in language and cohesion across written and spoken registers. In K. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th Annual Conference of the Cognitive Science Society (pp. 843–848). Mahwah, NJ: Erlbaum.

    Google Scholar 

  25. Maas, H. D. (1972). Zusammenhang zwischen Wortschatzumfang und Länge eines Textes. Zeitschrift für Literaturwissenschaft und Linguistik, 8, 73–79.

    Google Scholar 

  26. Malvern, D. D., Richards, B. J., Chipere, N., & Durán, P. (2004). Lexical diversity and language development: Quantification and assessment. Houndmills, NH: Palgrave Macmillan.

    Google Scholar 

  27. McCarthy, P. M., Dufty, D., Hempelman, C., Cai, Z., Graesser, A. C., & McNamara, D. S. (in press). Evaluating givenness/newness. Discourse Processes.

  28. McCarthy, P. M., & Jarvis, S. (2007). A theoretical and empirical evaluation of vocd. Language Testing, 24, 459–488.

    Article  Google Scholar 

  29. McCarthy, P. M., Myers, J. C., Briner, S. W., Graesser, A. C., & McNamara, D. S. (2009). A psychological and computational study of genre recognition. Journal for Language Technology & Computational Linguistics, 24, 23–55.

    Google Scholar 

  30. McEnery, T. (2003). Corpus linguistics. In R. Mitkov (Ed.), Handbook of computational linguistics (pp. 448–463). Oxford: Oxford University Press.

    Google Scholar 

  31. McKee, G., Malvern, D., & Richards, B. (2000). Measuring vocabulary diversity using dedicated software. Literary & Linguistic Computing, 15, 323–337.

    Article  Google Scholar 

  32. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57–86.

    Article  Google Scholar 

  33. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (in press). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes.

  34. Miller, D. P. (1981). The depth/breadth trade-off in hierarchical computer menus. In Proceedings of the Human Factors Society 25th Annual Meeting (pp. 296–300). Santa Monica, CA: HFES.

    Google Scholar 

  35. Morse, J. M. (1995). The significance of saturation. Qualitative Health Research, 5, 147–149.

    Article  Google Scholar 

  36. Olney, A. M. (2007). Latent semantic grammar induction: Context, projectivity, and prior distributions. In R. Dragomir & R. Mihalcea (Eds.), Proceedings of TextGraphs-2: Graph-based algorithms for natural language processing (pp. 45–52). Rochester, NY: Association for Computational Linguistics.

    Google Scholar 

  37. Ong, A. D., & van Dulmen, M. H. M. (2006). Oxford handbook of methods in positive psychology. Oxford: Oxford University Press.

    Google Scholar 

  38. Orlov, Y. K. (1983). Ein Model der Häufigekeitsstruktur des Vokabulars. In H. Guiter & M. V. Arapov (Eds.), Studies on Zipf’s law (pp. 154–233). Bochum: Brockmeyer.

    Google Scholar 

  39. Owen, A. J., & Leonard, L. B. (2002). Lexical diversity in the spontaneous speech of children with specific language impairment: Application of D. Journal of Speech & Hearing Research, 45, 927–937.

    Article  Google Scholar 

  40. Silverman, S. W., & Bernstein Ratner, N. (2000). Word frequency distributions and type-token characteristics. Mathematical Scientist, 11, 45–72.

    Google Scholar 

  41. Somers, H. H. (1966). Statistical methods in literary analysis. In J. Leeds (Ed.), The computer and literary style (pp. 128–140). Kent, OH: Kent State University.

    Google Scholar 

  42. Templin, M. (1957). Certain language skills in children. Minneapolis: University of Minnesota Press.

    Google Scholar 

  43. Tuldava, J. (1993). The statistical structure of a text and its readability. In L. Hrebícek & G. Altmann (Eds.), Quantitative text analysis (pp. 215–227). Trier: Wissenschaftlicher Verlag.

    Google Scholar 

  44. Tweedie, F. J., & Baayen, R. H. (1998). How variable may a constant be? Measures of lexical richness in perspective. Computers & the Humanities, 32, 323–352.

    Article  Google Scholar 

  45. Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.

    Google Scholar 

  46. Wu, T. (1993). An accurate computation of the hypergeometric distribution function. ACM Transactions on Mathematical Software, 19, 33–43.

    Article  Google Scholar 

  47. Yule, G. U. (1944). The statistical study of literary vocabulary. Cambridge: Cambridge University Press.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Philip M. McCarthy.

Additional information

This research was supported in part by the Institute for Education Sciences (IES; Grants R305GA080589, R305G020018-02, and R305G040046) and in part by the National Science Foundation (NSF; Grant IIS-0735682). The views expressed in this article do not necessarily reflect the views of the IES or the NSF.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

McCarthy, P.M., Jarvis, S. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods 42, 381–392 (2010). https://doi.org/10.3758/BRM.42.2.381

Download citation

Keywords

  • Specific Language Impairment
  • Latent Semantic Analysis
  • Factor Size
  • Divergent Validity
  • Hypergeometric Distribution