Behavior Research Methods

, Volume 49, Issue 5, pp 1639–1651 | Cite as

AGSuite: Software to conduct feature analysis of artificial grammar learning performance

  • Matthew T. Cook
  • Chrissy M. Chubala
  • Randall K. Jamieson


To simplify the problem of studying how people learn natural language, researchers use the artificial grammar learning (AGL) task. In this task, participants study letter strings constructed according to the rules of an artificial grammar and subsequently attempt to discriminate grammatical from ungrammatical test strings. Although the data from these experiments are usually analyzed by comparing the mean discrimination performance between experimental conditions, this practice discards information about the individual items and participants that could otherwise help uncover the particular features of strings associated with grammaticality judgments. However, feature analysis is tedious to compute, often complicated, and ill-defined in the literature. Moreover, the data violate the assumption of independence underlying standard linear regression models, leading to Type I error inflation. To solve these problems, we present AGSuite, a free Shiny application for researchers studying AGL. The suite’s intuitive Web-based user interface allows researchers to generate strings from a database of published grammars, compute feature measures (e.g., Levenshtein distance) for each letter string, and conduct a feature analysis on the strings using linear mixed effects (LME) analyses. The LME analysis solves the inflation of Type I errors that afflicts more common methods of repeated measures regression analysis. Finally, the software can generate a number of graphical representations of the data to support an accurate interpretation of results. We hope the ease and availability of these tools will encourage researchers to take full advantage of item-level variance in their datasets in the study of AGL. We moreover discuss the broader applicability of the tools for researchers looking to conduct feature analysis in any field.


Shiny application Artificial grammar learning Feature analysis Linear mixed effects analysis 


Author note

This research was supported by a University of Manitoba Undergraduate Research Award to M.T.C., an NSERC PGS-D to C.M.C., and an NSERC Discovery Grant to R.K.J.


  1. Baayen, R. (2008). Analyzing linguistic data: A practical introduction to statistics using R. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
  2. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390–412. doi: 10.1016/j.jml.2007.12.005 CrossRefGoogle Scholar
  3. Balota, D. A., Yap, M. J., Cortese, M. J., Hutchison, K. A., Kessler, B., Loftis, B., … Treiman, R. (2007). The English Lexicon Project. Behavior Research Methods, 39, 445–459. doi: 10.3758/BF03193014
  4. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. doi: 10.18637/jss.v067.i01 CrossRefGoogle Scholar
  5. Bodner, G. E., Taikh, A., & Fawcett, J. M. (2014). Assessing the costs and benefits of production in recognition. Psychonomic Bulletin & Review, 21, 149–154. doi: 10.3758/s13423-013-0485-1 CrossRefGoogle Scholar
  6. Brooks, L. R., & Vokey, J. R. (1991). Abstract analogies and abstracted grammars: Comments of Reber (1989) and Matthews et al. (1989). Journal of Experimental Psychology: General, 120, 316–323.CrossRefGoogle Scholar
  7. Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41, 977–990. doi: 10.3758/BRM.41.4.977 CrossRefPubMedGoogle Scholar
  8. Burgess, C., & Lund, K. (2000). The dynamics of meaning in memory. In E. Dietrich & A. B. Markman (Eds.), Cognitive dynamics: Conceptual and representational change in humans and machines (pp. 117–156). Mahwah, NJ: Erlbaum.Google Scholar
  9. Coltheart, M. (1981). The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology, 33, 497–505. doi: 10.1080/14640748108400805 CrossRefGoogle Scholar
  10. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671–684. doi: 10.1016/S0022-5371(72)80001-X CrossRefGoogle Scholar
  11. Dienes, Z. (1992). Connectionist and memory array models of artificial grammar learning. Cognitive Science, 16, 41–79.CrossRefGoogle Scholar
  12. Higham, P. A., & Brooks, L. (1997). Learning the experimenter’s design tacit: Sensitivity to the structure of memory lists. Quarterly Journal of Experimental Psychology, 50A, 199–215.CrossRefGoogle Scholar
  13. Jamieson, R. K., & Mewhort, D. J. K. (2010). Applying an exemplar model to the artificial-grammar task: String-completion and performance for individual items. Quarterly Journal of Experimental Psychology, 63, 1014–1039.CrossRefGoogle Scholar
  14. Jamieson, R. K., Mewhort, D. J. K., & Hockley, W. E. (2016). A computational account of the production effect: Still playing twenty questions with nature. Canadian Journal of Experimental Psychology, 70, 154–164.CrossRefPubMedGoogle Scholar
  15. Jamieson, R. K., Nevzorova, U., Lee, G., & Mewhort, D. J. K. (2016). Information theory and artificial grammar learning: Inferring grammaticality from redundancy. Psychological Research, 80, 195–211.CrossRefPubMedGoogle Scholar
  16. Jamieson, R. K., & Spear, J. (2014). The offline production effect. Canadian Journal of Experimental Psychology, 68, 20–28.CrossRefPubMedGoogle Scholar
  17. Jamieson, R. K., Vokey, J. R., & Mewhort, D. J. K. (2017). Implicit learning is order dependent. Psychological Research, 81, 204–218.CrossRefPubMedGoogle Scholar
  18. Johnstone, T., & Shanks, D. (1999). Two mechanisms in implicit artificial grammar learning? Comment on Meulemans and Van Der Linden (1997). Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 524–531.Google Scholar
  19. Johnstone, T., & Shanks, D. (2001). Abstractionist and processing accounts of implicit learning. Cognitive Psychology, 42, 61–112.CrossRefPubMedGoogle Scholar
  20. Jones, M. N., & Mewhort, D. J. K. (2007). Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114, 1–37. doi: 10.1037/0033-295X.114.1.1 CrossRefPubMedGoogle Scholar
  21. Kinder, A. (2000). The knowledge acquired during artificial grammar learning: Testing the predictions of two connectionist models. Psychological Research, 63, 95–105.CrossRefPubMedGoogle Scholar
  22. Kinder, A., & Lotz, A. (2009). Connectionist models of artificial grammar learning: What type of knowledge is acquired? Psychological Research, 73, 659–673.CrossRefPubMedGoogle Scholar
  23. Knowlton, B. J., & Squire, L. R. (1996). Artificial grammar learning depends on implicit acquisition of both abstract and exemplar-specific information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 169–181. doi: 10.1037/0278-7393.22.1.169 PubMedGoogle Scholar
  24. Ko, E., Soderstrom, M., & Morgan, J. (2009). Development of perceptual sensitivity to extrinsic vowel duration in infants learning American English. Journal of the Acoustical Society of America, 126, EL134–9.CrossRefPubMedPubMedCentralGoogle Scholar
  25. Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44, 978–990. doi: 10.3758/s13428-012-0210-4 CrossRefPubMedGoogle Scholar
  26. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240. doi: 10.1037/0033-295X.104.2.211 CrossRefGoogle Scholar
  27. Lorch, R. F., Jr., & Myers, J. L. (1990). Regression analyses of repeated measures data in cognitive research. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 149–157. doi: 10.1037/0278-7393.16.1.149 PubMedGoogle Scholar
  28. MacLeod, C. M., Gopie, N., Hourihan, K. L., Neary, K. R., & Ozubko, J. D. (2010). The production effect: Delineation of a phenomenon. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 671–685. doi: 10.1037/a0018785 PubMedGoogle Scholar
  29. McAndrews, M. P., & Moscovitch, M. (1985). Rule-based and exemplar-based classification in artificial grammar learning. Memory & Cognition, 13, 469–475.CrossRefGoogle Scholar
  30. McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New York, NY: Wiley-Interscience.Google Scholar
  31. Miller, G. A. (1958). Free recall of redundant strings of letters. Journal of Experimental Psychology, 56, 433–491.Google Scholar
  32. Perruchet, P., & Pacteau, C. (1990). Synthetic grammar learning: Implicit rule abstraction or explicit fragmentary knowledge. Journal of Experimental Psychology: General, 119, 264–275. doi: 10.1037/0096-3445.119.3.264 CrossRefGoogle Scholar
  33. Reber, A. S. (1967). Implicit learning of artificial grammars. Journal of Verbal Learning and Verbal Behavior, 6, 855–863. doi: 10.1016/S0022-5371(67)80149-X CrossRefGoogle Scholar
  34. Redington, M., & Chater, N. (1996). Transfer in artificial grammar learning: A reevaluation. Journal of Experimental Psychology: General, 125, 123–138.CrossRefGoogle Scholar
  35. Scott, R. B., & Dienes, Z. (2008). The conscious, the unconscious, and familiarity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1264–1288.PubMedGoogle Scholar
  36. Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance components. New York, NY: Wiley.CrossRefGoogle Scholar
  37. Van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. Quarterly Journal of Experimental Psychology, 67, 1176–1190. doi: 10.1080/17470218.2013.850521 CrossRefGoogle Scholar
  38. Vokey, J. R., & Brooks, L. R. (1992). Salience of item knowledge in learning artificial grammars. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 328–344.Google Scholar
  39. Vokey, J. R., & Jamieson, R. K. (2014). A visual familiarity account of evidence for orthographic processing in baboons (Papio papio). Psychological Science, 25, 991–996.CrossRefPubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • Matthew T. Cook
    • 1
  • Chrissy M. Chubala
    • 1
  • Randall K. Jamieson
    • 1
  1. 1.Department of PsychologyUniversity of ManitobaWinnipegCanada

Personalised recommendations