Attention, Perception, & Psychophysics

, Volume 76, Issue 7, pp 2117–2135

“Plateau”-related summary statistics are uninformative for comparing working memory models

Article
  • 338 Downloads

Abstract

Performance on visual working memory tasks decreases as more items need to be remembered. Over the past decade, a debate has unfolded between proponents of slot models and slotless models of this phenomenon (Ma, Husain, Bays (Nature Neuroscience 17, 347-356, 2014). Zhang and Luck (Nature 453, (7192), 233-235, 2008) and Anderson, Vogel, and Awh (Attention, Perception, Psychophys 74, (5), 891-910, 2011) noticed that as more items need to be remembered, “memory noise” seems to first increase and then reach a “stable plateau.” They argued that three summary statistics characterizing this plateau are consistent with slot models, but not with slotless models. Here, we assess the validity of their methods. We generated synthetic data both from a leading slot model and from a recent slotless model and quantified model evidence using log Bayes factors. We found that the summary statistics provided at most 0.15 % of the expected model evidence in the raw data. In a model recovery analysis, a total of more than a million trials were required to achieve 99 % correct recovery when models were compared on the basis of summary statistics, whereas fewer than 1,000 trials were sufficient when raw data were used. Therefore, at realistic numbers of trials, plateau-related summary statistics are highly unreliable for model comparison. Applying the same analyses to subject data from Anderson et al. (Attention, Perception, Psychophys 74, (5), 891-910, 2011), we found that the evidence in the summary statistics was at most 0.12 % of the evidence in the raw data and far too weak to warrant any conclusions. The evidence in the raw data, in fact, strongly favored the slotless model. These findings call into question claims about working memory that are based on summary statistics.

Keywords

Visual working memory Working memory Model selection Statistical methods 

References

  1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.CrossRefGoogle Scholar
  2. Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set both by visual information load and by number of objects. Psych Science, 15, 106–111.CrossRefGoogle Scholar
  3. Anderson, D. E., & Awh, E. (2012). The plateau in mnemonic resolution across large set sizes indicates discrete resource limits in visual working memory. Atten Percept Psychophys, 74(5), 891–910.PubMedCrossRefGoogle Scholar
  4. Anderson, D. E., Vogel, E. K., & Awh, E. (2011). Precision in visual working memory reaches a stable plateau when individual item limits are exceeded. J Neurosci, 31(3), 1128–1138.PubMedCrossRefGoogle Scholar
  5. Anderson, D. E., Vogel, E. K., & Awh, E. (2013). Selection and storage of perceptual groups is constrained by a discrete resource in working memory. J Exp Psych Hum Percept Perform, 39(3), 824–835.CrossRefGoogle Scholar
  6. Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. J Vision, 9(10), 1–11.CrossRefGoogle Scholar
  7. Bays, P. M., Gorgoraptis, N., Wee, N., Marshall, L., & Husain, M. (2011). Temporal dynamics of encoding, storage, and reallocation of visual working memory. J Vis, 11(10). doi: 11.10.6 [pii] 10.1167/11.10.6
  8. Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321(5890), 851–854.PubMedCrossRefPubMedCentralGoogle Scholar
  9. Buschman, T. J., Siegel, M., Roy, R. E., & Miller, E. K. (2011). Neural substrates of cognitive capacity limitations. Proc Natl Acad Sci, 108(27), 11252–11255.PubMedCrossRefPubMedCentralGoogle Scholar
  10. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: John Wiley & Sons.CrossRefGoogle Scholar
  11. Donkin, C., Nosofsky, R. M., Gold, J. M., & Shiffrin, R. M. (2013). Discrete-slots models of visual working-memory response times. Psych Rev, 120, 873--902.Google Scholar
  12. Elmore, L. C., Ma, W. J., Magnotti, J. F., Leising, K. J., Passaro, A. D., & Katz, J. S. (2011). Visual short-term memory compared in rhesus monkeys and humans. Curr Biol, 21(11), 975–979.PubMedCrossRefGoogle Scholar
  13. Fougnie, D., Suchow, J. W., & Alvarez, G. A. (2012). Variability in the quality of visual working memory. Nat Commun, 3, 1229.PubMedCrossRefPubMedCentralGoogle Scholar
  14. Fukuda, K., Awh, E., & Vogel, E. K. (2010). Discrete capacity limits in visual working memory. Curr Opin Neurobiol, 20(2), 177–182.PubMedCrossRefPubMedCentralGoogle Scholar
  15. Heyselaar, E., Johnston, K., & Pare, M. (2011). A change detection approach to study visual working memory of the macaque monkey. J Vision, 11(3), 11: 1–10.Google Scholar
  16. Jeffreys, H. (1961). The theory of probability (3rd ed.). Oxford: Oxford University Press.Google Scholar
  17. Jones, H. F. (Ed.). (1912). The notebooks of Samuel Butler. London: A.C. Fifield.Google Scholar
  18. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.CrossRefGoogle Scholar
  19. Keshvari, S., Van den Berg, R., & Ma, W. J. (2012). Probabilistic computation in human perception under variability in encoding precision. PLoS ONE, 7(6), e40216.PubMedCrossRefPubMedCentralGoogle Scholar
  20. Keshvari, S., Van den Berg, R., & Ma, W. J. (2013). No evidence for an item limit in change detection. PLoS Comp Biol, 9(2), e1002927.CrossRefGoogle Scholar
  21. Kullback, S. (1997). Information theory and statistics. New York: Courier Dover Publications. Google Scholar
  22. Lara, A. H., & Wallis, J. D. (2012). Capacity and precision in an animal model of short-term memory. J Vision, 12(3), 1–12.CrossRefGoogle Scholar
  23. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature 390(6657), 279--281.Google Scholar
  24. Luck, S. J., & Vogel, E. K. (2013). Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn Sci, 17(8), 391–400.PubMedCrossRefPubMedCentralGoogle Scholar
  25. Lynch, S. M. (2007). Introduction to applied Bayesian statistics and estimation for social scientists. New York: Springer.Google Scholar
  26. Ma WJ, Husain M, Bays PM (2014) Changing concepts of working memory. Nature Neuroscience 17, 347-56. doi:10.1038/nn.3655 Google Scholar
  27. Mardia, K. V., & Jupp, P. E. (1999). Directional statistics. England: John Wiley and Sons.Google Scholar
  28. Palmer, J. (1990). Attentional limits on the perception and memory of visual information. J Exp Psychol Hum Percept Perform, 16(2), 332–350.PubMedCrossRefGoogle Scholar
  29. Pashler, H. (1988). Familiarity and visual change detection. Percept Psychophys 44(4), 369--378.Google Scholar
  30. Rouder, J., Morey, R., Cowan, N., Morey, C., & Pratte, M. (2008). An assessment of fixed-capacity models of visual working memory. Proc Natl Acad Sci U S A, 105(16), 5975–5979.PubMedCrossRefPubMedCentralGoogle Scholar
  31. Schwartz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464.CrossRefGoogle Scholar
  32. Shaw, M. L. (1980). Identifying attentional and decision-making components in information processing. In R. S. Nickerson (Ed.), Attention and Performance (Vol. VIII, pp. 277–296). Hillsdale: Erlbaum.Google Scholar
  33. Sims, C. R., Jacobs, R. A., & Knill, D. C. (2012). An ideal-observer analysis of visual working memory. Psychological Review, 119(4), 807–830.PubMedCrossRefPubMedCentralGoogle Scholar
  34. Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society, Series B, 64(4), 583–639.CrossRefGoogle Scholar
  35. Van den Berg, R., Shin, H., Chou, W.-C., George, R., & Ma, W. J. (2012). Variability in encoding precision accounts for visual short-term memory limitations. Proc Natl Acad Sci U S A, 109(22), 8780–8785.PubMedCrossRefPubMedCentralGoogle Scholar
  36. Van den Berg, R., Awh, E., & Ma, W. J. (2014). Factorial comparison of working memory models. Psych Rev, 21(1), 124--149.Google Scholar
  37. Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748–751.PubMedCrossRefGoogle Scholar
  38. Wilken, P., & Ma, W. J. (2004). A detection theory account of change detection. J Vision, 4(12), 1120–1135.CrossRefGoogle Scholar
  39. Zhang, W., & Luck, S. J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233–235.PubMedCrossRefPubMedCentralGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  1. 1.University of CambridgeCambridgeUK
  2. 2.New York UniversityNew YorkUSA

Personalised recommendations