Model discrimination through adaptive experimentation
An ideal experiment is one in which data collection is efficient and the results are maximally informative. This standard can be difficult to achieve because of uncertainties about the consequences of design decisions. We demonstrate the success of a Bayesian adaptive method (adaptive design optimization, ADO) in optimizing design decisions when comparing models of the time course of forgetting. Across a series of testing stages, ADO intelligently adapts the retention interval in order to maximally discriminate power and exponential models. Compared with two different control (non-adaptive) methods, ADO distinguishes the models decisively, with the results unambiguously favoring the power model. Analyses suggest that ADO’s success is due in part to its flexibility in adjusting to individual differences. This implementation of ADO serves as an important first step in assessing its applicability and usefulness to psychology.
- Brown, J. A. (1958). Some tests of the decay theory of immediate memory. The Quarterly Journal of Experimental Psychology, 10, 12–21. CrossRef
- Brown, S., & Heathcote, A. J. (2003). Bias in exponential and power function fits due to noise: Comments on Myung, Kim, and Pitt. Memory & Cognition, 31, 656–661. CrossRef
- Cavagnaro, D. R., Myung, J. I., Pitt, M. A., & Kujala, J. V. (2010). Adaptive design optimization: A mutual information based approach to model discrimination in cognitive science. Neural Computation, 22, 887–905. CrossRef
- Chaloner, K., & Verdinelli, I. (1995). Bayesian experimental design: A review. Statistical Science, 10, 273–304. CrossRef
- Ebbinghaus, H. (1964). Memory: A contribution to experimental psychology. New York: Dover. (Original work published in 1885.)
- Edwards, W., Lindman, H., & Savage, L. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242. CrossRef
- Heathcote, A. J., Brown, S., & Mewhort, D. J. K. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin & Review, 7, 185–207. CrossRef
- Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795. CrossRef
- Laming, K. (1992). Analysis of short-term retention: Models for Brown-Peterson experiments. Journal of Experimental Psychology. Learning, Memory, and Cognition, 18, 1342–1365. CrossRef
- Lee, M. D. (2004). A Bayesian analysis of retention functions. Journal of Mathematical Psychology, 48, 310–321. CrossRef
- Lindley, D. V. (2000). The philosophy of statistics. The Statistician, 49, 293–337.
- McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2, 3–19. CrossRef
- Müller, P., Berry, D., Grieve, A., Smith, M., & Krams, M. (2007). Simulation-based sequential Bayesian design. Journal of Statistical Planning and Inference, 137, 3140–3150. CrossRef
- Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190–204. CrossRef
- Myung, J. I., & Pitt, M. (2009). Optimal experimental design for model discrimination. Psychological Review, 58, 193–198.
- Rubin, D., Hinton, S., & Wenzel, A. (1999). The precise time course of retention. Journal of Experimental Psychology, 25, 1161–1176.
- Rubin, D., & Wenzel, A. (1996). One hundred years of forgetting: A quantitative description of retention. Psychological Review, 103, 734–760. CrossRef
- Squire, L. R. (1989). On the course of forgetting in very long term memory. Journal of Experimental Psychology. Learning, Memory, and Cognition, 15, 241–245. CrossRef
- Wixted, J., & Ebbesen, E. (1991). On the form of forgetting. Psychological Science, 2, 409–415. CrossRef
- Model discrimination through adaptive experimentation
Psychonomic Bulletin & Review
Volume 18, Issue 1 , pp 204-210
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links
- Active learning
- Model discrimination
- Experimental design
- Adaptive testing
- Industry Sectors