Abstract
Computational cognitive models aim to simulate the cognitive processes humans go through when performing a particular task. In this chapter, we discuss a machine learning approach that can discover such cognitive processes in M/EEG data. The method uses a combination of multivariate pattern analysis (MVPA) and hidden semi-Markov models (HsMMs), to take both the spatial extent and the temporal duration of cognitive processes into account. In the first part of this chapter, we will introduce the HsMM-MVPA method and demonstrate its application to an associative recognition dataset. Next, we will use the results of the analysis to inform a high-level cognitive model developed in the ACT-R (adaptive control of thought – rational) architecture. Finally, we will discuss how the HsMM-MVPA method can be extended and how it can inform other modeling paradigms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The dataset and analysis scripts can be found on www.jelmerborst.nl/models.
- 2.
In the original experiment, there were also new foils, foils consisting of entirely new words. Because such stimuli result in a different sequence of cognitive stages, we disregard them for this chapter.
- 3.
The starting parameters influence the EM algorithm used to find the optimal model. An alternative method that we have regularly applied is to use 100 sets of random starting parameters and select the best result, but the method described here is more robust.
- 4.
Available for download at www.jelmerborst.nl/models.
- 5.
Note that our timing estimates are slightly different than in the original model due to our reanalysis of the data.
- 6.
This retrieval constitutes a familiarity process; if this were a slow retrieval due to unfamiliar words in the experimental context, this would trigger a production that directly proceeds to a response stage indicating a completely new foil, as described in Borst and Anderson (2015b) and Anderson et al. (2016).
References
Anderson, J. R. (2005). Human symbol manipulation within an integrated cognitive architecture. Cognitive Science, 29, 313–341.
Anderson, J. R. (2007). How can the human mind occur in the physical universe? Oxford University Press.
Anderson, J. R., & Fincham, J. M. (2014). Extending problem-solving procedures through reflection. Cognitive Psychology, 74, 1–34. https://doi.org/10.1016/j.cogpsych.2014.06.002
Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Lawrence Erlbaum.
Anderson, J. R., & Reder, L. M. (1999). The fan effect: New results and new theories. Journal of Experimental Psychology: General, 128(2), 186–197. https://doi.org/10.1037/0096-3445.128.2.186
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111(4), 1036–1060. https://doi.org/10.1037/0033-295X.111.4.1036
Anderson, J. R., Zhang, Q., Borst, J. P., & Walsh, M. M. (2016). The discovery of processing stages: Extension of Sternberg’s method. Psychological Review, 123(5), 481–509.
Anderson, J. R., Borst, J. P., Fincham, J. M., Ghuman, A. S., Tenison, C., & Zhang, Q. (2018). The common time course of memory processes revealed. Psychological Science, 32, 1463–1474. https://doi.org/10.1177/0956797618774526
Berberyan, H., Van Maanen, L., Van Rijn, H., & Borst, J. P. (2021). EEG-based identification of evidence accumulation stages in decision making. Journal of Cognitive Neuroscience, 33(3), 510–527.
Borst, J. P., & Anderson, J. R. (2015a). Using the cognitive architecture ACT-R in combination with fMRI data. In B. U. Forstmann & E.-J. Wagenmakers (Eds.), Model-based cognitive neuroscience. Springer.
Borst, J. P., & Anderson, J. R. (2015b). The discovery of processing stages: Analyzing EEG data with hidden semi-Markov models. NeuroImage, 108, 60–73.
Borst, J. P., & Anderson, J. R. (2017). A step-by-step tutorial on using the cognitive architecture ACT-R in combination with fMRI data. Journal of Mathematical Psychology, 76, 94–103.
Borst, J. P., Taatgen, N. A., & Van Rijn, H. (2010). The problem state: A cognitive bottleneck in multitasking. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(2), 363–382. https://doi.org/10.1037/a0018106
Borst, J. P., Schneider, D. W., Walsh, M. M., & Anderson, J. R. (2013). Stages of processing in associative recognition: Evidence from behavior, electroencephalography, and classification. Journal of Cognitive Neuroscience, 25(12), 2151–2166.
Borst, J. P., Nijboer, M., Taatgen, N. A., Van Rijn, H., & Anderson, J. R. (2015). Using data-driven model-brain mappings to constrain formal models of cognition. PLoS One, 10(3), e0119673. https://doi.org/10.1371/journal.pone.0119673
Borst, J. P., Ghuman, A. S., & Anderson, J. R. (2016). Tracking cognitive processing stages with MEG: A spatio-temporal model of associative recognition in the brain. NeuroImage, 141, 416–430. https://doi.org/10.1016/j.neuroimage.2016.08.002
Donders, F. C. (1868). De snelheid van psychische processen (On the speed of mental processes).
Eliasmith, C. (2013). How to build a brain: A neural architecture for biological cognition. Oxford University Press.
Forstmann, B. U., Dutilh, G., Brown, S., Neumann, J., von Cramon, D. Y., Ridderinkhof, K. R., & Wagenmakers, E.-J. (2008). Striatum and pre-SMA facilitate decision-making under time pressure. Proceedings of the National Academy of Sciences of the United States of America, 105(45), 17538–17542. https://doi.org/10.1073/pnas.0805903105
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2007). Towards an executive without a homunculus: Computational models of the prefrontal cortex/basal ganglia system. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1485), 1601–1613.
Just, M. A., & Varma, S. (2007). The organization of thinking: What functional brain imaging reveals about the neuroarchitecture of complex cognition. Cognitive, Affective, & Behavioral Neuroscience, 7(3), 153–191.
Kriete, T., Noelle, D. C., Cohen, J. D., & O’Reilly, R. C. (2013). Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences of the United States of America, 110(41), 16390–16395. https://doi.org/10.1073/pnas.1303547110
Lebiere, C. (1999). The dynamics of cognition: An ACT-R model of cognitive arithmetic. Kognitionswissenschaft, 8(1), 5–19.
Luck, S. J. (2005). An introduction to the event-related potential technique. MIT Press.
Makeig, S., Westerfield, M., Jung, T.-P., Enghoff, S., Townsend, J., Courchesne, E., & Sejnowski, T. J. (2002). Dynamic brain sources of visual evoked responses. Science, 295(5555), 690–694.
Malmberg, K. J. (2008). Recognition memory: A review of the critical findings and an integrated theory for relating them. Cognitive Psychology, 57(4), 335–384. https://doi.org/10.1016/j.cogpsych.2008.02.004
McElree, B. (2001). Working memory and focal attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(3), 817–835.
Nijboer, M., Borst, J. P., Van Rijn, H., & Taatgen, N. A. (2016). Contrasting single and multi-component working-memory systems in dual tasking. Cognitive Psychology, 86, 1–26. https://doi.org/10.1016/j.cogpsych.2016.01.003
O’Reilly, R. C., & Frank, M. J. (2006). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18(2), 283–328.
Oberauer, K. (2009). Design for a working memory. In B. H. Ross (Ed.), Psychology of learning and motivation (Vol. 51, pp. 45–100). Academic Press.
Portoles, O., Borst, J. P., & van Vugt, M. K. (2018). Characterizing synchrony patterns across cognitive task stages of associative recognition memory. European Journal of Neuroscience, 48(8), 2759–2769.
Redgrave, P., Prescott, T. J., & Gurney, K. (1999). The basal ganglia: A vertebrate solution to the selection problem? Neuroscience, 89(4), 1009–1023.
Rektor, I., Kaňovský, P., Bareš, M., Brázdil, M., Streitová, H., Klajblová, H., et al. (2003). A SEEG study of ERP in motor and premotor cortices and in the basal ganglia. Clinical Neurophysiology, 114(3), 463–471. https://doi.org/10.1016/S1388-2457(02)00388-7
Rektor, I., Bareš, M., Kaňovský, P., Brázdil, M., Klajblová, I., Streitová, H., et al. (2004). Cognitive potentials in the basal ganglia—Frontocortical circuits. An intracerebral recording study. Experimental Brain Research, 158(3), 289–301. https://doi.org/10.1007/s00221-004-1901-6
Rotello, C. M., & Heit, E. (2000). Associative recognition: A case of recall-to-reject processing. Memory and Cognition, 28(6), 907–922. https://doi.org/10.3758/BF03209339
Rotello, C. M., MacMillan, N. A., & Van Tassel, G. (2000). Recall-to-reject in recognition: Evidence from ROC curves. Journal of Memory and Language, 43, 67–88. https://doi.org/10.1006/jmla.1999.2701
Salvucci, D. D. (2006). Modeling driver behavior in a cognitive architecture. Human Factors, 48(2), 362–380.
Salvucci, D. D., & Anderson, J. R. (2001). Automated eye-movement protocol analysis. Human-Computer Interaction, 16(1), 39–86.
Schneider, D. W., & Anderson, J. R. (2012). Modeling fan effects on the time course of associative recognition. Cognitive Psychology, 64(3), 127–160. https://doi.org/10.1016/j.cogpsych.2011.11.001
Shah, A. S., Bressler, S. L., Knuth, K. H., Ding, M., Mehta, A. D., Ulbert, I., & Schroeder, C. E. (2004). Neural dynamics and the fundamental mechanisms of event-related brain potentials. Cerebral Cortex, 14(5), 476–483. https://doi.org/10.1093/cercor/bhh009
Sternberg, S. (1969). The discovery of processing stages: Extensions of Donders’ method. Acta Psychologica, 30, 276–315. https://doi.org/10.1016/0001-6918(69)90055-9
Stewart, T. C., Bekolay, T., & Eliasmith, C. (2012). Learning to select actions with spiking neurons in the basal ganglia. Frontiers in Neuroscience, 6, 2. https://doi.org/10.3389/fnins.2012.00002
Stocco, A. (2017). A biologically plausible action selection system for cognitive architectures: Implications of basal ganglia anatomy for learning and decision-making models. Cognitive Science, 12(10), 366.
Stocco, A., & Anderson, J. R. (2008). Endogenous control and task representation: An fMRI study in algebraic problem-solving. Journal of Cognitive Neuroscience, 20(7), 1300–1314. https://doi.org/10.1162/jocn.2008.20089
Stocco, A., Lebiere, C., & Anderson, J. R. (2010). Conditional routing of information to the cortex: A model of the basal ganglia’s role in cognitive coordination. Psychological Review, 117(2), 541–574. https://doi.org/10.1037/a0019077
Tenison, C., & Anderson, J. R. (2015). Modeling the distinct phases of skill acquisition. Journal of Experimental Psychology-Learning Memory and Cognition, 42(5), 749–767.
van Maanen, L., Brown, S. D., Eichele, T., Wagenmakers, E.-J., Ho, T., Serences, J., & Forstmann, B. U. (2011). Neural correlates of trial-to-trial fluctuations in response caution. The Journal of Neuroscience, 31(48), 17488–17495. https://doi.org/10.1523/jneurosci.2924-11.2011
Yeung, N., Bogacz, R., Holroyd, C. B., Nieuwenhuis, S., & Cohen, J. D. (2007). Theta phase resetting and the error-related negativity. Psychophysiology, 44(1), 39–49. https://doi.org/10.1111/j.1469-8986.2006.00482.x
Yu, S. Z. (2010). Hidden semi-Markov models. Artificial Intelligence, 174, 215–243. https://doi.org/10.1016/j.artint.2009.11.011
Zhang, Q., Walsh, M. M., & Anderson, J. R. (2017). The effects of probe similarity on retrieval and comparison processes in associative recognition. Journal of Cognitive Neuroscience, 29(2), 352–367.
Zhang, Q., van Vugt, M., Borst, J. P., & Anderson, J. R. (2018a). Mapping working memory retrieval in space and in time: A combined electroencephalography and electrocorticography approach. NeuroImage, 174, 472–484. https://doi.org/10.1016/j.neuroimage.2018.03.039
Zhang, Q., Walsh, M. M., & Anderson, J. R. (2018b). The impact of inserting an additional mental process. Computational Brain & Behavior, 38(4), 1–14. https://doi.org/10.1007/s42113-018-0002-8
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Exercises
-
1.
While the high temporal resolution of EEG is typically given as an advantage as compared to, for example, fMRI, it does have its drawbacks.
-
(a)
Explain why the fourth peak in Fig. 1 disappears when doing a standard ERP analysis.
-
(b)
Would this also be a problem when measuring the same process with fMRI?
-
(a)
-
2.
In the HsMM-MVPA analysis, bumps are supposed to be identical across different trials, while the stage lengths can vary. Explain the rationale behind this.
-
3.
To determine the number of stages in the data, we use leave-one-out cross validation, combined with a sign test. This test evaluates whether the log-likelihood of a significant number of subjects improve when we add a stage. Alternatively, one could also simply take the highest log-likelihood, given that we already use an LOOCV procedure. What is the advantage of using the sign test, and what would be the advantage of using log-likelihood to determine the number of stages?
-
4.
Imagine that participants solve a factorial like 4! = 4 × 3 × 2 × 1. What would the expected stage topology look like?
-
5.
One limitation of using the HsMM-MVPA method with M/EEG is that we only measure the top levels of the cortex. What does this mean for the cognitive stages that we can discover?
-
6.
The ACT-R model that we developed in Sect. 3 of this chapter follows the discovered stages very closely. One could almost say that this is “just a fit” of the discovered stages and does not explain the underlying process.
-
(a)
What speaks against such an interpretation?
-
(b)
How could one further evaluate this model?
-
(a)
-
7.
Take your favorite modeling approach, and discuss how the results of an HsMM-MVPA analysis could inform or help to evaluate such models.
Answers
-
1.
-
(a)
Because the fourth peak appears at a different moment in each trial, it will disappear when averaging across trials or result in a very broad peak with a very low amplitude.
-
(b)
No. This brief peak would add to the slow hemodynamic response on each trial, which is more or less the summation of all processes during the last several seconds. However, the exact timing of the peak would be lost.
-
(a)
-
2. The bumps signify the cognitive processes that participants go through when performing a task, which are assumed to be the same on each trial (at least within a condition). However, the length of these processes can vary on a trial-by-trial basis, which is why the analysis allows the stage duration to differ between trials.
-
3. Sign test: this focuses on generalization across subjects. When applying the sign test, we make sure that this is the best solution for the majority of the subjects. Log-likelihood: this focuses on the best account of all data points. If we would take the highest log-likelihood, we would choose the model that explains the EEG across all subjects best, which is not necessarily the best model for the majority of the subjects.
-
4. For solving n!: Visual encoding – n times a retrieval process for the result followed by an update of working memory – entering the response.
-
5. We can only discover cognitive processes that cause bumps in the top levels of the cortex. It is certainly possible that this does not hold for all cognitive processes, which would be missed by the current method.
-
6.
-
(a)
The mechanism that explains the length of the associative retrieval stage was not developed to account for the results of the HsMM-MVPA analysis. In addition, it is a single mechanism that accounts for the duration in four different conditions, suggesting it might be general.
-
(b)
The next step would be to use this model to predict data in a different experiment, for example, with higher-fan conditions, and test this in an experiment.
-
(a)
-
7. –
Further Reading
-
The current HsMM-MVPA method for EEG data was first introduced by Anderson and colleagues in 2016. This paper has the most detailed description of the method, including extensive appendices in which assumptions underneath the method are tested using synthetic data.
-
In 2018, we applied the same method to MEG data (Anderson et al., 2018) and intracranial EEG data (Zhang et al., 2018a), allowing for greater spatial precision and better interpretation of the underlying cognitive processes.
-
Chapter 1 of Anderson (2007) gives a very clear introduction to cognitive architectures and ACT-R. In case you do not have the book available, Anderson (2005) provides an introduction to ACT-R and its mapping on brain regions.
Rights and permissions
Copyright information
© 2024 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Borst, J.P., Anderson, J.R. (2024). Discovering Cognitive Stages in M/EEG Data to Inform Cognitive Models. In: Forstmann, B.U., Turner, B.M. (eds) An Introduction to Model-Based Cognitive Neuroscience. Springer, Cham. https://doi.org/10.1007/978-3-031-45271-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-45271-0_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45270-3
Online ISBN: 978-3-031-45271-0
eBook Packages: Behavioral Science and PsychologyBehavioral Science and Psychology (R0)