• Brandon M. Turner
  • Birte U. Forstmann
  • Mark Steyvers
Part of the Computational Approaches to Cognition and Perception book series (CACP)


In this chapter we highlight four applications of joint modeling to various contexts where the integration of neural and behavioral data proved useful in unveiling the underlying cognitive dynamics that were not apparent with behavioral data alone. In the first application, we discuss how structural information in the form of diffusion weighted imaging (DWI) can be used to better constrain a model of evidence accumulation fit to data from a perceptual decision making task with a speed accuracy manipulation. Second, we discuss a recent extension of this approach to functional imaging data obtained using magnetic resonance imaging (fMRI) on a trial-by-trial level. In particular, we discuss how pre-stimulus information can be used to enhance predictions about the speed and accuracy of decision making processes once the stimulus is presented. Third, we discuss how the joint modeling approach can be used to exploit the temporal resolution provided by electroencephalography (EEG) and the spatial resolution provided by fMRI to better constrain a cognitive model. We show that while these two measures provide similar information about the decision making process, the information is not identical within a subject, and as a result, having more neural covariates can enhance model predictions for withheld data. Finally, we discuss how joint modeling can be applied to single-unit recording data from monkeys, highlighting that our approach is not modality specific.


Structural connectivity Single-trial parameters Multimodal integration 


  1. 11.
    L. van Maanen, S.D. Brown, T. Eichele, E.J. Wagenmakers, T. Ho, J. Serences, J. Neurosci. 31, 17488 (2011)CrossRefGoogle Scholar
  2. 12.
    B.M. Turner, B.U. Forstmann, E.J. Wagenmakers, S.D. Brown, P.B. Sederberg, M. Steyvers, NeuroImage 72, 193 (2013)CrossRefGoogle Scholar
  3. 13.
    M.L. Mack, A.R. Preston, B.C. Love, Curr. Biol. 23, 2023 (2013)CrossRefGoogle Scholar
  4. 18.
    B.M. Turner, L. Van Maanen, B.U. Forstmann, Psychol. Rev. 122, 312 (2015)CrossRefGoogle Scholar
  5. 33.
    M.D. Nunez, R. Srinivasan, J. Vandekerckhove, Front. Psychol. 8–18 (2015)Google Scholar
  6. 34.
    M. Frank, C. Gagne, E. Nyhus, S. Masters, T.V. Wiecki, J.F. Cavanagh, D. Badre, J. Neurosci. 35(2), 485 (2015)CrossRefGoogle Scholar
  7. 35.
    M.D. Nunez, J. Vandekerckhove, R. Srinivasan, How attention influences perceptual decision making: single-trial EEG correlates of drift-diffusion model parameters (2016, in press)Google Scholar
  8. 36.
    S. Brown, A. Heathcote, Cogn. Psychol. 57, 153 (2008)CrossRefGoogle Scholar
  9. 37.
    R. Ratcliff, Psychol. Rev. 85, 59 (1978)CrossRefGoogle Scholar
  10. 40.
    D. van Ravenzwaaij, A. Provost, S.D. Brown, J. Math. Psychol. 76, 131 (2017)CrossRefGoogle Scholar
  11. 44.
    C.A. Rodriguez, B.M. Turner, T. Van Zandt, S.M. McClure, Eur. J. Neurosci. 1–11 (2015)Google Scholar
  12. 45.
    B.M. Turner, in An Introduction to Model-Based Cognitive Neuroscience, ed. by B.U. Forstmann, E.J. Wagenmakers (Springer, New York, 2015), pp. 199–220Google Scholar
  13. 46.
    B.M. Turner, C.A. Rodriguez, T. Norcia, M. Steyvers, S.M. McClure, NeuroImage 128, 96 (2016)CrossRefGoogle Scholar
  14. 67.
    B.M. Turner, P.B. Sederberg, S.D. Brown, M. Steyvers, Psychol. Methods 18, 368 (2013)CrossRefGoogle Scholar
  15. 68.
    B.M. Turner, T. Wang, E. Merkel, NeuroImage 153, 28 (2017)CrossRefGoogle Scholar
  16. 70.
    J.J. Palestro, G. Bahg, P.B. Sederberg, Z.L. Lu, M. Steyvers, B.M. Turner, J. Math. Psychol. 84, 20 (2018)CrossRefGoogle Scholar
  17. 75.
    I.J. Myung, J. Math. Psychol. 44, 190 (2000)CrossRefGoogle Scholar
  18. 76.
    I.J. Myung, M. Forster, M.W. Browne, J. Math. Psychol. 44, 1 (2000)CrossRefGoogle Scholar
  19. 83.
    P. Cassey, G. Gaut, M. Steyvers, S.D. Brown, A generative joint model for spike trains and saccades during perceptual decision making. Psychon. Bull. Rev. 23, 1757–1778 (2016)CrossRefGoogle Scholar
  20. 84.
    J.W. Mink, Prog. Neurobiol. 50, 381 (1996)CrossRefGoogle Scholar
  21. 85.
    Y. Smith, M.D. Bevan, E. Shink, J.P. Bolam, Neuroscience 86, 353 (1998)CrossRefGoogle Scholar
  22. 86.
    B.U. Forstmann, G. Dutilh, S. Brown, J. Neumann, D.Y. von Cramon, K.R. Ridderinkhof, E.J. Wagenmakers, Proc. Natl. Acad. Sci. 105, 17538 (2008)CrossRefGoogle Scholar
  23. 87.
    R. Bogacz, E.J. Wagenmakers, B.U. Forstmann, S. Nieuwenhuis, Trends Neurosci. 33, 10 (2010)CrossRefGoogle Scholar
  24. 88.
    B.U. Forstmann, A. Anwander, A. Schäfer, J. Neumann, S. Brown, E.J. Wagenmakers, R. Bogacz, R. Turner, Proc. Natl. Acad. Sci. 107, 15916 (2010)CrossRefGoogle Scholar
  25. 89.
    B.U. Forstmann, M. Tittgemeyer, E.J. Wagenmakers, J. Derrfuss, D. Imperati, S. Brown, J. Neurosci. 31, 17242 (2011)CrossRefGoogle Scholar
  26. 90.
    T. Behrens, H. Johansen-Berg, M.W. Woolrich, S.M. Smith, C.A. Wheeler-Kingshott, P.A. Boulby, G.J. Barker, E.L. Sillery, K. Sheehan, O. Ciccarelli, A.J. Thompson, J.M. Brady, P.M. Matthews, Nat. Neurosci. 6, 750 (2003)CrossRefGoogle Scholar
  27. 91.
    C. Donkin, S. Brown, A. Heathcote, E.J. Wagenmakers, Psychon. Bull. Rev. 18, 61 (2011)CrossRefGoogle Scholar
  28. 92.
    M. Stone, Psychometrika 25, 251 (1960)CrossRefGoogle Scholar
  29. 93.
    C. Donkin, A. Heathcote, S. Brown, in 9th International Conference on Cognitive Modeling – ICCM2009, Manchester, ed. by A. Howes, D. Peebles, R. Cooper (2009)Google Scholar
  30. 94.
    D.J. Spiegelhalter, N.G. Best, B.P. Carlin, A. van der Linde, J. R. Stat. Soc. B 64, 583 (2002)CrossRefGoogle Scholar
  31. 95.
    A. Tomohiro, Biometrika 94, 443 (2007)CrossRefGoogle Scholar
  32. 96.
    M.A. Pitt, I.J. Myung, S. Zhang, Psychol. Rev. 109, 472 (2002)CrossRefGoogle Scholar
  33. 97.
    B. Purcell, R. Heitz, J. Cohen, J. Schall, G. Logan, T. Palmeri, Psychol. Rev. 117, 1113 (2010)CrossRefGoogle Scholar
  34. 98.
    R. Ratcliff, Psychol. Rev. 120, 281 (2013)CrossRefGoogle Scholar
  35. 99.
    M. Jones, E.N. Dzhafarov, Psychol. Rev. 121, 1 (2014)CrossRefGoogle Scholar
  36. 100.
    A. Heathcote, E.J. Wagenmakers, S.D. Brown, Psychol. Rev. 121, 676 (2014)CrossRefGoogle Scholar
  37. 101.
    M. Peruggia, T. Van Zandt, M. Chen, Case Stud. Bayesian Stat. VI, 319 (2002)Google Scholar
  38. 102.
    P. Craigmile, M. Peruggia, T.V. Zandt, Psychometrika 75, 613 (2010)CrossRefGoogle Scholar
  39. 103.
    M.E. Raichle, A.M. MacLeod, A.Z. Snyder, W.J. Powers, D.A. Gusnard, G.L. Shulman, Proc. Natl. Acad. Sci. U. S. 98, 676 (2001)CrossRefGoogle Scholar
  40. 107.
    T. Eichele, S. Debener, V.D. Calhoun, K. Specht, A.K. Engel, K. Hugdahl, D.Y. von Cramon, M. Ullsperger, Proc. Natl. Acad. Sci. U. S. 16, 6173 (2008)CrossRefGoogle Scholar
  41. 109.
    M. Mittner, W. Boekel, A.M. Tucker, B.M. Turner, A. Heathcote, B.U. Forstmann, J. Neurosci. 34, 16286 (2014)CrossRefGoogle Scholar
  42. 110.
    C.P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer, New York, 2004)CrossRefGoogle Scholar
  43. 111.
    M.J. Mulder, E.J. Wagenmakers, R. Ratcliff, W. Boekel, B.U. Forstmann, J. Neurosci. 32, 2335 (2012)CrossRefGoogle Scholar
  44. 114.
    C.N. White, E. Congdon, J.A. Mumford, K.H. Karlsgodt, F.W. Sabb, N.B. Freimer, E.D. London, T.D. Cannon, R.M. Bilder, R.A. Poldrack, J. Cogn. Neurosci. 26, 1601 (2014)CrossRefGoogle Scholar
  45. 115.
    K. Amano, N. Goda, S. Nishida, Y. Ejima, T. Takeda, Y. Ohtani, J. Neurosci. 26, 3981 (2006)CrossRefGoogle Scholar
  46. 116.
    F.H. Lin, A.R. McIntosh, J.A. Agnew, G.F. Eden, T.A. Zeffiro, J.W. Belliveau, NeuroImage 20, 625 (2003)CrossRefGoogle Scholar
  47. 117.
    E. Martinez-Montes, P.A. Valdes-Sosa, F. Miwakeichi, R.I. Goldman, M.S. Cohen, NeuroImage 22, 1023 (2004)CrossRefGoogle Scholar
  48. 118.
    C.F. Beckmann, S.M. Smith, NeuroImage 25, 294 (2005)CrossRefGoogle Scholar
  49. 127.
    V.D. Calhoun, T. Adali, IEEE Trans. Inf. Technol. Biomed. 13, 711 (2009)CrossRefGoogle Scholar
  50. 128.
    L. Astolfi, F. Cincotti, D. Mattia, S. Salinari, C. Babiloni, A. Basilisco, P.M. Rossini, L. Ding, Y. Ni, B. He, M.G. Marciani, F. Babiloni, Magn. Reson. Imaging 22, 1457 (2004)CrossRefGoogle Scholar
  51. 129.
    K. Hamandi, H.W. Powell, H. Laufs, M.R. Symms, G.J. Barker, G.J. Parker, L. Lemieux, J.S. Duncan, J. Neurol. Neurosurg. Psychiatry 79, 594 (2008)CrossRefGoogle Scholar
  52. 130.
    F. De Martino, G. Valente, A.W. de Borst, F. Esposito, A. Roebroeck, R. Goebel, E. Formisano, Magn. Reson. Imaging 28, 1104 (2010)CrossRefGoogle Scholar
  53. 131.
    T. Eichele, K. Specht, M. Moosmann, M.L. Jongsma, R.Q. Quiroga, H.E.A. Nordby, Proc. Natl. Acad. Sci. U. S. 102, 17798 (2005)CrossRefGoogle Scholar
  54. 132.
    N.M. Correa, T. Adali, Y. Li, V.D. Calhoun, Signal Process. Mag. 27, 39 (2010)CrossRefGoogle Scholar
  55. 134.
    N.M. Correa, Y. Li, T. Adali, V.D. Calhoun, J. Sel. Top. Signal Process. 2, 998 (2008)CrossRefGoogle Scholar
  56. 135.
    J. Sui, T. Adali, Q. Yu, J. Chen, V.D. Calhoun, J. Neurosci. Methods 204, 68 (2012)CrossRefGoogle Scholar
  57. 136.
    S. Dähne, F. Bieβmann, W. Samek, S. Haufe, D. Goltz, C. Gundlach, A. Villringer, S. Fazli, K.R. Müller, Proc. IEEE 103, 1507 (2015)CrossRefGoogle Scholar
  58. 137.
    A. Krishnan, L.J. Williams, A. McIntosh, H. Abdi, NeuroImage 56, 455 (2015)CrossRefGoogle Scholar
  59. 138.
    C.A. Rodriguez, B.M. Turner, , S.M. McClure, PLoS ONE e90138, 9 (2014)Google Scholar
  60. 139.
    B.A. Purcell, J.D. Schall, G.D. Logan, T.J. Palmeri, J. Neurosci. 32(10), 3433 (2012)CrossRefGoogle Scholar
  61. 140.
    B.A. Purcell, T.J. Palmeri, Relating accumulator model parameters and neural dynamics. J. Math. Psychol. 76, 156–171 (2017)CrossRefGoogle Scholar
  62. 141.
    P.W. Glimcher, Annu. Rev. Neurosci. 26, 133 (2003)CrossRefGoogle Scholar
  63. 142.
    J.D. Roitman, M.N. Shadlen, J. Neurosci. 22, 9475 (2002)CrossRefGoogle Scholar
  64. 144.
    M. Mulder, L. Van Maanen, B. Forstmann, Neuroscience 277, 872 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Brandon M. Turner
    • 1
  • Birte U. Forstmann
    • 2
  • Mark Steyvers
    • 3
  1. 1.Department of PsychologyThe Ohio State UniversityColumbusUSA
  2. 2.Cognitive Science CenterUniversity of AmsterdamAmsterdamThe Netherlands
  3. 3.Department of Cognitive SciencesUniversity of California, IrvineIrvineUSA

Personalised recommendations