Skip to main content

Decision theory, motor planning, and visual memory: deciding where to reach when memory errors are costly


Limitations in visual working memory (VWM) have been extensively studied in psychophysical tasks, but not well understood in terms of how these memory limits translate to performance in more natural domains. For example, in reaching to grasp an object based on a spatial memory representation, overshooting the intended target may be more costly than undershooting, such as when reaching for a cup of hot coffee. The current body of literature lacks a detailed account of how the costs or consequences of memory error influence what we encode in visual memory and how we act on the basis of remembered information. Here, we study how externally imposed monetary costs influence behavior in a motor decision task that involves reach planning based on recalled information from VWM. We approach this from a decision theoretic perspective, viewing decisions of where to aim in relation to the utility of their outcomes given the uncertainty of memory representations. Our results indicate that subjects accounted for the uncertainty in their visual memory, showing a significant difference in their reach planning when monetary costs were imposed for memory errors. However, our findings indicate that subjects memory representations per se were not biased by the imposed costs, but rather subjects adopted a near-optimal post-mnemonic decision strategy in their motor planning.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  • Bays PM (2014) Noise in neural populations accounts for errors in working memory. J Neuro-Sci 34(10):3632–3645

    CAS  Google Scholar 

  • Brouwer A-M, Knill DC (2007) The role of memory in visually guided reaching. J Vis 7(5):6

    Article  PubMed  Google Scholar 

  • Brouwer A-M, Knill DC (2009) Humans use visual and remembered information about object location to plan pointing movements. J Vis 9(1):24

    Article  PubMed  PubMed Central  Google Scholar 

  • Franconeri SL, Alvarez GA, Cavanagh P (2013) Flexible cognitive resources: competitive content maps for attention and memory. Trends Cogn Sci 17(3):134–141

    Article  PubMed  Google Scholar 

  • Goldstone RL, Hendrickson AT (2010) Categorical perception. Wiley Interdiscip Rev Cogn Sci 1(1):69–78

    Article  PubMed  Google Scholar 

  • Hayhoe MM, Shrivastava A, Mruczek R, Pelz JB (2003) Visual memory and motor planning in a natural task. J Vis 3(1):6

    Article  Google Scholar 

  • Hayhoe MM, Rothkopf CA (2011) Vision in the natural world. Wiley Interdiscip Rev Cogn Sci 2(2):158–166

    Article  PubMed  Google Scholar 

  • Hollingworth A, Richard AM, Luck SJ (2008) Understanding the function of visual short-term memory: transsaccadic memory, object correspondence, and gaze correction. J Exp Psychol Gen 137(1):163

    Article  PubMed  PubMed Central  Google Scholar 

  • Hudson TE, Wolfe U, Maloney LT (2012) Speeded reaching movements around invisible obstacles. P-LOS Computational Biology

  • Körding K (2007) Decision theory: what “should” the nervous system do? Science 318(5850):606–610

    Article  PubMed  Google Scholar 

  • Luck SJ (2008) Visual short-term memory. In: Luck SJ, Hollingworth A (eds) Visual memory. Oxford University Press, Oxford, pp 43–85

    Chapter  Google Scholar 

  • Luck SJ, Vogel EK (2013) Visual working memory capacity: From psychophysics and neurobiology to individual differences. Trends Cogn Sci 17(8):391–400

    Article  PubMed  PubMed Central  Google Scholar 

  • Lu Z-L, Dosher B (2014) Visual psychophysics: from laboratory to theory. MIT Press, Massachusetts

    Google Scholar 

  • Ma WJ, Husain M, Bays PM (2014) Changing concepts of working memory. Nat Neurosci 17(3):347–356

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  • Maloney LT, Zhang H (2010) Decision-theoretic models of visual perception and action. Vis Res 50(23):2362–2374

    Article  PubMed  Google Scholar 

  • Orhan AE, Sims CR, Jacobs RA, Knill DC (2014) The adaptive nature of visual working memory. Curr Dir Psychol Sci 23(3):164–170

    Article  Google Scholar 

  • Richards W, Rubin JM (2015) Color vision: Representing material categories. Encyclopedia of Color Science and Technology

  • Roberson D, Davies I, Davidoff J (2000) Color categories are not universal: replications and new evidence from a stone-age culture. J Exp Psychol Gen 129(3):369

    CAS  Article  PubMed  Google Scholar 

  • Sims CR, Jacobs RA, Knill DC (2012) An ideal observer analysis of visual working memory. Psychol Rev 119(4):807

    Article  PubMed  PubMed Central  Google Scholar 

  • Sims CR (2015) The cost of misremembering: Inferring the loss function in visual working memory. J Vis 15(3):2

    Article  PubMed  Google Scholar 

  • Trommershäuser J, Maloney LT, Landy MS (2008) Decision making, movement planning and statistical decision theory. Trends Cogn Sci 12(8):291–297

    Article  PubMed  Google Scholar 

  • Wolpert DM, Landy MS (2012) Motor control is decision-making. Curr Opin Neurobiol 22(6):996–1003

    CAS  Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Rachel A. Lerch.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lerch, R.A., Sims, C.R. Decision theory, motor planning, and visual memory: deciding where to reach when memory errors are costly. Exp Brain Res 234, 1589–1597 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Visuospatial memory
  • Visual memory
  • Decision making
  • Motor planning