Biomimetics of Choice Behaviour for Autonomous Agents

  • Christopher M. Harris
  • Jonathan Waddington
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7429)

Abstract

Autonomy is inseparable from choice. Natural agents, including humans, make choices with probabilities that are proportional to the average reward from each choice (the ‘matching law’), rather than always choosing the alternative with maximum reward. Imbuing artificial agents with this flexibility is non-trivial. We introduce anti-competition mapping to overcome the tendency for high reward choices to dominate (mask) those with lower reward. The simplest solution is to map reward to new choice signals that generate proportional hazard functions. We demonstrate a procedure based on the reciprocal Normal distribution, a popular model of human reaction times. We then discuss the value of the matching law in synthetic agents.

Keywords

Matching law decision making reward choices rate of response hazard rate exploitation exploration 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Herrnstein, R.J.: Relative and absolute strength of responses as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267–272 (1961)CrossRefGoogle Scholar
  2. 2.
    Graft, D.A., Lea, S.E.G., Whitworth, T.L.: The matching law in and within groups of rats. J. Exp. Anal. Behav. 27, 183–194 (1977)CrossRefGoogle Scholar
  3. 3.
    Bradshaw, C.M., Szabadi, E., Bevan, P.: Behavior of humans in variable-interval schedules of reinforcement. J. Exp. Anal. Behav. 26, 135–141 (1976)CrossRefGoogle Scholar
  4. 4.
    Herrnstein, R.J.: On the law of effect. J. Exp. Anal. Behav. 13, 243–266 (1970)CrossRefGoogle Scholar
  5. 5.
    Baum, W.M.: On two types of deviation from the matching law: bias and undermatching. J. Exp. Anal. Behav. 22, 231–242 (1974)CrossRefGoogle Scholar
  6. 6.
    Sugrue, L.P.: Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004)CrossRefGoogle Scholar
  7. 7.
    Soltani, A., Wang, X.-J.: A biophysically based neural model of matching law behavior: melioration by stochastic synapses. J. Neurosci. 26, 3731–3744 (2006)CrossRefGoogle Scholar
  8. 8.
    Loewenstein, Y.: Robustness of learning that is based on covariance-driven synaptic plasticity. Plos. Comput. Biol. 4, e1000007 (2008)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Fretwell, S.: Populations in seasonal environments. Princeton University Press, Princeton (1972)Google Scholar
  10. 10.
    Carpenter, R.H., Williams, M.L.: Neural computation of log likelihood in control of saccadic eye movements. Nature 377, 59–62 (1995)CrossRefGoogle Scholar
  11. 11.
    Harris, C.M., Waddington, J.: On the convergence of time interval moments: caveat sciscitator. J. Neurosci. Meth. 205, 345–356 (2012)CrossRefGoogle Scholar
  12. 12.
    Pieron, H.: The sensations: their functions, processes and mechanisms. Frederick Muller Ltd., London (1952)Google Scholar
  13. 13.
    Cox, D.R., Oakes, D.: Analysis of survival data. Chapman and Hall, London (1984)Google Scholar
  14. 14.
    Smith, P.L., Ratcliff, R.: Psychology and neurobiology of simple decisions. Trends. Neurosci. 27, 161–168 (2004)CrossRefGoogle Scholar
  15. 15.
    Gold, J.L., Shadlen, M.N.: The neural basis of decision making. Ann. Rev. Neurosci. 30, 535–574 (2007)CrossRefGoogle Scholar
  16. 16.
    Hanes, D.P., Schall, J.D.: Neural control of voluntary movement initiation. Science 274, 427–430 (1996)CrossRefGoogle Scholar
  17. 17.
    Everling, S., Munoz, D.P.: Neuronal correlates for preparatory set associated with pro-saccades and anti-saccades in the primate frontal eye field. J. Neurosci. 20, 387–400 (2000)Google Scholar
  18. 18.
    Harris, C.M.: Biomimetics of human movement: functional or aesthetic? Bioinspir. Biomim. 4, e0033001 (2009)CrossRefGoogle Scholar
  19. 19.
    Seth, A.K.: The ecology of action selection: insights from artificial life. Philos. T. Roy. Soc. B. 362, 1545–1558 (2007)CrossRefGoogle Scholar
  20. 20.
    Houston, A.I., McNamara, J.M., Steer, M.D.: Do we expect natural selection to produce rational behaviour? Phil. Trans. R. Soc. B 362, 1531–1543 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Christopher M. Harris
    • 1
  • Jonathan Waddington
    • 1
  1. 1.Centre for Robotics and Neural SystemsPlymouth UniversityPlymouthUK

Personalised recommendations