Bulletin of Mathematical Biology

, Volume 77, Issue 5, pp 857–877 | Cite as

Stochastic Dynamic Programming Illuminates the Link Between Environment, Physiology, and Evolution

Special Issue Article


I describe how stochastic dynamic programming (SDP), a method for stochastic optimization that evolved from the work of Hamilton and Jacobi on variational problems, allows us to connect the physiological state of organisms, the environment in which they live, and how evolution by natural selection acts on trade-offs that all organisms face. I first derive the two canonical equations of SDP. These are valuable because although they apply to no system in particular, they share commonalities with many systems (as do frictionless springs). After that, I show how we used SDP in insect behavioral ecology. I describe the puzzles that needed to be solved, the SDP equations we used to solve the puzzles, and the experiments that we used to test the predictions of the models. I then briefly describe two other applications of SDP in biology: first, understanding the developmental pathways followed by steelhead trout in California and second skipped spawning by Norwegian cod. In both cases, modeling and empirical work were closely connected. I close with lessons learned and advice for the young mathematical biologists.


Stochastic dynamic programming Parasitoids 


  1. Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38:716–719MATHCrossRefGoogle Scholar
  2. Bellman R (1954) The theory of dynamic programming. Bull Am Math Soc 60:503–515MATHCrossRefGoogle Scholar
  3. Bellman R (1956) Dynamic programming and Lagrange multipliers. Proc Am Acad Sci USA 42:767–769MATHCrossRefGoogle Scholar
  4. Bellman R (1957) Dynamic programming. Princeton University Press, PrincetonGoogle Scholar
  5. Cayuela H, Besnard A, Bonnaire E, Perrt H, Rivoalen J, Miaud C, Joly P (2014) To breed or not to breed: past reproductive status and environmental cues drive current breeding decisions in a long-lived amphibian. Oecologia (in press)Google Scholar
  6. Charnov EL, Skinner SW (1984) Evolution of host selection and clutch size in parasitoid wasps. Fla Entomol 67:5–21CrossRefGoogle Scholar
  7. Charnov EL, Skinner SW (1985) Complementary approaches to understanding parasitoid oviposition decisions. Environ Entomol 14:383–391CrossRefGoogle Scholar
  8. Clark CW, Mangel M (2000) Dynamic state variable models in ecology. Methods and applications. Oxford University Press, New YorkGoogle Scholar
  9. Courant R, Hilbert D (1962) Methods of mathematical physics. Wiley, New YorkMATHGoogle Scholar
  10. Dayton PK, Sala E (2001) Natural history: the sense of wonder, creativity and progress in ecology. Sci Marine 65(Suppl 2):199–206Google Scholar
  11. Engelhard GH, Heino M (2005) Scale analysis suggests frequent skipping of the second reproductive season in Atlantic herring. Biol Lett 1:172–175CrossRefGoogle Scholar
  12. Feynman RP, Leighton RB, Sands M (1963) The Feynman lectures on physics. Mainly mechanics, radiation, and heat. Addison-Wesley Publishing Company, ReadingGoogle Scholar
  13. Giske J, Eliassen S, Fiksen O, Jakobsen PJ, Aksnes DL, Jørgensen C, Mangel M (2013) Effects of the emotion system on adaptive behavior. Am Nat 182:689–703CrossRefGoogle Scholar
  14. Hilborn R, Mangel M (1997) The ecological detective. Confronting models with data. Princeton University Press, PrincetonGoogle Scholar
  15. Houston AI, McNamara JM (1999) Models for adaptive behavior. An approach based on state. Cambridge University Press, CambridgeGoogle Scholar
  16. Houston A, Clark CW, McNamara JM, Mangel M (1988) Dynamic models in behavioural and evolutionary ecology. Nature 332:29–34CrossRefGoogle Scholar
  17. Jørgensen C, Fiksen Ø (2006) State-dependent energy allocation in cod (Gadus morhua). Can J Fish Aquat Sci 63:186–199CrossRefGoogle Scholar
  18. Jørgensen C, Ernande B, Fiksen Ø, Dieckmann U (2006) The logic of skipped spawning in fish. Can J Fish Aquat Sci 63:200–211CrossRefGoogle Scholar
  19. Karlin S, Taylor HM (1981) A second course in stochastic processes. Academic Press, New YorkMATHGoogle Scholar
  20. Kennedy J, Skjæraasen JE, Nash RDM, Slotte A, Geffen AJ, Kjesbu OS (2011) Evaluation of the frequency of skipped spawning in Norwegian spring-spawning herring. J Sea Res 65:327–332CrossRefGoogle Scholar
  21. Krebs JR, Davies NB (1978) Behavioural ecology. An evolutionary approach. Blackwell Scientific Publications, OxfordGoogle Scholar
  22. Krener AJ (1979) A formal approach to stochastic integration and differential equations. Stochastics 3:105–125MATHMathSciNetCrossRefGoogle Scholar
  23. Lima SL, Dill LM (1990) Behavioral decisions made under the risk of predation: a review and prospectus. Can J Zool 68:619–640CrossRefGoogle Scholar
  24. Mangel M (1971) A treatment of complex ions in sea water. Marine Geol 11:M24–26CrossRefGoogle Scholar
  25. Mangel M (1987) Oviposition site selection and clutch size in insects. J Math Biol 25:1–22MATHMathSciNetCrossRefGoogle Scholar
  26. Mangel M (1992) Rate maximizing and state variable theories of diet selection. Bull Math Biol 54:413–422MATHCrossRefGoogle Scholar
  27. Mangel M (2006) The theoretical biologist’s toolbox. Quantitative methods for ecology and evolutionary biology. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  28. Mangel M, Clark CW (1986) Towards a unified foraging theory. Ecology 67:1127–1138CrossRefGoogle Scholar
  29. Mangel M, Clark CW (1988) Dynamic modeling in behavioral ecology. Princeton University Press, PrincetonGoogle Scholar
  30. Mangel M, Ludwig D (1992) Definition and evaluation of behavioral and developmental programs. Ann Rev Ecol Syst 23:507–536CrossRefGoogle Scholar
  31. Mangel M, Rosenheim JA, Adler FR (1994) Clutch size, offspring performance, and intergenerational fitness. Behav Ecol 5:412–417CrossRefGoogle Scholar
  32. McNamara JM (2000) A classification of dynamic optimization problems in fluctuating environments. Evol Ecol Res 2:457–471Google Scholar
  33. McNamara JM, Houston AI (1986) The common currency for behavioral decisions. Am Nat 127:358–378CrossRefGoogle Scholar
  34. Nonacs P, Dill LM (1990) Mortality risk vs. food quality trade-offs in a common currency: ant patch preferences. Ecology 71:1886–1892CrossRefGoogle Scholar
  35. Railsback SF, Grimm V (2012) Agent-based and individual-based modeling. A practical introduction. Princeton University Press, PrincetonGoogle Scholar
  36. Rashevsky N (1969) Mathematical biophysics. Physico-mathematics foundations of biology, vol 1 and 2. Dover Press, New YorkGoogle Scholar
  37. Rideout RM, Rose GA, Burton MPM (2005) Skipped spawning in female iteroparous fishes. Fish Fish 6:50–72CrossRefGoogle Scholar
  38. Roitberg BD (2008) Gold Medal Address. Bull Entomol Soc Can 40:162–166Google Scholar
  39. Roitberg BD, Mangel M, Lalonde R, Roitberg CA, van Alphen JJM, Vet L (1992) Seasonal dynamic shifts in patch exploitation by parasitic wasps. Behav Ecol 3:156–165CrossRefGoogle Scholar
  40. Roitberg BD, Sircom J, Roitberg CA, van Alphen JJM, Mangel M (1993) Life expectancy and reproduction. Nature 364:351CrossRefGoogle Scholar
  41. Rosenheim JA, Rosen D (1991) Foraging and oviposition decisions in the parasitoid Aphytis lingnanensis: distinguishing the influences of egg load and experience. J Anim Ecol 60:873–893Google Scholar
  42. Rosenheim JA, Rosen D (1992) Influence of egg load and host size on host-feeding behaviour of the parasitoid Aphytis lingnanensis. Ecol Entomol 17:263–272CrossRefGoogle Scholar
  43. Satterthwaite WH, Beakes MP, Collins E, Swank DR, Merz JE, Titus RG, Sogard SM, Mangel M (2009) Steelhead life history on California’s Central Coast: insights from a state dependent model. Trans Am Fish Soc 132:532–548CrossRefGoogle Scholar
  44. Satterthwaite WH, Beakes MP, Collins E, Sawnk DR, Merz JE, Titus RG, Sogard SM, Mangel M (2010) State-dependent life history models in a changing (and regulated) environment: steelhead in the California Central Valley. Evol Appl 3:221–243CrossRefGoogle Scholar
  45. Skjæraasen JE, Nash RDM, Korsbrekke K, Fonn M, Nilsen T, Kennedy J, Nedreaas KH, Thorsen A, Witthames PR, Geffen AJ, Høiea H, Sigurd Kjesbu O (2012) Frequent skipped spawning in the worlds largest cod population. Proc Natl Acad Sci USA 109:8995–8999CrossRefGoogle Scholar
  46. Sogard SM, Merz JE, Satterthwaite WH, Beakes MP, Swank DR, Collins EM, Titus RG, Mangel M (2012) Contrasts in habitat characteristics and life history patterns of Onchorhynkus mykiss in California’s Central Coast and Central Valley. Trans Am Fish Soc 141:747–760CrossRefGoogle Scholar
  47. Stokes DE (1997) Pasteur’s quadrant: basic science and technological innovation. Brookings Institution, WashingtonGoogle Scholar
  48. Thorpe JE, Mangel M, Metcalfe NB, Huntingford FA (1998) Modelling the proximate basis of life history variation, with application to Atlantic salmon, Salmo salar L. Evol Ecol 121:581–600CrossRefGoogle Scholar
  49. Yaragina NA (2010) Biological parameters of immature, ripening, and non-reproductive mature northeast Arctic cod in 1984–2006. ICES J Marine Sci 67:2033–2041CrossRefGoogle Scholar

Copyright information

© Society for Mathematical Biology 2014

Authors and Affiliations

  1. 1.Center for Stock Assessment Research and Department of Applied Mathematics and StatisticsUniversity of CaliforniaSanta CruzUSA
  2. 2.Department of BiologyUniversity of BergenBergenNorway

Personalised recommendations