Skip to main content
SpringerLink
Log in

Stochastic Dynamic Programming Illuminates the Link Between Environment, Physiology, and Evolution

  • Special Issue Article
  • Published:
Bulletin of Mathematical Biology Aims and scope Submit manuscript

Abstract

I describe how stochastic dynamic programming (SDP), a method for stochastic optimization that evolved from the work of Hamilton and Jacobi on variational problems, allows us to connect the physiological state of organisms, the environment in which they live, and how evolution by natural selection acts on trade-offs that all organisms face. I first derive the two canonical equations of SDP. These are valuable because although they apply to no system in particular, they share commonalities with many systems (as do frictionless springs). After that, I show how we used SDP in insect behavioral ecology. I describe the puzzles that needed to be solved, the SDP equations we used to solve the puzzles, and the experiments that we used to test the predictions of the models. I then briefly describe two other applications of SDP in biology: first, understanding the developmental pathways followed by steelhead trout in California and second skipped spawning by Norwegian cod. In both cases, modeling and empirical work were closely connected. I close with lessons learned and advice for the young mathematical biologists.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38:716–719

    Article  MATH  Google Scholar 

  • Bellman R (1954) The theory of dynamic programming. Bull Am Math Soc 60:503–515

    Article  MATH  Google Scholar 

  • Bellman R (1956) Dynamic programming and Lagrange multipliers. Proc Am Acad Sci USA 42:767–769

    Article  MATH  Google Scholar 

  • Bellman R (1957) Dynamic programming. Princeton University Press, Princeton

  • Cayuela H, Besnard A, Bonnaire E, Perrt H, Rivoalen J, Miaud C, Joly P (2014) To breed or not to breed: past reproductive status and environmental cues drive current breeding decisions in a long-lived amphibian. Oecologia (in press)

  • Charnov EL, Skinner SW (1984) Evolution of host selection and clutch size in parasitoid wasps. Fla Entomol 67:5–21

    Article  Google Scholar 

  • Charnov EL, Skinner SW (1985) Complementary approaches to understanding parasitoid oviposition decisions. Environ Entomol 14:383–391

    Article  Google Scholar 

  • Clark CW, Mangel M (2000) Dynamic state variable models in ecology. Methods and applications. Oxford University Press, New York

    Google Scholar 

  • Courant R, Hilbert D (1962) Methods of mathematical physics. Wiley, New York

    MATH  Google Scholar 

  • Dayton PK, Sala E (2001) Natural history: the sense of wonder, creativity and progress in ecology. Sci Marine 65(Suppl 2):199–206

    Google Scholar 

  • Engelhard GH, Heino M (2005) Scale analysis suggests frequent skipping of the second reproductive season in Atlantic herring. Biol Lett 1:172–175

    Article  Google Scholar 

  • Feynman RP, Leighton RB, Sands M (1963) The Feynman lectures on physics. Mainly mechanics, radiation, and heat. Addison-Wesley Publishing Company, Reading

    Google Scholar 

  • Giske J, Eliassen S, Fiksen O, Jakobsen PJ, Aksnes DL, Jørgensen C, Mangel M (2013) Effects of the emotion system on adaptive behavior. Am Nat 182:689–703

    Article  Google Scholar 

  • Hilborn R, Mangel M (1997) The ecological detective. Confronting models with data. Princeton University Press, Princeton

    Google Scholar 

  • Houston AI, McNamara JM (1999) Models for adaptive behavior. An approach based on state. Cambridge University Press, Cambridge

    Google Scholar 

  • Houston A, Clark CW, McNamara JM, Mangel M (1988) Dynamic models in behavioural and evolutionary ecology. Nature 332:29–34

    Article  Google Scholar 

  • Jørgensen C, Fiksen Ø (2006) State-dependent energy allocation in cod (Gadus morhua). Can J Fish Aquat Sci 63:186–199

    Article  Google Scholar 

  • Jørgensen C, Ernande B, Fiksen Ø, Dieckmann U (2006) The logic of skipped spawning in fish. Can J Fish Aquat Sci 63:200–211

    Article  Google Scholar 

  • Karlin S, Taylor HM (1981) A second course in stochastic processes. Academic Press, New York

    MATH  Google Scholar 

  • Kennedy J, Skjæraasen JE, Nash RDM, Slotte A, Geffen AJ, Kjesbu OS (2011) Evaluation of the frequency of skipped spawning in Norwegian spring-spawning herring. J Sea Res 65:327–332

    Article  Google Scholar 

  • Krebs JR, Davies NB (1978) Behavioural ecology. An evolutionary approach. Blackwell Scientific Publications, Oxford

    Google Scholar 

  • Krener AJ (1979) A formal approach to stochastic integration and differential equations. Stochastics 3:105–125

    Article  MATH  MathSciNet  Google Scholar 

  • Lima SL, Dill LM (1990) Behavioral decisions made under the risk of predation: a review and prospectus. Can J Zool 68:619–640

    Article  Google Scholar 

  • Mangel M (1971) A treatment of complex ions in sea water. Marine Geol 11:M24–26

    Article  Google Scholar 

  • Mangel M (1987) Oviposition site selection and clutch size in insects. J Math Biol 25:1–22

    Article  MATH  MathSciNet  Google Scholar 

  • Mangel M (1992) Rate maximizing and state variable theories of diet selection. Bull Math Biol 54:413–422

    Article  MATH  Google Scholar 

  • Mangel M (2006) The theoretical biologist’s toolbox. Quantitative methods for ecology and evolutionary biology. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Mangel M, Clark CW (1986) Towards a unified foraging theory. Ecology 67:1127–1138

    Article  Google Scholar 

  • Mangel M, Clark CW (1988) Dynamic modeling in behavioral ecology. Princeton University Press, Princeton

    Google Scholar 

  • Mangel M, Ludwig D (1992) Definition and evaluation of behavioral and developmental programs. Ann Rev Ecol Syst 23:507–536

    Article  Google Scholar 

  • Mangel M, Rosenheim JA, Adler FR (1994) Clutch size, offspring performance, and intergenerational fitness. Behav Ecol 5:412–417

    Article  Google Scholar 

  • McNamara JM (2000) A classification of dynamic optimization problems in fluctuating environments. Evol Ecol Res 2:457–471

    Google Scholar 

  • McNamara JM, Houston AI (1986) The common currency for behavioral decisions. Am Nat 127:358–378

    Article  Google Scholar 

  • Nonacs P, Dill LM (1990) Mortality risk vs. food quality trade-offs in a common currency: ant patch preferences. Ecology 71:1886–1892

    Article  Google Scholar 

  • Railsback SF, Grimm V (2012) Agent-based and individual-based modeling. A practical introduction. Princeton University Press, Princeton

    Google Scholar 

  • Rashevsky N (1969) Mathematical biophysics. Physico-mathematics foundations of biology, vol 1 and 2. Dover Press, New York

    Google Scholar 

  • Rideout RM, Rose GA, Burton MPM (2005) Skipped spawning in female iteroparous fishes. Fish Fish 6:50–72

    Article  Google Scholar 

  • Roitberg BD (2008) Gold Medal Address. Bull Entomol Soc Can 40:162–166

    Google Scholar 

  • Roitberg BD, Mangel M, Lalonde R, Roitberg CA, van Alphen JJM, Vet L (1992) Seasonal dynamic shifts in patch exploitation by parasitic wasps. Behav Ecol 3:156–165

    Article  Google Scholar 

  • Roitberg BD, Sircom J, Roitberg CA, van Alphen JJM, Mangel M (1993) Life expectancy and reproduction. Nature 364:351

    Article  Google Scholar 

  • Rosenheim JA, Rosen D (1991) Foraging and oviposition decisions in the parasitoid Aphytis lingnanensis: distinguishing the influences of egg load and experience. J Anim Ecol 60:873–893

  • Rosenheim JA, Rosen D (1992) Influence of egg load and host size on host-feeding behaviour of the parasitoid Aphytis lingnanensis. Ecol Entomol 17:263–272

    Article  Google Scholar 

  • Satterthwaite WH, Beakes MP, Collins E, Swank DR, Merz JE, Titus RG, Sogard SM, Mangel M (2009) Steelhead life history on California’s Central Coast: insights from a state dependent model. Trans Am Fish Soc 132:532–548

    Article  Google Scholar 

  • Satterthwaite WH, Beakes MP, Collins E, Sawnk DR, Merz JE, Titus RG, Sogard SM, Mangel M (2010) State-dependent life history models in a changing (and regulated) environment: steelhead in the California Central Valley. Evol Appl 3:221–243

    Article  Google Scholar 

  • Skjæraasen JE, Nash RDM, Korsbrekke K, Fonn M, Nilsen T, Kennedy J, Nedreaas KH, Thorsen A, Witthames PR, Geffen AJ, Høiea H, Sigurd Kjesbu O (2012) Frequent skipped spawning in the worlds largest cod population. Proc Natl Acad Sci USA 109:8995–8999

    Article  Google Scholar 

  • Sogard SM, Merz JE, Satterthwaite WH, Beakes MP, Swank DR, Collins EM, Titus RG, Mangel M (2012) Contrasts in habitat characteristics and life history patterns of Onchorhynkus mykiss in California’s Central Coast and Central Valley. Trans Am Fish Soc 141:747–760

    Article  Google Scholar 

  • Stokes DE (1997) Pasteur’s quadrant: basic science and technological innovation. Brookings Institution, Washington

    Google Scholar 

  • Thorpe JE, Mangel M, Metcalfe NB, Huntingford FA (1998) Modelling the proximate basis of life history variation, with application to Atlantic salmon, Salmo salar L. Evol Ecol 121:581–600

    Article  Google Scholar 

  • Yaragina NA (2010) Biological parameters of immature, ripening, and non-reproductive mature northeast Arctic cod in 1984–2006. ICES J Marine Sci 67:2033–2041

    Article  Google Scholar 

Download references

Acknowledgments

My work on insect behavioral ecology was financially supported over the years by the NSF and the USDA and intellectually supported by the wonderful collaborations with Bernie Roitberg and Jay Rosenheim. The work on steelhead (which was supported by the Cal-Fed Science Program) was preceded by a decade of collaboration with Felicity Hungtingford, Neil Metcalfe, and John Thorpe on Atlantic salmon and supported by a NATO travel grant, California Sea Grant, and the US National Marine Fisheries Service. Preparation of this paper was partially supported by NSF Grant EF-0924195. I thank Suzanne Alonzo, Charles Fisher, Holly Kindsvater, Susan Mangel, Bernard Roitberg, Jay Rosenheim, and Katriona Shea for comments on a previous version of the manuscript, Simone Vincenzi for a careful and thoughtful review of the penultimate version, and two anonymous referees for their comments on the submission. To a very real extent, my career was shaped by Morton Brussels and Donald Ginsberg, who taught that course in electricity and magnetism in academic year 1969–1970 and by my first scientific mentor, William W. Hay, who guided me through my first paper in which models were used to illuminate a scientific question (Mangel 1971).

Author information

Authors and Affiliations

  1. Center for Stock Assessment Research and Department of Applied Mathematics and Statistics, University of California, Santa Cruz, CA, USA

    Marc Mangel

  2. Department of Biology, University of Bergen, Bergen, Norway

    Marc Mangel

Authors
  1. Marc Mangel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marc Mangel.

Additional information

Dedicated to my mentor Donald Ludwig on the occasion of his 80th birthday.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mangel, M. Stochastic Dynamic Programming Illuminates the Link Between Environment, Physiology, and Evolution. Bull Math Biol 77, 857–877 (2015). https://doi.org/10.1007/s11538-014-9973-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11538-014-9973-3

Keywords

Search

Navigation