Skip to main content
SpringerLink
Log in

Stochastic Games with Endogenous Transitions

  • Chapter
  • First Online:
Mathematical Programming and Game Theory

Part of the book series: Indian Statistical Institute Series ((INSIS))

Abstract

We present and analyze a stochastic game in which transition probabilities between states are not fixed as in standard stochastic games, but depend on the history of the play, i.e., the players’ past action choices. For the limiting average reward criterion we determine the set of jointly convergent pure-strategy rewards which can be supported by equilibria involving threats. For expository purposes we analyze a stylized fishery game. Each period, two agents choose between catching with restraint or without. The resource is in either of two states, High or Low. Restraint is harmless to the fish, but it is a dominated action at each stage. The less restraint shown during the play, the higher the probabilities that the system moves to or stays in Low. The latter state may even become “absorbing temporarily,”’ i.e., transition probabilities to High temporarily become zero while transition probabilities to Low remain nonzero.

I thank J. Flesch, F. Thuijsman, E. Solan and A. Laruelle for advice. Audiences in Enschede, Tilburg, Maastricht, Tel Aviv, Bilbao and Istanbul are also thanked for feedback. Last but by no means least, I thank the referee for extremely careful reading and for excellent suggestions for improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A word play on Levhari and Mirman [50] who show that strategic interaction in a fishery may induce a “tragedy of the commons” [27].

  2. 2.

    ‘Engineered’ as in Aumann [4]. Stochastic games were introduced by Shapley [69], see also Amir [1] for links to difference and differential games to which much work on fisheries belongs, cf., e.g., Haurie et al. [29], Long [51] for overviews.

  3. 3.

    So, the Markov property of standard stochastic games [69] is lost.

  4. 4.

    Right now, we do not need the actual payoffs and focus on the transition probabilities.

  5. 5.

    Figures 12.2 and 12.3 are based on Matlab graphs generated by an algorithm yielding 6 million pairs of rewards which took several days. Memory restrictions corrupt image quality as we experienced. The algorithm and output are available on request.

  6. 6.

    They are available on request, of course.

  7. 7.

    Hordijk et al. [34] show that a stationary strategy suffices as a best reply against a fixed stationary strategy, and we may write the next sequence as a deterministic one.

  8. 8.

    In earlier versions of our paper we were too quick to conclude that the associated optimization problems yield jointly convergent strategies. A referee pointed out a flaw in our reasoning, which by the way, makes to problem of finding an optimal strategy against a fixed strategy even much harder to solve. If jointly convergent strategies do not yield a solution, play never settles down measured in the space of the relative frequency vectors and the sequence of relative frequency vectors induced is essentially stochastic.

  9. 9.

    At several presentations the question was raised whether our games should not be presented as stochastic games with infinitely many states. We agree that our games fall into this class, as they can be rewritten as such. We prefer our presentation because of its simplicity and the circumstance that we were able to generate a number of results. Moreover, we are very sceptic about which known results from the analysis of stochastic games with infinitely many states would be helpful to obtain results for ours.

  10. 10.

    We like our rather complex model to resemble repeated games for psychological reasons and for reasons of ease of communication for instance with less mathematically inclined people (politicians, civil servants). Many people have learned about the repeated prisoners’ dilemma in educational programs, so offering our model in a simple fashion may offer windows of opportunity for communication with the general public. To present our model as a stochastic game with infinitely many states might scare researchers but more likely less mathematically inclined people away.

  11. 11.

    Our agent is not the individual fisherman, but rather countries, regions, villages or cooperatives. Whether or not the latter care for the future sufficiently to induce sustainability (see e.g., Ostrom [58], Ostrom et al. [59] for optimistic views), individual fisherman’s preferences seem too myopic (cf., e.g., Hillis and Wheelan [32]). Next to impatience of the agents, their number, communication, punishment possibilities and the observability of actions taken influence the likelihood that the tragedy of the commons can be averted (cf., e.g., Komorita and Parks [47], Ostrom [58, 59], Steg [70]).

References

  1. Amir, R.: Stochastic games in economics and related fields: an overview. In: Neyman, A., Sorin, S. (eds.) Stochastic Games and Applications. NATO Advanced Study Institute, Series D, pp. 455–470. Kluwer, Dordrecht (2003)

    Chapter  Google Scholar 

  2. Anderson, T., Carstensen, J., Hernández-Garcia, E., Duarte, C.M.: Ecological thresholds and regime shifts: approaches to identification. Trends Ecol. Evol. 24, 49–57 (2008)

    Article  Google Scholar 

  3. Armstrong, M.J., Connolly, P., Nash, R.D.M., Pawson, M.G., Alesworth, E., Coulahan, P.J., Dickey-Collas, M., Milligan, S.P., O’Neill, M., Witthames, P.R., Woolner, L.: An application of the annual egg production method to estimate spawning biomass of cod (Gadus morhua L.), plaice (Pleuronectes platessa L.) and sole (Solea solea L.) in the Irish Sea. ICES J. Mar. Sci. 58, 183–203 (2001)

    Article  Google Scholar 

  4. Aumann, R.: Game engineering. In: Neogy, S.K., Bapat, R.B., Das, A.K., Parthasarathy, T. (eds.) Mathematical Programming and Game Theory for Decision Making, pp. 279–286. World Scientific, Singapore (2008)

    Chapter  Google Scholar 

  5. BenDor, T., Scheffran, J., Hannon, B.: Ecological and economic sustainability in fishery management: a multi-agent model for understanding competition and cooperation. Ecol. Econ. 68, 1061–1073 (2009)

    Article  Google Scholar 

  6. Bewley, T., Kohlberg, E.: The asymptotic theory of stochastic games. Math Oper Res. 1, 197–208 (1976)

    Article  MathSciNet  Google Scholar 

  7. Bewley, T., Kohlberg, E.: The asymptotic solution of a recursive equation occuring in stochastic games. Math. Oper. Res. 1, 321–336 (1976)

    Article  MathSciNet  Google Scholar 

  8. Billingsley, P.: Probability and Measure. Wiley, New York (1986)

    MATH  Google Scholar 

  9. Blanchard, O., Summers, L.: Hysteresis and the European unemployment problem. In: Fisher, S. (ed.) NBER Macroecon. Annu., pp. 15–78. MIT Press, Cambridge (1986)

    Google Scholar 

  10. Brooks, S.E., Reynolds, J.D., Allison, A.E.: Sustained by snakes? seasonal livelihood strategies and resource conservation by Tonle Sap fishers in Cambodia. Hum. Ecol. 36, 835–851 (2008)

    Article  Google Scholar 

  11. Bulte, E.H.: Open access harvesting of wildlife:the poaching pit and conservation of endangered species. Agric. Econ. 28, 27–37 (2003)

    Article  Google Scholar 

  12. Carpenter, S.R., Ludwig, D., Brock, W.A.: Management of eutrophication for lakes subject to potentially irreversible change. Ecol. Appl. 9, 751–771 (1999)

    Article  Google Scholar 

  13. Courchamp, F., Angulo, E., Rivalan, P., Hall, R.J., Signoret, L., Meinard, Y.: Rarity value and species extinction: the anthropogenic Allee effect. PLoS Biol. 4, 2405–2410 (2006)

    Article  Google Scholar 

  14. Cross, J.G., Guyer, M.J.: Social Traps. University of Michigan Press, Ann Arbor (1980)

    Google Scholar 

  15. Ehtamo, H., Hämäläinen, R.P.: On affine incentives for dynamic decision problems. In: Başar, T. (ed.) Dynamic Games and Applications in Economics, pp. 47–63. Springer, Berlin (1986)

    Chapter  Google Scholar 

  16. Ehtamo, H., Hämäläinen, R.P.: Incentive strategies and equilibria for dynamic games with delayed information. JOTA 63, 355–369 (1989)

    Article  MathSciNet  Google Scholar 

  17. Ehtamo, H., Hämäläinen, R.P.: A cooperative incentive equilibrium for a resource management problem. J. Econ. Dyn. Control. 17, 659–678 (1993)

    Article  MathSciNet  Google Scholar 

  18. Ehtamo, H., Hämäläinen, R.P.: Credibility of linear equilibrium strategies in a discrete-time fishery management game. Group Decis. Negot. 4, 27–37 (1995)

    Article  Google Scholar 

  19. Filar, J., Raghavan, T.E.S.: A matrix game solution to a single-controller stochastic game. Math. Oper. Res. 9, 356–362 (1984)

    Article  MathSciNet  Google Scholar 

  20. Filar, J., Vrieze, O.J.: Competitive Markov Decision Processes. Springer, Berlin (1996)

    Book  Google Scholar 

  21. Flesch, J.: Stochastic games with the average reward. Ph.D. thesis, Maastricht University, ISBN 90-9012162-5 (1998)

    Google Scholar 

  22. Flesch, J., Schoenmakers, G., Vrieze, O.J.: Loss of skills in coordination games. Int. J. Game Theory 40, 769–789 (2011)

    Article  MathSciNet  Google Scholar 

  23. Forges, F.: An approach to communication equilibria. Econometrica 54, 1375–1385 (1986)

    Article  MathSciNet  Google Scholar 

  24. Hall, R.J., Milner-Gulland, E.J., Courchamp, F.: Endangering the endangered: the effects of perceived rarity on species exploitation. Conserv. Lett. 1, 75–81 (2008)

    Article  Google Scholar 

  25. Hämäläinen, R.P., Haurie, A., Kaitala, V.: Equilibria and threats in a fishery management game. Optim. Control. Appl. Methods 6, 315–333 (1985)

    Article  MathSciNet  Google Scholar 

  26. Hamburger, H.: N-person prisoner’s dilemma. J. Math. Psychol. 3, 27–48 (1973)

    MathSciNet  MATH  Google Scholar 

  27. Hardin, G.: The tragedy of the commons. Science 162, 1243–1248 (1968)

    Google Scholar 

  28. Hart, S.: Nonzero-sum two-person repeated games with incomplete information. Math. Oper. Res. 10, 117–153 (1985)

    Article  MathSciNet  Google Scholar 

  29. Haurie, A., Krawczyk, J.B., Zaccour, G.: Games and Dynamic Games. World Scientific, Singapore (2012)

    Book  Google Scholar 

  30. Heckathorn, D.D.: The dynamics and dilemmas of collective action. Am. Sociol. Rev. 61, 250–277 (1996)

    Article  Google Scholar 

  31. Herings P.J.J., Predtetchinski, A.: Voting in collective stopping games, working paper Maastricht University (2012)

    Google Scholar 

  32. Hillis, J.F., Wheelan, J.: Fisherman’s time discounting rates and other factors to be taken into account in planning rehabilitation of depleted fisheries. In: Antona, M., et al. (eds.) Proceedings of the 6th Conference of the International Institute of Fisheries Economics Trade, pp. 657–670. IIFET-Secretariat, Paris (1994)

    Google Scholar 

  33. Holden, M.: The Common Fisheries Policy: Origin, Evaluation and Future. Fishing News Books, Blackwell (1994)

    Google Scholar 

  34. Hordijk, A., Vrieze, O.J., Wanrooij, L.: Semi-Markov strategies in stochastic games. Int. J. Game Theory 12, 81–89 (1983)

    Article  MathSciNet  Google Scholar 

  35. Joosten, R.: Dynamics, Equilibria, and Values. Ph.D. thesis, Faculty of Economics and Business Administration, Maastricht University (1996)

    Google Scholar 

  36. Joosten, R.: A note on repeated games with vanishing actions. Int. Game Theory Rev. 7, 107–115 (2005)

    Article  MathSciNet  Google Scholar 

  37. Joosten, R.: Small Fish Wars: a new class of dynamic fishery-management games. ICFAI J. Manag. Econ. 5, 17–30 (2007a)

    Google Scholar 

  38. Joosten, R.: Small Fish Wars and an authority. In: Prinz, A. (ed.) The Rules of the Game: Institutions, Law, and Economics, pp. 131–162. LIT, Berlin (2007)

    Google Scholar 

  39. Joosten, R.: Strategic advertisement with externalities: a new dynamic approach. In: Neogy, S.K., Das, A.K., Bapat, R.B. (eds.) Modeling, Computation and Optimization. ISI Platinum Jubilee Series, vol. 6, pp. 21–43. World Scientific Publishing Company, Singapore (2009)

    Chapter  Google Scholar 

  40. Joosten, R.: Long-run strategic advertisement and short-run Bertrand competition. Int. Game Theory Rev. 17, 1540014 (2015). https://doi.org/10.1142/S0219198915400149

    Article  MathSciNet  MATH  Google Scholar 

  41. Joosten, R.: Strong and weak rarity value: resource games with complex price-scarcity relationships. Dyn. Games Appl. 16, 97–111 (2016)

    Article  MathSciNet  Google Scholar 

  42. Joosten, R., Brenner, T., Witt, U.: Games with frequency-dependent stage payoffs. Int. J. Game Theory 31, 609–620 (2003)

    Article  MathSciNet  Google Scholar 

  43. Joosten, R., Samuel, L.: On stochastic fishery games with endogenous stage payoffs and transition probabilities. In: Proceedings of 3rd Joint Chinese-Dutch Workshop on Game Theory and Applications and 7th China Meeting on Game Theory and Applications. CCIS-series. Springer, Berlin (2017)

    Chapter  Google Scholar 

  44. Joosten, R., Samuel, L.: On the computation of large sets of rewards in ETP-ESP-games with communicating states. Research memorandum, Twente University, The Netherlands (2017)

    Google Scholar 

  45. Joosten, R., Thuijsman, F., Peters, H.: Unlearning by not doing: repeated games with vanishing actions. Games Econ. Behav. 9, 1–7 (1993)

    Article  MathSciNet  Google Scholar 

  46. Kelly, C.J., Codling, E.A., Rogan, E.: The Irish Sea cod recovery plan: some lessons learned. ICES J. Mar. Sci. 63, 600–610 (2006)

    Article  Google Scholar 

  47. Komorita, S.S., Parks, C.D.: Social Dilemmas. Westview Press, Boulder (1996)

    Google Scholar 

  48. Krawczyk, J.B., Tołwinski, B.: A cooperative solution for the three nation problem of exploitation of the southern bluefin tuna. IMA J. Math. Appl. Med. Biol. 10, 135–147 (1993)

    Article  Google Scholar 

  49. Lenton, T.M., Livina, V.N., Dakos, V., Scheffer, M.: Climate bifurcation during the last deglaciation? Clim. Past 8, 1127–1139 (2012)

    Article  Google Scholar 

  50. Levhari, D., Mirman, L.: The great fish war: an example using a dynamic Cournot-Nash solution. Bell J. Econ. 11, 322–334 (1980)

    Article  MathSciNet  Google Scholar 

  51. Long, N.V.: A Survey of Dynamic Games in Economics. World Scientific, Singapore (2010)

    Book  Google Scholar 

  52. Mäler, K.-G., Xepapadeas, A., de Zeeuw, A.: The economics of shallow lakes. Environ. Resour. Econ. 26, 603–624 (2003)

    Article  Google Scholar 

  53. Marwell, G., Oliver, P.: The Critical Mass in Collective Action: A Micro-Social Theory. Cambridge University Press, Cambridge (1993)

    Book  Google Scholar 

  54. Messick, D.M., Brewer, M.B.: Solving social dilemmas: a review. Annu. Rev Pers. Soc. Psychol. 4, 11–43 (1983)

    Google Scholar 

  55. Messick, D.M., Wilke, H., Brewer, M.B., Kramer, P.M., Zemke, P.E., Lui, L.: Individual adaptation and structural change as solutions to social dilemmas. J Pers. Soc. Psychol. 44, 294–309 (1983)

    Article  Google Scholar 

  56. Mertens, J.F., Neyman, A.: Stochastic games. Int. J. Game Theory. 10, 53–66 (1981)

    Article  MathSciNet  Google Scholar 

  57. Oosthuizen, E., Daan, N.: Egg fecundity and maturity of North Sea cod, gadus morhua. Neth. J. Sea Res. 8, 378–397 (1974)

    Article  Google Scholar 

  58. Ostrom, E.: Governing the Commons. Cambridge University Press, Cambridge (1990)

    Book  Google Scholar 

  59. Ostrom, E., Gardner, R., Walker, J.: Rules, Games, and Common-Pool Resources. Michigan University Press, Ann Arbor (1994)

    Book  Google Scholar 

  60. Parthasarathy, T., Raghavan, T.E.S.: An orderfield property for stochastic games when one player controls the transition probabilities. J. Optim. Theory Appl. 33, 375–392 (1981)

    Article  MathSciNet  Google Scholar 

  61. Platt, J.: Social traps. Am. Psychol. 28, 641–651 (1973)

    Article  Google Scholar 

  62. Raghavan, T.E.S., Filar, J.: Algorithms for stochastic games, a survey. Z. Oper. Res. 33, 437–472 (1991)

    MathSciNet  MATH  Google Scholar 

  63. Rose, G.A., Bradbury, I.R., de Young, B., Fudge, S.B., Lawson, G.L., Mello, L.G.S., Robichaud, D., Sherwood, G., Snelgrove, P.V.R., Windle, M.J.S.: Rebuilding Atlantic Cod: Lessons from a Spawning Ground in Coastal Newfoundland. In: Kruse, G.H., et al. (eds.) 24th Lowell Wakefield Fisheries Symposium on Resiliency of gadid stocks to fishing and climate change, pp. 197–219 (2008)

    Google Scholar 

  64. Sanchirico, J.N., Smith, M.D., Lipton, D.W.: An empirical approach to ecosystem-based fishery management. Ecol. Econ. 64, 586–596 (2008)

    Article  Google Scholar 

  65. Scheffer, M.: The Ecology of Shallow Lakes. Chapman & Hall, London (1998)

    Google Scholar 

  66. Scheffer, M., Carpenter, S., Foley, J.A., Folke, C., Walker, B.: Catastrophic shifts in ecosystems. Nature 413, 591–596 (2001)

    Article  Google Scholar 

  67. Schoenmakers, G.M.: The profit of skills in repeated and stochastic games. Ph.D. thesis Maastricht University (2004)

    Google Scholar 

  68. Schoenmakers, G.M., Flesch, J., Thuijsman, F.: Coordination games with vanishing actions. Int. Game Theory Rev. 4, 119–126 (2002)

    Article  MathSciNet  Google Scholar 

  69. Shapley, L.: Stochastic games. Proc. Natl. Acad. Sci. USA 39, 1095–1100 (1953)

    Article  MathSciNet  Google Scholar 

  70. Steg, L.: Motives and behavior in social dilemmas relevant to the environment. In: Hendrickx, L., Jager, W., Steg, L. (eds.) Human Decision Making and Environmental Perception. Understanding and Assisting Human Decision Making in Real-Life Settings, pp. 83–102 (2003)

    Google Scholar 

  71. Thuijsman, F., Vrieze, O.J.: The power of threats in stochastic games. In: Bardi, M., et al. (eds.) Stochastic and Differential Games, Theory and Numerical Solutions, pp. 343–358. Birkhauser, Boston (1998)

    MATH  Google Scholar 

  72. Tołwinski, B.: A concept of cooperative equilibrium for dynamic games. Automatica 18, 431–441 (1982)

    Article  MathSciNet  Google Scholar 

  73. Tołwinski, B., Haurie, A., Leitmann, G.: Cooperative equilibria in differential games. JOTA 119, 182–202 (1986)

    MathSciNet  MATH  Google Scholar 

  74. Van Damme, E.E.C.: Stability and Perfection of Nash Equilibria. Springer, Berlin (1992)

    MATH  Google Scholar 

  75. Vrieze, O.J.: Linear programming and undiscounted games in which one player controls transitions. OR Spektrum 3, 29–35 (1981)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IEBIS, BMS, University of Twente, POB 217, 7500 AE, Enschede, The Netherlands

    Reinoud Joosten & Robin Meijboom

Authors
  1. Reinoud Joosten
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robin Meijboom
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Reinoud Joosten .

Editor information

Editors and Affiliations

  1. Indian Statistical Institute, New Delhi, India

    S.K. Neogy

  2. Indian Statistical Institute, New Delhi, India

    Ravindra B. Bapat

  3. Indian Statistical Institute, New Delhi, India

    Dipti Dubey

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Joosten, R., Meijboom, R. (2018). Stochastic Games with Endogenous Transitions. In: Neogy, S., Bapat, R., Dubey, D. (eds) Mathematical Programming and Game Theory. Indian Statistical Institute Series. Springer, Singapore. https://doi.org/10.1007/978-981-13-3059-9_12

Download citation

Publish with us

Policies and ethics

Search

Navigation