Synthesis of Optimal Strategies for Differential Games by Neural Networks

  • H. J. Pesch
  • I. Gabler
  • S. Miesbach
  • M. H. Breitner
Part of the Annals of the International Society of Dynamic Games book series (AISDG, volume 3)


The paper deals with the numerical approximation of optimal strategies for two-person zero-sum differential games of pursuit-evasion type by neural networks. Thereby, the feedback strategies can be computed in real-time after the training of appropriate neural networks. For this purpose, sufficiently many optimal trajectories and their associated open-loop representations of the optimal feedback strategies must be computed, to provide data for training and cross-validation of the neural networks. All the precomputations can be carried through in a highly parallel way. This approach turns out to be applicable for differential games of more general type.

The method is demonstrated for a modified cornered rat game where a pursuing cat and an evading rat, both moving in simple motion, are constrained to a rectangular arena. Two holes in the walls surrounding the arena enable the rat to evade once and for all, if the rat is not too far from these holes. The optimal trajectories in the escape zone can be computed analytically. In the capture zone, a game of degree is employed with terminal time as payoff. To compute optimal trajectories for this secondary game, the time evolution of the survival region for the rat is determined via a sequence of discretized games.

The combination of these methods permits the computation of more than a thousand trajectories leading to some ten thousand sample patterns which relate the state variables to the values of the optimal strategies. These data exhibit characteristic properties of the optimal strategies. It is shown that these properties can be extracted from the data by use of neural networks. By means of the trained networks, about 200 trajectories arc finally simulated. The pursuer as well as the evader acts according to the controls proposed by the neural networks. Despite the simple structure of the neural networks used in this study, the strategics based upon them show a reasonable, close to optimal performance in a large variety of simulations of the pursuit-evasion game under consideration.


Differential games neural networks real-time computation pursuit-evasion games optimal strategies numerical solution of differential games cornered rat game 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Anderson, B. D. O., Exponential Stability of Linear Equations Arising in Adaptive Identification, IEEE Transactions on Automatic Control 22, 1977, 83–88.zbMATHCrossRefGoogle Scholar
  2. [2]
    Ba§ar, T., and Olsder, G. J., Dynamic Noncooperative Game Theory, Academic Press, London, 1982, 2nd ed., 1994.Google Scholar
  3. [3]
    Benveniste, A., Metivier, M., and Priouret, P., Algorithmes adaptatifs et approximations stochastique, Masson, Paris, 1987.Google Scholar
  4. [4]
    Blum, E. K., and Li, L. K., Approximation Theory and Feedforward Networks, Neural Networks 4, 1990, 511 – 515.CrossRefGoogle Scholar
  5. [5]
    Breakwell, J. V., Zero-Sum Differential Games with Terminal Payoff, in: P. Hagedorn, H. W. Knobloch, and G. J. Olsder (Eds.), Differential Games and Applications, Lecture Notes in Control and Information Sciences 3, Springer, Berlin, 1977, 70 – 95.Google Scholar
  6. [6]
    Breitner, M. H., Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances, in: R. Bulirsch, and D. Kraft (Eds.), Computational Optimal Control, International Series of Numerical Mathematics 115, Birkhauser, Basel, 1994, 147 – 162.Google Scholar
  7. [7]
    Breitner, M. H., Real-Time Capable Approximation of Optimal Strategies in Complex Differential Games, in: M. Breton, G. Zaccour (Eds.), Proceedings of the Sixth International Symposium on Dynamic Games and Applications, St-Jovite, Quebec, GERAD, Ecole des Hautes Etudes Commerciales, Montreal, 1994, 370 – 374.Google Scholar
  8. [8]
    Breitner, M. H., and Pesch H. J., Reentry Trajectory Optimization under Atmospheric Uncertainty as a Differential Game, in: T. Ba§ar, and A. Haurie (Eds.), Advances in Dynamic Games and Applications, Annals of the International Society of Dynamic Games 1, Birkhauser, Boston, 1994, 70 – 88.Google Scholar
  9. [9]
    Breitner, M. H., Pesch, H. J., and Grimm, W., Complex Differential Games of Pursuit-Evasion Type with State Constraints, Part 1: Necessary Conditions for Optimal Open-Loop Strategies, Part 2: Numerical Computation of Optimal Open-Loop Strategies, Journal of Optimization Theory and Applications 78, 1993, 419–441, 443 – 463.MathSciNetCrossRefGoogle Scholar
  10. [10]
    Bryson, A. E., and Ho, Y.-C., Applied Optimal Control, Hemisphere, New York, 1975.Google Scholar
  11. [11]
    Finnoff, W., Diffusion Approximations for the Constant Learning Rate Backpropagation Algorithm and Resistance to Local Minima, Neural Computation 6, 285 – 295, 1994.CrossRefGoogle Scholar
  12. [12]
    Gabler, I., Numerische Berechnung optimaler Strategien eines Diffe-rentialspiels und Approximation durch Neuronale Netze, Diploma thesis, Department of Mathematics, Munich University of Technology, Munich, 1993.Google Scholar
  13. [13]
    Gabler, I., Miesbach, S., Breitner, M. H., and Pesch, H. J., Synthesis of Optimal Strategies for Differential Games by Neural Networks, Report No. 468, Deutsche Forschungsgemeinschaft, Schwer-punkt “Anwendungsbezogene Optimierung und Steuerung”, Department of Mathematics, Munich University of Technology, Munich, 1993.Google Scholar
  14. [14]
    Goh, C. J., and Edwards, N. J., Feedback Control of Minimum-Time Optimal Control Problems Using Neural Networks, Optimal Control Applications & Methods 14, 1993, 1 – 16.MathSciNetzbMATHCrossRefGoogle Scholar
  15. [15]
    Hertz, J., Krogh, A., and Palmer, R. G., Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City, 1991.Google Scholar
  16. [16]
    Hornik, K., Stinchcombe, M., and White, H., Multi-Layer Feedforward Networks are Universal Approximators, Neural Networks 2, 1989, 359 – 366.CrossRefGoogle Scholar
  17. [17]
    Isaacs, R., Differential Games, John Wiley & Sons, New York, 1965, Krieger, New York, 1975.Google Scholar
  18. [18]
    Jarmark, B., and Bengtsson, H., Near-Optimal Flight Trajectories Generated by Neural Networks, Internal Report, Saab/Scania Aircraft Division, Linkoping, 1989.Google Scholar
  19. [19]
    Koslik, B., Pesch, H. J., Breitner, M. H., and von Stryk, O., Optimal Control of a Complex Concern Model by Direct and Indirect Optimization Methods, to appear in: H. Engl, and H. Neunzert (Eds.), Proceedings of the 8th Conference of the European Consortium for Mathematics in Industry, ECMI Series, Teubner, Stuttgart, 1995.Google Scholar
  20. [20]
    Kugelmann, B., and Pesch, H. J., New General Guidance Method in Constrained Optimal Control, Part 1: Numerical Method, Part 2: Application to Space Shuttle Guidance, Journal of Optimization Theory and Applications 67, 1990, 421–435, 437 – 446.MathSciNetCrossRefGoogle Scholar
  21. [21]
    Lachner, R., Breitner, M. H., and Pesch H. J., Three-Dimensional Air Combat: Numerical Solution of Complex Differential Games, in: G. J. Olsder (Ed.), New Trends in Dynamic Games and Applications, this volume, Birkhauser, Boston, 1995, 165 – 190.Google Scholar
  22. [22]
    Lachner, R., Breitner, M.H., and Pesch H.J., Optimal Strategies of a Complex Pursuit-Evasion Game, to appear in Journal of Computing and Information, 1995.Google Scholar
  23. [23]
    Miesbach, S., Bahnfuhrung von Robotern mit Neuronalen Netzen, PhD thesis, Department of Mathematics, Munich University of Technology, Munich, 1995.Google Scholar
  24. [24]
    Miesbach, S., and Schurmann, B., Wenn Roboter arbeiten lernen: Ideen und Methoden der Neuroinformatik zur Regelung und Steuerung, Informationstechnik 33, 1991, 300 – 309.Google Scholar
  25. [25]
    Miller, W. T., Sutton, R. S., and Werbos, P. J. (Eds.), Neural Networks for Control, MIT Press, Cambridge, 1990.Google Scholar
  26. [26]
    Nguyen, D., and Widrow, B., The Truck Backer-Upper: An Example of Self-Learning in Neural Networks, in: W. T. Miller, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control, The MIT Press, Cambridge, 1990, 287 – 299.Google Scholar
  27. [27]
    Pesch, H. J., Real-Time Computation of Feedback Controls for Constrained Optimal Control Problems, Part 1: Neighboring Extremals, Part 2: A Correction Method Based on Multiple Shooting, Optimal Control Applications & Methods 10, 1989, 129–145, 147–171.MathSciNetzbMATHCrossRefGoogle Scholar
  28. [28]
    Pesch, H. J., Offline and Online Computation of Optimal Trajectories in the Aerospace Field, in: A. Miele and A. Salvetti (Eds.), Applied Mathematics in Aerospace Science and Engineering, Plenum, New York, 1994, 165–219.Google Scholar
  29. [29]
    Pesch, H. J., Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity, in: R. Bulirsch, and D. Kraft (Eds.), Computational Optimal Control, International Series of Numerical Mathematics 115, Birkhauser, Basel, 1994, 43–64.Google Scholar
  30. [30]
    Proceedings of the NIPS Neural Information Processing Systems, Morgan Kaufmann, San Mateo, CA, annually since 1987.Google Scholar
  31. [31]
    Ritter, H., Martinetz, T., and Schulten, K., Neuronale Netze: Eine Einfiihrung in die Neuroinformatik selbstorganisierender Netz-werke, Addison-Wesley, Bonn, 1991.Google Scholar
  32. [32]
    Rojas, R., Theorie der neuronalen Netze, Springer, Berlin, 1993.Google Scholar
  33. [33]
    Rosenblatt, F., Principles of Neurodynamics, Spartan, New York, 1962.zbMATHGoogle Scholar
  34. [34]
    Saarinen, S., Bramley, R., and Cybenko, G., Ill-Conditioning in Neural Network Training Problems, SIAM Journal of Scientific and Computing 14, 693 – 714, 1993.MathSciNetzbMATHCrossRefGoogle Scholar
  35. [35]
    Stoer, J., and Bulirsch, R., Introduction to Numerical Analysis, Springer, New York, 2nd ed., 1993.zbMATHGoogle Scholar
  36. [36]
    von Stryk, O., Numerical Solution of Optimal Control Problems by Direct Collocation, in: R. Bulirsch, A. Miele, J. Stoer, and K.-H. Well (Eds.), Optimal Control, International Series of Numerical Mathematics 111, Birkhauser, Basel, 1993, 129 – 143.Google Scholar
  37. [37]
    von Stryk, O., and Bulirsch, R., Direct and Indirect Methods for Trajectory Optimization, Annals of Operations Research 37, 1992, 357 – 373.MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Birkhäuser Boston 1995

Authors and Affiliations

  • H. J. Pesch
    • 1
  • I. Gabler
    • 2
  • S. Miesbach
    • 3
  • M. H. Breitner
    • 1
  1. 1.Department of MathematicsMunich University of TechnologyMunichGermany
  2. 2.Bayerische Hypotheken- und Wechsel-Bank AGMunichGermany
  3. 3.Corporate Research and DevelopmentZFE ST SN 41, Siemens AG, MunichMunichGermany

Personalised recommendations