Skip to main content
SpringerLink
Log in

Suboptimal Solutions to Dynamic Optimization Problems via Approximations of the Policy Functions

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

The approximation of the optimal policy functions is investigated for dynamic optimization problems with an objective that is additive over a finite number of stages. The distance between optimal and suboptimal values of the objective functional is estimated, in terms of the errors in approximating the optimal policy functions at the various stages. Smoothness properties are derived for such functions and exploited to choose the approximating families. The approximation error is measured in the supremum norm, in such a way to control the error propagation from stage to stage. Nonlinear approximators corresponding to Gaussian radial-basis-function networks with adjustable centers and widths are considered. Conditions are defined, guaranteeing that the number of Gaussians (hence, the number of parameters to be adjusted) does not grow “too fast” with the dimension of the state vector. The results help to mitigate the curse of dimensionality in dynamic optimization. An example of application is given and the use of the estimates is illustrated via a numerical simulation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Montrucchio, L.: Lipschitz continuous policy functions for strongly concave optimization problems. J. Math. Econ. 16, 259–273 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bertsekas, D.P., Tsitsiklis, J.: Neuro-Dynamic Programming. Athena Scientific, Belmont (1996)

    MATH  Google Scholar 

  3. Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 2. Athena Scientific, Belmont (2007)

    Google Scholar 

  4. Powell, W.B.: Approximate Dynamic Programming—Solving the Curses of Dimensionality. John Wiley & Sons, Hoboken (2007)

    Book  MATH  Google Scholar 

  5. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)

    MATH  Google Scholar 

  6. Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 1. Athena Scientific, Belmont (2005)

    MATH  Google Scholar 

  7. Singer, I.: Best Approximation in Normed Linear Spaces by Elements of Linear Subspaces. Springer, Berlin (1970)

    MATH  Google Scholar 

  8. Ekeland, I., Turnbull, T.: Infinite-Dimensional Optimization and Convexity. The University of Chicago Press, Chicago (1983)

    MATH  Google Scholar 

  9. Daniel, J.W.: The Approximate Minimization of Functionals. Prentice Hall, Englewood Cliffs (1971)

    MATH  Google Scholar 

  10. Alt, W.: On the approximation of infinite optimization problems with an application to optimal control problems. Appl. Math. Optim. 12, 15–27 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  11. Zoppoli, R., Sanguineti, M., Parisini, T.: Approximating networks and extended Ritz method for the solution of functional optimization problems. J. Optim. Theory Appl. 112, 403–439 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  12. Kůrková, V., Sanguineti, M.: Error estimates for approximate optimization by the extended Ritz method. SIAM J. Optim. 18, 461–487 (2005)

    Article  Google Scholar 

  13. Giulini, S., Sanguineti, M.: Approximation schemes for functional optimization problems. J. Optim. Theory Appl. 140, 33–54 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  14. Gelfand, I.M., Fomin, S.V.: Calculus of Variations. Prentice Hall, Englewood Cliffs (1963)

    Google Scholar 

  15. Juditsky, A., Hjalmarsson, H., Benveniste, A., Delyon, B., Ljung, L., Sjöberg, J., Zhang, Q.: Nonlinear black-box models in system identification: Mathematical foundations. Automatica 31, 1725–1750 (1995)

    Article  MATH  Google Scholar 

  16. Narendra, K.S., Mukhopadhyay, S.: Adaptive control using neural networks and approximate models. IEEE Trans. Neural Netw. 8, 475–485 (1997)

    Article  Google Scholar 

  17. Barron, A.R.: Neural net approximation. In: Narendra, K. (ed.) Proc. 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press, New Haven (1992)

    Google Scholar 

  18. Girosi, F., Anzellotti, G.: Rates of convergence for radial basis functions and neural networks. In: Mammone, R.J. (ed.) Artificial Neural Networks for Speech and Vision, pp. 97–114. Chapman and Hall, London (1993)

    Google Scholar 

  19. Hornik, K., Stinchcombe, M., White, H., Auer, P.: Degree of approximation results for feedforward networks approximating unknown mappings and their derivatives. Neural Comput. 6, 1262–1275 (1994)

    Article  MATH  Google Scholar 

  20. Girosi, F.: Approximation error bounds that use VC-bounds. In: Proc. Int. Conf. on Artificial Neural Networks. pp. 295–302, EC2 & Cie, Paris (1995)

    Google Scholar 

  21. Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numer. 8, 143–195 (1999)

    Article  MathSciNet  Google Scholar 

  22. Kůrková, V., Sanguineti, M.: Comparison of worst-case errors in linear and neural network approximation. IEEE Trans. Inform. Theory 48, 264–275 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  23. Mhaskar, H.N.: On the tractability of multivariate integration and approximation by neural networks. J. Complex. 20, 561–590 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  24. Kainen, P.C., Kůrková, V., Sanguineti, M.: Complexity of Gaussian radial basis networks approximating smooth functions. J. Complex. 25, 63–74 (2009)

    Article  MATH  Google Scholar 

  25. Kainen, P.C., Kůrková, V., Sanguineti, M.: Estimates of approximation rates by Gaussian radial-basis functions. In: Lecture Notes in Computer Science, vol. 4432, pp. 11–18. Springer, Berlin (2007)

    Google Scholar 

  26. Kůrková, V., Sanguineti, M.: Geometric upper bounds on rates of variable-basis approximation. IEEE Trans. Inform. Theory 54, 5681–5688 (2008)

    Article  MathSciNet  Google Scholar 

  27. Kůrková, V., Sanguineti, M.: Approximate minimization of the regularized expected error over kernel models. Math. Oper. Res. 33, 747–756 (2008)

    Article  MathSciNet  Google Scholar 

  28. Alessandri, A., Parisini, T., Sanguineti, M., Zoppoli, R.: Neural strategies for nonlinear optimal filtering. In: Proc. IEEE Int. Conf. Syst. Eng., pp. 44–49, Kobe, Japan, 1992

  29. Zoppoli, R., Parisini, T.: Learning techniques and neural networks for the solution of N-stage nonlinear nonquadratic optimal control problems. In: Isidori, A., Tarn, T.J. (eds.) Systems, Models and Feedback: Theory and Applications, pp. 193–210. Birkhäuser, Boston (1992)

    Google Scholar 

  30. Zolezzi, T.: Condition numbers and Ritz type methods in unconstrained optimization. Control Cybern. 36, 811–822 (2007)

    MathSciNet  MATH  Google Scholar 

  31. Buhmann, M.D.: Radial Basis Functions: Theory and Implementations. Cambridge University Press, Cambridge (2003)

    Book  MATH  Google Scholar 

  32. Alessandri, A., Sanguineti, M., Maggiore, M.: Optimization-based learning with bounded error for feedforward neural networks. IEEE Trans. Neural Netw. 13, 261–273 (2002)

    Article  Google Scholar 

  33. Chow, T.W.S., Cho, S.-Y.: Neural Networks and Computing: Learning Algorithms and Applications. World Scientific, Singapore (2007)

    Google Scholar 

  34. Dacorogna, B.: Introduction to the Calculus of Variations. Imperial College Press, London (2004)

    MATH  Google Scholar 

  35. Rudin, W.: Real and Complex Analysis. McGraw-Hill, Singapore (1987)

    MATH  Google Scholar 

  36. Stein, E.M.: Singular Integrals and Differentiability Properties of Functions. Princeton University Press, Princeton (1970)

    MATH  Google Scholar 

  37. Cugno, F., Montrucchio, L.: Scelte Intertemporali. Teoria e modelli. Carocci Editore, Rome (1998)

    Google Scholar 

  38. Heinonen, J.: Lectures on Analysis on Metric Spaces. Springer, New York (2001)

    MATH  Google Scholar 

  39. Montrucchio, L.: Thompson metric, contraction property, and differentiability of policy functions. J. Econ. Behav. Organ. 33, 449–466 (1998)

    Article  Google Scholar 

  40. Gnecco, G., Sanguineti, M.: Value and policy function approximations in infinite-horizon optimization problems. J. Dyn. Syst. Geom. Theories 6, 123–147 (2008)

    MathSciNet  MATH  Google Scholar 

  41. Stokey, N.L., Lucas, R.E., Prescott, E.: Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge (1989)

    MATH  Google Scholar 

  42. Pinkus, A.: n-Widths in Approximation Theory. Springer, Berlin (1985)

    MATH  Google Scholar 

  43. Giulini, S., Sanguineti, M.: On dimension-independent approximation by neural networks and linear approximators. In: Proc. Int. Joint Conference on Neural Networks, pp. I283–I288 (2000)

  44. Gnecco, G., Sanguineti, M.: Approximation error bounds via Rademacher’s complexity. Appl. Math. Sci. 2, 153–176 (2008)

    MathSciNet  MATH  Google Scholar 

  45. Traub, J.F., Werschulz, A.G.: Complexity and Information. Cambridge University Press, Cambridge (1999)

    Google Scholar 

  46. Gnecco, G., Sanguineti, M.: Estimates of the approximation error via Rademacher complexity: Learning vector-valued functions. J. Inequal. Appl. 2008, 640758 (2008)

    Article  MathSciNet  Google Scholar 

  47. Ljungqvist, L., Sargent, T.: Recursive Macroeconomic Theory. MIT Press, Cambridge (2004)

    Google Scholar 

  48. Pervozvanskii, A.A., Gaitsgori, V.G.: Theory of Suboptimal Decisions: Decomposition and Aggregation. Kluwer Academic, Norwell (1988)

    MATH  Google Scholar 

  49. Li, D., Sun, X.: Nonlinear Integer Programming. Springer, New York (2006)

    MATH  Google Scholar 

  50. Burger, M., Neubauer, A.: Error bounds for approximation with neural networks. J. Approx. Theory 112, 235–250 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  51. Araujo, A.: The once but not twice differentiability of the policy function. Econometrica 59, 1383–1393 (1991)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Communications, Computer, and System Sciences (DIST), University of Genoa, Via Opera Pia 13, 16145, Genova, Italy

    G. Gnecco & M. Sanguineti

Authors
  1. G. Gnecco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Sanguineti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Sanguineti.

Additional information

Communicated by F. Zirilli.

The authors were partially supported by a grant “Progetti di Ricerca di Ateneo 2008” of the University of Genoa, Project “Solution of Functional Optimization Problems by Nonlinear Approximators and Learning from Data”.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gnecco, G., Sanguineti, M. Suboptimal Solutions to Dynamic Optimization Problems via Approximations of the Policy Functions. J Optim Theory Appl 146, 764–794 (2010). https://doi.org/10.1007/s10957-010-9680-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-010-9680-7

Keywords

Search

Navigation