Abstract
We present a new Hessian estimator based on the simultaneous perturbation procedure, that requires three system simulations regardless of the parameter dimension. We then present two Newton-based simulation optimization algorithms that incorporate this Hessian estimator. The two algorithms differ primarily in the manner in which the Hessian estimate is used. Both our algorithms do not compute the inverse Hessian explicitly, thereby saving on computational effort. While our first algorithm directly obtains the product of the inverse Hessian with the gradient of the objective, our second algorithm makes use of the Sherman–Morrison matrix inversion lemma to recursively estimate the inverse Hessian. We provide proofs of convergence for both our algorithms. Next, we consider an interesting application of our algorithms on a problem of road traffic control. Our algorithms are seen to exhibit better performance than two Newton algorithms from a recent prior work.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Chong, E.K.P., Ramadge, P.J.: Optimization of queues using an infinitesimal perturbation analysis-based stochastic algorithm with general update times. SIAM J. Control Optim. 31(3), 698–732 (1993)
Ho, Y.C., Cao, X.R.: Perturbation Analysis of Discrete Event Dynamical Systems. Kluwer, Boston (1991)
Andradóttir, S.: Optimization of the transient and steady-state behavior of discrete event systems. Manag. Sci. 42(5), 717–737 (1996)
Kiefer, E., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Stat. 23, 462–466 (1952)
Spall, J.C.: Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Autom. Control 37(3), 332–341 (1992)
Spall, J.C.: A one-measurement form of simultaneous perturbation stochastic approximation. Automatica 33, 109–112 (1997)
Bhatnagar, S., Fu, M.C., Marcus, S.I., Bhatnagar, S.: Two-timescale algorithms for simulation optimization of hidden Markov models. IIE Trans. 33(3), 245–258 (2001)
Bhatnagar, S., Fu, M.C., Marcus, S.I., Wang, I.: Two-timescale simultaneous perturbation stochastic approximation using deterministic perturbation sequences. ACM Trans. Model. Comput. Simul. 13(2), 180–209 (2003)
Bhatnagar, S., Borkar, V.S.: Multiscale chaotic SPSA and smoothed functional algorithms for simulation optimization. Simulation 79(10), 568–580 (2003)
Bhatnagar, S.: Adaptive Newton-based smoothed functional algorithms for simulation optimization. ACM Trans. Model. Comput. Simul. 18(1), 2:1–2:35 (2007)
Bhatnagar, S., Prasad, H.L., Prashanth, L.A.: Stochastic Recursive Algorithms for Optimization: Simultaneous Perturbation Methods. Lecture Notes in Control and Information Sciences. Springer, London (2013)
Fabian, V.: Stochastic approximation. In: Rustagi, J.J. (ed.) Optimizing Methods in Statistics, pp. 439–470. Academic Press, New York (1971)
Spall, J.C.: Adaptive stochastic approximation by the simultaneous perturbation method. IEEE Trans. Autom. Control 45, 1839–1853 (2000)
Bhatnagar, S.: Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization. ACM Trans. Model. Comput. Simul. 15(1), 74–107 (2005)
Prashanth, L.A., Bhatnagar, S.: Reinforcement learning with function approximation for traffic signal control. IEEE Trans. Intell. Transp. Syst. 12(2), 412–421 (2011)
Prashanth, L.A., Bhatnagar, S.: Threshold tuning using stochastic optimization for graded signal control. IEEE Trans. Veh. Technol. 61(9), 3865–3880 (2012)
Bhatnagar, S., Prashanth, L.A.: Simultaneous perturbation Newton algorithms for simulation optimization. Technical report, Stochastic Systems Lab., IISc, (2013). http://stochastic.csa.iisc.ernet.in/www/research/files/IISc-CSA-SSL-TR-2013-4.pdf
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1999)
Zhu, X., Spall, J.C.: A modified second-order SPSA optimization algorithm for finite samples. Int. J. Adapt. Control Signal Process. 16, 397–409 (2002)
Borkar, V.S.: Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, Cambridge (2008)
Borkar, V.S.: Probability Theory: An Advanced Course. Springer, New York (1995)
Hirsch, M.W.: Convergent activation dynamics in continuous time networks. Neural Netw. 2, 331–349 (1989)
Borkar, V.S., Meyn, S.P.: The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim. 38(2), 447–469 (2000)
Kushner, H.J., Clark, D.S.: Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer, New York (1978)
Lasalle, J.P., Lefschetz, S.: Stability by Liapunov’s Direct Method with Applications. Academic Press, New York (1961)
Kushner, H.J., Yin, G.G.: Stochastic Approximation Algorithms and Applications. Springer, New York (1997)
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Ilio Galligani.
Rights and permissions
About this article
Cite this article
Bhatnagar, S., Prashanth, L.A. Simultaneous Perturbation Newton Algorithms for Simulation Optimization. J Optim Theory Appl 164, 621–643 (2015). https://doi.org/10.1007/s10957-013-0507-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-013-0507-1