A two-level approach to learning in nonstationary environments
A nonstationary environment is one in which the suitability of the strategies available to a learning element changes with time. Since the optimal action in such a case is not fixed, the learning problem (i.e., the determination of the optimal strategy) becomes considerably difficult. In this paper, a two-level approach is presented for a learning automaton operating in a nonstationary environment. The lower level consists of a standard absolutely expedient learning algorithm for stationary environments. The higher level on the other hand is a tracking algorithm, based on Bayesian decision theory, for detecting changes in the environment and reinitializing the lower level algorithm in a suitable manner. Simulation studies empirically demonstrate the clear superiority of the two-level approach over the single-level learning in nonstationary environments.
Unable to display preview. Download preview PDF.
- 1.K. S. Narendra and M. A. L. Thathachar. Learning Automata — an Introduction. Prentice Hall, Englewood Cliffs, New Jersey, 1989.Google Scholar
- 2.M. L. Tsetlin. Automaton Theory and Modeling of Biological Systems. Academic Press, New York, 1973.Google Scholar
- 3.V. I. Varshavskii and I. P. Vorontsova. “On the Behavior of Stochastic Automata with a Variable Structure”. Automation and Remote Control, vol. 24, pp. 327–333, 1963.Google Scholar
- 4.B. J. Oommen and H. Masum. “Switching Models for Nonstationary Random Environments”. IEEE Trans. Syst., Man, and Cybern., SMC-25, pp. 1334–1339, 1995.Google Scholar
- 5.P. R. Srikantakumar and K. S. Narendra. “A Learning Model for Routing in Telephone Networks”. SIAM Journal of Control and Optimization, vol. 20, pp. 34–57, 1982.Google Scholar
- 6.O. V. Nedzelnitsky, Jr. and K. S. Narendra. “Nonstationary Models of Learning Automata Routing in Data Communication Networks”. IEEE Trans. Syst., Man, and Cybern., SMC-17, pp. 1004–1015, 1987.Google Scholar
- 7.M. Herbster and M. Warmuth. “Tracking the Best Expert”. Proceedings of the Twelfth International Conference on Machine Learning, ICML95, pp. 286–294, 1995.Google Scholar
- 8.N. Baba and Y. Sawaragi. “On the Learning Behavior of Stochastic Automata under a Nonstationary Environment”. IEEE Trans. Syst., Man, and Cybern., SMC-5, pp. 273–275, 1975.Google Scholar
- 9.A. G. Barto and P. Anandan. “Pattern-recognizing Stochastic Learning Automata”. IEEE Trans. Syst., Man, and Cybern., SMC-15, pp. 360–375, 1985.Google Scholar
- 10.S. Zacks and Z. Barzily. “Bayes Procedures for Detecting a Shift in the Probability of Success in a Series of Bernoulli Trials”. Journal of Statistical Planning and Inference, No. 5, pp. 107–119, 1981.Google Scholar
- 11.M. A. L. Thathachar and P. S. Sastry. “A New Approach to the Design of Reinforcement Schemes for Learning Automata”. IEEE Trans. Syst., Man, and Cybern., SMC-15, pp. 168–175, 1985.Google Scholar
- 12.S. Mukhopadhyay and M.A.L. Thathachar. “Associative Learning of Boolean Functions”. IEEE Trans. Syst., Man and Cybern., SMC-19, pp. 1008–1015, 1989.Google Scholar
- 13.W. Lam and S. Mukhopadhyay. “A Two-Level Approach to Learning in Nonstationary Environments”. Management Sciences Department Technical Report, The University of Iowa, Iowa City, Iowa, 1995.Google Scholar