A two-level approach to learning in nonstationary environments

  • Wai Lam
  • Snehasis Mukhopadhyay
Learning II: Challenging Domains and Problems
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1081)


A nonstationary environment is one in which the suitability of the strategies available to a learning element changes with time. Since the optimal action in such a case is not fixed, the learning problem (i.e., the determination of the optimal strategy) becomes considerably difficult. In this paper, a two-level approach is presented for a learning automaton operating in a nonstationary environment. The lower level consists of a standard absolutely expedient learning algorithm for stationary environments. The higher level on the other hand is a tracking algorithm, based on Bayesian decision theory, for detecting changes in the environment and reinitializing the lower level algorithm in a suitable manner. Simulation studies empirically demonstrate the clear superiority of the two-level approach over the single-level learning in nonstationary environments.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    K. S. Narendra and M. A. L. Thathachar. Learning Automata — an Introduction. Prentice Hall, Englewood Cliffs, New Jersey, 1989.Google Scholar
  2. 2.
    M. L. Tsetlin. Automaton Theory and Modeling of Biological Systems. Academic Press, New York, 1973.Google Scholar
  3. 3.
    V. I. Varshavskii and I. P. Vorontsova. “On the Behavior of Stochastic Automata with a Variable Structure”. Automation and Remote Control, vol. 24, pp. 327–333, 1963.Google Scholar
  4. 4.
    B. J. Oommen and H. Masum. “Switching Models for Nonstationary Random Environments”. IEEE Trans. Syst., Man, and Cybern., SMC-25, pp. 1334–1339, 1995.Google Scholar
  5. 5.
    P. R. Srikantakumar and K. S. Narendra. “A Learning Model for Routing in Telephone Networks”. SIAM Journal of Control and Optimization, vol. 20, pp. 34–57, 1982.Google Scholar
  6. 6.
    O. V. Nedzelnitsky, Jr. and K. S. Narendra. “Nonstationary Models of Learning Automata Routing in Data Communication Networks”. IEEE Trans. Syst., Man, and Cybern., SMC-17, pp. 1004–1015, 1987.Google Scholar
  7. 7.
    M. Herbster and M. Warmuth. “Tracking the Best Expert”. Proceedings of the Twelfth International Conference on Machine Learning, ICML95, pp. 286–294, 1995.Google Scholar
  8. 8.
    N. Baba and Y. Sawaragi. “On the Learning Behavior of Stochastic Automata under a Nonstationary Environment”. IEEE Trans. Syst., Man, and Cybern., SMC-5, pp. 273–275, 1975.Google Scholar
  9. 9.
    A. G. Barto and P. Anandan. “Pattern-recognizing Stochastic Learning Automata”. IEEE Trans. Syst., Man, and Cybern., SMC-15, pp. 360–375, 1985.Google Scholar
  10. 10.
    S. Zacks and Z. Barzily. “Bayes Procedures for Detecting a Shift in the Probability of Success in a Series of Bernoulli Trials”. Journal of Statistical Planning and Inference, No. 5, pp. 107–119, 1981.Google Scholar
  11. 11.
    M. A. L. Thathachar and P. S. Sastry. “A New Approach to the Design of Reinforcement Schemes for Learning Automata”. IEEE Trans. Syst., Man, and Cybern., SMC-15, pp. 168–175, 1985.Google Scholar
  12. 12.
    S. Mukhopadhyay and M.A.L. Thathachar. “Associative Learning of Boolean Functions”. IEEE Trans. Syst., Man and Cybern., SMC-19, pp. 1008–1015, 1989.Google Scholar
  13. 13.
    W. Lam and S. Mukhopadhyay. “A Two-Level Approach to Learning in Nonstationary Environments”. Management Sciences Department Technical Report, The University of Iowa, Iowa City, Iowa, 1995.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1996

Authors and Affiliations

  • Wai Lam
    • 1
  • Snehasis Mukhopadhyay
    • 2
  1. 1.Department of Management SciencesThe University of IowaIowa CityUSA
  2. 2.Department of Computer and Information SciencePurdue School of Science at IndianapolisIndianapolisUSA

Personalised recommendations