Two-time scale learning automata: an efficient decision making mechanism for stochastic nonlinear resource allocation

  • Anis YazidiEmail author
  • Hugo L. Hammer
  • Tore M. Jonassen


The Stochastic Non-linear Fractional Equality Knapsack (NFEK) problem is a substantial resource allocation problem which admits a large set of applications such as web polling under polling constraints, and constrained estimation. The NFEK problem is usually solved by trial and error based on noisy feedback information from the environment. The available solutions to NFEK are based on the traditional family of Reward-Inaction Learning Automata (LA) scheme where the action probabilities are updated based on only the last feedback. Such an update form seems counterproductive for two reasons: 1) it only uses the last feedback and does not consider the whole history of the feedback and 2) it ignores updates whenever the last feedback does not correspond to a reward. In this paper, we rather suggest instead a learning solution that resorts to the whole history of feedback using the theory of two time-scale separation. Through comprehensive experimental results we show that the proposed solution is not only superior to the state-of-the-art in terms of peak performance but is also robust to the choice of the tuning parameters.


Decision making under uncertainty Continuous learning automata Two-time scale Stochastic non-linear fractional equality knapsack Resource allocation 



A very preliminary conference version of this work appeared in IEA/AIE 2017, the 30th International Conference on Industrial, Engineering, Other Applications of Applied Intelligent Systems, held in Paris, June 2017. Prof. Tore Jonassen passed away on February 04, 2018 and the authors dedicate this manuscript to his memory.


  1. 1.
    Al Islam AA, Alam SI, Raghunathan V, Bagchi S (2012) Multi-armed bandit congestion control in multi-hop infrastructure wireless mesh networks. In: IEEE 20th international symposium on modeling, analysis & simulation of computer and telecommunication systems (MASCOTS), IEEE, pp 31–40Google Scholar
  2. 2.
    Benveniste A, Priouret P, Métivier M (1990) Adaptive algorithms and stochastic approximations. Springer, New YorkCrossRefzbMATHGoogle Scholar
  3. 3.
    Black PE (2004) Fractional knapsack problem Dictionary of algorithms and data structuresGoogle Scholar
  4. 4.
    Ghavipour M, Meybodi MR (2017) Trust propagation algorithm based on learning automata for inferring local trust in online social networks. Knowl-Based Syst 33(1):3–20Google Scholar
  5. 5.
    Granmo O-C, Oommen BJ (2010) Optimal sampling for estimation with constrained resources using a learning automaton-based solution for the nonlinear fractional knapsack problem. Appl Intell 33(1):3–20CrossRefGoogle Scholar
  6. 6.
    Granmo O-C, Oommen BJ (2010) Solving stochastic nonlinear resource allocation problems using a hierarchy of twofold resource allocation automata. IEEE Trans Comput 59(4):545–560MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Granmo O-C, Oommen BJ, Myrer SA, Olsen MG (2007) Learning automata-based solutions to the nonlinear fractional knapsack problem with applications to optimal resource allocation. IEEE Trans Syst Man Cybern B Cybern 37(1):166–175CrossRefGoogle Scholar
  8. 8.
    Liu K, Zhao Q, Swami A (2013) Dynamic probing for intrusion detection under resource constraints. In: Proceedings of IEEE international conference on communications, ICC. Budapest, Hungary, June 9-13, 2013, pp 1980–1984Google Scholar
  9. 9.
    Ma Z, Wang H, Shi K, Wang X (2018) Learning automata based caching for efficient data access in delay tolerant networks. Wirel Commun Mob Comput 2018(2018):1–19Google Scholar
  10. 10.
    Malboubi M, Wang L, Chuah C-N, Sharma P (2014) Intelligent sdn based traffic (de) aggregation and measurement paradigm (istamp). In: Proceedings IEEE INFOCOM. IEEE, 2014, pp 934–942Google Scholar
  11. 11.
    Narendra KS, Thathachar MAL (2012) Learning automata: an introduction. Courier CorporationGoogle Scholar
  12. 12.
    Nicopolitidis P, Papadimitriou GI, Pomportsis AS (2003) Learning automata-based polling protocols for wireless lans. IEEE Trans Commun 51(3):453–463CrossRefGoogle Scholar
  13. 13.
    Nicopolitidis P, Papadimitriou GI, Pomportsis AS (2004) Distributed protocols for ad hoc wireless lans: a learning-automata-based approach. Ad Hoc Netw 2(4):419–431CrossRefGoogle Scholar
  14. 14.
    Obaidat MS, Papadimitriou GI, Pomportsis AS (2001) An efficient adaptive bus arbitration scheme for scalable shared-medium atm switch. Comput Commun 24(9):790–797CrossRefGoogle Scholar
  15. 15.
    Obaidat MS, Papadimitriou GI, Pomportsis AS, Laskaridis H (2002) Learning automata-based bus arbitration for shared-medium atm switches. IEEE Trans Syst Man Cybern B Cybern 32(6):815–820CrossRefGoogle Scholar
  16. 16.
    Papadimitriou GI, Pomportsis AS (2000) Dynamic bandwidth allocation in wdm passive star networks with asymmetric traffic. Photon Netw Commun 2(4):383–391CrossRefGoogle Scholar
  17. 17.
    Papadimitriou GI, Pomportsis AS (2000) Learning-automata-based tdma protocols for broadcast communication systems with bursty traffic. IEEE Commun Lett 4(3):107–109CrossRefGoogle Scholar
  18. 18.
    Papadimitriou GI, Pomportsis AS (2000) On the use of learning automata in medium access control of single-hop lightwave networks. Comput Commun 23(9):783–792CrossRefGoogle Scholar
  19. 19.
    Rezvanian A, Meybodi MR (2017) Sampling algorithms for stochastic graphs: a learning automata approach. Knowl-Based Syst 127:126–144CrossRefGoogle Scholar
  20. 20.
    Rezvanian A, Saghiri AM, Vahidipour SM, Esnaashari M, Meybodi MR (2018) Recent advances in learning automata, vol 754. Springer, BerlinCrossRefzbMATHGoogle Scholar
  21. 21.
    Rezvanian A, Vahidipour SM, Esnaashari M (2018) New applications of learning automata-based techniques in real-world environments. J Comput Sci 24:287–289CrossRefGoogle Scholar
  22. 22.
    Saghiri AM, Meybodi MR (2018) Open asynchronous dynamic cellular learning automata and its application to allocation hub location problem. Knowl-Based Syst 139:149–169CrossRefGoogle Scholar
  23. 23.
    Seyyedi SH, Minaei-Bidgoli B (2018) Estimator learning automata for feature subset selection in high-dimensional spaces, case study: email spam detection. Int J Commun Syst 31(8):31:e3541CrossRefGoogle Scholar
  24. 24.
    Tsetlin ML (1973) Automaton theory and modeling of biological systems. Academic Press, New YorkzbMATHGoogle Scholar
  25. 25.
    Yazidi A, Hammer H (2018) Solving stochastic nonlinear resource allocation problems using continuous learning automata. Applied Intelligence 48(11):4392–4411CrossRefGoogle Scholar
  26. 26.
    Yazidi A, Jonassen TM, Herrera-Viedma E (2018) An aggregation approach for solving the non-linear fractional equality knapsack problem. Expert Systems with Applications 110:323–334CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceOslo Metropolitan UniversityOsloNorway

Personalised recommendations