Coordination Model with Reinforcement Learning for Ensuring Reliable On-Demand Services in Collective Adaptive Systems

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11246)


Context-aware and pervasive systems are growing in the market segments. This is due to the expansion of Internet of things (IoT) devices. Current solutions rely on centralized services provided by servers gathering all requests and performing pre-defined computations involving pre-defined devices. Large-scale IoT scenarios, involving adaptation and unanticipated devices, call for alternative solutions. We propose here a new type of services, built and composed on-demand, arising from the interaction of multiple sensors and devices working together as a decentralized collective adaptive system. Our solution relies on a bio-inspired coordination model providing a communication platform among multi-agent systems working on behalf of these devices. Each device provides few simple services and data regarding its environment. On-demand services derive from the collective interactions among multiple sensors and devices. In this article, we investigate the design and implementation of such services and define a new approach that combines coordination model and reinforcement learning, in order to ensure reliable services and expected quality of services (QoS), namely convergence of composition, of coherent result and convergence of learning. We present an IoT scenario showing the feasibility of the approach and preliminary results.


Reliable services Coordination model Collective adaptive system Bio-inspired systems On-demand services Multi-agent learning Reinforcement Learning 


  1. 1.
    Algirdas, A., Laprie, J.-C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. IEEE Computer Society (2004)Google Scholar
  2. 2.
    Banicescu, I., Ciorba, F.M., Srivastava, S.: Performance optimization of scientific applications using an autonomic computing approach. In: Scalable Computing: Theory and Practice, pp. 437–466. Wiley (2013)Google Scholar
  3. 3.
    Baresi, L., Ghezzi, C., Guinea, S.: Smart monitors for composed services. In: Proceedings of the 2nd International Conference on Service Oriented Computing, ICSOC 2004, pp. 193–202. ACM, New York (2004)Google Scholar
  4. 4.
    Beal, J., Bachrach, J.: Infrastructure for engineered emergence on sensor/actuator networks. IEEE Intell. Syst. 21, 10–19 (2006)CrossRefGoogle Scholar
  5. 5.
    Boulmier, A., Banicescu, I., Ciorba, F.M., Abdennadher, N.: An autonomic approach for the selection of robust dynamic loop scheduling techniques. In: 16th International Symposium on Parallel and Distributed Computing, ISPDC 2017, Innsbruck, Austria, 3–6 July 2017, pp. 9–17 (2017)Google Scholar
  6. 6.
    Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent reinforcement learning: an overview. In: Srinivasan, D., Jain, L.C. (eds.) Innovations in Multi-Agent Systems and Applications-1. SCI, vol. 310, pp. 183–221. Springer, Heidelbeg (2010). Scholar
  7. 7.
    Canfora, G., Di Penta, M., Esposito, R., Villani, M.L.: An approach for QoS-aware service composition based on genetic algorithms. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, GECCO 2005, pp. 1069–1075. ACM, New York (2005)Google Scholar
  8. 8.
    Ciatto, G., Mariani, S., Louvel, M., Omicini, A., Zambonelli, F.: Twenty years of coordination technologies state-of-the-art and perspectives. COORDINATION 2018. LNCS, vol. 10852. Springer, Cham (2018). Scholar
  9. 9.
    Cook, P.R.: Limitations and extensions of the WoLF-PHC algorithm (2007)Google Scholar
  10. 10.
    Cruz Torres, M.H., Holvoet, T.: Composite service adaptation: a QoS-driven approach. In: Proceedings of the 5th International Conference on COMmunication System softWAre and MiddlewaRE (COMSWARE 2011). ACM (2011)Google Scholar
  11. 11.
    Fernandez-Marquez, J.L., Di Marzo Serugendo, G., Sara Montagna, Viroli, M., Arcos, J.L.: Description and composition of bio-inspired design patterns: a complete overview. Nat. Comput. 1–25 (2012)Google Scholar
  12. 12.
    Di Marzo Serugendo, G., De Angelis, F., Fernandez-Marquez, J.L.: Self-composition of services with chemical reactions. In: 29th Annual ACM Symposium on Applied Computing (SAC), Gyeongju, Republic of Korea, March 2014Google Scholar
  13. 13.
    De Angelis, F.L., Fernandez-Marquez, J.L., Di Marzo Serugendo, G.: Self-composition of services in pervasive systems: a chemical-inspired approach. In: Jezic, G., Kusek, M., Lovrek, I., J. Howlett, R., Jain, L.C. (eds.) Agent and Multi-Agent Systems: Technologies and Applications. AISC, vol. 296, pp. 37–46. Springer, Cham (2014). Scholar
  14. 14.
    Gelernter, D.: Generative communication in Linda. ACM Trans. Program. Lang. Syst. (TOPLAS) 7, 80–112 (1985)CrossRefGoogle Scholar
  15. 15.
    Boes, J., Nigon, J., Verstaevel, N., Gleizes, M.-P., Migeon, F.: The self-adaptive context learning pattern: overview and proposal. In: Christiansen, H., Stojanovic, I., Papadopoulos, G.A. (eds.) CONTEXT 2015. LNCS (LNAI), vol. 9405, pp. 91–104. Springer, Cham (2015). Scholar
  16. 16.
    Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)CrossRefGoogle Scholar
  17. 17.
    Mamei, M., Zambonelli, F.: Programming pervasive and mobile computing applications: the TOTA approach. ACM Trans. Softw. Eng. Methodol. 18(4), 1–56 (2009)CrossRefGoogle Scholar
  18. 18.
    Mazac, S., Armetta, F., Hassas, S.: Bootstrapping sensori-motor patterns for a constructivist learning system in continuous environments. In: 14th International Conference on the Synthesis and Simulation of Living Systems (Alife 2014), New York, NY, USA (2014)Google Scholar
  19. 19.
    McKinley, P.K., Sadjadi, S.M., Kasten, E.P., Cheng, B.H.C.: Composing adaptive software. Computer 37(7), 56–64 (2004)Google Scholar
  20. 20.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  21. 21.
    Montagna, S., Viroli, M., Pianini, D., Fernandez-Marquez, J.L.: Towards a comprehensive approach to spontaneous self-composition in pervasive ecosystems. In: De Paoli, F., Vizzari, V. (eds.) Proceedings of the 13th Workshop on Objects and Agents. CEUR-WS (2012)Google Scholar
  22. 22.
    Omicini, A., Zambonelli, F.: TuCSoN: a coordination model for mobile information agents. In: Internet Research: Electronic Networking Applications and Policy, pp. 59–79 (1999)Google Scholar
  23. 23.
    Panait, L., Luke, S.: Cooperative multi-agent learning : the state of the art. Auton. Agents Multi-Agent Syst. 11(3), 387–434 (2005)CrossRefGoogle Scholar
  24. 24.
    Peltz, C.: Web services orchestration and choreography. IEEE Comput. 36, 46–52 (2003)CrossRefGoogle Scholar
  25. 25.
    Rabanal, P., Mateo, J.A., Rodríguez, I., Díaz, G.: Data-aware automatic derivation of choreography-conforming systems of services. Comput. Stand. Interfaces 53, 59–79 (2017)CrossRefGoogle Scholar
  26. 26.
    Di Marzo Serugendo, G., Abdennadher, N., Mahfoudh, H.B., De Angelis, F.L., Tomaylla, R.: Spatial edge services. In: Global IoT Summit (2017)Google Scholar
  27. 27.
    Shi, W., Cao, J., Zhang, Q., Youhuizi, L., Xu, L.: Edge computing: Vision and challenges. IEEE (2016)Google Scholar
  28. 28.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  29. 29.
    Ter Beek, M., Bucchiarone, A., Gnesi, S.: Web service composition approaches: from industrial standards to formal methods. In: Proceedings of the Second International Conference on Internet and Web Applications and Services, ICIW 2007, p. 15. IEEE Computer Society, Washington, DC (2007)Google Scholar
  30. 30.
    Watkins, C.J.C.H.: Learning from Delayed Rewards. Ph.D. thesis, King’s College, Cambridge, UK, May 1989Google Scholar
  31. 31.
    Wu, Z., Ranabahu, A., Gomadam, K., Sheth, A.P., Miller, J.A.: Automatic composition of semantic web services using process and data mediation. In: Proceedings of the 9th International Conference on Enterprise Information Systems, pp. 453–461. Academic Press (2007)Google Scholar
  32. 32.
    Zambonelli, F.: Self-aware pervasive service ecosystems. Procedia Comput. Sci. 7, 197–199 (2011)CrossRefGoogle Scholar
  33. 33.
    Zambonelli, F., et al.: Developing pervasive multi-agent systems with nature-inspired coordination. Pervasive Mob. Comput. 17(Part B), 236–252 (2015). 10 years of Pervasive Computing’ In Honor of Chatschik BisdikianCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of GenevaGenevaSwitzerland
  2. 2.University of Applied Science of Western SwitzerlandGenevaSwitzerland

Personalised recommendations