Autonomous Agents and Multi-Agent Systems

, Volume 31, Issue 6, pp 1578–1609 | Cite as

When do agents outperform centralized algorithms?

A systematic empirical evaluation in logistics
  • Rinde R. S. van Lon
  • Tom Holvoet


Multi-agent systems (MAS) literature often assumes decentralized MAS to be especially suited for dynamic and large scale problems. In operational research, however, the prevailing paradigm is the use of centralized algorithms. Present paper empirically evaluates whether a multi-agent system can outperform a centralized algorithm in dynamic and large scale logistics problems. This evaluation is novel in three aspects: (1) to ensure fairness both implementations are subject to the same constraints with respect to hardware resources and software limitations, (2) the implementations are systematically evaluated with varying problem properties, and (3) all code is open source, facilitating reproduction and extension of the experiments. Existing work lacks a systematic evaluation of centralized versus decentralized paradigms due to the absence of a real-time logistics simulator with support for both paradigms and a dataset of problem instances with varying properties. We extended an existing logistics simulator to be able to perform real-time experiments and we use a recent dataset of dynamic pickup-and-delivery problem with time windows instances with varying levels of dynamism, urgency, and scale. The OptaPlanner constraint satisfaction solver is used in a centralized way to compute a global schedule and used as part of a decentralized MAS based on the dynamic contract-net protocol (DynCNET) algorithm. The experiments show that the DynCNET MAS finds solutions with a relatively lower operating cost when a problem has all following three properties: medium to high dynamism, high urgency, and medium to large scale. In these circumstances, the centralized algorithm finds solutions with an average cost of 112.3% of the solutions found by the MAS. However, averaged over all scenario types, the average cost of the centralized algorithm is 94.2%. The results indicate that the MAS performs best on very urgent problems that are medium to large scale.


Multi-agent systems Agents Centralized Decentralized Empirical Evaluation Dynamism Urgency Scale Operational research Logistics 



This research is partially funded by the Research Fund KU Leuven.


  1. 1.
    Wooldridge, M. (2002). An introduction to multiagent systems. New York: Wiley.Google Scholar
  2. 2.
    Weiss, G. (1999). Multiagent systems: A modern approach to distributed artificial intelligence. Cambridge: MIT press.Google Scholar
  3. 3.
    Pěchouček, M., & Mařík, V. (2008). Industrial deployment of multi-agent technologies: Review and selected case studies. Autonomous Agents and Multi-Agent Systems, 17(3), 397–431. doi: 10.1007/s10458-008-9050-0. (ISSN 1573-7454).CrossRefGoogle Scholar
  4. 4.
    Weyns, D., Schelfthout, K., Holvoet, T., & Lefever, T. (2005). Decentralized control of e’gv transportation systems. In Proceedings of the fourth international joint conference on autonomous agents and multiagent systems, AAMAS ’05 (pp 67–74), New York, NY, USA, ACM. ISBN 1-59593-093-0. doi: 10.1145/1082473.1082806.
  5. 5.
    Dorer, K., & Calisti, M. (2005). An adaptive solution to dynamic transport optimization. In Proceedings of the fourth international joint conference on autonomous agents and multiagent systems, AAMAS ’05, (pp. 45–51), New York, NY, USA, 2005. ACM. ISBN 1-59593-093-0. doi: 10.1145/1082473.1082803.
  6. 6.
    Berbeglia, G., Cordeau, J.-F., & Laporte, G. (2010). Dynamic pickup and delivery problems. European Journal of Operational Research, 202(1), 8–15. doi: 10.1016/j.ejor.2009.04.024. (ISSN 03772217).CrossRefMATHGoogle Scholar
  7. 7.
    Fischer, K., Müller, J. P., & Pischel, M. (1995). A model for cooperative transportation scheduling. In Proceedings of the 1st international conference on multiagent systems (ICMAS’95) (pp. 109–116), San Francisco.Google Scholar
  8. 8.
    Smith, R. G. (1980). The contract net protocol: High-level communication and control in a distributed problem solver. IEEE Transactions on Computers, 29(12), 1104–1113. doi: 10.1109/TC.1980.1675516. (ISSN 0018-9340).CrossRefGoogle Scholar
  9. 9.
    Mes, M., van der Heijden, M., & van Harten, A. (2007). Comparison of agent-based scheduling to look-ahead heuristics for real-time transportation problems. European Journal of Operational Research, 181(1), 59–75. doi: 10.1016/j.ejor.2006.02.051. (ISSN 0377-2217).MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Máhr, T., Srour, J. F., de Weerdt, M., & Zuidwijk, R. (2008). Agent performance in vehicle routing when the only thing certain is uncertainty. In Proceedings of 7th international conference on autonomous agents and multiagent systems (AAMAS), Estorial, Portugal.Google Scholar
  11. 11.
    van Lon, R. R. S., & Holvoet, T. (2015). Towards systematic evaluation of multi-agent systems in large scale and dynamic logistics. In Q. Chen, P. Torroni, S. Villata, J. Hsu & A. Omicini (Eds.), PRIMA 2015: Principles and practice of multi-agent systems: 18th international conference, Bertinoro, Italy, October 26–30, 2015, Proceedings (pp. 248–264). Springer International Publishing, Cham, ISBN 978-3-319-25524-8. doi: 10.1007/978-3-319-25524-8_16.
  12. 12.
    Máhr, T., Srour, J. F., de Weerdt, M., & Zuidwijk, R. (2010). Can agents measure up? A comparative study of an agent-based and on-line optimization approach for a drayage problem with uncertainty. Transportation Research: Part C, 18(1), 99–119. doi: 10.1016/j.trc.2009.04.018.CrossRefGoogle Scholar
  13. 13.
    Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012). The case for open computer programs. Nature, 482(7386), 485–488. doi: 10.1038/nature10836. (ISSN 1476-4687).CrossRefGoogle Scholar
  14. 14.
    van Lon, R. R. S., & Holvoet, T. (2013). Evolved multi-agent systems and thorough evaluation are necessary for scalable logistics. In 2013 IEEE workshop on computational intelligence in production and logistics systems (CIPLS) (pp. 48–53). doi: 10.1109/CIPLS.2013.6595199.
  15. 15.
    Pillac, V., Gendreau, M., Gueret, C., & Medaglia, A. L. (2013). A review of dynamic vehicle routing problems. European Journal of Operational Research, 225(1), 1–11. doi: 10.1016/j.ejor.2012.08.015. (ISSN 0377-2217).MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Gendreau, M., Guertin, F., Potvin, J.-Y., & Séguin, R. (2006). Neighborhood search heuristics for a dynamic vehicle dispatching problem with pick-ups and deliveries. Transportation Research Part C: Emerging Technologies, 14(3), 157–174. doi: 10.1016/j.trc.2006.03.002. (ISSN 0968090X).CrossRefGoogle Scholar
  17. 17.
    Madsen, O. B. G., Ravn, H. F., & Rygaard, J. M. (1995). A heuristic algorithm for a dial-a-ride problem with time windows, multiple capacities, and multiple objectives. Annals of Operations Research, 60(1), 193–208. doi: 10.1007/BF02031946. (ISSN 02545330).CrossRefMATHGoogle Scholar
  18. 18.
    Yang, J., Jaillet, P., & Mahmassani, H. (2004). Real-time multivehicle truckload pickup and delivery problems. Transportation Science, 38(2), 135–148. doi: 10.1287/trsc.1030.0068. (ISSN 0041-1655).CrossRefGoogle Scholar
  19. 19.
    van Lon, R. R. S., Ferrante, E., Turgut, A. E., Wenseleers, T., Berghe, G. V., & Holvoet, T. (2016). Measures of dynamism and urgency in logistics. European Journal of Operational Research, 253(3), 614–624. doi: 10.1016/j.ejor.2016.03.021. (ISSN 0377-2217).MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    van Lon, R. R. S. & Holvoet, T. (2012). RinSim: A simulator for collective adaptive systems in transportation and logistics. In Proceedings of the 6th IEEE international conference on self-adaptive and self-organizing systems (SASO 2012) (pp. 231–232), Lyon, France, doi: 10.1109/SASO.2012.41.
  21. 21.
    Law, A. M. (2007). Simulation modeling and analysis (4th ed.). New York: McGraw-Hill.Google Scholar
  22. 22.
    Preisler, T., Dethlefs, T., & Renz, W. (2015). Data-adaptive simulation: Cooperativeness of users in bike-sharing systems. In W. Kersten, T. Blecker, & C. M. Ringle (Eds.), Innovations and strategies for logistics and supply chains (pp. 1765–1772). epubli GmbH.Google Scholar
  23. 23.
    Preisler, T., Dethlefs, T., & Renz, W. (2016). Self-organizing redistribution of bicycles in a bike-sharing system based on decentralized control. In Federated conference on computer science and information systems (Vol. 8, pp. 1471–1480). ACSIS, 2016. doi: 10.15439/2016F126.
  24. 24.
    Merlevede, J., van Lon, R. R. S., & Holvoet, T. (2014). Neuroevolution of a multi-agent system for the dynamic pickup and delivery problem. In International joint workshop on optimisation in multi-agent systems and distributed constraint reasoning (co-located with AAMAS).Google Scholar
  25. 25.
    van Lon, R. R. S., Holvoet, T., Berghe, G. V., Wenseleers, T., & Branke, J. (2012). Evolutionary synthesis of multi-agent systems for dynamic dial-a-ride problems. In GECCO companion ’12 proceedings of the fourteenth international conference on genetic and evolutionary computation conference companion (pp. 331–336), Philadelphia, USA, ISBN 9781450311786. doi: 10.1145/2330784.2330832.
  26. 26.
    Dinh, H. T., van Lon, R. R. S., & Holvoet, T. (2016). Multi-agent route planning using delegate MAS. In ICAPS Proceedings of the 4th workshop on distributed and multi-agent planning (DMAP-2016) (pp. 24–32).Google Scholar
  27. 27.
    De Smet G. et al. OptaPlanner user guide. Red Hat and the community. OptaPlanner is an open source constraint satisfaction solver in Java.
  28. 28.
    Shen, W., Hao, Q., Yoon, H. J., & Norrie, D. H. (2006). Applications of agent-based systems in intelligent manufacturing: An updated review. Advanced Engineering Informatics, 20(4), 415–431. doi: 10.1016/j.aei.2006.05.004. (ISSN 1474-0346).CrossRefGoogle Scholar
  29. 29.
    Weyns, D., Boucké, N., Holvoet, T., & Demarsin, B. (2007). DynCNET: A protocol for dynamic task assignment in multiagent systems. In First international conference on self-adaptive and self-organizing systems, SASO (pp. 281–284). doi: 10.1109/SASO.2007.20.
  30. 30.
    van Lon, R. R. S. (2014a). When do agents outperform centralized algorithms? A systematic empirical evaluation in logistics—datasets and results v1.1.0, May 2017. doi: 10.5281/zenodo.576345.
  31. 31.
    van Lon, R. R. S. (2016a). RinSim v4.3.0, December 2016. doi: 10.5281/zenodo.192106.
  32. 32.
    van Lon, R. R. S. (2017b). RinLog v3.2.2, May 2017. doi: 10.5281/zenodo.571180.
  33. 33.
    van Lon, R. R. S. (2016b). PDPTW dataset dataset: v1.1.0, August 2016. doi: 10.5281/zenodo.59259.
  34. 34.
    van Lon, R. R. S. (2017c). When do agents outperform centralized algorithms? A systematic empirical evaluation in logistics—code v1.1.0, May 2017. doi: 10.5281/zenodo.576389.
  35. 35.
    Holvoet, T., Weyns, D., & Valckenaers, P. (2009). Patterns of delegate mas. In 2009 Third IEEE international conference on self-adaptive and self-organizing systems (pp. 1–9). doi: 10.1109/SASO.2009.31.
  36. 36.
    van Lon, R. R. S., Branke, J., & Holvoet, T. (2017). Optimizing agents with genetic programming: An evaluation of hyper-heuristics in dynamic real-time logistics. Genetic programming and evolvable machines (pp. 1–28). US: Springer. ISSN 1573-7632. doi: 10.1007/s10710-017-9300-5.

Copyright information

© The Author(s) 2017

Authors and Affiliations

  1. 1.imec-DistriNetKU LeuvenLeuvenBelgium

Personalised recommendations