Journal of Intelligent Manufacturing

, Volume 23, Issue 6, pp 2513–2529 | Cite as

Dynamic scheduling for multi-site companies: a decisional approach based on reinforcement multi-agent learning

  • N. AissaniEmail author
  • A. Bekrar
  • D. Trentesaux
  • B. Beldjilali


In recent years, most companies have resorted to multi-site or supply-chain organization in order to improve their competitiveness and adapt to existing real conditions. In this article, a model for adaptive scheduling in multi-site companies is proposed. To do this, a multi-agent approach is adopted in which intelligent agents have reactive learning capabilities based on reinforcement learning. This reactive learning technique allows the agents to make accurate short-term decisions and to adapt these decisions to environmental fluctuations. The proposed model is implemented on a 3-tier architecture that ensures the security of the data exchanged between the various company sites. The proposed approach is compared to a genetic algorithm and a mixed integer linear program algorithm to prove its feasibility and especially, its reactivity. Experimentations on a real case study demonstrate the applicability and the effectiveness of the model in terms of both optimality and reactivity.


Production control Scheduling Multi-agent system Reinforcement learning Multi-site company 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aissani N., Trentesaux D., Beldjilali B. (2008) Use of machine learning for continuous improvement of the real time manufacturing control system performances. International Journal of Industrial System Engineering 3(4): 474–497CrossRefGoogle Scholar
  2. Aissani, N., Beldjilali, B., & Trentesaux, D. (2008b). Efficient and effective reactive scheduling of manufacturing system using SARSA-multi-objective-agents. In Proceedings of the 7th international conference MOSIM, Paris, pp. 698–707.Google Scholar
  3. Aissani N., Trentesaux D., Beldjilali B. (2009) Dynamic scheduling of maintenance tasks in the petroleum industry: A reinforcement approach. EAAI: Engineering Applications of Artificial Intelligence 22: 1089–1103CrossRefGoogle Scholar
  4. Ait Si Larbi, E. Y., Aissani, N., & Beldjilali, B. (2008). Un Modèle de Planification pour les Entreprises Multi Sites basé sur les Systèmes Multi Agents et les Algorithmes Génétiques. In Proceedings of the 10th maghrebian conference on information technologies, Oran, Algeria, pp. 506–511.Google Scholar
  5. Bousbia S., Trentesaux D. (2002) Self-organization in distributed manufacturing control: State-of-the-art and future trends. IEEE International Conference on Systems, Man & Cybernetics 5: 6CrossRefGoogle Scholar
  6. Brucker P., Schlie R. (1990) Job shop scheduling with multi-purpose machines. Computing 45: 369–375CrossRefGoogle Scholar
  7. Brandimarte P. (1993) Routing and scheduling in a flexible job shop by tabu search. Annals of Operations Research 41: 157–183CrossRefGoogle Scholar
  8. Chuin Lau H., Agussurja L., Thangarajoo R. (2008) Real-time supply chain control via multi-agent adjustable autonomy. Computers and Operations Research 35(11): 3452–3464CrossRefGoogle Scholar
  9. Conway R. W., Maxwell W. L., Miller L. W. (1967) Theory of scheduling. Addison-Wesley, Reading, MAGoogle Scholar
  10. Dabbene, F., Gay, P., Tortia, C., & Sacco, N. (2005). Optimization of fresh–food supply chains in uncertain environments: An application to the meat-refrigeration process, decision and control, European Control Conference. CDC-ECC;05. 44th IEEE Conference on Volume 12, Issue 15, pp. 2077–2082.Google Scholar
  11. Dauzere-Peres S., & Paulli, J. (1994). Solving the general job-shop scheduling problem. Management. Report Series, vol. 182. Erasmus University Rotterdam, Rotterdam School of Management, Rotterdam.Google Scholar
  12. Dauzère-Pérès S., Paulli J. (1997) An integrated approach for modeling and solving the general multiprocessor job-shop scheduling problem using tabu search. Annals of Operations Resarch 70: 281–306CrossRefGoogle Scholar
  13. Dongbing G., Yang E. (2007) Fuzzy policy reinforcement learning in cooperative multi-robot systems. Journal of Intelligent and Robotic Systems 48(1): 7–22CrossRefGoogle Scholar
  14. Fattahi P., Saidi Mehrabad M., Jolai F. (2007) Mathematical modelling and heuristic approaches to flexible job shop scheduling problems. Journal of Intelligent and Manufacturing 18: 331–342CrossRefGoogle Scholar
  15. Ferber, J., & Gutknecht, O. (1998). A meta-model for the analysis and design of organizations in multi-agent systems. In Proceedings ICMAS’98, pp. 128–135.Google Scholar
  16. Fontan, G., Merce, C., & Erschler, J. (2001). La planification des flux de production, Performance industrielle et gestion des flux, Hermes Lavoisier, Traité IC2 Information-Commande-Communication, 2001, Chap 3, pp. 69–112.Google Scholar
  17. Galliano, D., & Soulie, N. (2007). Organisational and spatial determinants of the multiunit firm: Evidence from the French indus, Cahiers du GRES, No 17.Google Scholar
  18. Gao J., Gen M., Sun L., Zhao X. (2007) A hybrid of genetic algorithm and bottleneck shifting for multiobjective flexible job shop scheduling problems. Computers and Industrial Engineering 53: 149–162CrossRefGoogle Scholar
  19. Goldberg, D. E., & Lingle, R. (1985). Alleles, loci and the traveling salesman problem. In Proceedings of the first international conference on Genetic Algorithms, pp. 10–19.Google Scholar
  20. Haruno, M., & Kawato, M. (2006). Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: FMRI examination in stimulus-action-reward association learning. Neural Networks, 19(Special Issue), 1242–1254.Google Scholar
  21. Holland J. H. (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor, ILGoogle Scholar
  22. Hong, G., Chuan, L., Zhang, L., & Xianming, Z. (2007) Study on supply chain optimization scheduling of networked manufacturing, ACOS’07: In Proceedings of WSEAS: 6th international conference on applied computer science, Vol 6.Google Scholar
  23. IBM. (2010). IBM ILOG CPLEX optimizer, high performance mathematical optimization engines.
  24. James, T. L., Brown, E. C., & Keeling, K. B. (2007). A hybrid grouping genetic algorithm for the cell formation problem. Computers and operations research. No 34, pp. 2059–2079.Google Scholar
  25. Kacem I., Hammadi S., Borne P. (2002) Approach by localization and multiobjective evolutionary optimization for flexible job shop scheduling problems. IEEE Transactions on Systems, Man and Cybernetics, Part C 32(1): 408–419CrossRefGoogle Scholar
  26. Katalinic, B., & Kordic, V. (2004). Bionic assembly system: Concept, structure and function. In Proceedings of the 5th IDMME, Bath, UK.Google Scholar
  27. Lee H.L., Rosenblatt J. (1986) A generalized quantity discount pricing model to increase supplier’s profits. Management Science 33(9): 1167–1185Google Scholar
  28. Leitao P., Restivo F. (2008) A holonic approach to dynamic manufacturing scheduling. Robotics and Computer-Integrated Manufacturing 24: 625–634CrossRefGoogle Scholar
  29. Marquès, G., Lamothe, J., Thierry, C., & Gourc, D. (2009). A supply chain performance analysis of a pull inspired supply strategy faced to demande uncertainties. Journal of Intel Manufacturing, 20(6). doi: 10.1007/s10845-009-0337-z.
  30. Mastrolilli M., Gambardella L. M. (2000) Effective neighbourhood functions for the flexible job shop problem. Journal of Scheduling 3: 3–20CrossRefGoogle Scholar
  31. Mati Y., Rezg N., Xie X.L. (2001) Geometric approach and taboo search for scheduling flexible manufacturing systems. IEEE Transactions on Robotics and Automation 17: 805–818CrossRefGoogle Scholar
  32. Mati, Y., Lahlou, C., & Dauzère-Pérès, S.(2010). Modelling and solving a practical flexible job-shop scheduling problem with blocking constraints. International Journal of Production Research, 1366-588X, First published on 23 Sep. 2010.Google Scholar
  33. McCulloch W. S. (1945) A heterarchy of values determined by the topology of nervous nets. Bulletin of Mathematical Biology 7: 89–93Google Scholar
  34. Miles, R. E., & Snow, C. C. (1992). Managing 21st century network organisations. Organizational dynamics, winter session.Google Scholar
  35. Mintzberg, H. (1980). Structure in 5’s: A synthesis of the research. Organization Design Management Science, 26(3).Google Scholar
  36. Monahan J. P. (1984) A quantity pricing model to increase vendor profits. Management Science 30(6): 720–726CrossRefGoogle Scholar
  37. Monostori L., Csáji B. Cs., Kádár B. (2004) Adaptation and learning in distributed production control. CIRP Annals-Manufacturing Technology 53(1): 349–352CrossRefGoogle Scholar
  38. Mourtzis, D., Papakostas, N., Makris, S., Xanthakis, V., & Chryssolouris, G. (2008). Supply chain modeling and control for producing highly customized products. General assembly of CIRP No58, Manchester, UK (24/08/2008), Vol. 57, No 1, p. 588.Google Scholar
  39. Narasimhan R., Mahapatra S. (2004) Decision models in global supply chain management. Industrial Marketing Management 33: 21–27CrossRefGoogle Scholar
  40. Ouhimmou M., D’Amours S., Beauregard R., Ait-Kadi D., Singh Chauhan S. (2008) Furniture supply chain tactical planning optimization using a time decomposition approach. European Journal of Operational Research 189(3): 952–970CrossRefGoogle Scholar
  41. Ounnar, F., & Pujo, P. (2009). Pull control for job shop: Holonic manufacturing system approach using multicriteria decision-making. Journal of Intelligent Manufacturing. doi: 10.1007/s10845-009-0288-4.
  42. Pezzella, F., Morganti, G., & Ciaschetti, G. (2008). A genetic algorithm for the flexible job-shop scheduling problem. Computers & Operations Research, 35(10), Oct. 2008.Google Scholar
  43. Prabhu V. V. (2003) Stability and fault adaptation in distributed control of heterarchical manufacturing job shops. IEEE Transactions on Robotics and Automation 19(1): 142–147CrossRefGoogle Scholar
  44. Rummery, G., & Niranjan, M. (1994). On-line q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166. Cambridge University, Engineering Department.Google Scholar
  45. Russell, S., & Norvig, P. (1995). Artificial intelligence: A modern approach, the intelligent agent book. Prentice Hall Series in Artificial Intelligence.Google Scholar
  46. Sallez Y., Berger T., Trentesaux D. (2009) A stigmergic approach for dynamic routing of active products in FMS. Computers in Industry 60(3): 204–216CrossRefGoogle Scholar
  47. Sauer, J., Freese, T., & Teschke, T. (2000). Towards agent-based multi-site scheduling. ECAI 2000 workshop on new results in planning, scheduling and design, pp. 123–130.Google Scholar
  48. Silva, N., Sousa, P., & Ramos, C. (1998). A holonic manufacturing system implementation. Advanced Summer Institute(ASI’98). Bremen, Germany; 14–17 June 1998.Google Scholar
  49. Swaminathan, J., Smith, S., & Sadeh-Koniecpol, N. (1997). Modeling supply chain dynamics: A multiagent approach. Decision Sciences.Google Scholar
  50. Takadama, K., & Fujita, H. (2004). Lessons learned from comparison between Q-learning and sarsa agents in bargaining game. In North American association for computational social and organizational science (NAACSOS 2004), June 27–29, Pittsburgh, PA.Google Scholar
  51. Tang C. S. (1990) The impact of uncertainty on a production line. Management Science 36(12): 1518–1531CrossRefGoogle Scholar
  52. Tarantilis C. D. (2008) Topics in real-time supply chain management. Computers & Operations Research 35(11): 3393–3396CrossRefGoogle Scholar
  53. TehraniNik Nejad H., Sugimura N., Iwamura K., Tanimizu Y. (2008) Multi agent architecture for dynamic incremental process planning in the flexible manufacturing system. Journal of Intelligent Manufacturing 21(4): 487–499CrossRefGoogle Scholar
  54. Thierry, C. (2003). gestion de chaînes logistiques, modèles et mise en œuvre pour l’aide à la décision à moyen terme. HDR thesis, Toulouse 2 university.Google Scholar
  55. Trentesaux D., Pesin P., Tahon C. (2000) Distributed artificial intelligence for FMS scheduling, control and design support. Journal of Intelligent Manufacturing 11: 573–589CrossRefGoogle Scholar
  56. Trentesaux D. (2009) Distributed control of production systems. Engineering Applications of Artificial Intelligence 22(7): 971–978CrossRefGoogle Scholar
  57. Tsai J.-F. (2007) An optimization approach for supply chain management models with quantity discount policy. European Jounal of Operational Research 177(2): 982–994CrossRefGoogle Scholar
  58. Vaario J., Ueda K. (1998) An emergent modelling method for dynamic scheduling. Journal of Intelligent Manufacturing 9: 129–140CrossRefGoogle Scholar
  59. Van Brussel H., Wyns J., Valckenaers P., Bongaerts L., Peeters L. (1998) Reference architecture for holonic manufacturing systems: PROSA. Computers in Industry 37(3): 255–274CrossRefGoogle Scholar
  60. Vilcot, G. (2007). Algorithmes approchés pour des problèmes d’ordonnancement multicritère de type job shop flexible et job shop multiressource, PhD thesis, University of François-Rabelais, Tours, France.Google Scholar
  61. Voss S., Woodruff D. L. (2006) Introduction to computational optimization models for production planning in supply chain. Springer, BerlinGoogle Scholar
  62. Watkins, C. J. C. H. (1989). Learning from delayed rewards, PhD thesis, Cambridge University, Cambridge, England.Google Scholar
  63. Weng Z. K., Wong R. T. (1993) General models for the supplier’s all-unit quantity discount policy. Naval Research Logistics 40(6): 971–991CrossRefGoogle Scholar
  64. Yang P. C. (2004) Pricing strategy for deteriorating items using quantity discount when demand is price sensitive. European Journal of Operational Research 157: 389–397CrossRefGoogle Scholar
  65. Zambrano, G., Aissani, N., Pach, C., Berger, T., & Trentesaux, D. (2011). An approach for temporal myopia reduction in heterarchical control architectures. In Proceedings of 20th IEEE inter symposium on industrial electronics, 27–30 June 2011.Google Scholar
  66. Zbib, N., Pach, C., Sallez, Y., & Trentesaux, D. (2010). Heterarchical production control in manufacturing systems using the potential fields concept. Journal of Intelligent Manufacturing. doi: 10.1007/s10845-010-0467-3.
  67. Zhang H., Gen M. (2005) Multistage-based genetic algorithm for flexible job-shop scheduling problem. Journal of Complexity International 11: 223–232Google Scholar
  68. Zobolas G. I., Tarantilis C. D., Ioannou G. (2008) Exact, Heuristic and Meta-heuristic Algorithms for Solving Shop Scheduling Problems, Studies in Computational Intelligence (SCI). Springer, Berlin, pp 1–40Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • N. Aissani
    • 1
    Email author
  • A. Bekrar
    • 2
    • 3
  • D. Trentesaux
    • 2
    • 3
  • B. Beldjilali
    • 1
  1. 1.LIO, Department of Computer SciencesUniversity of OranOranAlgeria
  2. 2.University Lille Nord de FranceLilleFrance
  3. 3.UVHC, TEMPO Lab.ValenciennesFrance

Personalised recommendations