Skip to main content
SpringerLink
Log in

Design and calibration of a DRL algorithm for solving the job shop scheduling problem under unexpected job arrivals

  • Published:
Flexible Services and Manufacturing Journal Aims and scope Submit manuscript

Abstract

This paper proposes a Deep Reinforcement Learning(DRL)—based approach to solve the real-time Job Shop Scheduling Problem (JSSP) facing unexpected job arrivals. The approach combines the use of a DRL algorithm, Proximal Policy Optimization Actor and Critic (PPO-AC) algorithm, with an event-driven rescheduling strategy for solving a bi-objective decision problem. PPO-AC models an agent in interaction with its environment, aiming to achieve a predefined goal by maximizing the total cumulative reward. In this work, the total cumulative reward is designed as the opposite of the optimization objective function, which is expressed as the weighted sum of generated schedule completion time (efficiency criterion), and deviation from an initially generated schedule (stability criterion). The agent minimizes the objective function by maximizing the total cumulative reward. To the best of our knowledge, no prior work focused on scheduling stability while using DRL algorithms. Graph Neural Network (GNN) architecture is exploited to model environment states, enhancing the approach’s adaptability. Training experiments are conducted to calibrate the algorithm. A sensitivity analysis is conducted on the deviation weight parameter to evaluate its variation impact on the proposed model. Results indicate that the proposed model is robust to such variation. For a fixed deviation weight value, the algorithm is compared to CP-optimizer, IBM constraint programming method, and a Mixed Integer Program, to assess its performance. Results reveal that for arriving small-job batches, PPO-AC succeeds in solving the problem in real-time with low gaps to the optimal solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Code availability

Code available in the following Github repository: https://github.com/Nour0602/DRL_ADD_JSSP.

References

  • Adibi M, Zandieh M, Amiri M (2010) Multi-objective scheduling of dynamic job shop using variable neighborhood search. Expert Syst Appl 37(1):282–287

    Article  Google Scholar 

  • Agnetis A, Murgia G, Sbrilli S (2014) A job shop scheduling problem with human operators in handicraft production. Int J Prod Res 52(13):3820–3831

    Article  Google Scholar 

  • Baer S, Bakakeu J, Meyes R, Meisen T (2019) Multi-agent reinforcement learning for job shop scheduling in flexible manufacturing systems. In: 2019 second international conference on artificial intelligence for industries (AI4I). IEEE, pp 22–25

  • Bagheri Rad N, Behnamian J (2023) Multi-objective collaborative job shop scheduling in a dynamic environment: non-dominated sorting memetic algorithm. J Ambient Intell Humaniz Comput 14(3):2657–2671

    Article  Google Scholar 

  • Baldea M, Harjunkoski I (2014) Integrated production scheduling and process control: a systematic review. Comput Chem Eng 71:377–390

    Article  Google Scholar 

  • Beck JC, Feng T, Watson JP (2011) Combining constraint programming and local search for job-shop scheduling. INFORMS J Comput 23(1):1–14

    Article  MathSciNet  Google Scholar 

  • Błażewicz J, Pesch E, Sterna M (2000) The disjunctive graph machine representation of the job shop scheduling problem. Eur J Oper Res 127(2):317–331

    Article  Google Scholar 

  • Boukedroun M, Duvivier D, Ait-el Cadi A, Poirriez V, Abbas M (2023) A hybrid genetic algorithm for stochastic job-shop scheduling problems. RAIRO Oper Res 57(4):1617–1645

    Article  MathSciNet  Google Scholar 

  • Da Col, G, Teppan EC (2019) Industrial size job shop scheduling tackled by present day cp solvers. In: Principles and practice of constraint programming: 25th international conference, CP 2019, Stamford, CT, USA, September 30–October 4, 2019, Proceedings 25. Springer, pp 144–160

  • Dolgui A, Ivanov D, Sethi SP, Sokolov B (2019) Scheduling in production, supply chain and industry 4.0 systems by optimal control: fundamentals, state-of-the-art and applications. Int J Prod Res 57(2):411–432

    Article  Google Scholar 

  • Dong YH, Jang J (2012) Production rescheduling for machine breakdown at a job shop. Int J Prod Res 50(10):2681–2691

    Article  Google Scholar 

  • Frihat M, Hadj-Alouane AB, Sadfi C (2022) Optimization of the integrated problem of employee timetabling and job shop scheduling. Comput Oper Res 137:105332

    Article  MathSciNet  Google Scholar 

  • Garey MR, Johnson DS, Sethi R (1976) The complexity of flowshop and jobshop scheduling. Math Oper Res 1(2):117–129

    Article  MathSciNet  Google Scholar 

  • Ghaleb M, Zolfagharinia H, Taghipour S (2020) Real-time production scheduling in the industry-4.0 context: addressing uncertainties in job arrivals and machine breakdowns. Comput Oper Res 123:105031

    Article  MathSciNet  Google Scholar 

  • Hammami NEH, Lardeux B, Hadj-Alouane AB, Jridi M (2022) Job shop scheduling: a novel drl approach for continuous schedule-generation facing real-time job arrivals. IFAC-PapersOnLine 55(10):2493–2498

    Article  Google Scholar 

  • Han BA, Yang JJ (2020) Research on adaptive job shop scheduling problems based on dueling double dqn. IEEE Access 8:186474–186495

    Article  Google Scholar 

  • Herrmann JW (2006) Rescheduling strategies, policies, and methods. Handbook of production scheduling, pp 135–148

  • Huang JP, Gao L, Li XY, Zhang CJ (2023) A cooperative hierarchical deep reinforcement learning based multi-agent method for distributed job shop scheduling problem with random job arrivals. Comput Ind Eng 109650

  • Larsen R, Pranzo M (2019) A framework for dynamic rescheduling problems. Int J Prod Res 57(1):16–33

    Article  Google Scholar 

  • Lei K, Guo P, Wang Y, Zhang J, Meng X, Qian L (2023) Large-scale dynamic scheduling for flexible job-shop with random arrivals of new jobs by hierarchical reinforcement learning. IEEE Trans Ind Inform

  • Lei K, Guo P, Zhao W, Wang Y, Qian L, Meng X, Tang L (2022) A multi-action deep reinforcement learning framework for flexible job-shop scheduling problem. Expert Syst Appl 117796

  • Liang Z, Liu M, Zhong P, Zhang C (2023) Application research of a new neighbourhood structure with adaptive genetic algorithm for job shop scheduling problem. Int J Prod Res 61(2):362–381

    Article  Google Scholar 

  • Liu CL, Chang CC, Tseng CJ (2020) Actor-critic deep reinforcement learning for solving job shop scheduling problems. IEEE Access 8:71752–71762

    Article  Google Scholar 

  • Liu SQ, Ong H, Ng KM (2005) Metaheuristics for minimizing the makespan of the dynamic shop scheduling problem. Adv Eng Soft 36(3):199–205

    Article  Google Scholar 

  • Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: International conference on machine learning. PMLR, pp 1928–1937

  • Niu S, Ong S, Nee A (2012) An improved intelligent water drops algorithm for achieving optimal job-shop scheduling solutions. Int J Prod Res 50(15):4192–4205

    Article  Google Scholar 

  • Ojha VK, Abraham A, Snášel V (2017) Metaheuristic design of feedforward neural networks: a review of two decades of research. Eng Appl Artif Intell 60:97–116

    Article  Google Scholar 

  • Ouelhadj D, Petrovic S (2009) A survey of dynamic scheduling in manufacturing systems. J Sched 12(4):417–431

    Article  MathSciNet  Google Scholar 

  • Park J, Chun J, Kim SH, Kim Y, Park J (2021) Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning. Int J Prod Res 59(11):3360–3377

    Article  Google Scholar 

  • Pfeiffer A, Kádár B, Monostori L (2007) Stability-oriented evaluation of rescheduling strategies, by using simulation. Comput Ind 58(7):630–643

    Article  Google Scholar 

  • Rangsaritratsamee R, Ferrell WG Jr, Kurz MB (2004) Dynamic rescheduling that simultaneously considers efficiency and stability. Comput Ind Eng 46(1):1–15

    Article  Google Scholar 

  • Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

  • Song L, Eldin NN (2012) Adaptive real-time tracking and simulation of heavy construction operations for look-ahead scheduling. Autom Constr 27:32–39

    Article  Google Scholar 

  • Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press, Cambridge

    Google Scholar 

  • Tassel P, Gebser M, Schekotihin K (2021) A reinforcement learning environment for job-shop scheduling. arXiv preprint arXiv:2104.03760

  • Tighazoui A, Sauvey C, Sauer N (2020) New efficiency-stability criterion in a rescheduling problem with dynamic jobs weights. In: 2020 7th international conference on control, decision and information technologies (CoDIT). IEEE, Vol. 1, pp 475–480

  • van Hezewijk L, Dellaert N, Van Woensel T, Gademann N (2022) Using the proximal policy optimisation algorithm for solving the stochastic capacitated lot sizing problem. Int J Prod Res 1–24

  • Wang H, Sarker BR, Li J, Li J (2021) Adaptive scheduling for assembly job shop with uncertain assembly times based on dual q-learning. Int J Prod Res 59(19):5867–5883

    Article  Google Scholar 

  • Wang L, Hu X, Wang Y, Xu S, Ma S, Yang K, Liu Z, Wang W (2021) Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning. Comput Netw 190:107969

    Article  Google Scholar 

  • Wang S, Li X, Liu Q (2023) An effective neighborhood solution clipping method for large-scale job shop scheduling problem. CMES-Comput Model Eng Sci 137(2)

  • Wang YF (2020) Adaptive job shop scheduling strategy based on weighted q-learning algorithm. J Intell Manuf 31(2):417–432

    Article  Google Scholar 

  • Wang Z, Zhang J, Yang S (2019) An improved particle swarm optimization algorithm for dynamic job shop scheduling problems with random job arrivals. Swarm Evolut Comput 51:100594

    Article  Google Scholar 

  • Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24

    Article  MathSciNet  Google Scholar 

  • Yang Z, Bi L, Jiao X (2023) Combining reinforcement learning algorithms with graph neural networks to solve dynamic job shop scheduling problems. Processes 11(5):1571

    Article  Google Scholar 

  • Zhang C, Song W, Cao Z, Zhang J, Tan PS, Chi X (2020) Learning to dispatch for job shop scheduling via deep reinforcement learning. Adv Neural Inf Process Syst 33:1621–1632

    Google Scholar 

  • Zhang J, Ding G, Zou Y, Qin S, Fu J (2019) Review of job shop scheduling research and its new perspectives under industry 4.0. J Intell Manuf 30(4):1809–1830

    Article  Google Scholar 

  • Zhang Y, Zhu H, Tang D, Zhou T, Gui Y (2022) Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems. Robot Comput Integr Manuf 78:102412

    Article  Google Scholar 

  • Zhao M, Gao L, Li X (2019) A random forest-based job shop rescheduling decision model with machine failures. J Ambient Intell Humaniz Comput 1–11

  • Zheng P, Wang J, Zhang J, Yang C, Jin Y (2019) An adaptive cgan/irf-based rescheduling strategy for aircraft parts remanufacturing system under dynamic environment. Robot Comput Integr Manuf 58:230–238

    Article  Google Scholar 

Download references

Funding

This work was supported by L@bISEN Laboratory of ISEN Yncréa Ouest and the Ministry of Higher Education and Scientific Research of Tunisia.

Author information

Authors and Affiliations

  1. LR-OASIS, University of Tunis El Manar, Tunis, Tunisia

    Nour El Houda Hammami & Atidel B. Hadj-Alouane

  2. L@bISEN, Vision-AD, ISEN Yncréa Ouest, 33 Quatre Chemin du Champ de Manoeuvre, 44470, Carquefou, France

    Nour El Houda Hammami, Benoit Lardeux & Maher Jridi

  3. College of Engineering, University of Sharjah, Sharjah, United Arab Emirates

    Atidel B. Hadj-Alouane

Authors
  1. Nour El Houda Hammami
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benoit Lardeux
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Atidel B. Hadj-Alouane
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maher Jridi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nour El Houda Hammami.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Consent to participate

The authors consent to submit this paper in this journal, and that they obtained consent from the responsible authorities at the institute/organization where the work has been carried out

Consent for publication

The authors obtained consent from the responsible authorities at the institute/organization where the work has been carried out.

Ethical approval

The authors declare compliance with the Ethical Standards required by this journal.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hammami, N.E.H., Lardeux, B., B. Hadj-Alouane, A. et al. Design and calibration of a DRL algorithm for solving the job shop scheduling problem under unexpected job arrivals. Flex Serv Manuf J (2024). https://doi.org/10.1007/s10696-024-09540-2

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10696-024-09540-2

Keywords

Search

Navigation