Skip to main content

Zone scheduling optimization of pumps in water distribution networks with deep reinforcement learning and knowledge-assisted learning


This article studies the pump scheduling optimization problem in water distribution networks (WDNs) through a novel algorithm that combines knowledge learning and deep reinforcement learning. The optimization problem is modeled as a Markov decision process by taking three objectives in pressure management into consideration. Knowledge-assisted learning is incorporated into the reinforcement learning framework (KA-RL) to help evaluate the state value and guide the design of reward function, since the proposed KA-RL framework leverages the notion that historical data of WDNs could be utilized to produce optimal trajectories with respect to parametric variations. The knowledge-assisted proximal policy optimization (KA-PPO) algorithm, which only uses nodal pressure data, is proposed based on the KA-RL framework to address the arbitrary WDN topology and time-varying water demand. The effectiveness and applicability of the proposed algorithm are illustrated by virtue of the 22-node network with two pumps in a pump station. Empirical results demonstrate that KA-PPO works well in practice and compares favorably to Nelder–Mead method.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. Abdallah M, Kapelan Z (2019) Fast pump scheduling method for optimum energy cost and water quality in water distribution networks with fixed and variable speed pumps. J Water Resour Plann Manag 145(12):04019055

    Article  Google Scholar 

  2. Barán B, Von Lücken C, Sotelo A (2005) Multi-objective pump scheduling optimisation using evolutionary strategies. Adv Eng Softw 36(1):39–47

    Article  Google Scholar 

  3. Bazrafshan M, Gatsis N, Giacomoni M, Taha A (2018) A fixed-point iteration for steady-state analysis of water distribution networks. In: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), IEEE, pp 880–884

  4. Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38(8):716

    MathSciNet  Article  Google Scholar 

  5. Bene J, Selek I, Hős C (2013) Comparison of deterministic and heuristic optimization solvers for water network scheduling problems. Water Sci Technol Water Supply 13(5):1367–1376

    Article  Google Scholar 

  6. Broad D, Maier HR, Dandy GC (2010) Optimal operation of complex water distribution systems using metamodels. J Water Resour Plann Manag 136(4):433–443

    Article  Google Scholar 

  7. Guo G, Liu S, Wu Y, Li J, Zhou R, Zhu X (2018) Short-term water demand forecast based on deep learning method. J Water Resour Plann Manag 144(12):04018076

    Article  Google Scholar 

  8. Hajgató G, Paál G, Gyires-Tóth B (2020) Deep reinforcement learning for real-time optimization of pumps in water distribution systems. J Water Resour Plann Manag 146(11):04020079

    Article  Google Scholar 

  9. Hashemi SS, Tabesh M, Ataeekia B (2014) Ant-colony optimization of pumping schedule to minimize the energy cost using variable-speed pumps in water distribution networks. Urban Water J 11(5):335–347

    Article  Google Scholar 

  10. Kougias IP, Theodossiou NP (2013) Multiobjective pump scheduling optimization using harmony search algorithm (hsa) and polyphonic hsa. Water Resour Manag 27(5):1249–1261

    Article  Google Scholar 

  11. Levine S, Finn C, Darrell T, Abbeel P (2016) End-to-end training of deep visuomotor policies. The J Mach Learn Res 17(1):1334–1373

    MathSciNet  MATH  Google Scholar 

  12. Liu D, Li S, Zheng Y, Wu J (2019) Distributed zone mpc of pressure management for water distribution network systems. IET Control Theory Appl 13(11):1704–1717

    MathSciNet  Article  Google Scholar 

  13. Liu D, Zheng Y, Wu J, Li S (2020) Zone model predictive control for pressure management of water distribution network. Asian J Control 22(4):1522–1536

    MathSciNet  Article  Google Scholar 

  14. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Article  Google Scholar 

  15. Mala-Jetmarova H, Sultanova N, Savic D (2017) Lost in optimisation of water distribution systems? A literature review of system operation. Environ Modell Softw 93:209–254

    Article  Google Scholar 

  16. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  17. Odan FK, Ribeiro Reis LF, Kapelan Z (2015) Real-time multiobjective optimization of operation of water supply systems. J Water Resour Plann Manag 141(9):04015011

    Article  Google Scholar 

  18. Reddy T, RM SP, Parimala M, Chowdhary CL, Hakak S, Khan WZ, et al (2020) A deep neural networks based model for uninterrupted marine environment monitoring. Comput Commun 157:64–75

  19. Ritter J, Corzo G, Solomatine DP, Angarita H (2020) Multiobjective direct policy search using physically based operating rules in multireservoir systems. J Water Resour Plann Manag 146(4):05020002

    Article  Google Scholar 

  20. Rossman L (1993) Epanet–an advanced water quality modeling package for distribution systems. In: Proceeding management and regulations, american water works annual conference, pp 41–418

  21. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms

  22. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  23. Singh MK, Kekatos V (2020) On the flow problem in water distribution networks: uniqueness and solvers. IEEE Trans Control Netw Syst

  24. Stokes CS, Maier HR, Simpson AR (2015) Water distribution system pumping operational greenhouse gas emissions minimization by considering time-dependent emissions factors. J Water Resour Plann Manag 141(7):04014088

    Article  Google Scholar 

  25. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT press, Cambridge

    MATH  Google Scholar 

  26. Walski TM, Brill ED Jr, Gessler J, Goulter IC, Jeppson RM, Lansey K, Lee HL, Liebman JC, Mays L, Morgan DR et al (1987) Battle of the network models: epilogue. J Water Resour Plan Manag 113(2):191–203

    Article  Google Scholar 

  27. Wang S, Taha A, Gatsis N, Giacomoni M (2020) Receding horizon control for drinking water networks: the case for geometric programming. IEEE Trans Control Netw Syst

  28. Wang Y, Ocampo-Martinez C, Puig V (2016) Stochastic model predictive control based on gaussian processes applied to drinking water networks. IET Control Theory Appl 10(8):947–955

    MathSciNet  Article  Google Scholar 

  29. Wang Y, Puig V, Cembrano G (2017) Non-linear economic model predictive control of water distribution networks. J Process Control 56:23–34

    Article  Google Scholar 

  30. Ye D, Chen G, Zhang W, Chen S, Yuan B, Liu B, Chen J, Liu Z, Qiu F, Yu H, et al. (2020) Towards playing full moba games with deep reinforcement learning. Adv Neural Inform Process Syst 33

  31. Zhao H, Zhao J, Qiu J, Liang G, Dong ZY (2020) Cooperative wind farm control with deep reinforcement learning and knowledge assisted learning. IEEE Trans Ind Inform

  32. Zhu Y, Mottaghi R, Kolve E, Lim JJ, Gupta A, Fei-Fei L, Farhadi A (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 3357–3364

Download references


This work is supported by National Natural Science Foundation of China (61633019).

Author information



Corresponding author

Correspondence to Jingcheng Wang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interests regarding the publication of this paper.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Data Availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xu, J., Wang, H., Rao, J. et al. Zone scheduling optimization of pumps in water distribution networks with deep reinforcement learning and knowledge-assisted learning. Soft Comput 25, 14757–14767 (2021).

Download citation


  • Deep reinforcement learning
  • Optimal control
  • Knowledge-assisted learning
  • Pump schedule
  • Water distribution networks