Skip to main content
SpringerLink
Log in

Dissolved oxygen concentration control in wastewater treatment process based on reinforcement learning

  • Article
  • Published:
Science China Technological Sciences Aims and scope Submit manuscript

Abstract

In this article, the dissolved oxygen (DO) concentration control problem in wastewater treatment process (WWTP) is studied. Unlike existing control strategies that control DO concentration at a fixed value, here we develop a different control framework. Under the proposed control framework, an intelligent control method of DO concentration based on reinforcement learning (RL) algorithm is presented to resolve the DO concentration control problem. By using the deep deterministic policy gradient (DDPG) algorithm, the DO concentration of the fifth tank in the activated sludge reactor can be adjusted dynamically. In addition, by designing two different reward functions and by analysing the relationships among effluent quality, energy consumption, and DO concentration, the target of energy-saving and emission-reducing is achieved. The simulation results indicate that the designed control method can reduce energy consumption while ensuring that the effluent quality meet the specified standards.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Löwenberg J, Zenker A, Baggenstos M, et al. Comparison of two PAC/UF processes for the removal of micropollutants from wastewater treatment plant effluent: Process performance and removal efficiency. Water Res, 2014, 56: 26–36

    Article  Google Scholar 

  2. Von Borstel Luna F D, de la Rosa Aguilar E, Suarez Naranjo J, et al. Robotic system for automation of water quality monitoring and feeding in aquaculture shadehouse. IEEE Trans Syst Man Cybern Syst, 2016, 47: 1575–1589

    Article  Google Scholar 

  3. Hou Y, Wu Y L, Liu Z, et al. Dynamic multi-objective differential evolution algorithm based on the information of evolution progress. Sci China Tech Sci, 2021, 64: 1676–1689

    Article  Google Scholar 

  4. Xie Y B, Wang D, Qiao J F. Dynamic multi-objective intelligent optimal control toward wastewater treatment processes. Sci China Tech Sci, 2022, 65: 569–580

    Article  Google Scholar 

  5. Wahab N A, Katebi R, Balderud J. Multivariable PID control design for activated sludge process with nitrification and denitrification. Biochem Eng J, 2009, 45: 239–248

    Article  Google Scholar 

  6. Du S, Yan Q, Qiao J. Event-triggered PID control for wastewater treatment plants. J Water Process Eng, 2020, 38: 101659

    Article  Google Scholar 

  7. Vrečko D, Hvala N, Stare A, et al. Improvement of ammonia removal in activated sludge process with feedforward-feedback aeration controllers. Water Sci Tech, 2006, 53: 125–132

    Article  Google Scholar 

  8. Song X, Zhao Y, Song Z, et al. Dissolved oxygen control in wastewater treatment based on robust PID controller. Int J Modell Ident Contrl, 2012, 15: 297–303

    Article  Google Scholar 

  9. Ferrer J, Rodrigo M A, Seco A, et al. Energy saving in the aeration process by fuzzy logic control. Water Sci Tech, 1998, 38: 209–217

    Article  Google Scholar 

  10. Zhu G, Peng Y, Ma B, et al. Optimization of anoxic/oxic step feeding activated sludge process with fuzzy control model for improving nitrogen removal. Chem Eng J, 2009, 151: 195–201

    Article  Google Scholar 

  11. Peng Y, Zeng W, Wang S. DO concentration as a fuzzy control parameter for organic substrate removal in SBR processes. Environ Eng Sci, 2004, 21: 606–616

    Article  Google Scholar 

  12. Han H G, Li Y, Qiao J F. A fuzzy neural network approach for online fault detection in waste water treatment process. Comput Electrical Eng, 2014, 40: 2216–2226

    Article  Google Scholar 

  13. Santín I, Barbu M, Pedret C, et al. Fuzzy logic for plant-wide control of biological wastewater treatment process including greenhouse gas emissions. ISA Trans, 2018, 77: 146–166

    Article  Google Scholar 

  14. Han H G, Wu X L, Qiao J F. Real-time model predictive control using a self-organizing neural network. IEEE Trans Neural Netw Learn Syst, 2013, 24: 1425–1436

    Article  Google Scholar 

  15. Han H G, Zhang J C, Du S L, et al. Robust optimal control for anaerobic-anoxic-oxic reactors. Sci China Tech Sci, 2021, 64: 1485–1499

    Article  Google Scholar 

  16. Ruan Z W, Yang Q M, Ge S S, et al. Adaptive fuzzy fault tolerant control of uncertain mimo nonlinear systems with output constraints and unknown control directions. IEEE Trans Fuzzy Syst, 2021, 30: 1224–1238

    Article  Google Scholar 

  17. Cao W, Yang Q. Online sequential extreme learning machine based adaptive control for wastewater treatment plant. Neurocomputing, 2020, 408: 169–175

    Article  Google Scholar 

  18. Han H G, Qian H H, Qiao J F. Nonlinear multiobjective model-predictive control scheme for wastewater treatment process. J Process Control, 2014, 24: 47–59

    Article  Google Scholar 

  19. Han H G, Fu S J, Sun H Y, et al. Hierarchical nonlinear model predictive control with multi-time-scale for wastewater treatment process. J Process Control, 2021, 108: 125–135

    Article  Google Scholar 

  20. Zhang A, Yin X, Liu S, et al. Distributed economic model predictive control of wastewater treatment plants. Chem Eng Res Des, 2019, 141: 144–155

    Article  Google Scholar 

  21. Han H, Liu Z, Hou Y, et al. Data-driven multiobjective predictive control for wastewater treatment process. IEEE Trans Ind Inf, 2020, 16: 2767–2775

    Article  Google Scholar 

  22. Silver D, Huang A, Maddison C J, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529: 484–489

    Article  Google Scholar 

  23. Syafiie S, Tadeo F, Martinez E, et al. Model-free control based on reinforcement learning for a wastewater treatment problem. Appl Soft Comput, 2011, 11: 73–82

    Article  Google Scholar 

  24. Yang Q, Cao W, Meng W, et al. Reinforcement-learning-based tracking control of waste water treatment process under realistic system conditions and control performance requirements. IEEE Trans Syst Man Cybern Syst, 2022, 52: 5284–5294

    Article  Google Scholar 

  25. Filipe J, Bessa R J, Reis M, et al. Data-driven predictive energy optimization in a wastewater pumping station. Appl Energy, 2019, 252: 113423

    Article  Google Scholar 

  26. Alves Goulart D, Dutra Pereira R. Autonomous pH control by reinforcement learning for electroplating industry wastewater. Comput Chem Eng, 2020, 140: 106909

    Article  Google Scholar 

  27. Chen K, Wang H, Valverde-Peirez B, et al. Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning. Chemosphere, 2021, 279: 130498

    Article  Google Scholar 

  28. Alex J, Benedetti L, Copp J B, et al. Benchmark simulation model no. 1 (BSM1). IWA Taskgroup on benchmarking of control strategies for WWTPs. Lund University, 2008

  29. Qin S, Cheng L. A real-time tracking controller for piezoelectric actuators based on reinforcement learning and inverse compensation. Sustain Cities Soc, 2021, 69: 102822

    Article  Google Scholar 

  30. Yang R, Wang D, Qiao J. Policy gradient adaptive critic design with dynamic prioritized experience replay for wastewater treatment process control. IEEE Trans Ind Inf, 2022, 18: 3150–3158

    Article  Google Scholar 

  31. Dutta D, Upreti S R. A survey and comparative evaluation of actor-critic methods in process control. Can J Chem Eng, 2022, 100: 2028–2056

    Article  Google Scholar 

  32. Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning. arXiv: 1509.02971

  33. Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518: 529–533

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence and Automation, Beijing University of Technology, Beijing, 100124, China

    ShengLi Du, PeiXi Chen, HongGui Han & JunFei Qiao

  2. Engineering Research Center of Intelligent Perception and Autonomous Control, Ministry of Education, Beijing, 100124, China

    ShengLi Du, PeiXi Chen, HongGui Han & JunFei Qiao

  3. Beijing Laboratory of Smart Environmental Protection, Beijing, 100124, China

    ShengLi Du, PeiXi Chen, HongGui Han & JunFei Qiao

Authors
  1. ShengLi Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. PeiXi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. HongGui Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. JunFei Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ShengLi Du.

Additional information

This work was supported by the National Natural Science Foundation of China (Grant No. 62173009) and the National Key Research and Development Program of China (Grant No. 2021ZD0112302).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, S., Chen, P., Han, H. et al. Dissolved oxygen concentration control in wastewater treatment process based on reinforcement learning. Sci. China Technol. Sci. 66, 2549–2560 (2023). https://doi.org/10.1007/s11431-022-2403-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11431-022-2403-8

Keywords

Search

Navigation