Reinforcement Learning Techniques for the Control of WasteWater Treatment Plants

  • Felix Hernandez-del-Olmo
  • Elena Gaudioso
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6687)


Since water pollution is one of the most serious environmental problems today, control of wastewater treatment plants (WWTPs) is a crucial issue nowadays and stricter standards for the operation of WWTPs have been imposed by authorities. One of the main problems in the automation of the control of Wastewater Treatment Plants (WWTPs) appears when the control system does not respond as it should because of changes on influent load or flow. Thus, it is desirable the development of autonomous systems that learn from interaction with a WWTP and that can operate taking into account changing environmental circumstances. In this paper we present an intelligent agent using reinforcement learning for the oxygen control in the N-Ammonia removal process in the well known Benchmark Simulation Model no.1 (BSM1). The aim of the approach presented in this paper is to minimize the operation cost changing the set-points of the control system autonomously.


Expert System Wastewater Treatment Plant Reinforcement Learning Markov Decision Process Fuzzy Neural Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alex, J., Benedetti, L., Copp, J., Gernaey, K.V., Jeppsson, U., Nopens, I., Pons, M.N., Rosen, C., Steyer, J.P., Vanrolleghem, P.: Benchmark Simulation Model no. 1 (BSM1). In: IWA Taskgroup on Benchmarking of Control Strategies for WWTP, p. 61 (April 2008)Google Scholar
  2. 2.
    Belanche, L., Valdes, J.J., Comas, J., Roda, I.R., Poch, M.: Prediction of the bulking phenomenon in wastewater treatment plants. Artificial Intelligence in Engineering 14(4), 307–317 (2000)CrossRefGoogle Scholar
  3. 3.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. 2. Athenas Scientific (2007)Google Scholar
  4. 4.
    Carrasco, E.F., Rodriguez, J., Punyal, A., Roca, E., Lema, J.M.: Rule-based diagnosis and supervision of a pilot-scale wastewater treatment plant using fuzzy logic techniques. Expert Systems with Applications 22(1), 11–20 (2002)CrossRefGoogle Scholar
  5. 5.
    Carrasco, E.F., Rodriguez, J., Punyal, A., Roca, E., Lema, J.M.: Diagnosis of acidification states in an anaerobic wastewater treatment plant using a fuzzy-based expert system. Control Engineering Practice 12(1), 59–64 (2004)CrossRefGoogle Scholar
  6. 6.
    Chen, J.C., Chang, N.B., Shieh, W.K.: Assessing wastewater reclamation potential by neural network model. Engineering Applications of Artificial Intelligence 16(2), 149–157 (2003)CrossRefGoogle Scholar
  7. 7.
    Copp, J.: The COST Simulation Benchmark: Description and Simulator Manual, p. 154, Office for Official Publications of the European Community, Luxembourg (2002); ISBN: 92-894-1658-0Google Scholar
  8. 8.
    Cortes, U., Sanchez-Marre, M., Ceccaroni, L., R-Roda, I., Poch, M.: Artificial intelligence and environmental decision support systems. Applied Intelligence 13(1), 77–91 (2000)CrossRefGoogle Scholar
  9. 9.
    Huang, M., Wan, J., Ma, Y., Li, W., Sun, X., Wan, Y.: A fast predicting neural fuzzy model for on-line estimation of nutrient dynamics in an anoxic/oxic process. Bioresource Technology 101(6), 1642–1651 (2010)CrossRefGoogle Scholar
  10. 10.
    Huang, M., Wan, J., Yongwen, M., Yan, W., Weijiang, L., Xiaofei, S.: Control rules of aeration in a submerged biofilm wastewater treatment process using fuzzy neural networks. Expert Systems with Applications 36(7), 10428–10437 (2009)CrossRefGoogle Scholar
  11. 11.
    Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)Google Scholar
  12. 12.
    Liao, S.: Expert system methodologies and applications: a decade review from 1995 to 2004. Expert Systems with Appications 28, 93–103 (2005)CrossRefGoogle Scholar
  13. 13.
    Olsson, G., Nielsen, M., Yuan, Z., Lynggaard-Jensen, A., Steyer, J.P.: Instrumentation, Control and Automation in Wastewater Systems. IWA Publishing (2005)Google Scholar
  14. 14.
    Samuelsson, P., Halvarsson, B., Carlsson, B.: Cost-efficient operation of a denitrifying activated sludge process. Water Research 41, 2325–2332 (2007)CrossRefGoogle Scholar
  15. 15.
    Stare, A., Vrecko, D., Hvala, N., Strmcnik, S.: Comparison of control strategies for nitrogen removal in activated sludge process in terms of operating costs: A simulation study. Water Research 41, 2004–2014 (2007)CrossRefGoogle Scholar
  16. 16.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  17. 17.
    Syafiie, S., Tadeo, F., Martinez, E.: Model-free learning control of neutralization processes using reinforcement learning. Engineering Applications of Artificial Intelligence 20(6), 767–782 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Felix Hernandez-del-Olmo
    • 1
  • Elena Gaudioso
    • 1
  1. 1.Artificial Intelligence DepartmentE.T.S.I. Informatica, UNEDSpain

Personalised recommendations