Neural Computing and Applications

, Volume 31, Issue 3, pp 737–750 | Cite as

A fuzzy reinforcement learning approach to thermal unit commitment problem

  • Nandan Kumar NavinEmail author
  • Rajneesh Sharma
Original Article


Unit commitment problem (UCP) aims at optimizing generation cost for meeting a given load demand under several operational constraints. We propose to use fuzzy reinforcement learning (RL) approach for efficient and reliable solution to the unit commitment problem. In particular, we cast UCP as a multiagent fuzzy reinforcement learning task wherein individual generators act as players for optimizing the cost to meet a given load over a twenty-four-hour period. Unit commitment task has been fuzzified, and the most optimal unit commitment solution is generated by employing RL on this fuzzy multigenerator setup. Our proposed multiagent RL framework does not assume any a priori task or system knowledge, and the generators gradually learn to produce most optimal output solely based on their collective generation. We look at the UCP as a sequential decision-making task with reward/penalty to reduce the collective generation cost of generators. To the best of our knowledge, ours is a first attempt at solving UCP by employing fuzzy reinforcement learning. We test our approach on a ten-generating-unit system with several equality and inequality constraints. Simulation results and comparisons against several recent UCP solution methods prove superiority and viability of our proposed multiagent fuzzy reinforcement learning technique.


Unit commitment problem Fuzzy Q learning Economic load dispatch Multiagent fuzzy RL 

List of symbols


Output power of unit \(g\) at time h

\(FC_{g} (U_{g,h} )\)

Fuel cost of unit \(g\) when its output power is \(U_{g,h}\)

\(S_{\text{up}} (g,h)\)

Start-up cost of unit \(g\) at time \(h\)

\(S_{\text{down}} (g,h)\)

Shutdown cost of unit \(g\) at time \(h\)

\(HS_{\text{up}} (g,h)\)

Hot start-up cost of unit \(g\) at time \(h\)

\(CS_{\text{up}} (g,h)\)

Cold start-up cost of unit \(g\) at time \(h\)


ON/OFF state of unit \(g\) at time \(h\)

\(U_{g,\hbox{max} }\)

Maximum power generation of unit \(g\)

\(U_{g,\hbox{min} }\)

Minimum power generation of unit \(g\)


Continuously OFF time duration of unit \(g\) at time \(h\)


Continuously ON time duration of unit \(g\) at time \(h\)


Minimum uptime of unit \(g\)


Minimum downtime of unit \(g\)


Cold start hours of unit \(g\)

\(a_{g} ,b_{g} ,c_{g}\)

Fuel cost coefficients of unit \(g\)


System demand at time \(h\)


Spinning reserve at time \(h\)


Discount factor


Learning rate parameter


Conflict of interest

This research work does not have any financial or non-financial interest from any funding agencies. This work has carried out at Advanced Power and Control Research Lab of NSIT Delhi.


  1. 1.
    Wood AJ, Wollenberg BF (2012) Power generation, operation, and control. 2nd edn. John WileyGoogle Scholar
  2. 2.
    Padhy NP (2004) Unit commitment—a bibliographical survey. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2003.821611 Google Scholar
  3. 3.
    Mukherjee S, Adrian EC (1989) Implementation of a lagrangian relaxation based unit commitment problem. IEEE Trans Power Syst. doi 10(1109/59):41687Google Scholar
  4. 4.
    Ongsakul W, Petcharaks N (2004) Unit commitment by enhanced adaptive Lagrangian relaxation. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2003.820707 Google Scholar
  5. 5.
    Chandram K, Subrahmanyam N, Sydulu M (2011) Unit commitment by improved pre-prepared power demand table and Muller method. Int J Electr Power Energy Syst. doi: 10.1016/j.ijepes.2010.06.022 Google Scholar
  6. 6.
    Hosseini SH, Khodaei A, Aminifar F (2007) A novel straightforward unit commitment method for large-scale power systems. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2007.907443 Google Scholar
  7. 7.
    Cheng C-P, Liu C-W, Liu C-C (2000) Unit commitment by Lagrangian relaxation and genetic algorithms. IEEE Trans, POWER Syst, p 15Google Scholar
  8. 8.
    Venkatesh B, Yu P, Gooi HB, Choling D (2008) Fuzzy MILP unit commitment incorporating wind generators. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2008.2004724 Google Scholar
  9. 9.
    Liang R-H, Kang F-C (2000) Thermal generating unit commitment using an extended mean field annealing neural network. IEE proc-Gener Transm Distrib 147(3):164–170. doi: 10.1049/ipgtd:20000303 Google Scholar
  10. 10.
    Zhuang Galiana Senior Member FF (1990) Unit commitment by simulated annealing. IEEE Trans, Power Syst, p 5Google Scholar
  11. 11.
    Mantawy AH, Abdel-Magid YL, Selim SZ (1998) Unit commitment by tabu search. IEE Proc - Gener Transm Distrib 145:56. doi: 10.1049/ip-gtd:19981681 CrossRefGoogle Scholar
  12. 12.
    Logenthiran T, Srinivasan D (2010) Particle swarm optimization for unit commitment problem. PMAS. 642–647Google Scholar
  13. 13.
    Juste KA, Kitu H, Tunaka E, Hasegawa J (1999) An evolutionary programming solution to the unit commitment problem. IEEE Trans Power Syst 14:1452–1459CrossRefGoogle Scholar
  14. 14.
    Sisworahardjo NS, El-Keib AA (2002) Unit Commitment Using the Ant Colony Search Algorithm.Large Eng. syst.Conf. Power Eng. 2-6Google Scholar
  15. 15.
    Patra S, Goswami SK, Goswami B (2008) Differential evolution algorithm for solving unit commitment with ramp constraints. Electr Power Components Syst 36:771–787. doi: 10.1080/15325000801911377 CrossRefGoogle Scholar
  16. 16.
    Eslamian M, Hosseinian SH, Vahidi B (2009) Bacterial foraging-based solution to the unit-commitment problem. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2009.2021216 Google Scholar
  17. 17.
    Ebrahimi J, Hosseinian SH, Gharehpetian GB (2011) Unit commitment problem solution using shuffled frog leaping algorithm. IEEE Trans Power Syst 26:573–581. doi: 10.1109/TPWRS.2010.2052639 CrossRefGoogle Scholar
  18. 18.
    Roy PK (2013) Solution of unit commitment problem using gravitational search algorithm. Int J Electr Power Energy Syst 53:85–94. doi: 10.1016/j.ijepes.2013.04.001 CrossRefGoogle Scholar
  19. 19.
    Roy PK, Sarkar R (2014) Solution of unit commitment problem using quasi-oppositional teaching learning based algorithm. Int J Electr Power Energy Syst 60:96–106. doi: 10.1016/j.ijepes.2014.02.008 CrossRefGoogle Scholar
  20. 20.
    Rameshkumar J, Ganesan S, Abirami M, Subramanian S (2016) Cost, emission and reserve pondered pre-dispatch of thermal power generating units coordinated with real coded grey wolf optimisation. IET Gener Trans Distribut 10(4):972–985. doi: 10.1049/iet-gtd.2015.0726 Google Scholar
  21. 21.
    Srinivasan D, Chazelas J (2004) A priority list-based evolutionary algorithm to solve large scale unit commitment problem. IEEE international conference on power system technology (PowerCon 2004) pp 21–24Google Scholar
  22. 22.
    Saberl AY, Senjyul T, Miyagil T (2006) Fuzzy unit commitment using absolutely stochastic simulated annealing. IEEE Trans Power Syst 21(2):955–964CrossRefGoogle Scholar
  23. 23.
    Zhao B, Guo CX, Bai BR, Cao YJ (2006) An improved particle swarm optimization algorithm for unit commitment. Int J Electr Power Energy Syst 28:482–490. doi: 10.1016/j.ijepes.2006.02.011 CrossRefGoogle Scholar
  24. 24.
    Lau TW, Chung CY, Wong KP et al (2009) Quantum-inspired evolutionary algorithm approach for unit commitment. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2009.2021220 Google Scholar
  25. 25.
    Jeong YW, Park JB, Jang SH, Lee KY (2010) A new quantum-inspired binary PSO: application to unit commitment problems for power systems. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2010.2042472 Google Scholar
  26. 26.
    Damousis IG, Bakirtzis AG, Dokopoulos PS (2004) A solution to the unit-commitment problem using integer-coded genetic algorithm. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2003.821625 Google Scholar
  27. 27.
    Datta D, Dutta S (2012) A binary-real-coded differential evolution for unit commitment problem. Int J Electr Power Energy Syst 42:517–524. doi: 10.1016/j.ijepes.2012.04.048 CrossRefGoogle Scholar
  28. 28.
    Yuan X, Su A, Nie H et al (2009) Application of enhanced discrete differential evolution approach to unit commitment problem. Energy Convers Manag 50:2449–2456. doi: 10.1016/j.enconman.2009.05.033 CrossRefGoogle Scholar
  29. 29.
    Chandrasekaran K, Simon SP, Padhy NP (2013) Binary real coded firefly algorithm for solving unit commitment problem. Inf Sci (Ny) 249:67–84. doi: 10.1016/j.ins.2013.06.022 CrossRefGoogle Scholar
  30. 30.
    Farsangi MM, Barati M (2014) Solving unit commitment problem by a binary shuffled frog leaping algorithm. IET Gener Transm Distrib 8:1050–1060. doi: 10.1049/iet-gtd.2013.0436 CrossRefGoogle Scholar
  31. 31.
    Wu Z, Chow TWS (2012) Binary neighbourhood field optimisation for unit commitment problems. doi: 10.1049/iet-gtd.2012.0096 Google Scholar
  32. 32.
    Han D, Jian J, Yang L (2014) Outer approximation and outer-inner approximation approaches for unit commitment problem. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2013.2253136 Google Scholar
  33. 33.
    Niknam T, Bavafa F, Azizipanah-Abarghooee R (2013) New self-adaptive bat-inspired algorithm for unit commitment problem. doi: 10.1049/iet-smt.2013.0252 Google Scholar
  34. 34.
    Quan R, Jian J, Yang L (2015) An improved priority list and neighborhood search method for unit commitment. Int J Electr Power Energy Syst 67:278–285. doi: 10.1016/j.ijepes.2014.11.025 CrossRefGoogle Scholar
  35. 35.
    Yuan X, Ji B, Zhang S et al (2014) A new approach for unit commitment problem via binary gravitational search algorithm. Appl Soft Comput 22:249–260. doi: 10.1016/j.asoc.2014.05.029 CrossRefGoogle Scholar
  36. 36.
    Chen PH (2012) Two-level hierarchical approach to unit commitment using expert system and elite PSO. IEEE Trans Power Syst. doi: 10.1109/TPWRS.2011.2171197 Google Scholar
  37. 37.
    Quan H, Srinivasan D, Khosravi A (2015) Incorporating wind power forecast uncertainties into stochastic unit commitment using neural network-based prediction intervals. IEEE Trans Neural Networks Learn Syst. doi: 10.1109/TNNLS.2014.2376696 MathSciNetGoogle Scholar
  38. 38.
    Xie Y-G, Chiang H-D (2010) A novel solution methodology for solving large-scale thermal unit commitment problems. Electr Power Components Syst 38:1615–1634. doi: 10.1080/15325008.2010.492453 CrossRefGoogle Scholar
  39. 39.
    Ahmed MH, Bhattacharya K, Salama MMA (2012) Stochastic unit commitment with wind generation penetration. Electr Power Components Syst 40:1405–1422. doi: 10.1080/15325008.2012.694969 CrossRefGoogle Scholar
  40. 40.
    Govardhan M, Roy R (2015) Economic analysis of unit commitment with distributed energy resources. Int J Electr Power Energy Syst 71:1–14. doi: 10.1016/j.ijepes.2015.01.028 CrossRefGoogle Scholar
  41. 41.
    Kamboj VK, Bath SK, Dhillon JS (2016) Implementation of hybrid harmony/random search algorithm considering ensemble and pitch violation for unit commitment problem. Int J Electr Power Energy Syst 77:228–249. doi: 10.1016/j.ijepes.2015.11.045 CrossRefGoogle Scholar
  42. 42.
    Mahdavi MS, Vahidi B, Babamalek G et al (2016) A novel optimized fuzzy approach based on monte carlo method for system load, wind turbine and photovoltaic unit uncertainty modeling in unit commitment. Electr Power Components Syst 44:833–842. doi: 10.1080/15325008.2016.1138343 CrossRefGoogle Scholar
  43. 43.
    Tavakoli A, Sanjari MJ, Karami H et al (2015) Imperialistic competitive algorithm based unit commitment considering risk of cascading blackout. Electr Power Components Syst 43:374–383. doi: 10.1080/15325008.2014.963261 CrossRefGoogle Scholar
  44. 44.
    Abedinia O, Naslian MD, Bekravi M (2014) A new stochastic search algorithm bundled honeybee mating for solving optimization problems. Neural Comput Appl 25:1921–1939. doi: 10.1007/s00521-014-1682-1 CrossRefGoogle Scholar
  45. 45.
    Kamboj VK (2016) A novel hybrid PSO???GWO approach for unit commitment problem. Neural Comput Appl 27:1643–1655. doi: 10.1007/s00521-015-1962-4 CrossRefGoogle Scholar
  46. 46.
    Al-Betar MA, Awadallah MA, Khader AT et al (2016) Economic load dispatch problems with valve-point loading using natural updated harmony search. Neural Comput Appl. doi: 10.1007/s00521-016-2611-2 Google Scholar
  47. 47.
    Li F-D, Wu M, He Y, Chen X (2012) Optimal control in microgrid using multi-agent reinforcement learning. ISA Trans 51:743–751. doi: 10.1016/j.isatra.2012.06.010 CrossRefGoogle Scholar
  48. 48.
    Boubertakh H, Tadjine M, Glorennec P-Y, Labiod S (2010) Tuning fuzzy PD and PI controllers using reinforcement learning. ISA Trans 49:543–551. doi: 10.1016/j.isatra.2010.05.005 CrossRefGoogle Scholar
  49. 49.
    Treesatayapun C (2008) Fuzzy-rule emulated networks, based on reinforcement learning for nonlinear discrete-time controllers. ISA Trans 47:362–373. doi: 10.1016/j.isatra.2008.07.001 CrossRefGoogle Scholar
  50. 50.
    Yen GG, Hickey TW (2004) Reinforcement learning algorithms for robotic navigation in dynamic environments. ISA Trans 43:217–230. doi: 10.1016/S0019-0578(07)60032-9 CrossRefGoogle Scholar
  51. 51.
    Wiering M, van Otterlo M (2012) Reinforcement learning: State-of-the-Art. Adaptation, Learning, and Optimization, vol 12. Springer, Berlin. doi: 10.1007/978-3-642-27645-3 CrossRefGoogle Scholar
  52. 52.
    Buoniu L, Babuška R, De Schutter B, Ernst D (2010) Reinforcement learning and dynamic programming using function approximators. vol 39. CRC pressGoogle Scholar
  53. 53.
    Jouffe L (1998) Fuzzy inference system learning by reinforcement methods. IEEE Trans Syst Man Cybern Part C (Applications Rev 28:338–355. doi: 10.1109/5326.704563Google Scholar
  54. 54.
    Rahimiyan M, Mashhadi HR (2010) An adaptive -learning algorithm developed for agent-based computational modeling of electricity market. IEEE Trans Syst Man Cybern Part C Appl Rev 40:547–556. doi: 10.1109/TSMCC.2010.2044174 CrossRefGoogle Scholar
  55. 55.
    Rajabi Mashhadi H, Rahimiyan M (2011) Measurement of power supplier’s market power using a proposed fuzzy estimator. IEEE Trans Power Syst 26:1836–1844. doi: 10.1109/TPWRS.2011.2144626 CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2017

Authors and Affiliations

  1. 1.Division of Instrumentation and Control EngineeringNetaji Subhas Institute of TechnologyNew DelhiIndia

Personalised recommendations