A reinforcement learning approach to autonomous decision-making in smart electricity markets
- 1.3k Downloads
The vision of a Smart Electric Grid relies critically on substantial advances in intelligent decentralized control mechanisms. We propose a novel class of autonomous broker agents for retail electricity trading that can operate in a wide range of Smart Electricity Markets, and that are capable of deriving long-term, profit-maximizing policies. Our brokers use Reinforcement Learning with function approximation, they can accommodate arbitrary economic signals from their environments, and they learn efficiently over the large state spaces resulting from these signals. We show how feature selection and regularization can be leveraged to automatically optimize brokers for particular market conditions, and demonstrate the performance of our design in extensive experiments using real-world energy market data.
KeywordsEnergy brokers Feature selection Reinforcement learning Smart electricity grid Trading agents
We would like to thank three anonymous Machine Learning reviewers and three anonymous ECML-PKDD 2012 reviewers for their insightful comments on this work. The extensive exploration of alternative feature selection and regularization techniques we present here and the subsequent enhanced performance of the agent, was, among other things, inspired by their remarks.
- de Weerdt, M., Ketter, W., & Collins, J. (2011). A theoretical analysis of pricing mechanisms and broker’s decisions for real-time balancing in sustainable regional electricity markets. In Conference on information systems and technology, Charlotte (pp. 1–17). Google Scholar
- ETPSG (2010). European technology platform smart grids: strategic deployment document for Europe’s electricity networks of the future. Google Scholar
- European Commission (2011). EU energy country factsheet. Google Scholar
- Ghavamzadeh, M., Lazaric, A., Maillard, O., & Munos, R. (2010). LSTD with random projections. In Proceedings of the twenty-fourth annual conference on advances in neural information processing systems (pp. 721–729). Google Scholar
- Ketter, W., Collins, J., Reddy, P., & de Weerdt, M. (2012b). The 2012 power trading agent competition (Tech. Rep. ERS-2012-010-LIS). RSM Erasmus University, Rotterdam, The Netherlands. http://ssrn.com/paper=2144644.
- Kolter, J., & Ng, A. (2009). Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th annual international conference on machine learning (pp. 521–528). New York: ACM. Google Scholar
- Painter-Wakefield, C., & Parr, R. (2012). L1 regularized linear temporal difference learning (Tech. Rep. TR-2012-01) Duke University, Computer Science. Google Scholar
- Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., & Littman, M. L. (2008). An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the 25th international conference on machine learning (pp. 752–759). New York: ACM. CrossRefGoogle Scholar
- Peters, M., Ketter, W., Saar-Tsechansky, M., & Collins, J. (2012). Autonomous data-driven decision-making in smart electricity markets. In P. Flach, T. Bie, & N. Cristianini (Eds.), Lecture notes in computer science: Vol. 7524. Machine learning and knowledge discovery in databases (pp. 132–147). Berlin: Springer. CrossRefGoogle Scholar
- Petrik, M., Taylor, G., Parr, R., & Zilberstein, S. (2010). Feature selection using regularization in approximate linear programs for Markov decision processes. In International conference on machine learning (ICML). Google Scholar
- Pyeatt, L., Howe, A., et al. (2001). Decision tree function approximation in reinforcement learning. In Proceedings of the third international symposium on adaptive systems: evolutionary computation and probabilistic graphical models (Vol. 2, pp. 70–77). Google Scholar
- Reddy, P., & Veloso, M. (2011a). Learned behaviors of multiple autonomous agents in smart grid markets. In Proceedings of the twenty-fifth AAAI conference on artificial intelligence (AAAI-11). Google Scholar
- Reddy, P., & Veloso, M. (2011b). Strategy learning for autonomous agents in smart grid markets. In Proceedings of the twenty-second international joint conference on artificial intelligence (IJCAI) (pp. 1446–1451). Google Scholar
- Rummery, G., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Tech. Rep. CUED/F-INFENG/TR 166, University of Cambridge. Google Scholar
- Sutton, R., & Barto, A. (1998). Reinforcement learning: an introduction (Vol. 116). Cambridge: Cambridge University Press. Google Scholar
- Werbos, P. (2009). Putting more brain-like intelligence into the electric power grid: what we need and how to do it. In International joint conference on neural networks (pp. 3356–3359). New York: IEEE. Google Scholar
- Whiteson, S., Tanner, B., Taylor, M. E., & Stone, P. (2011). Protecting against evaluation overfitting in empirical reinforcement learning. In IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL). Google Scholar