Autonomous Data-Driven Decision-Making in Smart Electricity Markets

  • Markus Peters
  • Wolfgang Ketter
  • Maytal Saar-Tsechansky
  • John Collins
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7524)


For the vision of a Smart Grid to materialize, substantial advances in intelligent decentralized control mechanisms are required. We propose a novel class of autonomous broker agents for retail electricity trading that can operate in a wide range of Smart Electricity Markets, and that are capable of deriving long-term, profit-maximizing policies. Our brokers use Reinforcement Learning with function approximation, they can accommodate arbitrary economic signals from their environments, and they learn efficiently over the large state spaces resulting from these signals. Our design is the first that can accommodate an offline training phase so as to automatically optimize the broker for particular market conditions. We demonstrate the performance of our design in a series of experiments using real-world energy market data, and find that it outperforms previous approaches by a significant margin.


Agents Smart Electricity Grid Energy Brokers Reinforcement Learning 


  1. 1.
    Bichler, M., Gupta, A., Ketter, W.: Designing smart markets. Information Systems Research 21(4), 688–699 (2010)CrossRefGoogle Scholar
  2. 2.
    Busoniu, L., Babuska, R., De Schutter, B., Ernst, D.: Reinforcement learning and dynamic programming using function approximators. CRC (2010)Google Scholar
  3. 3.
    ETPSG: European Technology Platform Smart Grids: Strategic deployment document for Europe’s electricity networks of the future (April 2010)Google Scholar
  4. 4.
    European Commission: EU energy country factsheet (2011)Google Scholar
  5. 5.
    Gajjar, G., Khaparde, S., Nagaraju, P., Soman, S.: Application of actor-critic learning algorithm for optimal bidding problem of a genco. IEEE Transactions on Power Systems 18(1), 11–18 (2003)CrossRefGoogle Scholar
  6. 6.
    Gottwalt, S., Ketter, W., Block, C., Collins, J., Weinhardt, C.: Demand side management - a simulation of household behavior under variable prices. Energy Policy 39, 8163–8174 (2011)CrossRefGoogle Scholar
  7. 7.
    Herter, K., McAuliffe, P., Rosenfeld, A.: An exploratory analysis of california residential customer response to critical peak pricing of electricity. Energy 32(1), 25–34 (2007)CrossRefGoogle Scholar
  8. 8.
    Ketter, W., Collins, J., Reddy, P., Flath, C., de Weerdt, M.: The power trading agent competition. Tech. Rep. ERS-2011-027-LIS, RSM Erasmus University, Rotterdam, The Netherlands (2011),
  9. 9.
    Pardoe, D., Stone, P., Saar-Tsechansky, M., Keskin, T., Tomak, K.: Adaptive auction mechanism design and the incorporation of prior knowledge. INFORMS Journal on Computing 22(3), 353–370 (2010)zbMATHCrossRefGoogle Scholar
  10. 10.
    Rahimiyan, M., Mashhadi, H.: An adaptive q-learning algorithm developed for agent-based computational modeling of electricity market. IEEE Transactions on Systems, Man, and Cybernetics 40(5), 547–556 (2010)CrossRefGoogle Scholar
  11. 11.
    Reddy, P., Veloso, M.: Learned behaviors of multiple autonomous agents in smart grid markets. In: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011 (2011)Google Scholar
  12. 12.
    Reddy, P., Veloso, M.: Strategy learning for autonomous agents in smart grid markets. In: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI, pp. 1446–1451 (2011)Google Scholar
  13. 13.
    Sutton, R., Barto, A.: Reinforcement learning: An introduction, vol. 116. Cambridge Univ. Press (1998)Google Scholar
  14. 14.
    Szepesvári, C.: Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 4(1), 1–103 (2010)CrossRefGoogle Scholar
  15. 15.
    Venayagamoorthy, G.: Potentials and promises of computational intelligence for smart grids. In: Power & Energy Society General Meeting, pp. 1–6. IEEE (2009)Google Scholar
  16. 16.
    de Weerdt, M., Ketter, W., Collins, J.: A theoretical analysis of pricing mechanisms and broker’s decisions for real-time balancing in sustainable regional electricity markets. In: Conference on Information Systems and Technology, Charlotte, pp. 1–17 (November 2011)Google Scholar
  17. 17.
    Werbos, P.: Putting more brain-like intelligence into the electric power grid: What we need and how to do it. In: International Joint Conference on Neural Networks, pp. 3356–3359. IEEE (2009)Google Scholar
  18. 18.
    Whiteson, S., Tanner, B., Taylor, M.E., Stone, P.: Protecting against evaluation overfitting in empirical reinforcement learning. In: IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) (April 2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Markus Peters
    • 1
  • Wolfgang Ketter
    • 1
  • Maytal Saar-Tsechansky
    • 2
  • John Collins
    • 3
  1. 1.Erasmus UniversityRotterdamThe Netherlands
  2. 2.University of Texas at AustinUSA
  3. 3.University of MinnesotaUSA

Personalised recommendations