Abstract
Recent high profile developments of autonomous learning thermostats by companies such as Nest Labs and Honeywell have brought to the fore the possibility of ever greater numbers of intelligent devices permeating our homes and working environments into the future. However, the specific learning approaches and methodologies utilised by these devices have never been made public. In fact little information is known as to the specifics of how these devices operate and learn about their environments or the users who use them. This paper proposes a suitable learning architecture for such an intelligent thermostat in the hope that it will benefit further investigation by the research community. Our architecture comprises a number of different learning methods each of which contributes to create a complete autonomous thermostat capable of controlling a HVAC system. A novel state action space formalism is proposed to enable a Reinforcement Learning agent to successfully control the HVAC system by optimising both occupant comfort and energy costs. Our results show that the learning thermostat can achieve cost savings of 10% over a programmable thermostat, whilst maintaining high occupant comfort standards.
Keywords
- HVAC control
- Reinforcement learning
- Bayesian learning
Download conference paper PDF
References
Fuzzy logic inference temp. controller for air conditioner, September 1, 1999. https://www.google.fr/patents/CN2336254Y?cl=en. cN Patent 2,336,254
Honeywell evohome, January 01, 2015. http://evohome.honeywell.com/
Nest thermostat, January 01, 2015. https://nest.com/thermostat/life-with-nest-thermostat
Ahmed, O.: Method and apparatus for determining a thermal setpoint in a hvac system, November 9, 2004. https://www.google.fr/patents/CA2289237C?cl=en. cA Patent 2,289,237
Barrett, E., Duggan, J., Howley, E.: A parallel framework for bayesian reinforcement learning. Connection Science 26(1), 7–23 (2014)
Barrett, E., Howley, E., Duggan, J.: A learning architecture for scheduling workflow applications in the cloud. In: 2011 Ninth IEEE European Conference on Web Services (ECOWS), pp. 83–90. IEEE (2011)
Barrett, E., Howley, E., Duggan, J.: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurrency and Computation: Practice and Experience (2012)
Choi, S., Yeung, D.Y.: Predictive q-routing: a memory-based reinforcement learning approach to adaptive tra c control. In: Advances in Neural Information Processing Systems 8, pp. 945–951 (1996)
Dage, G., Davis, L., Matteson, R., Sieja, T.: Method and system for controlling an automotive hvac system, July 22, 1998. https://www.google.fr/patents/EP0706682B1?cl=en. eP Patent 0,706,682
Dorigo, M., Gambardella, L.: Ant-q: a reinforcement learning approach to the traveling salesman problem. In: Proceedings of ML-95, Twelfth Intern. Conf. on Machine Learning, pp. 252–260 (2014)
Doshi, P., Goodwin, R., Akkiraju, R., Verma, K.: Dynamic workflow composition using markov decision processes. International Journal of Web Services Research 2, 1–17 (2005)
Dutreilh, X., Kirgizov, S., Melekhova, O., Malenfant, J., Rivierre, N., Truck, I.: Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow. In: The Seventh International Conference on Autonomic and Autonomous Systems, ICAS 2011, pp. 67–74 (2011)
Fadell, A., Rogers, M., Satterthwaite, E., Smith, I., Warren, D., Palmer, J., Honjo, S., Erickson, G., Dutra, J., Fiennes, H.: User-friendly, network connected learning thermostat and related systems and methods, July 4, 2013. https://www.google.fr/patents/US20130173064. uS Patent App. 13/656,189
Grzes, M., Kudenko, D.: Learning shaping rewards in model-based reinforcement learning. In: Proc. AAMAS 2009 Workshop on Adaptive Learning Agents, vol. 115 (2009)
Karray, F.O., De Silva, C.W.: Soft computing and intelligent systems design: theory, tools, and applications. Pearson Education (2004)
Nau, D., Ghallab, M., Traverso, P.: Automated Planning: Theory & Practice. Morgan Kaufmann Publishers Inc., San Francisco (2004)
Russell, S., Norvig, P., Canny, J., Malik, J., Edwards, D.: Artificial intelligence: a modern approach, vol. 2. Prentice hall Englewood Cliffs, NJ (1995)
Scott, J., Bernheim Brush, A., Krumm, J., Meyers, B., Hazas, M., Hodges, S., Villar, N.: Preheat: controlling home heating using occupancy prediction. In: Proceedings of the 13th International Conference on Ubiquitous Computing, pp. 281–290. ACM (2011)
Spiegelhalter, D.J., Dawid, A.P., Lauritzen, S.L., Cowell, R.G.: Bayesian analysis in expert systems. Statistical science, 219–247 (1993)
Strens, M.: A bayesian framework for reinforcement learning, pp. 943–950 (2000)
Tesauro, G.: Temporal difference learning and td-gammon. Communications of the ACM 38(3), 58–68 (1995)
Tesauro, G., Kephart, J.O.: Pricing in agent economies using multi-agent q-learning. Autonomous Agents and Multi-Agent Systems 5(3), 289–304 (2002)
Watkins, C.: Learning from Delayed Rewards. Ph.D. thesis, University of Cambridge, England (1989)
Wiering, M.: Multi-agent reinforcement learning for traffic light control. In: ICML, pp. 1151–1158 (2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Barrett, E., Linder, S. (2015). Autonomous HVAC Control, A Reinforcement Learning Approach. In: , et al. Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science(), vol 9286. Springer, Cham. https://doi.org/10.1007/978-3-319-23461-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-23461-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23460-1
Online ISBN: 978-3-319-23461-8
eBook Packages: Computer ScienceComputer Science (R0)