Abstract
Occupant-centric controls (OCC) is an indoor climate control approach whereby occupant feedback is used in the sequence of operation of building energy systems. While OCC has been used in a wide range of building applications, an OCC category that has received considerable research interest is learning occupants’ thermal preferences through their thermostat interactions and adapting temperature setpoints accordingly. Many recent studies used reinforcement learning (RL) as an agent for OCC to optimize energy use and occupant comfort. These studies depended on predicted mean vote (PMV) models or constant comfort ranges to represent comfort, while only few of them used thermostat interactions. This paper addresses this gap by introducing a new off-policy reinforcement learning (RL) algorithm that imitates the occupant behaviour by utilizing unsolicited occupant thermostat overrides. The algorithm is tested with a number of synthetically generated occupant behaviour models implemented via the Python API of EnergyPlus. The simulation results indicate that the RL algorithm could rapidly learn preferences for all tested occupant behaviour scenarios with minimal exploration events. While substantial energy savings were observed with most occupant scenarios, the impact on the energy savings varied depending on occupants’ preferences and thermostat use behaviour stochasticity.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
ASHRAE (2020). Standard 55-2020—Thermal Environmental Conditions for Human Occupancy. Atlanta, GA, USA: American Society of Heating, Refrigeration, and Air-Conditioning Engineers.
Awalgaonkar N, Bilionis I, Liu X, et al. (2019). Learning personalized thermal preferences via Bayesian active learning with unimodality constraints. arXiv: 1903.09094.
Barrett E, Linder S (2015). Autonomous HVAC control, A reinforcement learning approach. In: Albert et al. (eds), Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science, Vol 9286. Cham: Springer.
Borgeson S, Brager G (2011). Comfort standards and variations in exceedance for mixed-mode buildings. Building Research and Information, 39: 118–133.
Cheng Z, Zhao Q, Wang F, et al. (2016). Satisfaction based Q-learning for integrated lighting and blind control. Energy and Buildings, 127: 43–55.
Daum D, Haldi F, Morel N (2011). A personalized measure of thermal comfort for building controls. Building and Environment, 46: 3–11.
DOE (2011). Buildings Energy Data Book. US Department of Energy.
Fanger PO (1970). Thermal Comfort: Analysis and Applications in Environmental Engineering. Copenhagen: Danish Technical Press.
Feldmeier M, Paradiso JA (2010). Personalized HVAC control system. In: Proceedings of 2010 Internet of Things (IOT), Tokyo, Japan.
Graesser L, Keng WL (2019). Foundations of Deep Reinforcement Learning: Theory and Practice in Python. Boston, MA, USA: Addison-Wesley Professional.
Guenther J, Sawodny O (2019). Feature selection and Gaussian Process regression for personalized thermal comfort prediction. Building and Environment, 148: 448–458.
Gunay HB, O’Brien W, Beausoleil-Morrison I (2013). A critical review of observation studies, modeling, and simulation of adaptive occupant behaviors in offices. Building and Environment, 70: 31–47.
Gunay HB, O’Brien W, Beausoleil-Morrison I, et al. (2016). implementation of an adaptive occupancy and building learning temperature setback algorithm. ASHRAE Transactions, 122(1): 179–192.
Gunay HB, O’Brien W, Beausoleil-Morrison I, et al. (2018). Development and implementation of a thermostat learning algorithm. Science and Technology for the Built Environment, 24: 43–56.
Haarnoja T, Tang H, Abbeel P, et al. (2017). Reinforcement learning with deep energy-based policies. arXiv: 1702.08165.
Haldi F, Robinson D (2008). On the behaviour and adaptation of office occupants. Building and Environment, 43: 2163–2177.
Han M, May R, Zhang X, et al. (2019). A review of reinforcement learning methodologies for controlling occupant comfort in buildings. Sustainable Cities and Society, 51: 101748.
Han M, May R, Zhang X, et al. (2020). A novel reinforcement learning method for improving occupant comfort via window opening and closing. Sustainable Cities and Society, 61: 102247.
Heidari A, Maréchal F, Khovalyg D (2022a). An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach. Applied Energy, 312: 118833.
Heidari A, Maréchal F, Khovalyg D (2022b). Reinforcement learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use. Applied Energy, 318: 119206.
Hoes P, Hensen JLM, Loomans MGLC, et al. (2009). User behavior in whole building simulation. Energy and Buildings, 41: 295–302.
Huchuk B, Sanner S, O’Brien W (2021). Development and evaluation of data-driven controls for residential smart thermostats. Energy and Buildings, 249: 111201.
IEA (2012). World Energy Balances for 2012. International Energy Agency
Jayathissa P, Quintana M, Abdelrahman M, et al. (2020). Humans-as-a-sensor for buildings—Intensive longitudinal indoor comfort models. Buildings, 10: 174.
Jung W, Jazizadeh F (2019). Comparative assessment of HVAC control strategies using personal thermal comfort and sensitivity models. Building and Environment, 158: 104–119.
Kazmi H, Mehmood F, Lodeweyckx S, et al. (2018). Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems. Energy, 144: 159–168.
Konis K, Annavaram M (2017). The Occupant Mobile Gateway: A participatory sensing and machine-learning approach for occupant-aware energy management. Building and Environment, 118: 1–13.
Lai D, Chen C (2019). Comparison of the linear regression, multinomial logit, and ordered probability models for predicting the distribution of thermal sensation. Energy and Buildings, 188–189: 269–277.
Lee S, Karava P, Tzempelikos A, et al. (2019). Inference of thermal preference profiles for personalized thermal environments with actual building occupants. Building and Environment, 148: 714–729.
Li D, Menassa CC, Kamat VR (2017). Personalized human comfort in indoor building environments under diverse conditioning modes. Building and Environment, 126: 304–317.
Li W, Zhang J, Zhao T (2019). Indoor thermal environment optimal control for thermal comfort and energy saving based on online monitoring of thermal sensation. Energy and Buildings, 197: 57–67.
Lork C, Li W, Qin Y, et al. (2020). An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management. Applied Energy, 276: 115426.
Mason K, Grijalva S (2019). A review of reinforcement learning for autonomous building energy management. Computers & Electrical Engineering, 78: 300–312.
McKee E, Du Y, Li F, et al. (2020). Deep reinforcement learning for residential HVAC control with consideration of human occupancy. In: Proceedings of 2020 IEEE Power & Energy Society General Meeting (PESGM), Montreal, Canada.
National Research Council of Canada (2017). National Energy Code of Canada for Buildings 2017.
O’Brien W, Wagner A, Schweiker M, et al. (2020). Introducing IEA EBC annex 79: Key challenges and opportunities in the field of occupant-centric building design and operation. Building and Environment, 178: 106738.
Odonkor P, Lewis K (2019). Automated design of energy efficient control strategies for building clusters using reinforcement learning. Journal of Mechanical Design, 141: 021704.
Ouf MM, Park JY, Gunay HB (2021). A simulation-based method to investigate occupant-centric controls. Building Simulation, 14: 1017–1030.
Park JY, Dougherty T, Fritz H, et al. (2019a). LightLearn: An adaptive and occupant centered controller for lighting based on reinforcement learning. Building and Environment, 147: 397–414.
Park JY, Ouf MM, Gunay B, et al. (2019b). A critical review of field implementations of occupant-centric building controls. Building and Environment, 165: 106351.
Park JY, Nagy Z (2020). HVACLearn: A reinforcement learning based occupant-centric control for thermostat set-points. In: Proceedings of the 11th ACM International Conference on Future Energy Systems.
Pazhoohesh M, Zhang C (2018). A satisfaction-range approach for achieving thermal comfort level in a shared office. Building and Environment, 142: 312–326.
Peng Y, Nagy Z, Schlüter A (2019). Temperature-preference learning with neural networks for occupant-centric building indoor climate controls. Building and Environment, 154: 296–308.
Sato K, Samejima M, Akiyoshi M, et al. (2012). A scheduling method of air conditioner operation using workers daily action plan towards energy saving and comfort at office. In: Proceedings of 2012 IEEE 17th International Conference on Emerging Technologies & Factory Automation (ETFA 2012).
Shi Z (2018). Building operation specialist: A probabilistic distributed fault detection, diagnostics and evaluation framework for building systems. PhD Thesis, Carleton University, Canada.
Sutton RS, Barto AG (2018). Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT press.
Tartarini F, Frei M, Schiavon S, et al. (2022). Cozie Apple: An iOS mobile and smartwatch application for environmental quality satisfaction and physiological data collection. arXiv: 2210.13977.
Valladares W, Galindo M, Gutiérrez J, et al. (2019). Energy optimization associated with thermal comfort and indoor air control via a deep reinforcement learning algorithm. Building and Environment, 155: 105–117.
Vázquez-Canteli J, Kämpf J, Nagy Z (2017). Balancing comfort and energy consumption of a heat pump using batch reinforcement learning with fitted Q-iteration. Energy Procedia, 122: 415–420.
Vázquez-Canteli JR, Nagy Z (2019). Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied Energy, 235: 1072–1089.
Vázquez-Canteli JR, Ulyanin S, Kämpf J, et al. (2019). Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities. Sustainable Cities and Society, 45: 243–257.
Wang Z, Hong T (2020). Reinforcement learning for building controls: The opportunities and challenges. Applied Energy, 269: 115036.
Yoshino H, Hong T, Nord N (2017). IEA EBC annex 53: Total energy use in buildings—Analysis and evaluation methods. Energy and Buildings, 152: 124–136.
Zhang Z, Chong A, Pan Y, et al. (2018). A deep reinforcement learning approach to using whole building energy model for HVAC optimal control. In: Proceedings of 2018 Building Performance Modeling Conference and SimBuild co-organized by ASHRAE and IBPSA-USA.
Zhang H, Tzempelikos A (2021). Thermal preference-based control studies: Review and detailed classification. Science and Technology for the Built Environment, 27: 1031–1039.
Zhang H, Tzempelikos A, Liu X, et al. (2023). The impact of personal preference-based thermal control on energy use and thermal comfort: Field implementation. Energy and Buildings, 284: 112848.
Zhou S, Hu Z, Gu W, et al. (2019). Artificial intelligence based smart energy community management: A reinforcement learning approach. CSEE Journal of Power and Energy Systems, 5: 1–10.
Acknowledgements
This research is supported by Brainbox AI Inc. This work was also developed thanks to the excellent research networking provided by IEA EBC Annex 79 “Occupant-Centric Building Design and Operation”.
Author information
Authors and Affiliations
Contributions
All authors contributed to the design of the study. Model and algorithm preparation and analysis were performed by Hussein Elehwany, Mohamed Ouf and Burak Gunay. The first draft of the manuscript was written by Hussein Elehwany and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Declaration of competing interest
The authors have no competing interests to declare that are relevant to the content of this article.
Ethical approval
This study does not contain any studies with human or animal subjects performed by any of the authors.
Rights and permissions
About this article
Cite this article
Elehwany, H., Ouf, M., Gunay, B. et al. A reinforcement learning approach for thermostat setpoint preference learning. Build. Simul. 17, 131–146 (2024). https://doi.org/10.1007/s12273-023-1056-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12273-023-1056-7