Abstract
In order to support decision-making problems on the energy sector, like energy forecasting and demand prediction, analytics services are developed that assist users in extracting useful inferences on energy related data. Such analytics services use AI techniques to extract useful knowledge on collected data from energy infrastructure like smart meters and sensors. The big data value chain describes the steps of big data life cycle from collecting, pre-processing, storing and querying energy consumption data for high-level user-driven services. With the exponential growth of networking capabilities and the Internet of Things (IoT), data from the energy sector is arriving with a high throughput taking the problem of calculating big data analytics to a new level. This research will review existing approaches for big data energy analytics services and will further propose a framework for facilitating AI-enabled energy analytics taking into consideration all the requirements of analytics services through the entire big data value chain from data acquisition and batch and stream data ingestion, to creating proper querying mechanisms. Those query mechanisms will in turn enable the execution of queries on huge volumes of energy consumption data with low latency, and establishing high-level data visualizations. The proposed framework will also address privacy and security concerns regarding the big data value chain and allow easy applicability and adjustment on various use cases on energy analytics.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Abbreviations
- Term:
-
Abbreviation
- Internet of things:
-
IoT
- Machine learning:
-
ML
- Neural network:
-
NN
- Support vector machine:
-
SVM
- Online analytical processing:
-
OLAP
- Deep learning:
-
DL
- Online transactional processing:
-
OLTP
- Atomicity, consistency, isolation and durability:
-
ACID
- Hadoop distributed file system:
-
HDFS
- International dataspaces:
-
IDS
- Artificial neural network:
-
ANN
- Multiple linear regression:
-
MLR
- Multiple layer perceptron:
-
MLP
- Building management system:
-
BMS
- Building information management:
-
BIM
- Infrastructure as a service:
-
IaaS
- Platform as a service:
-
PaaS
- Software as a service:
-
SaaS
- Identity access management:
-
IAM
- Generic enabler:
-
GE
- Context broker:
-
CB
- Machine to machine:
-
M2M
- Linear model:
-
LM
- K-nearest neighbour:
-
KNN
References
D. Koh, COVID-19 lockdowns throughout the world. Occup. Med. (Chic Ill) 70(5), 322–322 (2020)
P. Jiang, Y. Van Fan, J.J. Klemeš, Impacts of covid-19 on energy demand and consumption: challenges, lessons and emerging opportunities. Appl. Energy 285, 116441 (2021)
K. Ahmed Ali, M.I. Ahmad, Y. Yusup, Issues, impacts, and mitigations of carbon dioxide emissions in the building sector. Sustainability 12(18), 7427 (2020)
C.A. Horowitz, Paris agreement. Int. Leg. Mater. 55(4), 740–755 (2016)
E. Karakolis, K. Alexakis, P. Kapsalis, S. Mouzakitis, J. Psarras: An end-to-end approach for scalable real time anomaly detection in smart buildings, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–7
S. Pelekis et al, In search of deep learning architectures for load forecasting: a comparative analysis and the impact of the covid-19 pandemic on model performance, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–8
P. Skaloumpakas, E. Sarmas, Z. Mylona, A. Cavadenti, F. Santori, V. Marinakis, Predicting thermal comfort in buildings with machine learning and occupant feedback, in 2023 IEEE International Workshop on Metrology for Living Environment (MetroLivEnv) (IEEE, 2023), pp. 34–39
E. Sarmas, N. Dimitropoulos, S. Strompolas, Z. Mylona, V. Marinakis, A. Giannadakis, H. Doukas et al., A web-based building automation and control service, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, 2022), pp. 1–6
E. Sarmas, S. Strompolas, V. Marinakis, F. Santori, M.A. Bucarelli, H. Doukas, An incremental learning framework for photovoltaic production and load forecasting in energy microgrids. Electronics 11(23), 3962 (2022)
J. Anuradha, A brief introduction on big data 5Vs characteristics and hadoop technology. Proc. Comput. Sci. 48, 319–324 (2015)
B.W. Wirtz, J.C. Weyerer, C. Geyer, Artificial intelligence and the public sector—applications and challenges. Int. J. Public Adm. 42(7), 596–615 (2019)
D.E. O’Leary, Artificial intelligence and big data. IEEE Intell. Syst. 28(2), 96–99 (2013)
R. Mayer, H.A. Jacobsen, Scalable deep learning on distributed infrastructures. ACM Comput. Surv.Comput. Surv. 53(1), 1–37 (2021)
S. Dilmaghani, M.R. Brust, G. Danoy, N. Cassagnes, J. Pecero, P. Bouvry, Privacy and security of big data in AI systems: a research and standards perspective, in 2019 IEEE International Conference on Big Data (Big Data) (IEEE, Los Angeles, CA, USA, 2019), pp. 5737–5743
J. Hu, A.V. Vasilakos, Energy big data analytics and security: challenges and opportunities. IEEE Trans. Smart Grid 7(5), 2423–2436 (2016)
E. Sarmas, E. Spiliotis, V. Marinakis, G. Tzanes, J.K. Kaldellis, H. Doukas, ML-based energy management of water pumping systems for the application of peak shaving in small-scale islands. Sustain. Cities Soc. 82, 103873 (2022)
E. Sarmas, E. Spiliotis, E. Stamatopoulos, V. Marinakis, H. Doukas, Short-term photovoltaic power forecasting using meta-learning and numerical weather prediction independent Long Short-Term Memory models. Renew. Energy 216, 118997 (2023)
H. Plattner, The impact of columnar in-memory databases on enterprise systems. Proc. VLDB Endow. 7(13), 1722–1729 (2014)
S. Singh, A. Yassine, Big data mining of energy time series for behavioral analytics and energy consumption forecasting. Energies (Basel) 11(2), 452 (2018)
G. Hernández-Moral et al., Big data value chain: multiple perspectives for the built environment. Energies (Basel) 14(15), 4624 (2021)
E. Sarmas, N. Dimitropoulos, V. Marinakis, Z. Mylona, H. Doukas, Transfer learning strategies for solar power forecasting under data scarcity. Sci. Rep. 12(1), 14643 (2022)
A. Kongkanand, M.F. Mathias, The priority and challenge of high-power performance of low-platinum proton-exchange membrane fuel cells. J. Phys. Chem. Lett. 7(7), 1127–1137 (2016)
A.Z. Faroukhi, I. El Alaoui, Y. Gahi, A. Amine, Big data monetization throughout big data value chain: a comprehensive review. J. Big Data 7(1), 3 (2020)
N. Naik, Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP, in 2017 IEEE International Systems Engineering Symposium (ISSE) (IEEE, Vienna, Austria, 2017), pp. 1–7
E. Curry, The big data value chain: definitions, concepts, and theoretical approaches, in New Horizons for a Data-Driven Economy (Springer International Publishing, Cham, 2016), pp. 29–37
Y. Li, S. Manoharan, A performance comparison of SQL and NoSQL databases, in 2013 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM) (IEEE, Victoria, British Columbia, Canada 2013), pp. 15–19
Neo4j, https://neo4j.com/. Accessed 29 May 2023
MongoDB, https://www.mongodb.com/. Accessed 29 May 2023
Apache Handoop, https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html. Accessed 29 May 2023
Presto, https://prestodb.io/. Accessed 29 May 2023
Trino, https://trino.io/. Accessed 29 May 2023
K. Alexakis et al, Intelligent querying for implementing building aggregation pipelines, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–6
K. Touloumis, E. Karakolis, P. Kapsalis, V. Marinakis, J. Psarras, BD4NRG query engine–intuitive, efficient and federated querying on big data, in e-Society (Iadis, Lisbon, Portugal, 2023)
Grafana, https://grafana.com/. Accessed 29 May 2023
G. Kormpakis, P. Kapsalis, K. Alexakis, S. Pelekis, E. Karakolis, H. Doukas, An advanced visualisation engine with role-based access control for building energy visual analytics, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–8
P. Kapsalis, G. Kormpakis, K. Alexakis, D. Askounis, Leveraging graph analytics for energy efficiency certificates. Energies (Basel) 15(4), 1500 (2022)
P. Kapsalis, G. Kormpakis, K. Alexakis, E. Karakolis, S. Mouzakitis, D. Askounis, A reasoning engine architecture for building energy metadata management, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–7
N. Dimitropoulos, E. Sarmas, M. Lampkowski, V. Marinakis, A quantitative methodology to support local governments in climate change adaptation and mitigation actions, in International Symposium on Distributed Computing and Artificial Intelligence (Springer Nature Switzerland, Cham, 2023), pp. 99–108
European Dataspaces, https://dataspaces.info/common-european-data-spaces/#page-content. Accessed 29 May 2023
IDS Connector, https://www.isst.fraunhofer.de/en/business-units/data-business/technologies/Dataspace-Connector.html. Accessed 29 May 2023
True Connector, https://fiware-true-connector.readthedocs.io/en/latest/. Accessed 29 May 2023
L. Xuemei, D. Lixing, L. Jinhu, X. Gang, L. Jibin, A novel hybrid approach of KPCA and SVM for building cooling load prediction, in 2010 Third International Conference on Knowledge Discovery and Data Mining (IEEE, Washington DC, USA, 2010), pp. 522–526
J.P. Zhang, Z.W. Li, J. Yang, A parallel SVM training algorithm on large-scale classification problems, in 2005 International Conference on Machine Learning and Cybernetics (IEEE, 2005), pp. 1637–1641
R. Mena, F. RodrÃguez, M. Castilla, M.R. Arahal, A prediction model based on neural networks for the energy consumption of a bioclimatic building. Energy Build. 82, 142–155 (2014)
C. Tsolkas, E. Spiliotis, E. Sarmas, V. Marinakis, H. Doukas, Dynamic energy management with thermal comfort forecasting. Build. Environ. 237, 110341 (2023)
V. Marinakis, H. Doukas, An advanced IoT-based system for intelligent energy management in buildings. Sensors 18(2), 610 (2018)
S.S.K. Kwok, R.K.K. Yuen, E.W.M. Lee, An intelligent approach to assessing the effect of building occupancy on building cooling load prediction. Build. Environ. 46(8), 1681–1690 (2011)
S. Pelekis et al, A comparative assessment of deep learning models for day-ahead load forecasting: investigating key accuracy drivers (2023)
E. Sarmas, E. Spiliotis, N. Dimitropoulos, V. Marinakis, H. Doukas, Estimating the energy savings of energy efficiency actions with ensemble machine learning models. Appl. Sci. 13(4), 2749 (2023)
E. Sarmas, E. Spiliotis, V. Marinakis, T. Koutselis, H. Doukas, A meta-learning classification model for supporting decisions on energy efficiency investments. Energy Build. 258, 111836 (2022)
S. Pelekis et al, Targeted demand response for flexible energy communities using clustering techniques (2023)
R. Ahmadiahangar, T. Haring, A. Rosin, T. Korotko, J. Martins, Residential load forecasting for flexibility prediction using machine learning-based regression model, in 2019 IEEE International Conference on Environment and Electrical Engineering and 2019 IEEE Industrial and Commercial Power Systems Europe (EEEIC/I&CPS Europe) (IEEE, Genova, Italy, 2019), pp. 1–4
E. Karakolis, K. Alexakis, P. Kapsalis, S. Mouzakitis, S. Psarras, An end-to-end approach for scalable real time anomaly detection in smart buildings, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–7
M. Yamauchi, Y. Ohsita, M. Murata, K. Ueda, Y. Kato, Anomaly detection in smart home operation from user behaviors and home conditions. IEEE Trans. Consum. Electron. 66(2), 183–192 (2020)
J. Xu, H. Wu, J. Wang, M. Long, Anomaly transformer: time series anomaly detection with association discrepancy (2021)
M. Panthi, Anomaly detection in smart grids using machine learning techniques, in 2020 First International Conference on Power, Control and Computing Technologies (ICPC2T) (IEEE, Raipur, India, 2020), pp. 220–222
E. Karakolis et al, Artificial intelligence for next generation energy services across Europe–the I-Nergy project, in International Conferences e-Society 2022 and Mobile Learning 2022 (IADIS, Lisbon, Portugal, 2022)
E. Karakolis, S. Pelekis, S. Mouzakitis, G. Kormpakis, V. Michalakopoulos, J. Psarras, The I-Nergy reference architecture for the provision of next generation energy services through artificial intelligence, in e-Society (IADIS, Lisbon, Portugal, 2022)
M. Pau, P. Kapsalis, Z. Pan, G. Korbakis, D. Pellegrino, A. Monti, MATRYCS—a big data architecture for advanced services in the building domain. Energies (Basel) 15(7), 2568 (2022)
European Energy Data Exchange, https://energy.ec.europa.eu/system/files/2021-06/bridge_wg_data_management_eu_reference_architcture_report_2020-2021_0.pdf/. Accessed 29 May 2023
K.A. Wehrmeister, The BD4NRG reference architecture for big data driven energy applications, in 2022 13th International Conference on Information, Intelligence, Systems & Applications (IISA) (IEEE, Corfu, Greece, 2022), pp. 1–8
F. Gao, Frameworks for big data integration, warehousing, and analytics, in Big Data Application in Power Systems (Elsevier, 2018), pp. 57–73
L. Cai, Y. Zhu, The challenges of data quality and data quality assessment in the big data era. Data Sci. J. 14, 2 (2015)
B. Balaji et al, Brick, in Proceedings of the 3rd ACM International Conference on Systems for Energy-Efficient Built Environments (ACM, New York, NY, USA, 2016), pp. 41–50
M. Muniswamaiah, T. Agerwala, C.C. Tappert, Federated query processing for big data in data science, in 2019 IEEE International Conference on Big Data (Big Data) (IEEE, Los Angeles, CA, USA, 2019), pp. 6145–6147
A. Gandomi, M. Haider, Beyond the hype: big data concepts, methods, and analytics. Int. J. Inf. Manage. 35(2), 137–144 (2015)
D. Cheng, P. Schretlen, N. Kronenfeld, N. Bozowsky, W. Wright, Tile based visual analytics for Twitter big data exploratory analysis, in 2013 IEEE International Conference on Big Data (IEEE, Santa Clara, CA, USA, 2013), pp. 2–4
M. Tavana, A. Shaabani, F. Javier Santos-Arteaga, I. Raeesi Vanani, A review of uncertain decision-making methods in energy management using text mining and data analytics. Energies (Basel) 13(15), 3947 (2020)
V. Cevher, S. Becker, M. Schmidt, Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics. IEEE Signal Process. Mag. 31(5), 32–43 (2014)
J. Kampars, J. Grabis, Near real-time big-data processing for data driven applications, in 2017 International Conference on Big Data Innovations and Applications (Innovate-Data) (IEEE, Prague, Czech Republic, 2017), pp. 35–42
ENTSO-E, https://www.entsoe.eu/. Accessed 29 May 2023
V. Araujo, K. Mitra, S. Saguna, C. Åhlund, Performance evaluation of FIWARE: a cloud-based IoT platform for smart cities. J. Parallel Distr. Comput. 132, 250–261 (2019)
M.A. da Cruz, J.J. Rodrigues, P. Lorenz, P. Solic, J. Al-Muhtadi, V.H.C. Albuquerque, A proposal for bridging application layer protocols to HTTP on IoT solutions. Futur. Gener. Comput. Syst. 97, 145–152 (2019)
A. Braud, G. Fromentoux, B. Radier, O. Le Grand, The road to European digital sovereignty with Gaia-X and IDSA. IEEE Netw. 35(2), 4–5 (2021)
M. Mohammadi, A. Al-Fuqaha, S. Sorour, M. Guizani, Deep learning for IoT big data and streaming analytics: a survey. IEEE Commun. Surv. Tutor., 2923–2960 (2018)
K.M.M. Thein, Apache kafka: next generation distributed messaging system. Int. J. Sci. Eng. Technol. Res. 3(47), 9478–9483 (2014)
J.E. Luzuriaga, M. Perez, P. Boronat, J.C. Cano, C. Calafate, P. Manzoni, A comparative evaluation of AMQP and MQTT protocols over unstable and mobile networks, in 2015 12th Annual IEEE Consumer Communications and Networking Conference (CCNC) (IEEE, Las Vegas, Nevada, USA, 2015), pp. 931–936
SAREF, https://saref.etsi.org/. Accessed 29 May 2023
DataHub, https://datahub.io/. Accessed 29 May 2023
M. Brümmer, C. Baron, I. Ermilov, M. Freudenberg, D. Kontokostas, S. Hellmann, DataID: towards semantically rich metadata for complex datasets, in Proceedings of the 10th International Conference on Semantic Systems (ACM, New York, NY, USA, 2014), pp. 84–91
PostgreSQL, https://www.postgresql.org/. Accessed 29 May 2023
SQLite, https://www.sqlite.org/index.html. Accessed 29 May 2023
Jupyter, https://jupyter.org/. Accessed 29 May 2023
Scikit-Learn, https://scikit-learn.org/stable/. Accessed 29 May 2023
TensorFlow, https://www.tensorflow.org/. Accessed 29 May 2023
Mlflow, https://mlflow.org/. Accessed 29 May 2023
Apache Spark, https://spark.apache.org/. Accessed 29 May 2023
W. Wu, W. Lin, C.H. Hsu, L. He, Energy-efficient hadoop for big data analytics and computing: a systematic review and research insights. Futur. Gener. Comput. Syst. 86, 1351–1367 (2018)
Apache Flink, https://flink.apache.org/. Accessed 29 May 2023
S. Chintapalli et al.: Benchmarking streaming computation engines: storm, flink and spark streaming, in 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) (IEEE, Chicago, IL, USA, 2016), pp. 1789–1792
Fiware Keyrock, https://fiware-idm.readthedocs.io/en/latest/. Accessed 29 May 2023
Keycloack, https://www.keycloak.org/. Accessed 29 May 2023
M. Muthuprasanna, K. Wei, S. Kothari, Eliminating SQL injection attacks—a transparent defense mechanism, in 2006 Eighth IEEE International Symposium on Web Site Evolution (WSE’06) (IEEE, Philadelphia, PA, USA, 2006), pp. 22–32
S. Biswas, T. Carley, M. Simpson, B. Middha, R. Barua, Memory overflow protection for embedded systems using run-time checks, reuse, and compression. ACM Trans. Embed. Comput. Syst. 5(4), 719–752 (2006)
B. Biggio, B. Nelson, P. Laskov, Poisoning attacks against support vector machines (2012)
T. Nagata, H. Sasaki, A multi-agent approach to power system restoration. IEEE Trans. Power Syst. 17(2), 457–462 (2002)
V. Vasilyev, R. Shamsutdinov, Security analysis of wireless sensor networks using SIEM and multi-agent approach, in 2020 Global Smart Industry Conference (GloSIC) (IEEE, Chelyabinsk, Russia, 2020), pp. 291–296
K. Touloumis, A. Michalitsi-Psarrou, P. Kapsalis, A. Georgiadou, D. Askounis, Vulnerabilities manager, a platform for linking vulnerability data sources, in 2021 IEEE International Conference on Big Data (Big Data) (IEEE, Orlando, FL, USA, 2021), pp. 2178–2184
K. Touloumis, A. Michalitsi-Psarrou, A. Georgiadou, D. Askounis, A tool for assisting in the forensic investigation of cyber-security incidents, in 2022 IEEE International Conference on Big Data (Big Data) (IEEE, Osaka, Japan, 2022), pp. 2630–2636
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Touloumis, K., Karakolis, E., Kapsalis, P., Pelekis, S., Askounis, D. (2024). The Big Data Value Chain for the Provision of AI-Enabled Energy Analytics Services. In: Doukas, H., Marinakis, V., Sarmas, E. (eds) Machine Learning Applications for Intelligent Energy Management. Learning and Analytics in Intelligent Systems, vol 35. Springer, Cham. https://doi.org/10.1007/978-3-031-47909-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-47909-0_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47908-3
Online ISBN: 978-3-031-47909-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)