1 Introduction

Energy is an important resource for individuals, business entities as well as the macroeconomic development. Individuals use energies such as electricity, gas and petrol on a daily basis, while business entities reply to a large extent on different energy resources for their daily operations; finally, the macroeconomic development fueled by the transportation and logistics industries cannot be operated without energy. The price of the energy, obviously, is a factor that is concerned by different parties of the economy and is an important determinant of individual activities, industry-related activities as well as economic activities as a whole.

The spot price in the energy sector mainly means the price that retailers pay to the wholesales which experiences a level of volatility based on the demand and supply (Kaleta and Toczylowski, 2008), whereas futures contract is used by the investors to buy or sell energy at a price and date which are predetermined to reduce the price fluctuations (Huisman et al., 2009). Over the past few decades, there are a number of studies investigating the spot and futures prices in the energy sectors, (see literature review below in detail), very few pieces of research considered the spot demand and futures demand at the same time.

In the first half of the twentieth century, the Iberian energy sector, namely the energy sector of Portugal and Spain, was different from the one of other European or OECD countries from the perspective of its higher degree of dependence on external energy. Now Spain and Portugal are the two countries leading the energy transition promoted by electricity generation from renewable energy sources (Perez-Franco et al., 2020). There are several electricity markets among the European countries, one of which is operated by OMIE for the Iberian electricity market, the main responsibility of which is to manage the day-ahead and intraday electricity markets in Spain and Portugal (Canelas et al., 2020). Because of the 2007–2009 financial crisis hit the economy of both Portugal and Spain very hard and the disastrous effect was even stronger than other European counties, therefore, both of these two countries have been attempting to make transition toward a more sustainable and a resilient economy. One of the efforts made by the governments was to reduce the level of pollution from energy production. In Particular, Portugal committed to have electricity generation from renewable resource accounting for 60% of total production by 2020 (Gil-Alana et al., 2020). In addition, investment in renewable energy will play a positive impact on stabilizing climate, increase employment and economic growth (Pollin et al., 2015), it will also reduce GDP per capita emission as well as diminish the energy dependency (Perez-Franco et al., 2020), which is of particular importance for the Iberian Peninsula. Through the investigation of the relationship between energy price and energy demand under the propose innovative method, we would be able to not only establish a reasonable price mechanism in the spot and futures energy market in Iberian market, but also the price mechanism will provide a guidance and motivation for these two countries to engage in the endeavor in the process of decarbonization in the energy sector by producing energy from coal sources towards sustainable ones such as wind and solar.

There are quite a few questions need to be answered which have not been addressed by the previous literature: (1) will energy spot price be related to the futures price, spot demand and futures energy demand? (2) Will there be a linkage between futures energy price and spot demand, spot energy price as well as the futures energy demand? (3) Will both futures and spot energy prices as well as futures energy demand influence the spot energy demand? (4) Is there a potential influence of spot and futures energy price as well as spot energy demand on futures energy demand?. Answers these questions would be very important due to the consideration that this will fill in the gap of the empirical literature on this topic, and in addition, the answers to these questions will provide accurate and clear information and guidance related to the pricing mechanism in the spot and futures energy market in Iberian Penisula. This research idea can be applied or expanded to the whole European Market or other regional markets.

A Hidden Markov Model (HMM) is a type of graphical model often used to model temporal data. Unlike traditional Markov Models, HMMs assume that the data observed is not the actual state of the model, but is instead generated by underlying hidden states (Danisman & Kocer, 2021). HMMs have been widely applied to many different fields because of their flexibility and computational efficiency, such as energy (Cai et al., 2018; Ullah et al., 2018). The current study distinguishes itself from the previous literature in the energy sector by investigating the endogenous and temporal dependent relationship between energy prices and demands through a stochastic HMM derived from copula generation. The stochastic HMM method benefits from the advantages of being able to examine the temporal dependence and endogeneity on price and demand from the perspective of the initialization of state and probabilities of transition. Within this specific model, both Vector Auto Regressive (VAR) and Multinomial Logit (ML) models can be used under the bootstrapped procedure.

The Energy Information Administration of US department of Energy has developed a method called levelized avoid cost of electricity (LACE) to compare and select energy investment, this method compares the levelized costs (LCOE) with the avoided costs when implementing new projects. The net value (the difference between LACE and LCOE) can be thought of as the potential profit (or loss) per unit of energy production for the plant (EIA, 2013). Our proposed HMM model can be beneficial for a better predictability of the financial equations for assessing decarbonized alternative sources of energy, this being one of the motivations for the present work. This study departs from Gil-Alana et al. (2020) by digging further into the issue of endogeneity between prices and consumption in Iberian energy markets. While this previous paper used fractional integration to examine the degree of persistence of these series in spot and futures markets in Spain and Portugal, this research now uses a stochastic HMM approach to unveil the cause-effect relationships and feedback process that may exist within the temporal dependent structure previously analysed. By answering this remaining gap, it is possible to ascertain the better sequence of actions for improving market regulation and development in Iberian Peninsula. In addition, the objectives of this work are the following:

  1. (1)

    We investigate the autocorrelation of futures energy price and demand as well as the relationship between futures price and spot price. The investigation of this is very important due to the fact that the futures contracts play a role of price discovery and they are used to reduce particular risks (Nicolau & Palomba, 2015).

  2. (2)

    We examine the auto-correlation of spot energy price and energy demand. The examination of this issue is motivated by the fact that although the spot market only accounted for a very small percentage in the energy transactions, it has a degree of influence on the formation of government selling prices or official selling prices (Silvério & Szklo, 2012).

  3. (3)

    We evaluation the inter-dependence between energy price and energy demand in both the spot and futures market. The evaluation of this issue is very essential considering the fact that based on the energy demand, relevant policies can be made related to the price mechanism, while the price mechanism can be used as a regulatory tool to adjust the demand in the energy spot and futures market, which to a certain degree, contribute to the achievement of macroeconomic goals.

This paper will have the following five parts. Section 2 reviews the literature related to the spot and futures energy market, Sect. 3 presents the innovative methods adopted in the current study. Results are analyzed and discussed in Sect. 4. Conclusions follow in Sect. 5. The R code developed for this research is disclosed in the Appendix for the sake of research reproducibility.

2 Literature review

Academic researchers have been active in research in the energy sector from different perspectives, one stream of the studies investigated energy efficiency and its determinants (Bashir et al., 2020; Bian et al., 2017; Lv et al., 2015; Omrani et al., 2019; Zhou et al., 2017), the second stream of studies examined the issue of energy demand and energy consumption (Li et al., 2017; Shadzad, 2020; Shahzad et al., 2021; Talbi et al., 2020; Yang & Yang, 2015); while there is a growing awareness and concern during recent decades about climate change derived from carbon emission, therefore, relevant efforts have been made by academic research in the area of renewable energy and carbon emissions (Dogan et al., 2021; Farooq et al., 2019; Fatima et al., 2021; Ghazouani et al., 2020; Sarwar et al., 2019; Shahzad et al., 2020). Although relevant studies during recent couple of years focused on the evaluation of energy demand as illustrated before, however, none of the above-mentioned studies provided a detailed analysis about the energy spot market and futures market, let alone the inter-relationships among the price and demand in these two markets. However, over the past few decades, the literature has made effect in estimating the spot and futures energy markets from different perspectives. King and Cuc (1996) investigate the price convergence of the North American natural gas spot markets under the time-varying parameter analysis, which benefits from the ability to assess the strength of price convergence across different gas-producing basins, the other merit of the method lies to its explicit presence of time varying parameters. The findings show that the degree of price convergence becomes significantly higher after the price deregulation took place since the mid 1980s. Instead examining the degree of price convergence, Worthington et al. (2005) evaluate the transmission of spot electricity prices and price volatility in Australian under the multivariate generalised autoregressive conditional heteroskedasticity model. The findings suggest that the price volatility in some markets is affected by the shocks from other markets. Employing a VAR model, which has the advantage of allowing regularities in the data without imposing as many prior restrictions compared to structural model, Park et al. (2006) examine the relationship among 11 US spot market electricity prices. The findings show that the transmission effect is limited across different market in a long-time frame. The evidence of electricity market integration between Spain, Portugal, Austria, Germany, Switzerland and France is assessed by Ciarreta and Zarraga (2015). The relationship between spot prices in the national Australian market is investigated by Yan and Trück (2020) under the dynamic network analysis. the results from the principal component analysis as well as the granger causality network analysis show that there are few factors that are related to the derived measure of interdependence including unexpected high demand for electricity and sudden increase in the level of price volatility, among others.

The integration is mainly measured by the price convergence and spillovers. The investigation is facilitated by using multivariate Generalized Autoregressive Conditional Heteroscedastic models. They find dynamic correlations between Spain–Portugal, Germany–Austria and Switzerland–Austria. On the other hand, the integration between Spain–France and Germany–France is very weak.

Ewing et al. (2006) use three different markets in the United States (crude oil, heating oil and gasoline) to investigate the asymmetric relationship between spot price and futures price under the momentum-threshold autoregressive (M-TAR) model, the result suggests that the spot price and futures price for each of these three markets are cointegrated. The research of this topic was also undertaken by other studies. Maslyuk and Smyth (2009) use US WTI and the UK Brent to examine whether the crude oil spot and futures prices are cointegrated under a residual-based cointegration test. This method possesses the property of allowing for one structural break in the cointegrating vector and high frequency data. The findings show that the spot and futures crude prices under both the same grade and different grades are cointegrated.

Weron and Zator (2014) address the pitfalls of using the linear regression models to investigate the relationship between spot and futures prices in the electricity market (i.e. simultaneity problems, measurement errors and seasonality issues) by employing regression models with GARCH residuals, the results from the sample of a 13-year long spot and futures prices series in the Nord Pool electricity market suggest that there is a significant and positive influence of water reservoir level on the risk premium. Using the Dow Jones US commodity index, Beckmann et al. (2014) analyze the spot and futures prices in the energy sector. The long-run relationship between the spot and futures prices is examined in the first step followed by the examination of the adjustment pattern under a smooth transition model, which provide a higher degree of flexibility compared to the threshold model. The findings show that the present price discovery function is significantly influenced by the past relative volatility, while Zhang and Wang (2013) find that crude oil and gasoline futures market contribute more to the price discovery function compared to the spot energy market. This finding is also supported by Silvério and Szklo (2012). The relationship between spot prices and futures prices of crude oils from North America, Europe, Africa and the middle East is investigated by Kaufmann and Ullman (2009) under two methods, namely a two-step DOLS error correction model and a vector error correction model. The findings show that the linkage between spot and futures markets is relatively weak. Nicolau and Palomba (2015) investigate the relationship between spot and futures prices of natural gas and oil using daily data from 7th January 1997 to 31st May 2014. The results from the recursive bivariate VAR models suggest that there is a level of interactions between spot and futures prices. More specifically, it is found that the gas price in Europe is significantly affected by the crude oil price (Asche et al., 2018).

The US natural gas spot and futures prices and their responses to storage change surprises are examined by Chiou-Wei et al. (2014). The results suggest that there is a negative relationship between changes in futures price and surprises in the change in the natural gas in storage. The study further indicates that the change in the price level will be firstly reflected in the futures market and then the information will be flown to the spot market.

Efforts have been made in the literature to investigate the predictability of natural gas spot and futures prices, in particular, using daily data of Henry Hub natural gas spot and futures prices, Mishra and Smyth (2016) answer the question of whether futures natural gas prices can predict the natural gas spot prices. The finding show that the predictability of futures natural gas prices is not better than the one predicted from the random walk model.

Therefore, as far as we are concerned, we are the pioneer to use multivariate copulas together with HMM to capture the joint variations of energy prices and demands in future and spot markets in a stochastic manner. Not only the multivariate copulas modelling adopted here can represent the times series being examined in a more precise manner, but they also allow follow-on robustness analysis by means of bootstrapping in terms of the models (VAR and ML) used to unveil their time dependence and endogeneity. In summary, we have the following research hypotheses for the Iberian energy industry:

H1a

Spot energy prices are autocorrelated.

H1b

Spot energy prices are endogenously time-dependent with future energy prices, spot energy demand, and future energy demand.

H1a has been documented in the literature, as we can see that Park et al. (2006) present the same findings in terms of the relationship between spot prices in the US electricity market, therefore, we hypothesize that there would be the same case for the Iberian energy market. In terms of H1b, the causality runs from futures prices to spot prices is in accordance with Moose and Al-Loughani (1995), while Pindyck (2011) argues that most of the commodities including the energy, the price level will have a positive correlation with the condition of the macroeconomic environment due to the fact that an economic growth that will push up the demand level and further increase the level of energy price.

H2a

Future energy prices are autocorrelated.

H2b

Future energy prices are endogenously time-dependent with spot energy prices, spot energy demand, and future energy demand.

The research related to the relationship among futures energy prices is conducted by Sadorsky (2000) shows that the futures prices of crude oil, heating oil and unleaded gas are cointegrated with each other. The causality runs from spot energy prices to futures energy prices is in line with Bekiros and Diks (2008). Due to the fact that the futures prices are an indicator that reflects the traders’ expectations on future demand conditions (Demirer et al., 2012), this will be reflected from both the spot energy demand as well as futures energy demand.

H3a

Spot energy demands are autocorrelated.

H3b

Spot energy demands are endogenously time-dependent with future energy prices, spot energy prices, and future energy demand.

The research has illustrated that price would be one of the main factors that will influence the energy demand (Adeyemi & Hunt, 2007; Zhang & Kotani, 2012), therefore, we hypothesize that the spot energy demand will be influence by both the price level in the spot markets and futures market. As reviewed in the literature sector, there is evidence of price convergence in the spot energy market, and due to the consideration price is one of the main factors influencing the demand level, we hypothesize that the spot energy demand will be correlated with each other. Futures energy prices will significantly influence the level of futures energy demand, while futures energy prices will indirectly exert an impact of spot energy demand through spot energy prices, therefore, we argue that there should be an impact of futures energy demand on spot energy demand. The influence of price on demand in the spot energy market is evidence by Bonte et al. (2015) who find that the average price elasticity of demand in European Power Exchange spot market is − 0.43 between 2010 and 2014.

H4a

Future energy demands are autocorrelated.

H4b

Future energy demands are endogenously time-dependent with future energy prices, spot energy prices, and spot energy demand.

We hypothesize the autocorrelation among futures energy demand is based on the findings of Lin and Tamvakis (2001) who argue that spillover effects do exist in the futures energy market. As discussed previously, the price level will be an important factor affecting the level of demand in both the spot and futures markets, because there is a level of correlation between spot prices and futures prices, therefore, we argue that there would be a cross-influence of price level on the demand level between spot and futures energy markets. Finally, Silva et al. (2019) find that there is a positive correlation between spot and futures market in the Iberian electricity market, because a higher spot energy demand will increase the spot energy prices, the further positive influence on futures energy prices will impact on the futures energy demand.

Readers should recall that, as long as inertia in the time series itself is produced by the autocorrelation, thus smooth variations are obtained while sudden abrupt changes such as peaks and valleys are avoided, on the other hand, endogeneity reflects the fact that either there would be an unclear cause effect or there would be a simultaneous feedback mechanism, both of which have a probability of temporal dependence.

3 Methodology

The HMM approach developed in this paper is built on the premises that energy prices and consumptions are not-only endogenous but also time dependent. A number of approaches such as VAR, ML, and distributional modelling through copulas are employed in conjunction with the HMM approach as described next.

3.1 The data

Data were collected from the OMIP website (http://www.omip.pt/Downloads/tabid/104/language/pt-PT/Default.aspx) 2007 to 2017 for both Iberian countries. While Table 1 reports on the descriptive statistics, Table 2 shows the non-parametric Kendall´s Tau correlation coefficients, useful for modelling joint distributions using copulas.

Table 1 Descriptive statistics
Full size table
Table 2 Non-parametric Kendall’s Tau correlation results
Full size table

Table 1 indicates that both price and energy consumption are asymmetrically distributed and low dispersed (with the exception of future energy demand), Table 2 shows a relevant correlation pattern not only between energy prices in future and spot markets, but also between prices and energy in spot markets. Figure 1 sheds some light into the endogenic behavior among these variables: one can easily note that both prices and energy consumption reflect the ongoing feedback processes, for a given time t, that occur between future and spot markets. Besides, prices and energy are auto-correlated, at least between t and t − 1.

Fig. 1
figure 1

Endogeneity in prices and energy time series (spot and future markets)

Full size image

3.2 Vector auto-regressive (VAR) model

The VAR model is formed by a set of \(K\) endogenous variables \({y}_{t} = \left({y}_{1t},..., ykt,...,{y}_{Kt}\right)\) for \(k = 1,...,K\). The VAR(p)-process is then defined as (Pfaff, 2008):

$$ = A_{1} y_{t - 1} + , \ldots , + A_{p} y_{t - p} + u_{t} , $$
(1)

with \({A}_{i}\) are \(\left(K x K\right)\) coefficient matrices for \(i = 1,...,p\) and \({u}_{t}\) is a \(K-dimensional\) process with \(E\left({u}_{t}\right)= 0\) and time invariant positive definite covariance matrix \(E\left({u}_{t}{u}_{t}^{T}\right)={\sum }_{u}\) (white noise). Stability is an important characteristic of a VAR(p)-and can be checked by assessing the characteristic polynomial:

$$ \det \left( {I_{K} {-} A_{1} z - \cdots {-} A_{p} z^{p} } \right) \ne 0\quad {\text{for}}\;\; \left| z \right| \le 1. $$
(2)

In this research, the VAR(p)-process is reduced to a VAR(1)-process (Pfaff, 2008):

$$ \xi_{t} = A\xi_{t - 1} + v_{t} , $$
(3)

with

$$ \xi_{t} = \left[ {\begin{array}{*{20}c} {y_{t} } \\ \vdots \\ {y_{t - p + 1} } \\ \end{array} } \right],\quad A = \left[ {\begin{array}{*{20}c} {A_{1} } & {A_{2} } & \ldots & {A_{p - 1} } & {A_{p} } \\ I & 0 & \ldots & 0 & 0 \\ 0 & I & \ldots & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \ldots & I & 0 \\ \end{array} } \right], v_{t} = \left[ {\begin{array}{*{20}c} {u_{t} } \\ 0 \\ \vdots \\ 0 \\ \end{array} } \right], $$
(4)

where the dimensions of the stacked vectors \({\xi }_{t}\) and \({v}_{t}\) is \((KP x 1)\) and the dimension of the matrix \(A\) is \(\left({K}_{p}x {K}_{p}\right)\). Time series plots for energy demand and prices in spot and future markets are given in Fig. 2. Smoothed time series, such as in the case of future prices and spot energy may suggest not only the presence of autocorrelation but also endogeneity with other explanatory variables. Conversely, more nervous time series, such as spot price and future energy may indicate negligible autocorrelation and also a limited feedback process with other variables.

Fig. 2
figure 2

Time series for energy demands and prices in future and spot markets

Full size image

3.3 Multivariate copulas

Copulas are a useful tool for modelling multivariate dependence (Yan, 2007) with a special focus on the impact of tail dependence on data analytics and forecasting (Wang & Pham, 2012). In this research, the first step towards the multivariate copula modelling was the distributional fit for each one of the price and energy variables. The goodness of fit procedure adopted here observed two parts. The first part consisted of an indicative approach to the best fit by the Cullen & Frey graph, showed in Fig. 3 Cullen and Frey (1999). The second part consisted of a Maximum Likelihood Estimation (MLE) to compute the parameters of the chosen distributions in the first step (Delignette-Muller & Dutang, 2014).

Fig. 3
figure 3

Cullen & Frey graphs for the spot and future energy and demand prices

Full size image

A resample size of 100 was adopted here to generate the yellow dots presented in Fig. 3. Table 3 summarizes the best fitting probability distributions and their estimated parameters for each energy price and demand variables. Results suggest that beta distribution exhibited the best fit for energy demand in future and spot markets, while energy prices in future and spot markets were best represented by a logistic distribution.

Table 3 Results for the best distributional fit
Full size table

3.4 Multivariate HMM with trend

Let´s consider j time series {(Xt1, Xt2, …, Xtj): t = 1, …, T}, which are represented compactly as {Xt: t = 1, …, T}. Let´s also assume that, conditional on C(T) = {Ct: t = 1, …, T}, these random vectors are mutually independent (Zucchini et al., 2016). To specify an HMM for such time series it is necessary to postulate a model for the distribution of the random vector Xt in each of the m states of the parameter process. That is, one requires the following probabilities to be specified for t = 1, 2, …, T, i = 1, 2, …, m, and all relevant x: tpi(x) = P(Xt = x|Ct = i). Let´s keep the time index t, thus allowing the state dependent probabilities to change over time. It should be noted that it is not required that each of the time series have the same distribution.

In this research, we use multivariate copulas, as discussed in the previous section, to model stochastic variations in HMM. Once the required joint distributions have been selected, i.e. state-dependent probabilities tpi(xt) have been specified, the likelihood of a general multivariate HMM has the same form as that of basic HMM models:

$$ L_{T} = \delta_{1} P\left( {X_{1} } \right)\Gamma_{2} P\left( {X_{2} } \right) \ldots \Gamma_{T} P\left( {X_{T} } \right)1^{\prime } , $$
(5)

where \({\mathrm{X}}_{1},\dots ,{\mathrm{X}}_{\mathrm{T}}\) are the observations and the likelihood function is given as (Zucchini et al., 2016):

$$_{t} P\left( {X_{t} } \right) = {\text{diag}}\left( {_{t} p_{1} \left( {X_{t} } \right), \ldots , _{t} p_{m} \left( {X_{t} } \right)} \right) $$
(6)

Equations (5) and (6) hold for continuous-valued series, provided that probabilities are interpreted as densities where necessary. Therefore, in the stochastic multivariate HMM model adopted in this research, VAR and ML models are used in a complementary fashion. In fact, the model tpij = aij + bt is taken as the starting point for the pseudo-code further discussed, where tpij is the probability that time series j belongs to state m on day t, observing the following set definition given in the first paragraph of this section:

T = {1…days}, where days is the number of days of the sample;

J = {spot energy prices (sp), future energy prices (fp), spot energy demand (se), future energy demand (fe)};

M = {m1….m256}, where each m is given observing the quantile combinations for each individual time series of energy prices and demands, as given in Table 4.

Table 4 Multivariate HMM states
Full size table

4 Presentation and discussion of results

Figures 4 and 5 present the results of transition matrix, which includes fifteen more portable states and their respective subsets. we can see that there are 235 individual states out of 256 have more than zero probabilities, however, there are only 1580 possible combinations out of 55225 have more than zero probabilities that the results indicate that, when using Markov chains and their transition probabilities to analyze energy demands and prices, there is only a small proportion of possible movements. This can be possibly attributed to the strong feedback mechanisms between energy prices and demands in future and spot markets in parallel with autocorrelation that reduces the level of abrupt fluctuations. With regard to future and spot energy prices, prices are found to be presented in the same quantile for the 15 more probable individual states (cf. Table 4, footnote), while future and spot energy demands present a more detached behavior. This maybe the result of prospective and speculative movements with respect to future energy demand behavior. The detached behaviour between spot energy demand and futures energy demand can be partly attributed to the fact that investment in the spot market has a lower level of risk, the information can be more easily obtained compared to the investment in the futures market. In comparison, investment in the futures markets require a larger volume of funds and also it suffers from a higher level of risk, it requires the investors to have a certain level of skill and experience in the investment activities, because these two investments have different characteristics and feature, this determines the differentiated behaviour of demands in these two markets. In reality, Fig. 5 shows the probable movement between different states, a subtle movement in the quantile interval of future energy triggers the transition, all other three variables remaining at their respective quantiles. It is also interesting to note that the two more probable movements are related to transitions to the same state, thus suggesting inertia.

Fig. 4
figure 4

The most probable states’ Pareto plot

Full size image
Fig. 5
figure 5

Most probable state transitions’ heat map (zero probability is represented by blank cells)

Full size image

The convergence of spot prices in the current study is in accordance with the finding of King and Cuc (1996). This is in line with our hypothesis H1a. While the detached behaviour between spot energy demand and futures energy demand seems not be explicitly addressed by the previous literature yet. This investigation of this issue is very important due to the fact that understanding the demand behaviour in the spot and futures markets will provide a basis to further understand and formulate a reasonable price mechanism for the market. We further find that the in general spot prices and futures price are related with each other, this is also in line with the previous literature as reviewed in Sect. 2.

The results of the bootstrapped VAR model as well as the feedback mechanisms among energy price and demand in the spot and futures markets are reported in Figs. 6 and 7, respectively. In terms of Fig. 6, if the horizontal solid line in zero does not cross the dispersion diagram, that indicates that the variable is significant. We further find that the significant relationships between energy price and energy demand are positive. As reflected from both these two figures, there is a complex feedback mechanism between energy price and demand in both the spot and futures markets. In particular, Fig. 7 is the response to the hypotheses raised in the end of Sect. 2.

Fig. 6
figure 6

The bootstrapped VAR with 2 lagged periods’ coefficients

Full size image
Fig. 7
figure 7

Feedback mechanisms between energy prices and demands in future and spot markets

Full size image

It is interesting to note that, as suspected in Fig. 2, spot prices and future energy are not autocorrelated, thus implying smaller inertia in their fluctuations. As a matter of fact, spot prices are only time dependent on future prices and spot energy, this result partly supports our hypothesis H1b, however, our results are not exactly in line with the argument of Pindyck (2001), we find that the spot prices are not time dependent on future energy demand. While future energy is solely time dependent on spot energy behavior. These results suggest that the behaviour of futures energy demand is affected by the spot energy demand not the important factor futures energy prices as discussed previously. This indicates that in the Iberian energy markets, the investors normally focus more on the spot price movement in their investment decisions in the futures market.

On the other hand, future prices are not only autocorrelated but also time-dependent with spot energy and future energy demands level, thus suggesting that the price formation mechanisms differ between spot and future prices. The autocorrelation of future prices is in line with Lin and Tamvakis (2001) and in accordance with our hypothesis H2a. The findings also partly support our hypothesis H2b, however, the findings are different from Bekiros and Diks (2008) who report that there is a causality running from spot prices to future prices, this difference is mainly attributed to the fact that we adopt a different methodology and different data period was used. This has a number of policy implications for the Iberian energy industry, for instance, capacity expansion should be one of the focuses in the Iberian energy sector through introducing additional investment. This additional supply of energy in the market will not only increase the competitive conditions but also stabilize the price level on the futures market and reduce the risk for the companies purchasing the futures contract.

A similar behavior occurs with spot energy, which is autocorrelated and time-dependent with future prices and future energy. Results suggest that part of the behavior of spot energy time series is not only driven by current demand levels, but also by recent expectations on future behavior of energy and future prices. One of the scenarios is a higher level of futures price will induce an increase in the spot energy demand. We can also see from our results that the spot energy and futures energy demands are correlated with each other. This result is partly in line with our hypotheses H3b and H4b. this finding also accord with Silva et al. (2019).

Finally, Figs. 8 and 9 presents the results from the bootstrapped multinomial logit regression, which provides more information about the underlying more probable states between energy price and energy demand. We notice that there is a level of stability of the probabilities of belonging to a given state although there are few exceptions (cf. Figure 8), in addition, the figure shows that the significant trend components do not influence these states (cf. Figure 9), this finding can also be supported by Fig. 2. These results suggest that there is no via lotion on the main assumptions of HMM with regard to its application to the Iberian energy industry.

Fig. 8
figure 8

Results on bootstrapped multinomial logit model for the probability of 15 more probable individual states

Full size image
Fig. 9
figure 9

results of bootstrapped multinomial logit model on the trend coefficients of 15 most probable states

Full size image

Different from the previous literature, which finds that the price elasticity of demand is negative, indicating that there would be a negative relationship between price and demand level, the current study finds that the relationship is significant and positive, suggesting that in the Iberian energy industry, there is a large volume of demand without any influence by the price level, therefore, one of the recommendations to the Iberian energy market is to encourage investment in this energy sector, expand the energy producing capacity, which would be helpful to form a reasonable market trading mechanism.

5 Conclusions

The energy sector is very important for the economic and societal development for all countries in the world, and for the Iberian energy industry specifically, it suffers from a number of issues. For the Spanish energy sector, numerous sources of uncertainties arise because of the historically highly unpredictability of the sector itself. For the Portuguese case, a balancing mechanism is urgently needed to implement renewable resources and encourage new investment through providing more capacity incentives (Amorim et al., 2013). In terms of two specific trading patterns in the energy market, the spot market and future market, Ciarreta and Zarraga (2016) argue that The Iberian energy sector has the characteristics of price and transmission volatility, while Capitán-Herráiz and Rodríguez-Monroy (2013) find that the Iberian energy sector lacks of liquidity in the futures energy market. These issues can be solved thorough different ways from various perspectives, while one possible route is to provide more detailed and depth analysis about the price mechanism in the spot and futures energy markets by investigating the inter-temporary interactions between the price level and the demand conditions. In addition, for both of these two countries, sustainable economic development has been an issue concerned by the governments following the hard hit of the financial crisis, investment in the renewable energy will not only promote sustainable development, increase the employment and improve the economic growth, but also will reduce the energy external dependency of these two countries. Therefore, the process of decarbonization is the focus of these two countries in their process of moving toward a greener economy. Investigating the inter-temporal relationship between demand and price level in the spot and futures energy market will be helpful not only to establish a marketable price mechanism, but will also provide more guidance, information and motivation for these two countries to further increase their effort in reducing the level of pollution for energy generation.

The current study significantly contributes to the literature in terms of spot and futures energy markets by empirically investigating the inter-connections among spot energy prices, spot energy demand, futures energy prices as well as futures energy demand. Methodologically, the current study proposed a stochastic Hidden Markov Model, which benefits from the advantage of being able to examine the endogeneity and temporal dependence under the facilitation of bootstrapped vector auto-regressive model and multinominal logit model. Our findings suggest that when using Markov chains and their transition probabilities to analyze energy demand and prices, there is only a small proportion of possible movements; in addition, we find that all significant relationships between energy prices and demands are positive; it is shown that spot prices are only time dependent on future prices and spot energy, while future energy is solely time dependent on spot energy behavior; the results further report that future prices are not only autocorrelated but also time-dependent with spot energy and future energy demands level; finally, we find that spot energy is autocorrelated and time-dependent with future prices and future energy.

Our study generates policy implications to solve the issues faced by the Iberian energy market. In order to improve the liquidity in the futures market, the price setting up mechanism can be based on the spot and futures energy demand. This is supposed to set up a price level that will more accurately reflects the demand conditions and further improve the level of liquidity (the seller would be able to sell the energy in the futures markets more quickly). On the other hand, the positive relationship between energy price and energy demand indicates that higher energy price leads to an increase in the energy demand, this can be understood from the perspective that energy is necessity product, the price mechanism can not only reflect the demand condition, but it can be used to adjust the level of energy demand. In other words, different types of policies can be used during different time (i.e. during the period of energy supply shortage, the price level can be reduced to decrease energy demand, in comparison, during the time of energy supply surplus, the energy price can be lifted up). Obviously, the volatility in the price level of the spot market can be alleviated by considering the level in the spot and futures energy demand, while probably increasing the incentive for the incumbent firms to invest in the energy sector will not only increase the competitive conditions in the Iberian energy sector considering that there are only few suppliers in the market, but also plays an important role in stabilize the price level on the spot market. The main purpose of futures energy market is to reduce the price volatility and future reduce the risk, from our results, it shows that spot energy demand and future energy prices and futures energy demand are time dependent with each other, therefore, in order to reduce the price volatility in the energy futures market, the priority is the increase the stability in the spot and futures energy demand.

As discussed in the introduction, for the future research, we would make use of our proposed HMM model to predict and assess the decarbonized alternative sources of energy with the facilitation of financial equations.Footnote 1