Intraday imbalance optimization: incentives and impact of strategic intraday bidding behavior

Intraday markets are crucial to balance supply and demand in the very short-term, up to delivery. They are often designed as continuous auctions with a pay-as-bid pricing mechanism. While several studies assess trading strategies to balance different types of portfolios, they normally do not consider the incentives of the imbalance prices for portfolio management. This paper analyzes a strategy of taking positions in the German intraday market based on expected imbalance prices and examines its impact on system stability. Using a logistic regression model, it is possible to accurately predict the direction of the overall system balance and to apply a profitable trading strategy. For a period from 01/07/2017 to 30/06/2019, the strategy outperforms a simple approach by EUR 47 000 per MW. However, this behavior would predominantly not have been system supportive due to biased imbalance price incentives. These are asymmetric price spreads and insufficiently low imbalance prices compared to intraday prices. An efficient intraday price constraint would partly solve the problem. The overall share of system supportive imbalance positions would raise by ten percentage points. In situations with high system wide imbalances, up to three-quarters of the positions would stabilize the system. These findings are important for regulation in Germany and other countries with a single imbalance pricing as they provide an indication for crucial points of the imbalance pricing rules to incite appropriate market behavior.


BRP
Balancing responsible party Delta Deviation between a day-ahead forecast and the actual

Introduction
Electricity trading in European countries is done in a sequence of interrelated markets serving different purposes to managers of generation and consumption portfolios. Net suppliers and demanders use long-term contracts to hedge their price risks by buying and selling standardized base and peak products on an exchange (futures) or tailored bilateral contracts (forwards) [1]. The spot market is the main market for physical delivery in hourly or quarter-hourly resolution. Most of the volume is traded at the day-ahead market in a unique auction to aggregate liquidity [2]. The intraday market gives portfolio managers the opportunity to trade until a couple of minutes before delivery and thus react to latest information. This is crucial for portfolios dominated by fluctuating renewable sources such as wind and solar, as the forecast accuracy increases with less time to delivery [3,4]. These forecast deviations have a major impact on intraday prices as several studies show [5][6][7][8][9][10]. As intermittent electricity generation becomes a more significant part of the electric power grid, intraday markets are important for balancing supply and demand minute-to-minute.
Consequently, examining efficient market design comes to the fore of scientific analyses [11][12][13][14][15]. Next to the question of an appropriate market design, another research focus is on the development and assessment of bidding strategies [16][17][18][19][20][21][22]. It has a significant impact on the market value of both conventional and renewable power plants [23]. So far, these two research topics have been analyzed separately as in the aforementioned studies, but they are closely linked. On the one hand, market design affects the profitability of trading strategies. On the other hand, market behavior influences system stability and must be considered within the regulatory framework. Therefore, this study goes beyond an individual consideration of these research topics and combines them by examining a trading strategy and analyzing its impact on the system balance. The question is whether it is possible to generate a return with a speculation between current intraday and expected imbalance prices for Germany and if the underlying positions would stabilize the system or not.
There are papers investigating partly the incentives of balancing mechanisms. Just and Weber [24] discuss the opportunity of strategic optimization between spot market and balancing energy. But they focus only on situations where the imbalance price incentive is too low compared to the spot price. Based on empirical data, they try to proof a misconduct of market participants in such situations and draw conclusion on overall market data. The same holds for the study of Möller et al. [25] who try to model the balancing energy demand considering arbitrage opportunities. This analysis was done under a completely different market design as it is now. The intraday market was illiquid and there was no marketplace with trading products in quarter-hourly resolution. Therefore, their model includes only speculation on the day-ahead market and an important factor is the quarter-hourly difference of load compared to the hourly mean. Van der Veen et al. [26] analyze more generally the impact of different imbalance pricing mechanisms on a balancing strategy for portfolio managers. They focus on a day-ahead optimization looking for a general strategy without predicting the actual balancing situation.
In contrast to these studies, the present paper provides a holistic assessment of strategic intraday imbalance optimization. It is based on a unique data set of intraday trade and order book data for Germany from 01/07/2017 to 30/06/2019. As intraday trading is possible until a couple of minutes before delivery, it enables to estimate the imbalance prices and to individually decide in each quarter-hour whether to take a buy or sell position. So, the applied strategy simulates a decision with available information during active trading considering the current and not just average market prices. It turns out that an advanced strategy based on a logistic regression would have made a profit of EUR 42,000 per MW whereas a simple strategy based on lagged system balances would cause a loss of EUR 5000 per MW.
In a second step, it is possible to examine the impact of this strategy on the overall system. Even though the model was able to classify the system balance correctly in 68% of all quarter-hours, the strategy would not have been system supportive. More than half of the positions would have increased the actual system balance, because perverse price incentives would have caused some willful misconduct. The reasons are asymmetric price spreads between positive and negative system balances and insufficiently low imbalance prices compared to intraday prices. The implications of bad market behavior became apparent during some days in June 2019 when severe situations with balancing reserve activations of up to 7.5 GW have been aggravated by prohibited conduct of some market players [27]. Such situations can be avoided with a stricter intraday price constraint as part of the imbalance price calculation. The TSOs provide a new proposal [28] which is also analyzed in this paper. The willful misconduct would decrease by 12 percentage points and the positions based on the trading strategy would be predominantly system supportive -especially in situations with high system balances.
The paper is structured as follows: Sect. 2 provides background information about the intraday market and balancing system and the price incentives for intraday imbalance optimization. Section 3 introduces the trading strategy and the underlying methodological approach. Section 4 examines the success of the strategy and its impact on system stability. It also analyzes the effect of a stricter intraday price constraint with regard to profitability and the behavior related to the overall system. Section 5 concludes.

Background
This section provides some context to the analysis, first by introducing background information on the German intraday market and the institutional setup of the balancing system. It also contains a detailed explanation of the imbalance price calculation and its price incentives for strategic behavior on the intraday market.

The intraday market
The intraday market facilitates trading until a couple of minutes before delivery in order to balance portfolios of generators and/or loads. For the German market area, EPEX SPOT is the major exchange for short-term trading. The intraday trading period officially starts with a quarter-hourly auction at 3 pm the day before delivery. It was introduced in December 2014 and enables to trade systematic deviations to the hourly mean and thus scheduling in the required 15-min resolution. After this auction, the intraday market is designed as continuous auctions for hourly and quarter-hourly contracts. Market participants may submit their orders during the whole trading session and a transaction is executed immediately as soon as two entered orders match. The pricing mechanism is designed as a pay-as-bid scheme. Gate closure of continuous trading is at 30 min before delivery with an extended period up until five minutes before delivery for trading within the same control area [29]. Table 1 summarizes key figures for the German intraday market. There are 270 exchange members registered for the continuous trading and 133 for the quarterhourly auction [30]. The average prices are around 40 EUR/MWh for all three different product types showing that systematic arbitrage potential between the markets is hard to find. However, trading volumes are much higher on the hourly than on the quarter-hourly market. More than 5000 MW are traded on average per hourly contract, whereas it is 700 MW for the continuous quarter-hourly trading and 750 MW for the quarter-hourly auction. This shows that major deviations to the day ahead forecast are compensated with hourly contracts due to the lower effort. The quarterhourly markets are predominantly used to compensate deviations from the quarterhourly to the hourly mean.
A large share of continuous intraday trading is done close to delivery to process the best information about the portfolio balance. A common index is based on the ID1 trading period which covers all trades executed within the last trading hour of a contract up to 30 min before delivery start [31]. This period contains 38% of the hourly and even 48% of the quarter-hourly trading volume.

The balancing system
In AC electric grids, it is necessary to ensure the balance between demand and supply at every point of time to maintain a stable frequency and to ensure a reliable electricity supply. A number of regulations, processes and markets have been developed to prevent large deviations. In Europe, the core of this system is the active portfolio management of all market actors (generators, consumers, retail suppliers and traders). They take the role of a balancing responsible party (BRP) and are obligated to balance their portfolio through dispatch of physical assets or trading.
In Germany, there are more than one thousand BRPs [32 -35]. Each of them must send their schedules to the associated transmission system operator (TSO). The remaining deviations between schedules and the actual physical positions are called imbalances. Positive and negative imbalances offset each other and the TSOs compensate only the net position by the activation of balancing reserves.
This compensation is done in high time resolution, but the imbalance settlement period is 15 min in Germany. The average net imbalance of all balancing groups is called system balance ( SB ). It includes all measures of balancing the remaining deviations. These are the activation of automated and manual Frequency Restoration Reserve ( aFRR and mFRR ) and additional measures ( AM ) as well as emergency measures for or from foreign TSOs ( EM). 1 Furthermore, there is an International Grid Control Cooperation ( IGCC ) to prevent unnecessary balancing reserve activations and a cooperation with Austria to exchange aFRR capacity whenever possible [36]: The system balance is defined from the perspective of the predominant direction of activated measures. Thus, a negative system balance means an activation of negative balancing reserves and represents a power surplus in a grid. A positive system balance corresponds to a shortage of supply in the system.
The system balance is also a key element for the calculation of the imbalance price within the German imbalance settlement. The general idea is to distribute the costs of all balancing reserve activations for every settlement period to the BRPs, who caused the activations. The calculation reads: This basic determination is supplemented by different constraints to limit or increase the imbalance price. In case of a small quarter-hourly system balance, the fraction can lead to high imbalance prices even though the system is not at risk. This shall be avoided by a price cap to the highest working price of any activated asset within the particular quarter-hour. There is also a linear function limiting the imbalance price when the system balance is between − 500 MW and 500 MW.
However, the imbalance price must always incite active intraday trading to balance predictable deviations. Currently, the comparison price is the volume weighted average price of the corresponding hour. This constraint is under consultation because it does not prevent that quarter-hourly intraday prices are higher than the imbalance price and there is no focus on the market situation close to gate closure [37]. Section 4.3 examines the potential impacts on the behavior of market participants.
The last condition is a surcharge, if TSOs must activate 80% of the procured aFRR and mFRR within one quarter-hour. 2 It is the maximum of 100 EUR/MWh or half of the imbalance price [39].

Price incentives for intraday imbalance optimization
The main purpose of the imbalance price is to incite active trading on the in an appropriate manner. Van der Veen and Hakvoort [40] discuss some of the important design variables that must be considered to develop suitable frameworks. European countries follow two different ideas of imbalance pricing mechanisms: single and dual pricing systems [41]. In dual pricing systems, BRPs with imbalances mitigating the system balance only receive a lower imbalance price. It is typically based on the day-ahead price [42]. A single imbalance pricing means, that BRPs, whose imbalance position is opposed to the system balance, receive the same price as the BRPs pay, who reinforce the system balance. This is the imbalance pricing mechanism in Germany and it incites to take an intentional imbalance position [43,44]. If the expected imbalance price is lower than the current intraday market price, it might be beneficial to take a sell position and pay the imbalance price (and vice versa).
Such a behavior is not allowed in Germany. According to Sect. 4 (2) StromNZV and the balancing group contract, BRPs are obliged to keep imbalances to a minimum by taking reasonable measures [45]. Thus, it is not allowed to take intentional imbalance positions. However, empirical analyses indicate that at least some market participants react to imbalance price expectations [46]. It needs an appropriate imbalance price so that this activity stabilizes the system. The imbalance price must be higher than the intraday market price if the system balance is positive (shortage) and lower than the intraday price if the system balance is negative (surplus). This is especially important for trading close to intraday gate closure when market participants have the best indication for the own position and the overall system status of the associated settlement period. Empirical data show that there have been situations with inefficient imbalance prices. Between 01/07/2017 and 30/06/2019, the imbalance price incentive was too low in 13% of the quarter-hours (Table 2). Market participants will contemplate this, if they react to imbalance price incentives. Thus, it is important to consider it when modeling their behavior.

Methods and materials
This section presents the model approach to simulate the strategy of a market participant who takes intentional intraday positions at the EPEX SPOT market based on an imbalance price estimation. The imbalance price prediction is compared with the prices of the limit orders 3 at the decision point. If the expected imbalance price is higher than the best sell order, the market actor will take a buy position. If it is lower than the best buy order, the he or she will take a sell position. The period of our analysis spans from 01/07/2017 to 30/06/2019. Looking at the imbalance price formula presented in Eq. (2), the price estimation should start with predicting the system balance. With this information, it is possible to approximate the activation costs in the numerator by considering the potential working price payments using the merit order of the different measures. With these information, it would be possible to directly estimate the imbalance price (IP) depending on the system balance (SB): E[IP|SB]. The problem of this approach is the high volatility of the system balance. During the analyzed period, it has a standard deviation of 530 MW with an average absolute value of only 490 MW. Hüttinger [47] developed different models for a deterministic prediction. The root mean squared errors were not better than 350 MW proving the high uncertainty of such forecasts.
An alternative way is to use a binary classification model to estimate the probability of a positive system balance which is the concept being applied in the present paper. This approach is to calculate an expected profit for a sell and a buy position based on actual intraday bids and the expected imbalance price for a positive and a negative system balance. The advantage is that the trader is able to consider both the probability of the potential system status and the associated price spreads. Figure 1 shows an overview of the trading strategy which is explained in detail below.
The approach starts with a classification model to predict the probability of a positive system balance ( E Pr pos ). It is not appropriate to use a linear probability model as the fitted probabilities can be lower than zero or greater than one. This limitation can Fig. 1 Overview of the trading strategy. It considers the best buy and sell limit orders (IDP Buy and IDP Sell ) at the decision point (4 min to gate closure). The trading volume for the strategy is set to 1 MW be overcome by using a binary response model taking a function G with values strictly between zero and one [48]: We use x to denote the full set of explanatory variables and to indicate the set of parameters.
In the present case, the chosen model is a logistic regression model, also called logit model, with where z is the function to link the probability of a positive system balance P pos to the explanatory variables. It is defined for every quarter-hourly time period t as: The first group of regressors contains fundamental variables. SB t−60 is the latest published system balance and ΔSB t−60 the gradient to the previous system balance. These parameters are included because of the strong autocorrelation of the system balance [49]. Additionally, variables for load, wind and solar generation are included as these portfolios are the main drivers of imbalances. Devs are the deviation of the quarter-hourly day-ahead forecast from the mean of that hour. The Deltas for wind, solar and load are the deviations between the day-ahead forecast and the actuals. Unfortunately, there are no public data for the latest intraday forecast, which could be deployed to consider fundamental deviations of wind, solar and load. So, taking the actuals is necessary for calculating an unbiased estimator of the deviation between day-ahead and latest intraday forecast. Using these regressors should be only a little additional information because the deviation between intraday forecast and actuals should have the highest influence on the system balance. However, an additional model is applied without these Deltas to calculate a floor of the potential profit of the trading strategy. Power plant outages are not considered as these imbalances must be compensated quickly after occurrence and shall not influence the system balance [45].
The second class of regressors is based on intraday trading data. Vol IDQ is the trading volume of all trades for the corresponding quarter-hour being executed until the time the model calculation starts. Vol IDQ,last15 is the same for the last 15 min. p IDQ,last15 is the volume weighted average price of intraday trades for the corresponding quarter-hour carried out within the last 15 min. It is also considered in relation to the intraday price of the associated hourly product p IDH , as this is the price level  Table 3 provides an overview of all variables and their data sources.
To address the problem of potential overfitting, a backward selection is applied using the Akaike information criterion (AIC) [50]. It is a well-established estimator for the quality of statistical models dealing with the trade-off between goodness of fit and simplicity of the model. Comparing a set of different candidate models, the preferred model has the minimum AIC value. A backward selection function in programming language R calculates the AIC for the complete model specification and every model excluding one of the regressors. It drops the variable for which the associated model has the lowest AIC. This is done until there is no further improvement by deleting a regressor. To reduce computing time and speed up the decision process, highly insignificant quarter-hourly dummies are dropped upfront.
As the approach aims to simulate the actual behavior of a market participant, the model is built as a rolling forecast. It is run only for one day and the training period contains the data of a fixed amount of latest past days (in this case 30 days). Therefore, the model reduction with AIC and parameter estimation is done separately for every day of the analyzed period. 4 The second step is to estimate an imbalance price IP for a positive and negative system balance (see again Fig. 1). It is not possible to calculate expected activation costs without a deterministic forecast of the system balance. Thus, it needs a more general approach. The price estimation is based on the working price bids of aFRR as it is the mostly activated balancing reserve measure in Germany [51]. We calculate for every tendered period tp for both positive and negative balancing reserve the average activation costs (sum of working price WP times volume q of a bid) for a fixed volume Q . We consider the lowest working price bids i until Q is reached. The considered volume Q is kept constant for every tendered period. It shall lead to price predictions that are on average equal to the mean imbalance price separately for negative and positive system balances of the total analyzed period (July 2017 to June 2019). Table 4 shows the results for different levels of volume Q . It follows to take a volume of 250 MW for positive aFRR and 700 MW for negative aFRR.
Additionally, the intraday price constraint is respected as explained in Sect. 2.1. The final imbalance price forecast considers the volume weighted average intraday price of the corresponding hour including all trades being executed until the decision point.
With the estimated classification probability and imbalance prices, it is possible to calculate the expected profit of an intraday buy and sell position for every quarterhour (step three in Fig. 1). We need to define therefor the time of trading simulation. It depends on data availability and model calculation time. Gate closure of Germanwide intraday trading is 30 min before delivery. 5 The empirical study of Maskos [49] shows that German TSOs regularly publish the past system balance 10.65 min after the quarter-hour. So, the latest information is available 4.35 min before gate closure. It takes about 20 s to collect the latest data, calculate the model and take a market position. Consequently, the strategy simulates intraday trading at four minutes to gate closure.
In addition to the estimated imbalance payment, the expected profit depends on the available orders in the market. The strategy replicates a market participant executing a market order. A buy market order stands for accepting the sell limit orders with the best prices until the requested volume x is reached. The market participant pays the volume weighted average price of the sell orders IDP xMW Sell . If the trader initiates a sell market order, he or she will pay the volume weighted average price of the buy orders IDP xMW Buy . The trading strategy is based on a one Megawatt trade. According to the law of iterated expectations, the expected profits for every quarter-hour are: There is also the "Same Delivery Area Trading" for trading within one of the four German TSO areas. This is possible until five minutes before delivery. It is not considered here, since the liquidity is lower than in German-wide trading [52].
If the expected profit of a sell position is positive, the market actor will sell an additional volume of 1 MW at the market leading to a shorter portfolio. An expected profit of a buy position causes a buy trade and a surplus of the portfolio. It is also possible that both expected profits are negative because of the price spread between the best sell and buy orders. The actual profit is calculated afterwards based on the actual prices of the intraday market orders and the actual imbalance prices of the analyzed period. Impacts on the imbalance prices can be neglected because of the small volume of the simulated trading strategy.
So, the described approach simulates a trading decision based on available information and evaluates its profitability with real market price data.

Results and discussion
This section presents the results of the trading strategy and the potential impact on the system balance. It starts with a discussion of the model results including model assumptions, the coefficients of the independent variables and the model accuracy. It continues by analyzing the potential profit of the trading strategy to validate it as a realistic approach for market participants. If it is possible to apply a beneficial intraday imbalance optimization, it is appropriate to evaluate the effect on system stability under the current imbalance pricing scheme and the potential adaptions to improve the trading incentives.

Model results
Logistic regression models do not make many of the assumptions of linear regression models that are based on ordinary least squares algorithms. There is no need for a linear relationship between dependent and independent variable and no assumption for the distribution of the error terms. Homoscedasticity is also not required. However, there are some assumptions that still apply. The dependent variable must be binary which is obviously given here. The same goes for the independence of the observations. Lastly, logistic regression assumes linearity between the independent variables and the log odds. Otherwise the test underestimates the strength of the relation. A solution would be to transform metric variables to an ordinary level. But transforming variables before modeling might cause problems such as distortion of confidence intervals or lack of fit. Instead, it is appropriate to examine the effective sample size to allow complexity of the model [53]. The effective sample size is the minimum number of observations for any class. According to Concato et al. [54] and Peduzzi et al. [55], there must be at least 10 observations per independent variable.
The minimum ratio is 11.61 for any model run. So, the sample size is always sufficiently high. As the assumptions for the logistic regressions are fulfilled, it is possible to analyze the estimated impact of different variables on the system balance direction. Figure 2 shows the share of selections for the fundamental and intraday variables and the mostly selected dummies. All fitted models select the lagged system balance and the deviations between the day-ahead forecast and the actuals (abbreviation: delta) for wind and load. The gradient of the lagged system balance is chosen in 90% of the model runs.
Delta PV seems to have a lower influence. A potential explanation is that TSOs manage PV portfolios who are willing to close every open position. Therefore, it could be that the system balance is less influenced by day ahead forecast errors. The fitted models also select less often the deviations to the hourly mean for all three fundamental factors. Intraday trading volumes seem to be less important as well, whereas intraday prices are predicted to have a consistent impact. The model chooses the volume weighted average price of the last 15 min before in 95% of all cases.
The dummy for the day being a working day is considered in three-quarter of all model runs suggesting that there could be a difference in trading activity or forecast accuracy between working and non-working days. Even though the model indicates a regular pattern for some quarter-hours, most of the quarter-hourly dummies are chosen less frequently compared to the other regressors. The share is lower than 30% for 74 dummies.    The heterogeneous parameterization of the model confirms the approach of individual model fitting for every day. But the premise of a successful intraday imbalance optimization is a high accuracy of the classification model. Figure 4 shows the ROC curve for the logistic regression model (logit model) illustrating the true positive rate against the false positive rate for different thresholds for the classification. It shows a clear deviation from the diagonal which represents an uninformative classification. At any threshold the model detects more true positive than false positive observations. The area under the curve is 0.734.
To evaluate the accuracy of the logit model and later the profitability of the underlying strategy (see Sect. 4.2), it is compared to a reference strategy. A simple, but efficient estimation can be based on the latest published system balance because the system balance has a high autocorrelation [49]. So, the simple model is that the sign remains the same for the traded quarter-hour.
The true classification rate is 68.6% for the logit model with a cut-off of 0.5 for predicting a positive system balance. This is 2.4 percentage points better than a prediction solely based on the latest published system balance (see Table 5). So even though the accuracy of the logit model is also slightly better at other thresholds, there is only little additional information by including variables other than the lagged system balance. The latter is the most important information for predicting the direction of the system balance.

Trading strategy
The model accuracy indicates the potential of a profitable intraday imbalance optimization. The simulation of the trading strategy applies the approach presented in Sect. 3. It uses the opportunity of predicting a probability for the system balance direction and combines it with the expected price spread for a sell and buy position. Therefore, it may be that the decision is to take a position in the same direction of the expected system balance, if the expected price spread differs significantly. This strategy is compared with a simple one based on the lagged system balance. If the lagged system balance is positive (system shortage), the trader will take a buy position to be oversupplied for the traded period and vice versa for a negative system balance. This approach does not need a complex model and the decision can be done faster. Therefore, the results are calculated with market prices of five seconds after the latest system balance publication (compared to 20 s for the application of the logit model, see Sect. 3). Table 6 presents the results for both strategies. They vary widely even though the model accuracy is only 1.4 percentage points better  Figure 5 shows the development of profits and losses for both approaches over the analyzed period. The strategy using the logistic regression model is profitable on 503 out of 730 days (Fig. 6) and in 90 out of 106 weeks. So, its success is not caused by some outliers, but by constant return and an application holds a reasonable risk for portfolio managers.
As explained in Sect. 3, the logit model uses the actuals for wind, solar and load to calculate an unbiased estimator of the deviation between day-ahead and latest intraday forecast. However, taking this information could lead to slightly better forecasts of the probability for the system balance. Therefore, an additional model is applied without these Deltas to calculate a floor of the potential profit of the trading strategy. The percentage of right decisions is at 67.7% and thus 0.9 percent points worse than the complete logit model. The profit is positive over the whole period at EUR 18,700. So, the profit of a model including the deviation between day-ahead and latest intraday forecast can be expected to be between EUR 18,700 and EUR 42,000.

Impact on system balance
Section 4.2 shows the potential of an intraday imbalance optimization. The introduced strategy is able to generate profits at reasonable risk. From a regulators perspective, it is therefore necessary to examine whether such a behavior would support the system under the current imbalance price mechanism. We analyze therefore how often the market player would have taken a position opposed to the actual system balance. We compare again the results for both strategies to illustrate the differences. They are presented in Table 7.
A market player would have executed a trade in 86% of all quarter-hours by applying the logistic regression model. Less than half of these positions would have been opposed to the system balance (47%). As the model accuracy is 68%, there must be a significant number of situations, where the expected imbalance price leads to a position that is in the same direction as the model prediction for the system balance. We dub these situations as willful misconduct. Such a behavior cannot happen by applying the simple strategy looking at the latest published system balance without considering the current market situation. In this case, the market player would trade anytime, and every right prediction leads to a position that supports the system. Additionally, it needs a detailed look on high absolute system balances as it is even more important to take no positions in these quarter-hours that further destabilize the system. The limit is set to an absolute value of 1000 MW including around 6% of all quarter-hours. The price incentives were even worse in these situations. The intraday position was opposed to the actual system balance in only one third of the quarter-hours mainly because of increasing willful misconduct.
There are two potential reasons for a willful misconduct. The first one is, that the price spreads are asymmetric for positive and negative system balances. This is Fig. 7 Absolute imbalance price spread as a function of the absolute system balance (SB) Fig. 8 Comparison of intraday and imbalance prices for positive (left) and negative (right) system balances. The red area highlights quarter-hours with insufficient imbalance price incentives. The intraday price is calculated as the volume weighted average price of all trades within the last 4.35 min of trading already discussed in Koch and Hirth [51] whose empirical analysis shows a systematic shift towards a system shortage. If the system balance is short, the spreads tend to be smaller (Fig. 7). For the analyzed period, the average spread was 17.62 EUR/ MWh for positive and 28.44 EUR/MWh for negative system balances. From a BRP's perspective, being undersupplied leads to a low penalty in case of a system shortage, but to high profits when the system balance is oversupplied. So, if the estimated probability for a positive system balance is slightly above 0.5, the strategy would still be to take a sell position and be undersupplied as well. Such a bias (in any direction) caused a willful misconduct in 35% of all quarter-hours.
The second reason is, that weak imbalance prices can lead to wrong incentives. For example, if the expected imbalance price is higher than the current intraday price even when the system is oversupplied, it is always beneficial to buy additional volumes on the intraday market and be oversupplied as well. If the expected imbalance price is in any case below the intraday market price, it is profitable to take a sell position. Under the current imbalance price mechanism, this is possible due to the weak intraday price constraint. The comparison price is the volume weighted average price of the corresponding hour. This does not prevent that quarter-hourly intraday prices are higher than the imbalance price as Fig. 8 shows. With regard to the trading strategy, it caused a misconduct in 11% of all quarter-hours and even 38% in situations with a system balance above 1000 MW or below − 1000 MW.

Stricter intraday constraint
The analyses of Sects. 4.2 and 4.3 allow two broad conclusions: it is possible for market participants to apply a profitable intraday imbalance optimization and it has the potential to stabilize the system (68.6% of correct classified system balance directions). But biased balancing incentives influence the decisions of the underlying strategy so that the final impact is negative (only 47% of supportive positions). The problem of perverse imbalance price incentives became apparent during some days in June 2019. There were four days with severe situations and balancing reserve activations of up to 7.5 GW in Germany [56]. Day-ahead forecast errors led to a shortage of several portfolios. The intraday market reflected this situation with high prices which were higher than the expected maximum of the imbalance price. So, market parties with forecast errors got the incentive to pay the imbalance price rather than correcting their schedule at the intraday market [57]. The German regulator became aware of infringements and cautioned five different BRPs [27].
Avoiding such situations requires a reform of the imbalance price calculation. It is not necessary to change the whole calculation approach, but it needs stricter constraints to give better price incentives to portfolio managers. This section focuses on the imbalance price coupling to the intraday price. This restriction is currently under discussion in Germany. The TSOs started a consultation emphasizing its importance [37]-especially because the imbalance costs will potentially decrease with the introduction of a separate auction of balancing energy which is scheduled for June 2020 [58].
The current proposal is based on the study of Consentec [59] and is described in [28]. The approach is to consider the volume weighted average price of the quarterhourly trades with the shortest time lag to delivery and an aggregated volume of 500 MW ( ID500 ). If the total quarter-hourly trading volume is lower than 500 MW, the price index considers as much of the latest hourly trades to reach the 500 MW. The constraint is not applied if the combined trading volume of quarter-hourly and hourly trading is below 500 MW. Additionally, there is a price markup which is 25% of ID500 , but at least 10 EUR/MWh. The price markup is reduced if the absolute system balance is lower than 500 MW. Thus, the definition of the imbalance price including the new intraday constraint ( IP ID ) is: Positive system balance: Negative system balance: We rerun the analyses presented in Sects. 4.1 and 4.3 with this new price restriction to evaluate the impact on the trading strategy and its effect on system stability. The assessment includes also a calculation with a volume of 200 MW instead of the recommended 500 MW. The reason is that the total quarter-hourly trading volume is below 500 MW in 32% of the quarter-hours and it also includes trades that are executed way before the gate closure at potentially different price levels. Considering an accumulated volume of 200 MW, there is still a moderate risk that market participants try to influence the price constraint, as they would take a relatively high position that shifts the own balancing group deviation in an unfavorable direction. The intraday price constraint considered at the decision point (4.35 min to gate (12) closure) is calculated with a reduced volume (100 MW less) to reflect that a share of the trades is executed afterwards. The constraint always includes the complete price markup to provide a conservative estimation. The evaluation starts by looking at the effect on the behavior in relation to the overall system as it is the overarching goal of the constraint to provide better incentives. An application of the constraint as it is proposed by the TSOs would have significantly positive effects (see Table 8). The share of quarter-hours with trading positions opposed to the actual system balance would increase from 47 to 54% mainly because of the decrease of situations with willful misconduct (46% compared to 34%). The impact on situations with high absolute system balances is even better. Without the stricter intraday constraint, the predominant strategy was to take a position in the same direction as the expected system balance. This reduced significantly and now the positions would support the system in two third of all quarter-hours. The willful misconduct caused by insufficient price incentives would decrease from 38 to 7.4%.
However, there is still a considerable number of quarter-hours were the constraint is still inadequate. Literally, it applies on average to almost two quarter-hours per day. Reducing the accumulated volume of the considered intraday trades to 200 MW would lead to better incentives. The willful misconduct reduces to a percentage of 31% and the predicted imbalance price is sufficiently high to 99.3%. The share of supportive positions increases by 3 percentage points. There is even a higher impact on situations with large system balances with 74% supportive positions. The share of quarter-hours with insufficient imbalance price incentives drops by another 6% points.
But what is the impact on the profitability of the strategy? Table 9 presents the profit again compared to a simple strategy only based on the lagged system balance. It turns out that the new imbalance constraint would increase the potential return of the strategy. It was EUR 42,000 under the current regime and would grow to EUR 63,000 with the intraday constraint supposed by the TSOs and EUR 76,000 with a considered volume of 200 MW. There would be an even higher influence on the strategy based on taking positions opposed to the lagged system balance. It was unprofitable before but would make profits of EUR 40,000 or rather EUR 51,000. It shows again that the advantage of combining predictions of system balance direction and expected price spreads was caused mainly by the biased price incentives which partly vanish by applying a stricter intraday price constraint.

Conclusion
This paper demonstrates that it is possible to apply a profitable trading strategy by taking intraday positions considering imbalance price expectations. The basis is a logistic regression model to predict the probability of the system balance to be positive or negative. This information is combined with an expected imbalance price for positive and negative system balances to estimate the profits of an intraday buy and sell position. From July 2017 to June 2019, this approach would have been profitable on 503 out of 730 days leading to a total return of EUR 42,000. The results are better than applying a simple strategy of taking positions that are opposed to the latest published system balance.
From a system perspective, this behavior would not have been supportive. Less than half of the decisions would have been opposed to the actual system balance even though the model was able to classify it correctly in 68% of all quarter-hours. The reason is that biased imbalance price incentives lead to a willful misconduct. Firstly, the price spread between intraday and imbalance prices are systematically higher for a surplus than for a shortage of the system. So, being undersupplied will lead to a low penalty in case of a system shortage, but to high profits when the system is oversupplied. If the estimated probability for a system shortage is slightly above 0.5, the strategy would still be to take a sell position and be undersupplied as well.
Secondly, there are multiple situations with imbalance prices providing insufficient trading incentives. This is the case if the intraday price is higher than the imbalance price even though the system is undersupplied or if the intraday price is lower when the system is oversupplied. In both cases, the beneficial strategy would be to take a position in the same direction of the system. It caused a willful misconduct in 11% of all quarter-hours and even 38% when the absolute system balance was higher than 1000 MW. The latter one is problematic as wrong market behavior can destabilize the system when a large share of the procured balancing reserve is already activated.
A part of the described problem could be addressed by a stricter intraday price constraint. The TSOs provide a new proposal for coupling the imbalance price to the intraday price. It covers quarter-hourly intraday trading and is more related to trading close to gate closure by considering the last trades with an accumulated volume of 500 MW. A rerun of the strategy shows that the profitability of the intraday imbalance optimization raises when considering for the proposed intraday price constraint. The total profit would have been EUR 63,000-an increase of 50%. But it is important to mention that the adjustment yields to a more system supportive behavior. The share of positions opposed to the system balance increases from 47 to 54% and the willful misconduct reduces from 46 to 34%. The price incentive would be even better when reducing the considered volume to 200 MW as the coupling price would be closer to the market situation close to gate closure. This applies especially for the quarter-hours with high system balances. The adoption of the strategy would reduce the system balance in 74% of all situations when using the strictest intraday price coupling. This shows the importance of efficient regulation. A simple improvement of the current imbalance price calculation enables system supportive intraday imbalance optimization.
However, there would still be a considerable share of quarter-hours with bad price incentives due to the asymmetric price spreads. This bias causes a systematic shift towards a system shortage, if market participants consider it in their decision. It needs further research to analyze the impact of this problem on system stability and to find reasonable regulatory measures to improve the imbalance price incentive.
These findings are important for other regulators facing an increasing importance of the intraday market in their countries. This holds especially for countries with a single imbalance pricing mechanism such as UK, Belgium, Netherlands or Austria. An application of the presented strategy will show whether their imbalance price system also causes biased trading motives. Regardless of the pricing rule, it is necessary in any case to establish an efficient intraday price constraint to provide appropriate balancing incentives for market participants.