Theoretical study on demand-side management to reduce imbalance between electricity supply and demand

Abstract

The gap between electricity supply and demand, which is the electricity imbalance, can have negative economic impacts on retail electricity suppliers. This paper presents a methodology for designing demand response (DR) programs while mitigating the negative impacts. The DR programs require the retailers to set target electricity consumption and prices of economic incentives for customers who cooperate with the DR programs. First, the target electricity consumption is determined by solving a problem of social welfare maximization in which the objective function is the sum of economic surplus of the retailers and their customers and impacts of electricity imbalance. The optimal electricity prices or the optimal rebate levels are then calculated under the principle that rational customers act to maximize their economic surplus. Through numerical simulations, the authors confirmed that the proposed methodology mitigated the impacts of electricity imbalance. Meanwhile, results of the numerical simulations showed that in practical situations, there were cases where the retailers unable to reduce the electricity imbalance profitably. Therefore, the authors additionally discuss ways to assess the electricity imbalance, which motivates the retailers to reduce the imbalance, in such cases.

Article Highlights

• Change in electricity consuming patterns enables us to balance between electricity supply and demand.

• Proposal improves a reliability in power grid and benefits on both electricity sellers and buyers.

• Regulations motivating society to change its electricity demand are vital.

1 Introduction

In Japan, retail electricity suppliers are required to notify power transmission and distribution companies (T&Ds) of their daily electricity supply schedules by noon the day before. Each value in the notified schedules can be updated at 30-min intervals up to one hour before the actual electricity supply and demand balancing operation. The retailers then procure electricity based on their latest notified values from contracted power generation companies and electric power markets. Once the notified schedules are fixed, the T&Ds are responsible for making up the imbalance arising from the difference between the notified electricity and the actual electricity consumption. When making this adjustment, the retailers pay an extra charge to the T&Ds, known as the imbalance charge [1]. The imbalance charge is an incentive to reduce the electricity imbalance and can have negative economic impacts on the retailers.

To mitigate the negative impacts on the retailers, the authors focus on demand response (DR) programs, which are designed to encourage changes in electricity consumption by changing electricity prices or providing monetary rebates [2]. This is because the DR programs are available even after the retailers have finalized their supply schedules. However, how to set parameters of the DR programs, electricity prices or rebate levels, remains as an important issue for the retailers. In practical DR programs, the prices of economic incentives have been flat rates since the DR programs were designed to conserve the demand in peak-time [3]. Reference [4] also concludes that price variations in incentive-based DR programs are time-independent. However, as the DR programs grow in popularity, dynamic pricing is becoming increasingly important [3, 5, 6].

Recently, numerous studies have been carried out on designing DR programs [7], including those based on the framework of social welfare maximization (SWM). Reference [8] presents an incentive-based DR program that maximizes the retailers’ economic surplus per day, focusing on reduction of peak electricity consumption. In [9], a day-ahead power transaction under DR programs is represented as a multi-objective SWM problem. The authors of [10] propose an energy management strategy in microgrids, as part of which a Stackelberg game-based DR program is integrated. The subject of these studies is to provide outlines of the DR programs, so DR parameters, i.e. electricity prices and rebate levels, are treated as known.

On the other hand, several approaches are being conducted on the setting of DR parameters. Reference [11] introduces a coupon incentive-based DR program is introduced, and the authors evaluate benefits of the DR program in terms of social welfare, consumer surplus, and robustness of retail electricity prices. The authors of [12] analyze the attribution of customers towards electricity price and customers’ preferences. Reference [13] provides the rebate levels calculated with the aim of maximizing the retailers’ profits when price spikes are detected. In [14], electricity prices in a DR program are set using a quadratic transfer cost. References [15] and [16] discuss the pricing in DR programs based on game-theoretic approaches. The subject of these approaches is to evaluate the price of economic incentives for DR programs. However, there is no established methodology for setting DR parameters, and how to design the DR programs is an active topic of discussion.

Given these circumstances, the authors have proposed in [17] a design method for the retailers with the function of energy resource aggregators (RES-AGs). The method in [17] builds on Reference [18] and is applicable to both price-based and incentive-based DR programs. However, this study only considered to maximize the total economic surplus of the retailers and their customers and does not discuss the distribution of the total surplus. As a result, the DR parameters determined by [17] bring an increase in the total surplus but may impose an unfair burden on either the retailers or the customers. Therefore, this paper presents an improved methodology for designing DR programs considering the economic impacts of the electricity imbalance on both the retailers and their customers. Similar to the conventional studies already discussed, the authors’ proposal is constructed on the SWM framework. One of the distinctive features of this study, including [17], is that it is possible to measure the value of electricity consumption to the customers, thereby allowing the calculation of electricity prices and rebate levels.

Section 2 describes how to mathematically set the electricity prices and the rebate levels as the DR parameters. In Sect. 3, power transactions between the RES-AGs and their customers are formulated as SWM problems. Section 3 also presents the proposed design method for DR programs against the electricity imbalance. Sections 4 and 5 provides conditions and results of numerical simulations, respectively. Here, the results of numerical simulations not only verify the validity of the proposal, but also indicate an issue in the setting of actual imbalance charge. Therefore, in Sect. 6, a methodology for calculating the imbalance charge is discussed.

2 Optimal pricing in demand response programs

This section explains how to set the optimal electricity price and rebate level as the DR parameters, with reference to [17]. The customers’ surplus is defined as the difference between their utility gained in exchange for consuming electricity and retail electricity payment and is expressed as

$$CS\left({d}_{t}\right)=U\left({d}_{t}\right)-P\left({d}_{t}\right),$$

(1)

where $t$ is the time period, $CS(\cdot )$ is the customers’ surplus function, ${d}_{t}$ is the electricity consumption, $U(\cdot )$ is the customers’ utility function, and $P(\cdot )$ is the function of retail electricity payment.

Customers’ surplus function is generally represented as an upward-convex function. This is because the more services the customers already have, the smaller the marginal utility, while the retail electricity payments are calculated by multiplying the retail prices and the actual electricity consumption. The condition is often used, as in [13, 19]. Hence, the actual electricity consumption satisfies the following equation.

$$\frac{\partial CS({d}_{t})}{\partial {d}_{t}}=0.$$

(2)

Considering the customers’ characteristics, the optimal electricity price in price-based DR programs is defined in SubSect. 2.1, and the optimal rebate level in incentive-based DR programs is defined in SubSect. 2.2.

2.1 Derivation of optimal electricity price

In the price-based DR program, as electricity price changes, the customers’ surplus function is changed as

$${CS}_{p}^{\prime }\left({d}_{t}\right)=U\left({d}_{t}\right)-{p}_{t}^{\prime }{d}_{t},$$

(3)

where ${CS}_{p}^{\prime }(\cdot )$ is the customers’ surplus function in the price-based DR program, and ${p}_{t}^{\prime }$ is the optimal electricity price in the price-based DR program defined in this paper.

At the target electricity consumption in the DR programs, it is necessary to set the electricity price so that the customers’ surplus function ${CS}_{p}^{\prime }(\cdot )$ satisfies Eq. (2). Therefore, the optimal electricity price can be derived by Eq. (4). This equation is used in the problem formulation presented in Subsection 3.3.

$${p}_{t}^{\prime }=\frac{\partial U({d}_{t})}{\partial {d}_{t}}{|}_{{d}_{t}={d}_{t}^{\prime }},$$

(4)

where ${d}_{t}^{\prime }$ is the target electricity consumption in the DR program.

Here, Eq. (4) is invalid unless the customers’ utility function is differentiable at the target electricity consumption. This challenge is discussed as follows.

The actual electricity consumption is finite. If the actual electricity consumption were infinite, the customers’ surplus would positively diverge. On the other hand, if the actual electricity consumption is zero, power transaction cannot realize, and the customers’ surplus is then zero. Therefore, the electricity price needs to be set so that the customers’ surplus function reaches a local maximum value at the target electricity consumption. Since the customers’ surplus function was set as an upward-convex function, the customers’ surplus function reaches the local maximum value when its function satisfies the following conditions.

$$\underset{{d}_{t}\to {d}_{t}^{\prime }-0}{\text{lim}}\frac{\partial CS\left({d}_{t}\right)}{\partial {d}_{t}}\ge 0,$$

(5)

$$\underset{{d}_{t}\to {d}_{t}^{\prime }+0}{\text{lim}}\frac{\partial CS\left({d}_{t}\right)}{\partial {d}_{t}}\le 0.$$

(6)

Hence, the optimal electricity price can be represented as

$$\begin{gathered} p_{t}^{\prime } = p_{t} , \hfill \\ \mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } + 0}} \frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }} \le p_{t} \le \mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } - 0}} \frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }}, \hfill \\ \end{gathered}$$

(7)

Where ${p}_{t}$ is the electricity price.

2.2 Derivation of optimal rebate level

In the incentive-based DR program, as adding the rebate, the customers’ surplus function is changed as

$${CS}_{r}^{\prime }\left({d}_{t}\right)=U\left({d}_{t}\right)-P\left({d}_{t}\right)+R\left({r}_{t}^{\prime }, {d}_{t}\right),$$

(8)

$$R\left({r}_{t},{d}_{t}\right)=\left\{\begin{array}{c}{r}_{t}({\widehat{d}}_{t}-{d}_{t}), {\widehat{d}}_{t}>{d}_{t}^{\prime }\\ {r}_{t}({d}_{t}-{\widehat{d}}_{t}), {\widehat{d}}_{t}<{d}_{t}^{\prime }\end{array}\right.,$$

(9)

where ${CS}_{r}^{\prime }\left(\cdot \right)$ is the customers’ surplus function in the incentive-based DR program, $R(\cdot )$ is the function of rebate, ${r}_{t}^{\prime }$ is the optimal rebate level in the incentive-based DR program defined in this paper, ${r}_{t}$ is the rebate level, and ${\widehat{d}}_{t}$ is the estimated electricity consumption when someone requests the DR program, meaning that ${\widehat{d}}_{t}$ is the baseline of electricity consumption in the DR program.

As in SubSect. 2.1, for the target electricity consumption in the DR programs, the rebate level must be set so that the customers’ surplus function ${CS}_{r}^{\prime }(\cdot )$ satisfies Eq. (2). Therefore, Eq. (10) can be calculated, which is defined as the optimal rebate level. This equation is used in the problem formulation presented in Subsection 3.4.

$${r}_{t}^{\prime }=\left\{\begin{array}{c}\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={d}_{t}^{\prime }}-\frac{\partial P\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={d}_{t}^{\prime }}, {\widehat{d}}_{t}>{d}_{t}^{\prime }\\ \frac{\partial P\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={d}_{t}^{\prime }}-\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={d}_{t}^{\prime }}, {\widehat{d}}_{t}<{d}_{t}^{\prime }\end{array}\right..$$

(10)

Equation (10) also involves a mathematical challenge similar to the challenge discussed in SubSect. 2.1. However, unlike the price-based DR programs, we must take into account not only the customers’ utility but also the retail electricity payment. In general, the function of retail electricity payment can be represented as

$$P\left({d}_{t}\right)={p}_{n,t}\,\left({d}_{t}-{d}_{n-1,t}\right)+{P}_{n-1,t},$$

(11)

$${p}_{n,t}=\left\{\begin{array}{c}{p}_{1,t}, \,{d}_{t}\le {d}_{1,t}\\ {p}_{2,t}, {{d}_{1,t}<d}_{t}\le {d}_{2,t}\\ \vdots \\ {p}_{n,t},{{d}_{n-1,t}<d}_{t}\le {d}_{n,t}\\ \vdots \end{array}\right.,$$

(12)

$${d}_{0,t}=0,$$

(13)

$${P}_{0,t}=0,$$

(14)

where ${p}_{n,t}$ is the retail price in tier $n$ ($=1, 2, \cdots$), ${d}_{n,t}$ is the upper limit of the electricity consumption to which the retail price in tier $n$ is adapted, and ${P}_{n-1,t}$ is the total retail electricity payments from tier $1$ to tier $n-1$.

The customers’ surplus function in each tier (${{d}_{n-1,t}<d}_{t}\le {d}_{n,t}$) can be treated as an upward-convex function. Therefore, the authors derive the optimal rebate level as

$$\begin{gathered} r_{t}^{\prime } = r_{t} , \hfill \\ {\text{if}}\,\hat{d}_{t} > d_{t}^{\prime } \,{\text{then}}\,\mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } + 0}} \left( {\frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right) \le r_{t} \le \mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } - 0}} \left( {\frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right), \hfill \\ {\text{if}}\,\hat{d}_{t} < d^{\prime }_{t} \,{\text{then}}\,\mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } - 0}} \left( {\frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right) \le r_{t} \le \mathop {\lim }\limits_{{d_{t} \to d_{t}^{\prime } + 0}} \left( {\frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right). \hfill \\ \end{gathered}$$

(15)

However, depending on the retail electricity payment, the proposal may not be able to induce electricity consumption to the target value. The authors’ proposal functions appropriately when the function of retail electricity payment is a downward-convex function. Otherwise, even if the proposal changes the customers’ surplus function to reach a local maximum at the target electricity consumption, the value may not be a global maximum. This is because the customers’ surplus function is likely to have multiple local maxima.

3 Theoretical design of demand response programs

This study focuses on the changes in the surplus of the RES-AGs and their customers in the DR programs. Based on the SWM framework, the authors formulate power transactions between the RES-AGs and their customers. In this study, the society consists of the RES-AGs and their customers, and the total surplus of the RES-AGs and their customers is defined as social surplus.

In Subsection 3.1, first, the negative economic impact on the RES-AGs by the electricity imbalance is discussed. Next, the authors present the design methods of DR programs to mitigate the negative impact. Steps of the authors’ proposal are as follows:

• Step 1: The objective function to determine the target electricity consumption is formulated.

• Step 2: The constraints are derived that consider the economic impact on the RES-AGs and their customers.

• Step 3: The maximization problem consisting of the objective function and the constraints is solved to obtain the target electricity consumption.

• Step 4: The DR parameters are calculated.

Subsection 3.2 explains how to formulate the objective function to determine the target electricity consumption. Subsections 3.3 and 3.4 present methods for designing price-based and incentive-based DR programs, respectively.

3.1 Economic impact of imbalance between electricity supply and demand

The RES-AGs typically create electricity supply schedules according to electricity consumption by their contracted customers and notify them to the T&Ds. Then, the RES-AGs procure the notified electricity from the power generation companies and electric power markets. The RES-AGs also pay wheeling charges to the T&Ds. The ideal power transaction assumed by the RES-AGs is displayed in Fig. 1. In the scheduling phases, the RES-AGs expect their surplus as

$${\widehat{RS}}_{0, t}={RS}_{0}\left({\widehat{s}}_{0,t}\right),$$

(16)

$${RS}_{0}\left({s}_{t}\right)=P\left({s}_{t}\right)-W\left({s}_{t}\right)-C\left({s}_{t}\right),$$

(17)

where ${\widehat{RS}}_{0, t}$ is the RES-AGs’ surplus expected in scheduling phase, ${RS}_{0}\left(\cdot \right)$ is the RES-AGs’ surplus function if the electricity supply and demand are balanced, ${\widehat{s}}_{0,t}$ is the notified electricity, ${s}_{t}$ is the electricity that the RES-AGs procure in scheduling phase, $W\left(\cdot \right)$ is the wheeling charge, and $C\left(\cdot \right)$ is the electricity procurement cost.

Fig. 1

Power transaction that the RES-AGs expect in scheduling phase

However, in practice, the notified electricity does not always match the actual electricity consumption. When the electricity imbalance occurs, the RES-AGs must pay the imbalance charge. Figure 2 illustrates the power transaction with the electricity imbalance, and the actual RES-AGs’ surplus is calculated as

$${RS}_{t}^{*}=RS\left({d}_{t}^{*}\right),$$

(18)

$$RS\left({d}_{t}\right)=P\left({d}_{t}\right)-W\left({d}_{t}\right)-C\left({\widehat{s}}_{0,t}\right)-I\left({d}_{t}\right),$$

(19)

where ${RS}_{t}^{*}$ is the actual RES-AGs’ surplus, $RS\left(\cdot \right)$ is the RES-AGs’ surplus function with the electricity imbalance, ${d}_{t}^{*}$ is the actual electricity consumption, and $I\left(\cdot \right)$ is the imbalance charge.

Fig. 2

Power transaction with imbalance between electricity supply and demand

The decrease of the RES-AGs’ surplus from its expected value due to the electricity imbalance is calculated as

$${\Delta RS}_{t}={\widehat{RS}}_{0, t}-{RS}_{t}^{*},$$

(20)

where ${\Delta RS}_{t}$ is the decrease of the RES-AGs’ surplus from the expected value in the scheduling phase.

3.2 Objective function to determine target electricity consumption

In this paper, the DR programs are designed on the basis of the SWM framework. The maximization problem is formulated as

$$\underset{{d}_{t}}{\text{max}}\Delta {\widehat{SS}}_{1}({d}_{t}),$$

(21)

$$\Delta {\widehat{SS}}_{1}\left({d}_{t}\right)={\widehat{SS}}_{1}\left({d}_{t}\right)-{\widehat{SS}}_{1}\left({\widehat{d}}_{1, t}\right),$$

(22)

$${\widehat{SS}}_{1}\left({d}_{t}\right)={RS}_{1}\left({d}_{t}\right)+{\widehat{CS}}_{1}\left({d}_{t}\right),$$

(23)

where $\Delta {\widehat{SS}}_{1}(\cdot )$ is the expected increase of the social surplus by the DR program from the social surplus without DR program, ${\widehat{SS}}_{1}(\cdot )$ is the social surplus function with the DR program expected by RES-AGs, ${\widehat{d}}_{1, t}$ is the DR baseline, ${RS}_{1}(\cdot )$ is the RES-AGs’ surplus function with the DR program, and ${\widehat{CS}}_{1}\left(\cdot \right)$ is the customers’ surplus function with the DR program expected by RES-AGs.

The RES-AGs must operate so that the notified electricity is equal to the actual electricity consumption. This constraint is defined as

$${d}_{t}={\widehat{s}}_{0,t}.$$

(24)

If we simply treat the supply and demand balancing constraint in the maximization problem, the RES-AGs have no choice but to request the DR to prevent the electricity imbalance. However, in practice, the RES-AGs can make more profits than complying with the balancing regulations. This is why the authors formulate the objective function using the penalty method [20, 21] to express more strategies of the RES-AGs. The penalty method relaxes a constrained optimization problem as an unconstrained one. The objective function to determine the target electricity consumption is formulated as

$$F\left({d}_{t}\right., \left.\lambda \right)=\Delta {\widehat{SS}}_{1}\left({d}_{t}\right)-\lambda {\left({d}_{t}-{\widehat{s}}_{0,t}\right)}^{2},$$

(25)

where $F\left(\cdot \right)$ is the objective function to determine the target electricity consumption in the DR program, and $\lambda$ ($\ge 0$) is the penalty factor.

The relative importance between the increase of the social surplus and the reduction of the electricity imbalance is determined by the penalty factor $\lambda$. If the penalty factor $\lambda$ is zero, the objective of the DR programs is to maximize the social surplus. Otherwise, the higher the penalty factor $\lambda$, the more dominant the balancing constraint becomes.

3.3 Design of price-based demand response program

The power transaction in the price-based DR program is illustrated as Fig. 3. The RES-AGs’ surplus function and the estimated customers’ surplus function are then represented as

$${RS}_{1,p}\left({d}_{t}\right)={p}_{t}^{\prime }{d}_{t}-W\left({d}_{t}\right)-C\left({\widehat{s}}_{0,t}\right)-I\left({d}_{t}\right),$$

(26)

$${\widehat{CS}}_{1,p}\left({d}_{t}\right)={\widehat{U}}_{1}\left({d}_{t}\right)-{p}_{t}^{\prime }{d}_{t},$$

(27)

where ${RS}_{1,p}(\cdot )$ is the RES-AGs’ surplus function in the price-based DR program, ${\widehat{CS}}_{1,p}\left(\cdot \right)$ is the customers’ surplus function in the price-based DR program estimated by the RES-AGs, and ${\widehat{U}}_{1}\left(\cdot \right)$ is the customers’ utility function estimated by the RES-AGs.

Fig. 3

Power transaction in price-based DR program

By substituting Eqs. (26) and (27) into Eq. (23) and solving the maximization problem, the RES-AGs can obtain the target electricity consumption in the price-based DR programs. At this stage, the economic impacts on both the RES-AGs and their customers have not yet been discussed. To study the economic impacts, the acceptable range of electricity consumption in the price-based DR programs is introduced.

Generally, DR requests will not be activated unless the RES-AGs can expect their additional surplus. Hence, the RES-AGs request the price-based DR if Condition (28) holds.

$${RS}_{1,p}\left({d}_{t}^{\prime }\right)\ge RS\left({\widehat{d}}_{1, t}\right).$$

(28)

In the price-based DR programs, the customers’ surplus decreases as the customers conserve electricity. This is because Eq. (29) is the function calculating the rational customers’ surplus for electricity consumption.

$$G\left({d}_{t}\right)={\widehat{U}}_{1}\left({d}_{t}\right)-\frac{\partial {\widehat{U}}_{1}\left({d}_{t}\right)}{\partial {d}_{t}}{d}_{t},$$

(29)

where $G\left(\cdot \right)$ is the function calculating the rational customers’ surplus for electricity consumption in the price-based DR program.

$G\left(\cdot \right)$ is a monotonically increasing function unless electricity consumption is negative. This is because the customers’ utility function is an upward-convex function. That is, the more the customer conserve electricity, the smaller the rational customers’ surplus becomes. Under the circumstance, the authors modify the condition of the customers’ surplus function for the price-based DR program as

$${\widehat{CS}}_{1,p}\left({d}_{t}^{\prime }\right)\ge 0.$$

(30)

The price-based DR programs need to be designed under Condition (31) to satisfy Conditions (28) and (30).

$$P\left({\widehat{d}}_{1, t}\right)+\Delta {W}_{1}\left({d}_{t}^{\prime }\right)+\Delta {I}_{1}\left({d}_{t}^{\prime }\right)\le {p}_{t}^{\prime }{d}_{t}^{\prime }\le {\widehat{U}}_{1}\left({d}_{t}^{\prime }\right),$$

(31)

$$\Delta {W}_{1}\left({d}_{t}\right)=W\left({d}_{t}\right)-W\left({\widehat{d}}_{1, t}\right),$$

(32)

$$\Delta {I}_{1}\left({d}_{t}\right)=I\left({d}_{t}\right)-I\left({\widehat{d}}_{1, t}\right),$$

(33)

where $\Delta {W}_{1}\left(\cdot \right)$ is the increase of the wheeling charge due to the DR program, and $\Delta {I}_{1}\left(\cdot \right)$ is the increase of the imbalance charge paid from the RES-AGs due to the DR program.

Only the target electricity consumption satisfying Condition (31) is valid. In other words, the electricity consumption satisfying Condition (34) is acceptable as the target electricity consumption in the price-based DR programs.

$$P\left({\widehat{d}}_{1, t}\right)+\Delta {W}_{1}\left({d}_{t}\right)+\Delta {I}_{1}\left({d}_{t}\right)\le \frac{\partial {\widehat{U}}_{1}\left({d}_{t}\right)}{\partial {d}_{t}}{d}_{t}\le {\widehat{U}}_{1}\left({d}_{t}\right).$$

(34)

Since the RES-AGs can use Condition (34) as a constraint, the maximization problem to calculate the target electricity consumption is formulated as

$$\begin{gathered} \mathop {\max }\limits_{{d_{t} }} F\left( {d_{t} } \right., \left. \lambda \right), \hfill \\ {\text{s}}.{\text{ t}}.\,P\left( {\hat{d}_{1, t} } \right) + \Delta W_{1} \left( {d_{t} } \right) + \Delta I_{1} \left( {d_{t} } \right) \le \frac{{\partial \widehat{U}_{1} \left( {d_{t} } \right)}}{{\partial d_{t} }}d_{t} , \hfill \\ \frac{{\partial \widehat{U}_{1} \left( {d_{t} } \right)}}{{\partial d_{t} }}d_{t} \le \widehat{U}_{1} \left( {d_{t} } \right). \hfill \\ \end{gathered}$$

(35)

By substituting the target electricity consumption calculated from the maximization problem into Eq. (4), the RES-AGs can obtain the optimal electricity price.

If the RES-AGs adopt Eq. (7) as the optimal electricity price, the RES-AGs can obtain the target electricity consumption from the following maximization problem.

$$\begin{gathered} \mathop {\max }\limits_{{d_{t} }} F\left( {d_{t} } \right., \left. \lambda \right), \hfill \\ {\text{s}}.{\text{ t}}.\,P\left( {\hat{d}_{1, t} } \right) + \Delta W_{1} \left( {d_{t} } \right) + \Delta I_{1} \left( {d_{t} } \right) \le \widehat{U}_{1} \left( {d_{t} } \right), \hfill \\ \mathop {\lim }\limits_{\delta \to + 0} \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }d_{t} \le \mathop {\lim }\limits_{\delta \to - 0} \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }d_{t} , \hfill \\ \left( {\hat{d}_{{1,{ }t}} } \right) + \Delta W_{1} \left( {d_{t} } \right) + \Delta I_{1} \left( {d_{t} } \right) \le \mathop {\lim }\limits_{\delta \to - 0} \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }d_{t} , \hfill \\ \mathop {\lim }\limits_{\delta \to + 0} \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }d_{t} \le \widehat{U}_{1} \left( {d_{t} } \right), \hfill \\ \end{gathered}$$

(36)

Where $\delta$ is a variable.

3.4 Design of incentive-based demand response program

The RES-AGs present the rebate levels to their customers beforehand and pay the rebates in response to the DR cooperation. Figure 4 shows the power transaction in the incentive-based DR program, and the RES-AGs’ surplus function and the estimated customers’ surplus function are represented as

$${RS}_{1,r}\left({d}_{t}\right)=P\left({d}_{t}\right)-W\left({d}_{t}\right)-C\left({\widehat{s}}_{0,t}\right)-I\left({d}_{t}\right)-R\left({r}_{t}^{\prime }, {d}_{t}\right),$$

(37)

$${\widehat{CS}}_{1,r}\left({d}_{t}\right)={\widehat{U}}_{1}\left({d}_{t}\right)-P\left({d}_{t}\right)+R\left({r}_{t}^{\prime }, {d}_{t}\right),$$

(38)

where ${RS}_{1,r}(\cdot )$ is the RES-AGs’ surplus function in the incentive-based DR program, and ${\widehat{CS}}_{1,r}\left(\cdot \right)$ is the customers’ surplus function in the incentive-based DR program estimated by the RES-AGs.

Fig. 4

Power transaction in incentive-based DR program

Even in the incentive-based DR programs, the constraints considering the economic impacts on both the RES-AGs and their customers are important. The RES-AGs can design the incentive-based DR programs benefiting on both the RES-AGs and their customers. In contrast, the customers will not cooperate the DR requests if the customers cannot recover the decrease of their surplus. Therefore, the RES-AGs must compensate the decrease as the rebates, and this condition is represented as

$${\widehat{CS}}_{1,r}\left({d}_{t}^{\prime }\right)\ge {\widehat{CS}}_{1,r}\left({\widehat{d}}_{1, t}\right).$$

(39)

The other condition is related to the RES-AGs’ surplus and can be expressed as

$${RS}_{1,r}\left({d}_{t}^{\prime }\right)\ge RS\left({\widehat{d}}_{1, t}\right).$$

(40)

The RES-AGs need to set the target electricity consumption in the incentive-based DR programs that satisfies Condition (41).

$$\begin{gathered} {\text{If }}\hat{d}_{1, t} > d_{t}^{\prime } \,{\text{then}} - \left( {\Delta \widehat{U}_{1} \left( {d_{t}^{\prime } } \right) - \Delta P_{1} \left( {d_{t}^{\prime } } \right)} \right) \le r_{t}^{\prime } \left( {\hat{d}_{1, t} - d_{t}^{\prime } } \right) \le RS\left( {d_{t}^{\prime } } \right) - RS\left( {\hat{d}_{1, t} } \right), \hfill \\ {\text{if }}\hat{d}_{1, t} < d_{t}^{\prime } \, {\text{then}} - \left( {\Delta \widehat{U}_{1} \left( {d_{t}^{\prime } } \right) - \Delta P_{1} \left( {d_{t}^{\prime } } \right)} \right) \le r_{t}^{\prime } \left( {d_{t}^{\prime } - \hat{d}_{1, t} } \right) \le RS\left( {d_{t}^{\prime } } \right) - RS\left( {\hat{d}_{1, t} } \right), \hfill \\ \end{gathered}$$

(41)

$$\Delta {\widehat{U}}_{1}\left({d}_{t}\right)={\widehat{U}}_{1}\left({d}_{t}\right)-{\widehat{U}}_{1}\left({\widehat{d}}_{1, t}\right),$$

(42)

$$\Delta {P}_{1}\left({d}_{t}\right)=P\left({d}_{t}\right)-P\left({\widehat{d}}_{1, t}\right),$$

(43)

where $\Delta {\widehat{U}}_{1}\left(\cdot \right)$ is the increase of the customers’ utility owing to the DR program, and $\Delta {P}_{1}\left(\cdot \right)$ is the increase of the retail electricity payment due to the DR program.

The electricity consumption satisfying Condition (44) is acceptable as the target electricity consumption in the incentive-based DR programs.

$$-\left(\Delta {\widehat{U}}_{1}\left({d}_{t}\right)-\Delta {P}_{1}\left({d}_{t}\right)\right)\le \left(\frac{\partial {\widehat{U}}_{1}\left({d}_{t}\right)}{\partial {d}_{t}}-\frac{\partial P\left({d}_{t}\right)}{\partial {d}_{t}}\right)\left({\widehat{d}}_{1, t}-{d}_{t}\right)\le RS\left({d}_{t}\right)-RS\left({\widehat{d}}_{1, t}\right).$$

(44)

Therefore, the maximization problem for calculating the target electricity consumption in the incentive-based DR programs is represented as

$$\begin{gathered} \mathop {\max }\limits_{{d_{t} }} F\left( {d_{t} } \right.,~\left. \lambda \right), \hfill \\ {\text{s}}.{\text{ t}}.\, - \left( {\Delta \widehat{U}_{1} \left( {d_{t} } \right) - \Delta P_{1} \left( {d_{t} } \right)} \right) \le \left( {\frac{{\partial \widehat{U}_{1} \left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right)\left( {\hat{d}_{{1,t}} - d_{t} } \right), \hfill \\ \left( {\frac{{\partial \widehat{U}_{1} \left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial P\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right)\left( {\hat{d}_{{1,t}} - d_{t} } \right) \le RS\left( {d_{t} } \right) - RS\left( {\hat{d}_{{1,t}} } \right) \hfill \\ \end{gathered}$$

(45)

By substituting the target electricity consumption calculated from the maximization problem to Eq. (10), the RES-AGs can obtain the optimal rebate level.

The improved maximization problem addressing the mathematical challenge of the optimal pricing in the incentive-based DR programs can be expressed as follows:

If ${\widehat{d}}_{t}>{d}_{t}^{\prime }$ then.

$$\begin{gathered} \mathop {\max }\limits_{{d_{t} }} F\left( {d_{t} } \right.,{ }\left. \lambda \right) \hfill \\ {\text{S}}.{\text{t}}.\, - \left( {\Delta \widehat{U}_{1} \left( {d_{t} } \right) - \Delta P_{1} \left( {d_{t} } \right)} \right) \le RS\left( {d_{t} } \right) - RS\left( {\hat{d}_{1, t} } \right), \hfill \\ \mathop {\lim }\limits_{\delta \to + 0} \left( {\frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta } - \frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta }} \right)\left( {\hat{d}_{1, t} - d_{t} } \right) \hfill \\ \le \mathop {\lim }\limits_{\delta \to - 0} \left( {\frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta } - \frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta }} \right)\left( {\hat{d}_{{1,{ }t}} - d_{t} } \right), \hfill \\ - \left( {\Delta \widehat{U}_{1} \left( {d_{t} } \right) - \Delta P_{1} \left( {d_{t} } \right)} \right) \le \mathop {\lim }\limits_{\delta \to - 0} \left( {\frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta } - \frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta }} \right)\left( {\hat{d}_{{1,{ }t}} - d_{t} } \right), \hfill \\ \mathop {\lim }\limits_{\delta \to + 0} \left( {\frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta } - \frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta }} \right)\left( {\hat{d}_{{1,{ }t}} - d_{t} } \right) \le RS\left( {d_{t} } \right) - RS\left( {\hat{d}_{{1,{ }t}} } \right). \hfill \\ \end{gathered}$$

(46)

If ${\widehat{d}}_{t}<{d}_{t}^{\prime }$ then.

$$\begin{gathered} \mathop {\max }\limits_{{d_{t} }} F\left( {d_{t} } \right.,{ }\left. \lambda \right) \hfill \\ {\text{S}}.{\text{t}}.\, - \left( {\Delta \widehat{U}_{1} \left( {d_{t} } \right) - \Delta P_{1} \left( {d_{t} } \right)} \right) \le RS\left( {d_{t} } \right) - RS\left( {\hat{d}_{1, t} } \right), \hfill \\ \mathop {\lim }\limits_{\delta \to - 0} \left( {\frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta } - \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }} \right)\left( {d_{t} - \hat{d}_{1, t} } \right) \hfill \\ \le \mathop {\lim }\limits_{\delta \to + 0} \left( {\frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta } - \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }} \right)\left( {d_{t} - \hat{d}_{{1,{ }t}} } \right), \hfill \\ - \left( {\Delta \widehat{U}_{1} \left( {d_{t} } \right) - \Delta P_{1} \left( {d_{t} } \right)} \right) \le \mathop {\lim }\limits_{\delta \to + 0} \left( {\frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta } - \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }} \right)\left( {d_{t} - \hat{d}_{{1,{ }t}} } \right), \hfill \\ \mathop {\lim }\limits_{\delta \to - 0} \left( {\frac{{P\left( {d_{t} + \delta } \right) - P\left( {d_{t} } \right)}}{\delta } - \frac{{\widehat{U}_{1} \left( {d_{t} + \delta } \right) - \widehat{U}_{1} \left( {d_{t} } \right)}}{\delta }} \right)\left( {d_{t} - \hat{d}_{{1,{ }t}} } \right) \le RS\left( {d_{t} } \right) - RS\left( {\hat{d}_{{1,{ }t}} } \right). \hfill \\ \end{gathered}$$

(47)

4 Numerical simulation models

This paper shows the numerical simulation results when the RES-AGs’ surplus was most decreased from the expected one due to the electricity imbalance in fiscal year (FY) 2017. This section explains about numerical simulation models, and the results are shown in the next section. In the numerical simulations, the authors substituted the DR baseline with the actual electricity consumption as

$${\widehat{d}}_{1, t}={d}_{t}^{*}.$$

(48)

This equation indicates that discussion on uncertainties in estimating electricity consumption is beyond the scope of this paper. The authors will address the challenge in future works.

In the numerical simulations, 500 households were regarded as one aggregated customer, and their smart metering data was used as the actual electricity consumption, which is treated as the DR baseline by Eq. (48). The notified electricity was assumed as the standard baseline (different from the DR baseline) defined by the Ministry of Economy, Trade, and Industry (METI) [22]. Table 1 summarizes the DR baseline and the notified electricity.

Table 1 DR baseline and notified electricity

Each component of the RES-AGs’ surplus function excluding the rebate was designed as

$$C\left({s}_{t}\right)={c}_{t}^{*}{s}_{t},$$

(49)

$$I\left( {d_{t} } \right) = \left\{ {\begin{array}{*{20}c} {i_{t}^{ - } \left( {d_{t} - \hat{s}_{{0,t}} } \right),} & {\hat{s}_{{0,t}} < d_{t} } \\ \begin{gathered} _{t}^{ + } \left( {d_{t} - \hat{s}_{{0,t}} } \right),~ \hfill \\ 0~~~~~~~~,~ \hfill \\ \end{gathered} & \begin{gathered} \hat{s}_{{0,t}} > d_{t} \hfill \\ \hat{s}_{{0,t}} = d_{t} \hfill \\ \end{gathered} \\ \end{array} } \right.$$

(50)

$$W\left({d}_{t}\right)={w}_{t}^{*}{d}_{t},$$

(51)

$$P\left({d}_{t}\right)={p}_{t}^{*}{d}_{t},$$

(52)

where ${c}_{t}^{*}$ is the unit price of the electricity procurement cost, ${i}_{t}^{-}$ is the imbalance price for electricity shortage, so-called negative imbalance price, ${i}_{t}^{+}$ is the imbalance price for excess electricity, so-called positive imbalance price, ${w}_{t}^{*}$ is the wheeling price, and ${p}_{t}^{*}$ is the retail price.

The authors assumed that the RES-AGs procure electricity only from the Japan Electric Power Exchange, which is an electric power market in Japan. To be exact, the unit price of the electricity procurement cost is the weighted average price of the spot market price and intraday market price. For simplicity, the unit price was defined as the area price, which is the regional spot market price. The imbalance prices were calculated with the method that the METI had defined until FY 2021 [1]. Table 2 shows the area prices and the imbalance prices.

Table 2 Electric power market prices

The wheeling price was set to 9.02 JPY/kWh, which was the average wheeling price in the low voltage power service reported by the METI, during the calculation period. The retail prices should be set so that the RES-AGs can expect profits and contract to customers. The authors thus set the acceptable range of the retail price as

$$\overline{C }+\overline{W}\le {p }_{t}\le \overline{P },$$

(53)

where $\overline{C }$ is the average area price, $\overline{W }$ is the average wheeling price, and $\overline{P }$ is the average retail price that Japanese retail electricity suppliers set.

This study set the acceptable range so that the RES-AGs can satisfy Condition (53) throughout a year. As the retail price ${p}_{t}^{*}$, the authors adopted the average value of the upper and the lower prices in Condition (53), 22.28 JPY/kWh, during the calculation period. The RES-AGs’ surplus expected in the scheduling phase and the actual RES-AGs’ surplus are shown in Table 3.

Table 3 RES-AGs’ surplus expected in scheluling phase and actual RES-AGs’ surplus

Customers’ utility function is often represented as logarithmic, exponential, power, or quadratic function [23]-[26]. With reference to [25], the customers’ utility function was set as

$$U\left({d}_{t}\right)={\alpha }_{t}\text{log}({\beta }_{t}\left({d}_{t}-{\gamma }_{t}\right)),$$

(54)

where ${\alpha }_{t}$, ${\beta }_{t}$, ${\gamma }_{t}$ are the coefficients of the customers’ utility function.

This paper adopts different setting method of each coefficient of the customers’ surplus function from Reference [25]. To derive the coefficients, the authors used price elasticity of demand. The price elasticity of demand measures how much the quantity demanded responds to a change in price [27] and is defined as

$$e\left({p}_{t}\right)=\frac{\partial {d}_{t}}{\partial {p}_{t}}\frac{{p}_{t}}{{d}_{t}},$$

(55)

where $e(\cdot )$ is the price elasticity of demand.

The customers are generally inelastic for change in prices. References [28]-[30] show the analysis that customers are inelastic in their electricity consumption with respect to price changes. The authors thus changed the value of the price elasticity of demand at the trading point $e({p}_{t}^{*})$ per 0.01 from − 0.99 to − 0.01.

The customers consume electricity to maximize their surplus in response to the given retail prices. To obtain the relation between the electricity price and electricity consumption, the customers’ surplus function is replaced as

$$CS\left({p}_{t},{d}_{t}\right)={\alpha }_{t}\text{log}\left({\beta }_{t}\left({d}_{t}-{\gamma }_{t}\right)\right)-{p}_{t}{d}_{t}.$$

(56)

We can obtain the relation as Eq. (57) from Eqs. (2) and (56).

$${d}_{t}=\frac{{\alpha }_{t}}{{p}_{t}}+{\gamma }_{t}, \,{p}_{t}>0.$$

(57)

Equations (55) and (57) enable us to derive the coefficients ${\alpha }_{t}$ and ${\gamma }_{t}$. In addition, the customers’ utility function can be represented as Eq. (58) by integrating the customers’ marginal utility function, Eq. (59).

$$U\left({d}_{t}\right)={\alpha }_{t}\text{log}\left({d}_{t}-{\gamma }_{t}\right)+K, {d}_{t}>{\gamma }_{t},$$

(58)

$$\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}=\frac{{\alpha }_{t}}{{d}_{t}-{\gamma }_{t}}, {d}_{t}>{\gamma }_{t},$$

(59)

where $K$ is the constant of integration.

If (${d}_{t}\le {\gamma }_{t}$), power transaction cannot realize, and then the customers’ surplus is zero. By comparing Eqs. (54) and (58), the constant of integration $K$ can be expressed as

$$K={\alpha }_{t}\text{log}{\beta }_{t}.$$

(60)

When the customers’ surplus function is continuous, the coefficient ${\beta }_{t}$ is derived from Eq. (61).

$$0={\alpha }_{t}\text{log}({\underline{d}}_{t}-{\gamma }_{t})+{\alpha }_{t}\text{log}{\beta }_{t}-{p}_{t}^{*}{\underline{d}}_{t},$$

(61)

where ${\underline{d}}_{t}$ is the lower bound of electricity consumption that the power transaction realizes.

This paper defined the lower bound of electricity consumption as the electricity consumption when the price elasticity is small enough, $e\left(\cdot \right)=-0.0000001$. This is because the marginal utility function is infinite at the real lower bound of electricity consumption.

Under Eq. (48), Eq. (62) holds.

$${\widehat{U}}_{1}\left({d}_{t}\right)=U\left({d}_{t}\right).$$

(62)

5 Numerical simulation results

The authors assumed two scenarios for the objective of the DR requests. When the penalty factor $\lambda$ is zero, the objective is to maximize the social surplus. In contrast, a sufficiently large penalty factor, such as $1.0\times {10}^{16}$, acts to prevent the electricity imbalance. This section compares the numerical simulation results whether the constraints in the DR programs exist and validates the proposed DR programs. SubSects. 5.1 and 5.2 show the results of the price-based and incentive-based DR programs, respectively.

5.1 Price-based demand response programs

Figure 5 compares the target electricity consumption in the price-based DR programs for the price elasticity of demand at the trading point $e({p}_{t}^{*})$ whether the constraints exist. From 18:00 to 18:30 on February 5th, 2018, the constraints were effective in the price elasticity of demand ranged from -0.17 to -0.01, assuming the penalty factor $\lambda$ was zero. If the penalty factor $\lambda$ was $1.0\times {10}^{16}$, the constraints worked in the range of the price elasticity of demand from − 0.12 to − 0.01. The more inelastic the customers were for change in prices, the narrower the acceptable range of electricity consumption becomes. It is rational. From 8:30 to 9:00 on March 4th, 2018, the RES-AGs did not request the DR in all dates and times. This is because the RES-AGs’ surplus decreases by increasing demand in the price-based DR programs. It should be noted that the decrease only holds true in the numerical simulation models but not in every case. Table 4 displays some values of the target electricity consumption in the price-based DR programs with the constraints, extracted from Fig. 5.

Fig. 5

Comparison of target electricity consumption in price-based DR programs whether constraints exist

Table 4 Target electricity consumption in price-based DR programs with constraints

Figure 6 compares the optimal electricity prices in the price-based DR programs for the price elasticity of demand at the trading point $e({p}_{t}^{*})$ whether the constraints exist. When the constraints had an efficacy, the optimal electricity prices were calculated differently from the values in the price-based DR programs without any constraints. Table 5 shows some values of the optimal electricity prices in the price-based DR programs with the constraints, extracted from Fig. 6.

Fig. 6

Comparison of optimal electricity prices in price-based DR programs whether constraints exist

Table 5 Optimal electricity prices in price-based DR programs with constraits

Table 6 shows the amount of electricity imbalance and the increase of the social surplus owing to the DR programs without the constraints. Table 7 summarizes these results in the price-based DR programs with the constraints. These tables show that the amount of electricity imbalance in the price-based DR programs with the constraints was more than the one without the constraints. In addition, the increase of the social surplus in the price-based DR programs without the constraints was more than the one with the constraints. Therefore, the price-based DR programs without the constraints were suitable if the RES-AGs should only consider the impacts on the whole society.

Table 6 Results of DR programs without constraints

Table 7 Results of price-based DR programs with constraints

Here, the constraints make the customers’ surplus non-negative. From 18:00 to 18:30 on February 5th, 2018, the customers’ surplus without DR programs was 7338.56 JPY when the price elasticity of demand was assumed as − 0.10. Tables 8 and 9 compare the divisions of the social surplus increased by the price-based DR programs into the RES-AGs and their customers whether the constraints exist. The customers’ surplus in the price-based DR program with the constraints became positive though the value in the price-based DR program without the constraints was negative. The constraints also contributed to averting the RES-AGs’ surplus from decreasing due to the price-based DR programs.

Table 8 Divisions of total surplus of RES-AGs and their customers increased by price-based DR programs without constraints

Table 9 Divisions of total surplus of RES-AGs and their customers increased by price-based DR programs with constraints

These results suggest that the price-based DR programs with the constraints may not function well in the society. On the other hand, the price-based DR programs without the constraints are valid if the RES-AGs only consider the impacts on the whole society. This is because the results without the constraints show that the RES-AGs could achieve the objective of the DR programs. Furthermore, the results imply that the price-based DR programs are likely to increase the social surplus while reducing the electricity imbalance.

5.2 Incentive-based demand response programs

Figure 7 compares the target electricity consumption in the incentive-based DR programs for the price elasticity of demand at the trading point $e({p}_{t}^{*})$ whether the constraints exist. When the penalty factor $\lambda$ was zero, the target electricity consumption was calculated regardless that the constraints exist. This is because the constraints regarding the RES-AGs’ surplus and their customers’ surplus are satisfied in the numerical simulation models. If the penalty factor was $1.0\times {10}^{16}$, the constraints were effective when the notified electricity was out of the acceptable range of electricity consumption in the incentive-based DR programs. The RES-AGs then reduced the electricity imbalance as much as their surplus did not decrease. The condition regarding the customers’ surplus was satisfied in the numerical simulation models. Table 10 shows some values of the target electricity consumption in the incentive-based-DR programs with the constraints.

Fig. 7

Comparison of target electricity consumption in incentive-based DR programs whether constraints exist

Table 10 Target electricity consumption in incentive-based DR programs with constraints

Figure 8 compares the optimal rebate levels in the incentive-based DR programs for the price elasticity of demand at the trading point $e({p}_{t}^{*})$ whether the constraints exist. When the constraints have an efficacy, the optimal rebate levels were calculated differently from the values in the incentive-based DR programs without the constraints. Table 11 displays some values of the optimal rebate levels in the incentive-based DR programs with the constraints.

Fig. 8

Comparison of optimal rebate levels in incentive-based DR programs whether constraints exist

Table 11 Optimal rebate levels in incentive-based DR programs with constraits

Table 12 shows the amount of electricity imbalance and the increase of the social surplus owing to the incentive-based DR programs with the constraints. The results in the DR programs without the constraints are summarized in Table 6. In the incentive-based DR programs, by setting the constraints, the social surplus increased in exchange for leaving the electricity imbalance partially.

Table 12 Results of incentive-based DR programs with constraints

Moreover, by setting the constraints, the authors could design the incentive-based DR programs having no negative economic impacts on both the RES-AGs and their customers. Tables 13 and 14 compare the divisions of the social surplus increased by the incentive-based DR programs into the RES-AGs and their customers whether the constraints exist.

Table 13 Divisions of total surplus of RES-AGs and their customers increased by incentive-based DR programs without constraints

Table 14 Divisions of total surplus of RES-AGs and their customers increased by incentive-based DR programs with constraints

Throughout the numerical simulations, the authors suggest that the incentive-based DR programs should be adopted to mitigate the negative economic impact of the imbalance settlement on the RES-AGs. This is because, thanks to the constraints, the RES-AGs could design the incentive-based DR programs benefiting on both the RES-AGs and their customers. In the impacts on the whole society, the increase of social surplus and the reduction of electricity imbalance had a trade-off.

6 Discussion regarding calculation methods of imbalance prices

The authors concluded that the proposed DR programs functioned well. However, the picked-up dates and times were when the proposals were most essential for the RES-AGs. The percentage that the actual RES-AGs’ surplus was higher than the RES-AGs’ surplus expected in the scheduling phase was 47.69%.

Figure 9 compares the frequency distributions whether the RES-AGs requested the DR in all dates and times. The RES-AGs then requested the DR without the constraints to maximize the social surplus, and then the penalty factor $\lambda$ was set to zero. In Fig. 9, the price elasticity of demand at the trading point $e\left({p}_{t}^{*}\right)$ was − 0.20. Overall, the electricity shortage increased. The more elastic the customers were for change in prices, the more the propensity manifested. Table 15 shows the percentages of dates and times that the electricity imbalance was increased by the DR programs. Table 16 shows the total amount of electricity shortage and excessed electricity in FY 2017. The total amount of electricity shortage was increased by the DR programs. The more inelastic the customers were for change in prices, the more the total amount of electricity shortage increased.

Fig. 9

Frequency distributions of imbalance between electricity supply and demand ($e\left({p}_{t}^{*}\right)=-0.20$)

Table 15 Percentages of dates and times that imbalanace between electricity supply and demand was increased by DR programs

Table 16 Total amount of electricity shortage and excess electricity in FY 2017

These results imply that the RES-AGs can intentionally increase the amount of electricity imbalance to seek their surplus. It may degrade the reliability in power grids. To resolve the issue, the authors focus on the calculating methods of the imbalance prices. The imbalance prices should be set as the imbalance settlement becomes the incentive for the retailers to reduce the electricity imbalance. In addition, the imbalance prices should be set so that the T&Ds can recover the costs for eliminating the electricity imbalance.

There are two ideas to reduce the electricity imbalance: one is that each retail electricity suppliers prevent the electricity imbalance, and the other is that it is enough as long as the electricity supply and demand are balanced in entire power grids [31]. In the latter idea, it does not matter even if the retailers cannot procure the same electricity as the actual electricity consumption.

This section discusses the calculating methods of the imbalance prices motivating the retailers to reduce the electricity imbalance. Subsection 6.1 formulates what conditions the imbalance prices should hold true. In Subsection 6.2, the authors present the optimal pricing method of the imbalance prices.

6.1 Conditions for imbalance prices

The retailers will not reduce the electricity imbalance unless their actual surplus is less than the retailers’ surplus expected in the scheduling phase. This is why Condition (63) should hold true to motivate the retailers to reduce the electricity imbalance, and the condition derives Condition (64) for the imbalance prices.

$${\widehat{RS}}_{0, t}>{RS}_{t}^{*}.$$

(63)

$${i}_{t}^{+}<\frac{(P\left({d}_{t}^{*}\right)-P({\widehat{s}}_{0,t}))-(W\left({d}_{t}^{*}\right)-W({\widehat{s}}_{0,t}))}{{d}_{t}^{*}-{\widehat{s}}_{0,t}}<{i}_{t}^{-}, {d}_{t}^{*}\ne {\widehat{s}}_{0,t}.$$

(64)

In only 3.05% of dates and times in FY 2017, Condition (64) held true. When the electricity shortage occurred, the percentage satisfying Condition (63) was 12.97%. Moreover, the width between the negative and positive imbalance prices was 1.19 JPY/kWh during the calculation period. Therefore, cheap negative imbalance prices and narrow gap between both imbalance prices might be the issues of the conventional pricing method. As a simple resolution, we can use Condition (64) as follows:

$${i}_{t}^{-}=\text{max}\left({i}_{t}^{-},13.27\right).$$

(65)

$${i}_{t}^{+}=\text{min}\left({i}_{t}^{+},13.25\right).$$

(66)

The corrected imbalance prices contributed to motivating the RES-AGs to reduce the electricity imbalance. Hence, the imbalance prices should satisfy Condition (64). Note: the pricings that the imbalance settlement leads the retailers to take too economic burdens may interfere the new retailers from participating in the retail sector.

6.2 Optimal pricing method of imbalance prices

Market designers should set economic incentives so that society can maximize its surplus when the society becomes an ideal state. In this discussion, the society is consisted of the retailers and their customers. This is why the authors use the maximized condition of the social surplus function. The maximized condition is represented as

$$\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}=\frac{\partial W\left({d}_{t}\right)}{\partial {d}_{t}}+\frac{\partial I\left({d}_{t}\right)}{\partial {d}_{t}},$$

(67)

$$I\left({d}_{t}\right)={i}_{t}^{\prime }\left({d}_{t}-{\widehat{s}}_{0,t}\right),$$

(68)

where ${i}_{t}^{\prime }$ is the optimal imbalance price defined in this paper.

The ideal society is that the electricity supply and demand are balanced. The economic incentives are the imbalance charges. The authors thus define Eq. (69) as the optimal imbalance price.

$${i}_{t}^{\prime }=\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={\widehat{s}}_{0,t}}-\frac{\partial W\left({d}_{t}\right)}{\partial {d}_{t}}{|}_{{d}_{t}={\widehat{s}}_{0,t}}.$$

(69)

Table 17 shows the percentages that social surplus was maximized at the notified electricity. The optimal pricing method led the society to maximize its surplus because the RES-AGs could maximize the social surplus by preventing the electricity imbalance. When the notified electricity was less than the lower bound of electricity consumption, however, the pricing method was invalid. In Eq. (69), we then have a choice to substitute the notified electricity to the lower bound of electricity consumption. It can construct the society that reducing the electricity imbalance as much as possible is the most economical.

Table 17 Percentages that total surplus of retail electricity suppliers and their cusomers was maximized at notified electricity

The proposed pricing method contributes to satisfying Condition (64) in more dates and times. In the numerical simulation models, at least, Condition (64) holds true in every case. This is because Conditions (70) and (71), and Eq. (72) hold true.

$$i_{t}^{ + } < i_{t}^{\prime } \left( {d_{t}^{*} } \right) < i_{t}^{ - } ,$$

(70)

$$\begin{gathered} {\text{if }}\hat{s}_{0,t} < d_{t}^{*} {\text{ then }}i_{t}^{\prime } \left( {\hat{s}_{0,t} } \right) > i_{t}^{\prime } \left( {d_{t}^{*} } \right), \hfill \\ {\text{if }}\hat{s}_{0,t} > d_{t}^{*} \,{\text{then}}\,i_{t}^{\prime } \left( {\hat{s}_{0,t} } \right) < i_{t}^{\prime } \left( {d_{t}^{*} } \right), \hfill \\ \end{gathered}$$

(71)

$${i}_{t}^{\prime }\left({d}_{t}\right)=\frac{\partial U\left({d}_{t}\right)}{\partial {d}_{t}}-\frac{\partial W\left({d}_{t}\right)}{\partial {d}_{t}},$$

(72)

where ${i}_{t}^{\prime }\left(\cdot \right)$ is the function of the optimal imbalance price.

If (${\widehat{s}}_{0,t}<{d}_{t}^{*}$) then ${i}_{t}^{\prime }({\widehat{s}}_{0,t})$ is the negative imbalance price, and it satisfies Condition (64). The more the electricity shortage increases, the higher the imbalance prices since the function of the optimal imbalance price is a monotonically decreasing function. In contrast, if (${\widehat{s}}_{0,t}>{d}_{t}^{*}$) then ${i}_{t}^{\prime }({\widehat{s}}_{0,t})$ is the positive imbalance price and satisfies Condition (64). The more the excess electricity increases, the lower the imbalance prices.

Figure 10 compares the scatter plots between the ratio of the electricity imbalance and the imbalance price in all dates and times. Interestingly, although Eq. (72) was designed for each time period, the more the electricity shortage increased, the higher the proposed imbalance prices were. Moreover, the more the excess electricity increased, the lower the proposed imbalance prices were.

Fig. 10

Scatter plots between ratio of electricity imbalance for notified electricity and imbalance price ($e\left({p}_{t}^{*}\right)=-0.40$)

These results show that the proposed imbalance prices can motivate the retailers to reduce the electricity imbalance. However, it is too hard for the market designers to obtain the precise customers’ utility function. The proposed pricing method thus needs an improvement as the imbalance prices can be calculated with only the data that the market designers can obtain. Even so, the discussion can contribute to designing the imbalance prices to motivate the retailers to reduce the electricity imbalance.

If each function in Eq. (69) is non-differentiable, the pricing method can be improved as.

$$i^{\prime }_{t} = i_{t} ,\,\mathop {\lim }\limits_{{d_{t} \to \hat{s}_{0,t} + 0}} \left( {\frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial W\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right) \le i_{t} \le \mathop {\lim }\limits_{{d_{t} \to \hat{s}_{0,t} - 0}} \left( {\frac{{\partial U\left( {d_{t} } \right)}}{{\partial d_{t} }} - \frac{{\partial W\left( {d_{t} } \right)}}{{\partial d_{t} }}} \right)$$

(73)

where ${i}_{t}$ is the imbalance price.

7 Conclusions

The authors proposed an improved methodology for designing the DR programs considering the economic impacts of the electricity imbalance. The authors’ proposal is constructed on the SWM framework in which the objective function is the sum of economic surplus of the retailers and their customers and impacts of electricity imbalance. This makes it possible to measure the value of electricity consumption to the customers and calculate the DR parameters (electricity prices or rebate levels). The optimal DR parameters in this paper were derived under the principle that rational customers act to maximize their economic surplus. The results of numerical simulations showed that the proposed methodology functioned well and benefits on the RES-AGs as well as their customers. On the other hand, the numerical simulations also revealed that there are cases where the imbalance charge cannot motivate the retailers to reduce the electricity imbalance. Therefore, this paper additionally presented the pricing method of imbalance charge as a solution to this issue.

However, the authors proposal still requires an important premise that the DR baseline is given accurately to the RES-AGs, which is impractical. This challenge will be addressed in future works. Furthermore, the mathematical challenges discussed in Sect. 2 will be addressed as an important challenge.