1 Introduction

Demand response programs which are designed to re-shape customers’ electricity usage patterns to enhance the reliability and efficiency of the grid [1]. Demand response programmes are generally divided into two main categories which are incentive-based programmes [2,3,4] and price-based programmes [5,6,7,8]. Dynamic pricing tariff has been commonly considered as one of the most efficient and economic price-based programmes [9], in which the electricity price varies between time-periods in a day, and is usually released one hour-ahead or one day-ahead (DAP). In this paper, we consider DAP where prices for the 24 hours of the following day are announced on the evening before. Such tariff has been used in utility companies in the USA [10, 11]. Furthermore, a dedicated review of dynamic electricity pricing can be found in [12].

Most existing research on dynamic pricing based demand response optimization assume that customers are installed with smart meters and home energy management systems (HEMS), i.e. there is an optimization software which is able to help customers schedule their home appliances usages explicitly to maximize customers’ utility such as minimizing their payment bills [13,14,15,16].

On the other hand, for customers without smart meters, utility functions are usually adopted to represent customers’ electricity use behaviors and preferences [17,18,19]. It is assumed that customers will maximize the defined utility functions and the retailer knows the exact utility functions of customers. However, in reality, customers’ utility functions may not be able to be represented in a mathematical form such as those in [17,18,19] simply because customers usually use their energy heuristically and their behavior patterns could change with time. Furthermore, the retailer might not be able to know the utility function information due to data privacy issues.

Without knowing the exact utility functions of customers, modelling customers’ responses to time-varying prices becomes important. Schweppe et al. [20] developed the concept of spot pricing of electricity where they envisaged a system wherein the spot prices change in real time, and customers can adjust their demand depending on the spot prices. Three responsive load models, namely, linear, exponential and potential demand functions are proposed. [21] indicated that the assumption that customers can reduce their loads indefinitely is unrealistic; instead, customers will attempt to shift their consumption to other periods. Therefore, the concept of the price-elasticity of demand is important for modelling responses. The influence of demand elasticity on the shape of the load curve has been addressed in the past works including the concepts of self-elasticity (the relation between demand and price in one time interval) and cross-elasticity (the relation between demand and price in different time intervals) [22,23,24,25]. For instance, in [22], demand models were formulated as linear models, and the matrix of elasticities which includes all self-elasticities and cross-elasticities was evaluated. A demand responsive model was proposed in [23] based on price self-elasticity, which consists of four different demand functions, namely, linear, exponential, logarithmic and potential functions. An economic model for the demand response which can explain the change and cross-period shift in consumption pattern of customers is proposed in [25]. However, the above price elasticity based demand models are not ready for use in the pricing optimization problem. This is mainly due to the lack of market behavior constraints between the elasticities, which can lead to incorrect demand modelling where higher market prices lead to higher usages. Motivated by the above analysis, in this paper, we adopt a realistic demand model based on price elasticity including relevant market behavior constraints to be use in the pricing optimization model.

Although the above and other unlisted studies have provided valuable insights on how to model customers’ demand patterns in the context of dynamic pricing, they only consider one type of customers (either customers with smart meters or customers without smart meters). However, there are real situations where two different types of customers coexist in the same electricity market. Unfortunately, no existing literature have studied this problem. As a result, in this paper we take the initiate to investigate such a demand response problem with mixed types of customers where the retailer needs to make best dynamic pricing decisions by taking into account different customers’ responses at the same time. To this end, we firstly develop a hybrid demand modelling framework to capture electricity use behaviors of mixed types of customers. Secondly, the pricing optimization problem is formulated for the retailer to maximize its profit. The main contributions of this paper can be summarized as follows:

  1. 1)

    We develop a hybrid demand modelling framework for mixed types of customers including customers without smart meters and customers with smart meters based on our previous work [16] [26]. In particular, the present demand model for customers without smart meters extends [26] by a detailed analysis of customers’ usage behaviors in response to time varying electricity prices through price elasticity matrix evaluations.

  2. 2)

    A profit maximization based optimal dynamic pricing is formulated for the retailer in such a realistic and meaningful electricity market scenario where two types of customers coexist.

  3. 3)

    A comprehensive simulation study is conducted by evaluating the proposed demand model and the pricing optimization model.

The rest of this paper is organized as follows: a hybrid demand modelling for both types of customers is presented in Section 2. The profit maximization model for the retailer is presented in Section 3. Simulations studies are conducted in Section 4 and the paper is concluded in Section 5.

2 Hybrid demand modelling for customers

The hybrid demand modelling at each hour \(h \in {\mathcal{H}}\;\,\triangleq\,\;\left\{ {1,2, \ldots ,H} \right\}\) for the mixed customer pool can be achieved by aggregating the demand of each type of customers for that hour where the demand modelling of customers without/with smart meters are presented in subsection 2.1 and subsection 2.2 respectively.

2.1 Demand model of customers without smart meter

In this subsection, we consider that at the end of each day, the retailer announces electricity prices for the next 24 hours; then, based on the announced prices, customers determine and consume electricity based on their preferences. We also assume that the retailer does not have demand information from customers because of the unavailability of (mathematical or analytical) utility functions for customers. To solve this problem, our proposed solution is building hourly demand models (i.e., hourly reaction functions to the announced prices) from historical data in terms of varying prices over the next 24 hours.

Throughout this paper, we assume that the price and demand information for the last \(N\) days are available. Based on such information, our objective is to decide the prices on day \(N + 1\) to achieve the maximum profit for the retailer. We denote the prices of electricity on day \(n \in \aleph \;\,\triangleq\,\;\left\{ {1,2, \ldots ,N} \right\}\) as \(P\left( n \right) = \left[ {p_{1} \left( n \right),p_{2} \left( n \right), \ldots ,p_{h} \left( n \right), \ldots ,p_{H} \left( n \right)} \right]\) where \(p_{h} \left( n \right)\) represents the price at hour \(h \in {\mathcal{H}}\,\triangleq\,\left\{ {1,2, \ldots ,H} \right\}\) on day \(n \in \aleph\) and \(H\) represents the pricing horizon; in this study, \(H = 24\). We also denote the demand for electricity from all customers on day \(n \in \aleph\) as \(D\left( n \right) = \left[ {d_{1} \left( n \right),d_{2} \left( n \right), \ldots ,d_{h} \left( n \right), \ldots ,d_{H} \left( n \right)} \right]\) where \(d_{h} \left( n \right)\) is the load demand from customers at hour \(h \in {\mathcal{H}}\) on day \(n\).

The usage of electricity of customers during each hour depends not only on the price of electricity at that hour but also prices of other hours [22]. As a result, we consider the price- reaction function from all customers at each hour as:

$$y_{h} = R_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)$$
(1)

where \(y_{h}\) is the amount of electricity demanded by customers at hour \(h \in {\mathcal{H}}\). As aforementioned, the retailer cannot know \(R_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)\), and we need to find an estimated reaction function \(\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)\) that is as close to \(R_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)\) as possible. This requirement can be expressed as:

$${ \hbox{min} }\;\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {\hat{R}_{h} - R_{h} } \right)^{2}$$
(2)

In this paper, we use linear model to express the estimated reaction function [26] for each hour \(h \in {\mathcal{H}}\):

$$\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right) = \alpha_{h} + \beta_{h,1} p_{1} + \cdots + \beta_{h,H} p_{H}$$
(3)

where parameter \(\beta_{h,h}\) is direct elasticity and parameter \(\beta_{h,c}\) (\(h \ne c\)) is cross-price elasticity [22,23,24].

The self-elasticity measures the responsiveness of the demand for electricity at hour \(h \in {\mathcal{H}}\) to its own price\(p_{h}\). When the price at hour \(h\) increases but prices of the other hours remain unchanged, the demand at hour\(h\) will decrease. Therefore, the self-elasticity \(\beta_{h,h}\) is always negative. That is, the following inequality holds:

$$\beta_{h,h} < 0$$
(4)

The cross elasticity measures the responsiveness of the demand for electricity at hour \(h \in {\mathcal{H}}\) to changes in price at some other hour\(c \in {\mathcal{H}}\). When the price of electricity at hour \(c\) (i.e. \(p_{c}\)) increases but prices at other hours remain unchanged, some demand at hour \(c\) may be switched from hour \(c\) to hour \(h\) due to the price increase at hour \(c\). As a result, cross elasticity \(\beta_{h,c}\) is always positive [22]. That is:

$$\beta_{h,c} > 0\quad if \quad h \ne c$$
(5)

In addition, the necessary and sufficient condition that the electricity is a demand-consistent product in the considered retail market is also considered. That is, for each \(h,c \in {\mathcal{H}}\), the following inequality holds:

$$\beta_{h,h} + \mathop \sum \limits_{c = 1,c \ne h}^{N} \beta_{c,h} < 0$$
(6)

The detailed proof of necessary and sufficient of (6) can be found in our previously published paper [26]. Finally the demand estimation model is given by (2)–(6).

2.2 Demand model for customers with smart meter

For customers with smart meters, they are assumed to respond to dynamic electricity prices to minimize their electricity bill payments. It is assumed that home energy management systems (HEMS) are embedded in the smart meter where there is a two-way communication and control infrastructure between the smart meter control center and each home appliance. Such an HEMS can schedule the electricity consumptions of each appliance on behalf of customer. The resulted electricity consumption scheduling problem aims to determine when and how much electricity to consume in each time period for each home appliance within the scheduling window to minimize customer’s electricity bills. Throughout the rest of this paper, we denote \({\mathcal{K}}\) as the set of customers who have installed smart meters at their homes where \(K\,\triangleq\,\left| {\mathcal{K}} \right|\).

The home appliances are divided into three different categories: non-shiftable appliances (e.g., lights), shiftable appliances (e.g. washing machines), and curtailable appliances (e.g. air-conditioners) [16].

For each customer \(k \in {\mathcal{K}}\;\,\triangleq\,\;\left\{ {1,2, \ldots ,K} \right\}\), we denote \(A_{k}\) as the set of appliances in household \(k\). In addition, we define relevant sets of shiftable and non-shiftable appliances\(S_{k}\). The reason why we can use the same model to model all shiftable and non-shiftable appliance is that non-shiftable appliances can be treated as one special type of shiftable appliances for which the amount of electricity that can be shifted is zero. We also define the set of curtailable appliances\(C_{k}\). Therefore, we have\(A_{k} = S_{k} \cup C_{k}\), and \(S_{k} \cap C_{k} = \emptyset\). As a result, the bill payment minimization problem of each customer \(k\) is divided into two sub-problems because the appliances sets are non-overlapping.

  1. 1)

    Shiftable and non-shiftable appliances

For each customer \(k \in {\mathcal{K}}\) and each appliance \(s \in S_{k}\), we define an electricity consumption scheduling vector:

$$\varvec{x}_{k,s} = \left[ {x_{k,s}^{1} , \ldots ,x_{k,s}^{h} , \ldots x_{k,s}^{H} } \right]$$
(7)

where \(x_{k,s}^{h}\) represents the electricity consumption of appliance \(s\) at hour \(h\) for \(k\)-th customer.

Each customer needs to set a scheduling window \({\mathcal{H}}_{k,s}\,\triangleq\,\left\{ {\rho_{k,s} , \ldots ,\sigma_{k,s} } \right\}\) for each shiftable appliance \(s \in S_{k}\), in which \(\rho_{k,s}\) indicates the earliest possible beginning time of the scheduling window, and \(\sigma_{k,s}\) is the last acceptable ending time of the scheduling window.

In the scheduling window \({\mathcal{H}}_{k,s} ,\) the electricity consumption of appliance \(s\) can be shifted from periods with high electricity prices to periods with low electricity prices. However, the total electricity consumption needed to finish all the operations for appliance \(s\) is fixed and defined as \(E_{k,s}\). Therefore, we have:

$$\mathop \sum \limits_{{h = \rho_{k,s} }}^{{\sigma_{k,s} }} \;x_{k,s}^{h} = E_{k,s}$$
(8)
$$x_{k,s}^{h} = 0\quad \forall h \in {\mathcal{H}}/{\mathcal{H}}_{k,s}$$
(9)

For each shiftable appliance \(s \in S_{k}\), we also define the minimum electricity consumption level during each hour as \(\gamma_{k,s}^{ \hbox{min} }\) and the maximum electricity consumption level during each hour as \(\gamma_{k,s}^{ \hbox{max} }\). Therefore, we have:

$$\gamma_{k,s}^{ \hbox{min} }\,<\,x_{k,s}^{h}\,<\,\gamma_{k,s}^{ \hbox{max} }\quad \forall h \in {\mathcal{H}}_{k,s}$$
(10)

The minimum electricity consumption level \(\gamma_{k,s}^{ \hbox{min} }\) can be considered an electricity consumption of the appliance in the standby mode while \(\gamma_{k,s}^{ \hbox{max} }\) is the maximum electricity consumption level and usually stands for the rated electricity consumption level.

Finally, for each customer \(k \in {\mathcal{K}}\), the electricity bill minimization problem for shiftable and non-shiftable appliances can be modelled as follows, which a linear programming problem and can be solved easily using existing solvers.

$${ \hbox{min} } F_{{S_{k} }} = \mathop {\hbox{min} }\limits_{{x_{k,s}^{h} }} \mathop \sum \limits_{{s \in S_{k} }} \mathop \sum \limits_{{h = \rho_{k,s} }}^{{\sigma_{k,s} }} p_{h} x_{k,s}^{h}$$

s.t.

$$\mathop \sum \limits_{{h = \rho_{k,s} }}^{{\sigma_{k,s} }} \;\;x_{k,s}^{h} = E_{k,s}$$
(11)
$$x_{k,s}^{h} = 0\quad \forall h \in {\mathcal{H}}/{\mathcal{H}}_{k,s}$$
$$\gamma_{k,s}^{ \hbox{min} } < x_{k,s}^{h} < \gamma_{k,s}^{ \hbox{max} }\quad \forall h \in {\mathcal{H}}_{k,s}$$
  1. 2)

    Curtailable appliances

For each customer \(k \in {\mathcal{K}}\) and for each curtailable appliance \(c \in C_{k}\), we define an electricity consumption scheduling vector as follows:

$$\varvec{x}_{k,c} = \left[ {x_{k,c}^{1} , \ldots ,x_{k,c}^{h} , \ldots x_{k,c}^{H} } \right]$$
(12)

Similar to shiftable and non-shiftable appliances, for each appliance \(c \in C_{k}\), we assume that each customer needs to set a scheduling window \({ \mathcal{H}}_{k,c}\,\triangleq\,\left\{ {\rho_{k,c} , \ldots ,\sigma_{k,c} } \right\}\), in which \(\rho_{k,c}\) indicates the earliest beginning time of the scheduling window, and \(\sigma_{k,c}\) represents the latest ending time of the scheduling window.

For each curtailable appliance \(c \in C_{k}\), we define the minimum electricity consumption level during each hour as \(\gamma_{k,c}^{ \hbox{min} }\) and the maximum electricity consumption level during each hour as \(\gamma_{k,c}^{ \hbox{max} }\). Therefore, we have:

$$\gamma_{k,c}^{ \hbox{min} } < x_{k,c}^{h} < \gamma_{k,c}^{ \hbox{max} }\quad \forall h \in {\mathcal{H}}_{k,c}$$
(13)
$$x_{k,c}^{h} = 0\quad \forall h \in {\mathcal{H}}/{\mathcal{H}}_{k,c}$$
(14)

Furthermore, the electricity usage of curtailable appliance \(c \in C_{k}\) of each customer \(k \in {\mathcal{K}}\) at each hour \(h \in {\mathcal{H}}_{k,c}\) is modelled as a linear function of electricity price: \(f_{k,c} \left( {p_{h} } \right) = a_{k,c} p_{h} + b_{k,c}\). As a result, the electricity bill minimization problem for curtailable appliances for each customer can be modelled as follows:

$${ \hbox{min} } \; F_{{C_{k} }} = \mathop {\hbox{min} }\limits_{{x_{k,c}^{h} }} \mathop \sum \limits_{{s \in C_{k} }} \mathop \sum \limits_{{h = \rho_{k,c} }}^{{\sigma_{k,c} }} p_{h} x_{k,c}^{h}$$

s.t.

$$\gamma_{k,c}^{ \hbox{min} } < x_{k,c}^{h} < \gamma_{k,c}^{ \hbox{max} }\quad \forall h \in {\mathcal{H}}_{k,c}$$
(15)
$$x_{k,c}^{h} = f_{k,c} \left( {p_{h} } \right) = a_{k,c} p_{h} + b_{k,c}\quad \forall h \in {\mathcal{H}}_{k,c}$$

Finally, for each customer\(k \in {\mathcal{K}}\,\triangleq\,\left\{ {1,2, \ldots ,K} \right\}\), the optimal electricity consumption scheduling problem for the whole household with smart meter can be modelled as:

$${ \hbox{min} } \left( {F_{{C_{k} }} + F_{{S_{k} }} } \right)$$
(16)

3 Profit maximization model for the retailer

The profit modelled in our problem is defined by revenue minus electricity cost which includes wholesale cost of electricity, services costs, and etc.

In the real world, many factors exist that can affect the procurement cost of electricity for a retailer. For simplicity, a cost function \(C_{h} \left( {L_{h} } \right)\) is defined and used to represent such a cost of providing electricity at each hour \(h \in {\mathcal{H}}\,\triangleq\,\left\{ {1,2, \ldots ,H} \right\}\) where \(L_{h}\) represents the total amount of electricity provided to all customers in the retail electricity market at each hour. We assume that the cost function \(C_{h} \left( {L_{h} } \right)\) is convex, increasing in \(L_{h}\) for each hour \(h\) [16, 18]. We design the cost functions as follows:

$$C_{h} \left( {L_{h} } \right) = a_{h} L_{h}^{2} + b_{h} L_{h} + c_{h}$$
(17)

where\(a_{h} > 0\), and \(b_{h} \ge 0\), \(c_{h} \ge 0\) for each hour \(h \in {\mathcal{H}}\).In this paper, we consider two pricing optimization formulations for the retailer: ① a market only containing customers without smart meters; ② a market containing mixed customers (customers with smart meters and customers without smart meters).

3.1 Pricing optimization for market only containing customers without smart meters

For each hour\(h \in {\mathcal{H}}\), we define the minimal and maximum price that the retailer can offer to its customers as

$$p_{h}^{ \hbox{min} } \le p_{h} \le p_{h}^{ \hbox{max} }$$
(18)

where \(p_{h}^{ \hbox{min} }\)and \(p_{h}^{ \hbox{max} }\) are usually set based on factors such as the cost of wholesale electricity, customers’ willingness and affordability to pay, and political pressure. For instance, usually to avoid a loss, the retail price should be higher than the corresponding wholesale price. On the other hand, an upper bound of the retail price is often needed to create a reasonable long-term price image to withhold market share from competitors. Further, government policy and customers’ acceptability also force prices to be upper bounded.

We also set a maximum supply capacity for each hour\(h \in {\mathcal{H}}\), which is denoted as \(E_{h}^{ \hbox{max} }\). Therefore, we have

$$\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right) \le E_{h}^{ \hbox{max} }\quad \forall h \in {\mathcal{H}}$$
(19)

Furthermore, a constraint on the total revenue should exist due to pressure from government policy and customers’ acceptability. Thus, we have the constraint

$$\mathop \sum \limits_{{{{h}} \in {\mathcal{H}}}} p_{h} \left( {\alpha_{h} + \beta_{h,1} p_{1} + \ldots + \beta_{h,H} p_{H} } \right) \le R^{ \hbox{max} }$$
(20)

It should be noted that, the above revenue constraint is critically important in practice from the profit maximization perspective due to that electricity is a basic necessity and therefore less elastic. Based on our experiment, without such a constraint, the optimal prices will always increase to the upper price bounds because such maximum prices usually represent greatest profitability.

In addition, minimizing the peak-to-average ratio (PAR) is another request that retailers should consider and is commonly achieved by shifting the use of high-power household appliances from peak hours to off-peak hours. In this paper, we consider there is a maximum PAR constraint imposed on the retailer. By denoting the daily peak and average load levels as \(L_{peak} = \mathop {\hbox{max} }\limits_{{h \in {\mathcal{H}}}} L_{h}\) and \(L_{avg} = \frac{{\sum\limits_{{h \in {\mathcal{H}}}} {L_{h} } }}{H}\), the PAR is formulated as \(\frac{{L_{peak} }}{{L_{avg} }}\) by definition. Since the total load across all customers at each hour \(h \in {\mathcal{H}}\) in our considered problem is \(\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)\), the PAR can then be defined as \(\frac{{\mathop { \hbox{max} }\limits_{{h \in {\mathcal{H}}}} {\hat{R}_{h}} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)}}{{\sum\limits_{{h \in {\mathcal{H}}}} {\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)} /H}}\) Therefore, we have the following PAR constraint immediately.

$$\frac{{\mathop { \hbox{max} }\limits_{{h \in {\mathcal{H}}}} {\hat{R}_{h}} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)}}{{\sum\limits_{{h \in {\mathcal{H}}}} {\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)} /H}} \le PAR_{ \hbox{max} }$$

The above constraint with \({ \hbox{max} }\) function in the left hand side can then be reformulated to (21) to improve the mathematical tractability.

$$\frac{{H\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)}}{{\sum\limits_{{h \in {\mathcal{H}}}} {\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)} }} \le PAR_{ \hbox{max} } \;\;\;\;\forall h \in {\mathcal{H}}$$
(21)

Finally, the pricing optimization problem for the retailer can be expressed as the following quadratic programming with quadratic constraints problem (QPQCP), which can be solved using the SCIP solver in OPTI TOOLBOX [27].

$${ \hbox{max} }\;\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {\left( {p_{h} - c_{h} } \right)\hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right)} \right)$$
(22)

s.t. (18)–(21)

3.2 Pricing optimization for market containing mixed customers with and without smart meters

We first rewrite the maximum supply capacity constraint of (19) in the previous subsection to consider both types of customers. As a result, we have:

$$\mathop \sum \limits_{{k \in {\mathcal{K}}}} \mathop \sum \limits_{{a \in A_{k} }} x_{k,a}^{h} + \hat{R}_{h} \left( {p_{1} ,p_{2} , \ldots ,p_{H} } \right) \le E_{h}^{ \hbox{max} } \;\;\;\;\forall h \in {\mathcal{H}}$$
(23)

Furthermore, by rewriting the total revenue constraint of (20) in the previous subsection, we have:

$$\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {p_{h} \left( {\left( {\mathop \sum \limits_{{k \in {\mathcal{K}}}} \mathop \sum \limits_{{a \in A_{k} }} x_{k,a}^{h} } \right) + \left( {\alpha_{h} + \beta_{h,1} p_{1} + \ldots + \beta_{h,H} p_{H} } \right)} \right)} \right) \le Revenue^{ \hbox{max} }$$
(24)

Finally, the profit maximization for the retailer can be modelled as (25). As (25) is a nonlinear and non-convex problem and difficult to solve using conventional nonlinear programming methods, we use genetic algorithm (GA) in this study to solve the above profit maximization problem for the retailer.

$$\begin{aligned} { \hbox{max} }\left\{ {\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {p_{h} \left( {\left( {\mathop \sum \limits_{{k \in {\mathcal{K}}}} \mathop \sum \limits_{{a \in A_{k} }} x_{k,a}^{h} } \right) + \left( {\alpha_{h} + \beta_{h,1} p_{1} + \ldots + \beta_{h,H} p_{H} } \right)} \right)} \right) - } \right. \hfill \\ \;\;\;\;\;\;\left. {\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {C_{h} \left( {\left( {\mathop \sum \limits_{{k \in {\mathcal{K}}}} \mathop \sum \limits_{{a \in A_{k} }} x_{k,a}^{h} } \right) + \left( {\alpha_{h} + \beta_{h,1} p_{1} + \ldots + \beta_{h,H} p_{H} } \right)} \right)} \right)} \right\} \hfill \\ \end{aligned} $$
(25)

s.t. (18), (23) and (24)

4 Results and analysis

In this section, we first evaluate the demand modelling and pricing optimization in the electricity market only containing customers without smart meters. Second, the pricing optimization model for the electricity market including both types of customers will be evaluated.

4.1 Result and evaluation of demand modelling and pricing optimization in electricity market only containing customers without smart meters

  1. 1)

    Results and evaluation of demand modelling

As aforementioned, our demand model can not only be used to predict short-term demand accurately, but also be used to analyse customer’s behaviour of using electricity based on varying prices. Since the capability of short term demand forecasting of our demand model has been comprehensively evaluated in our previous paper [26], in this section, we will focus on analyzing all elasticity coefficients and the elasticity matrix to obtain a better understanding of the demand behaviors. The dataset being used to build the demand model for the evaluation is from the PJM and includes the day-ahead electricity prices and demand information between 01/01/2011 and 22/08/2012 (600 days) for a set of residential customers. The dataset is available to the public online at [28].

For a 24-hour scheduling period, the cross elasticity coefficients can be arranged in a 24-by-24 matrix as follows [24]:

$$\left[ {\begin{array}{*{20}c} {\begin{array}{*{20}c} {\beta_{1,1} } \\ {\beta_{2,1} } \\ \end{array} } & {\begin{array}{*{20}c} {\beta_{1,2} } \\ {\beta_{2,2} } \\ \end{array} } & {\begin{array}{*{20}c} {\begin{array}{*{20}c} \cdots & { \beta_{1,H} } \\ \end{array} } \\ {\begin{array}{*{20}c} { \cdots } & {\beta_{2,H} } \\ \end{array} } \\ \end{array} } \\ \vdots & \vdots & {\begin{array}{*{20}c} { \ddots } & { \vdots } \\ \end{array} } \\ {\beta_{H,1} } & {\beta_{H,2} } & { \begin{array}{*{20}c} { \cdots } & {\beta_{H,H} } \\ \end{array} } \\ \end{array} } \right]$$

Figure 1 shows that the absolute values of \(\beta_{h,h}\) during the period around midnight (11 pm to 6 am) and during working hours (9 am to 3 pm) are much higher than other periods. This phenomenon indicates that the usage of electricity is more relaxed during these two typical periods. Therefore, people are more willing to shift their usage of electricity from these hours to other hours if the price is excessive. However, during the heavy consumption hours between 7 to 8 am, as previously mentioned, people are waking up and preparing for work, and nearly all electricity consumption during this period is necessary and not easy to shift. In addition, during the hours between 4 to 10 pm when people are coming home from work, the usage of electricity for tasks such as cooking for dinner and watching TV and common usage of lighting are time constrained and difficult to shift.

Fig. 1
figure 1

Average \(\beta_{h,h}\)s of a day computed for the period between 01/01/2011 and 22/08/2012 (600 days)

Full size image

The elasticity matrix is illustrated as Fig. 2 and is based on the visualization of the evaluation of the elasticity matrix in [22]. We can see that, generally, for an hour c that is closer to h, the absolute values of \(\beta_{h,c}\) (\(h \ne c\)) is much greater than the absolute values of \(\beta_{h,c}\) (\(h \ne c\)) when hour c is farther away from hour h. This phenomenon indicates that, although some usage of electricity during a specific hour is shiftable, customers are more willing to shift this electricity consumption to an hour that is closer to this specific hour. This is because they do not want to wait too long, especially during the daytime (approximately 7 am to 8 pm); thus, the nonzero elements are clustered around the diagonal, which indicates that customers’ flexibility is limited, namely, they only can tolerate postponing or advancing the use of electricity for one or two hours. However, during off-peak hours (i.e., the period around midnight), especially between 11 pm and 2 am, the non-zero elements are spread widely over the column, which indicates that consumers are more willing to reschedule their usages over a longer period as the usage of electricity during this period is not as urgent.

Fig. 2
figure 2

Average estimated price elasticities (all \(\beta\)s) computed for the period between 01/01/2011 and 22/08/2012 (600 days)

Full size image
  1. 2)

    Results and evaluation of pricing optimization

Our pricing optimization model was evaluated using the following methods: first, we identify the demand models from the data between 01/01/2011 and 22/08/2012 (600 days), and based on the identified demand models, we use our pricing optimization model to generate the optimal prices of 24 hours for the 601st day (23/08/2012). We will then perform a comparison between our optimized prices \(X = \left[ {p_{1} ,p_{2} , \ldots ,p_{H} } \right]\) and the original prices \(X^{\prime} = \left[ {p_{1}^{'} ,p_{2}^{'} , \ldots ,p_{H}^{'} } \right]\) in the dataset to determine how great of an improvement can be achieved. This can be expressed as:

$$IMPV = \frac{{\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {\left( {p_{h} - c_{h} } \right)\hat{R}_{h} \left( X \right)} \right) - \mathop \sum \limits_{{{{h}} \in {\mathcal{H}}}} \left( {\left( {p_{h}^{'} - c_{h} } \right){\hat{R}_{h}} \left( {X^{'} } \right)} \right)}}{{\mathop \sum \limits_{{h \in {\mathcal{H}}}} \left( {( {p_{h}^{'} - c_{h} } )\hat{R}_{h} \left( {X^{'} } \right)} \right)}}$$

The parameter setting for our experiment is showed in Table 1. Figure 3 shows a comparison of original prices and optimized prices on 601st day, and Fig. 4 shows a comparison between optimized and original profit based on similar revenue of 601st day. From these two figures, we found that the optimized prices generated from our optimization model are more sensible, and could achieve higher profits on the 601st day (approximately 4.6%) even though the revenue does not increase. The above findings are further confirmed by Fig. 5 which shows a comparison between optimized and original profit based on same revenue setting between the 601st day and the 614st day.

Table 1 Parameters setting for pricing optimization
Full size table
Fig. 3
figure 3

Comparison between original prices and optimized prices on the 601st day

Full size image
Fig. 4
figure 4

Comparison between optimized and original revenue and profit of 601st day

Full size image
Fig. 5
figure 5

Comparison between optimized and profit between 601st day and 614st day

Full size image

4.2 Result and evaluation of pricing optimization in mixed electricity market containing customers with and without smart meters

In this experiment, we simulate an electricity market with 1000 households (including some households with/without smart meters), all of whom are served by one electricity retailer. For evaluation, we first set parameters and generate data for both types of customers (both with and without smart meters) separately. Second, we implement the optimization method proposed in Sect. 3.2 (which treats the two types of customers differently) on the dataset. Third, we implement the optimization method proposed in Sect. 3.1 (build the demand model for all customers, which includes both types of customers, as the customers without smart meters in order to provide a comparison case) on the same dataset. Fourth, we compare the results generated from both optimization methods to find out whether the retailer can obtain any improvement on the profit maximization using the detailed utility function from all or part of its customers (from the implementation of optimization proposed earlier in this section).

For customers with smart meters, we assume each customer has eight appliances (including shiftable, non-shiftable and curtailable appliances) at home. The scheduling horizon is set to 24 hours, from 8 am on one day to 8 am the next day. The parameter settings for each category of appliances for each customer with smart meter are listed in Tables 2, 3 and 4.

Table 2 Parameters setting of shiftable appliances for each household
Full size table
Table 3 Parameters setting of non-shiftable appliances for each household
Full size table
Table 4 Parameters setting of curtailable appliances for each household
Full size table

For customers without smart meters, we use the dataset from the PJM that includes both day-ahead electricity prices and demand information between 01/01/2012 and 30/11/2013 for a set of residential customers. The dataset is available to the public online at [28]. Before applying demand modelling to the dataset, we normalize the dataset to the demand from around 500 households (to ensure that the total demand from customers without smart meters is similar to the total demand from customers with smart meters).

As mentioned, we choose a quadric cost function: \(C_{h} \left( {L_{h} } \right) = a_{h} L_{h}^{2} + b_{h} L_{h} + c_{h}\) to model the cost of electricity for the retailer, where \(L_{h}\) represents the amount of electricity consumed by all existing customers in the market at each hour of the day. For simplicity, we set \(b_{h} = 0\) and \(c_{h} = 0\) for each hour\(h \in {\mathcal{H}}\). Also we set \(a_{h} = 5.5 \times 10^{ - 4}\) cents during the hours from 8 am to 12 am and \(a_{h} = 4.0 \times 10^{ - 4}\) cents during the hours from 12 am to 8 am the next day.

In our experiment, we design three test cases in which the proportions of households with/without smart meters are different as shown in Table 5.

Table 5 Parameters setting of curtailable appliances for each household
Full size table

The detailed results from the experiments on these three cases are shown in Figs. 6, 7, 8, 9, 10 and 11, in which Figs. 6, 8 and 10 show comparisons between the optimal prices obtained from the price optimization model proposed in subsection 3.2 and the model proposed in subsection 3.1 whereas Figs. 7, 9 and 11 show comparisons between the optimal profits obtained from the above two mentioned pricing optimization models. More precisely, as shown in Figs. 7, 9 and 11, the retailer under the model proposed in subsection 3.2 (including both types of customers) receives more accurate information from customers who have smart meters installed (the detailed customers’ utility functions) and therefore can gain more profit when compared with that under the model in subsection 3.1 (with only customer without smart meters). It is worth noting that with the proportion of smart meter installed households increases, the model in subsection 3.2 can gain further profits for the retailer (i.e. a 3.2% improvement for Case 1, a 4.9% improvement for Case 2 and a 6.2% improvement for Case 3).

Fig. 6
figure 6

Comparison of optimal prices obtained from proposed method from subsection 3.1 and subsection 3.2 (Case 1)

Full size image
Fig. 7
figure 7

Comparison of optimal profit obtained from proposed method from subsection 3.1 and subsection 3.2 (Case 1) (3.2% Improvement)

Full size image
Fig. 8
figure 8

Comparison of optimal prices obtained from proposed method subsection 3.1 and subsection 3.2 (Case 2)

Full size image
Fig. 9
figure 9

Comparison of optimal profit obtained from proposed method from subsection 3.1 and subsection 3.2 (Case 2). (3.2% Improvement)

Full size image
Fig. 10
figure 10

Comparison of optimal prices obtained from proposed method subsection 3.1 and subsection 3.2 (Case 3)

Full size image
Fig. 11
figure 11

Comparison of optimal profit obtained from proposed method subsection 3.1 and subsection 3.2 (Case 3). (6.3% Improvement)

Full size image

5 Conclusion

In this paper, we study the dynamic pricing optimization problem to support the retailer to make the best day-ahead dynamic pricing decision to maximize its profit under a realistic scenario that mixed types of customers coexist in the electricity market where some customers are installed with smart meters in their home whereas other customers have not. To this end, we firstly propose a hybrid demand model to capture the demand behaviors of both types of customers. Secondly, a profit maximization algorithm is developed based on the established hybrid demand model to obtain optimal prices for the retailer under realistic market constraints. The simulation results show that the retailer can achieve a 5% profit improvement by using our proposed pricing model.