Background

Recent research indicates that human society must begin reducing greenhouse gas emissions sharply by 2020 and reach zero net emissions by 2040–2055 in order to prevent dangerous climate change [1]. The electricity sector will likely play a key role in this transition, absorbing renewable power and distributing it to traditional loads and electrified heating and transport sectors [2]. Achieving this transition at least cost will require widespread analysis and optimization of high-renewable power system designs, integrated with the surrounding economy.

In recent years, several new, open-source or academic power system models have been introduced to aid in this effort [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]. These models are publicly documented and available at no cost, enabling a wide range of stakeholders to propose and assess transition plans, often integrated with broader economic modeling. However, most renewable energy integration studies to date have used closed-source, proprietary models, e.g., [19,20,21,22,23,24,25,26,27]. These proprietary models are generally written and run by well-known firms in the power industry, and over the years, they have built up a store of confidence among electricity planners. This confidence comes from a history of benchmarking against existing power systems, repeated use for a wide variety of studies, and backing by respected firms. However, established, commercial models are only available at high cost; do not use public, peer-reviewed formulations; and are not easily extendable for integration into broader economic modeling.

This leaves a gap in our ability to plan for high-renewable power systems. Open-source and academic models have sound, peer-reviewed formulations; are accessible and transparent enough to support consensus-building among a variety of stakeholders; and can be integrated with larger economic models to conduct advanced modeling of high-renewable economies [4, 28,29,30]. However, power system planners are not yet confident that the open and academic models accurately represent power system operations. On the other hand, power system operators are confident in using proprietary power system models, but those models are not transparent or accessible enough to support consensus-building and integrated energy-economic modeling.

This work seeks to bridge this gap, comparing results from one of the most capable open-source planning models—Switch 2.0—against one of the best-established proprietary power system models—GE Energy Consulting’s Multi-Area Production Simulation (MAPS)—when modeling operation of a high-renewable power system in Hawaii. The primary goal is to test whether the open-source model produces similar results to the established, benchmarked model, in order to help users judge whether it is accurate enough for their work. As a secondary goal, this research also tests the replicability of work previously done with the proprietary model, a key element of the scientific process [29, 31, 32].

In other fields, especially climate modeling, intercomparisons between models are common, either to benchmark the accuracy of different models [33,34,35] or to identify areas of consensus and disagreement among the models [36,37,38,39]. This study takes the former approach, focusing on the question of whether a new, open-source model is able to provide similar results to an established and validated commercial model when modeling a real-world power system. There has also been a serious effort in psychology and other social sciences fields in recent years to test the replicability of previously published findings, in order to ensure the soundness of the field’s work [31, 32]. This paper also makes a small contribution along these lines, testing whether results reported from a proprietary, industry-standard model can be replicated using an open, peer-reviewed model [29].

Switch is an open-source and peer-reviewed model for power systems with large shares of renewable power, storage, and demand response. It can run in either capacity-expansion mode (choosing an optimal power system design directly) or production-cost mode (evaluating the cost of running a pre-specified power system design). In this work, we focus on production-cost modeling, which is vitally important to power system operators and is one of the main strengths of existing, proprietary power system models. Switch was released as open-source software in 2008 [13, 40] and has subsequently been used for a number of long-term studies of renewable energy adoption [41,42,43,44,45,46,47]. Here we test Switch version 2.0, which was released in 2018 [3, 48].

MAPS is a commercial-grade power system production-cost model that has been widely used for renewable integration studies in Hawaii and elsewhere [19, 24, 25, 49,50,51,52,53,54,55,56]. It has also been calibrated against system operations in Oahu and Maui [57, 58], giving an extra element of “ground truth” to its findings.

We are not aware of any previous model intercomparison between open-source and commercial grid planning software, especially focusing on renewable energy integration. This gap may be partly because open-source grid planning models have only become widely available in recent years and partly because few researchers have access to the inputs and assumptions that are needed to replicate professional integration studies. It also requires a great deal of effort to set up a second model to perform a benchmarking study, and resources are rarely available to do this even if it can significantly improve confidence and understanding of both models and their findings. This study takes advantage of the unusual fact that comprehensive data are available on the inputs, assumptions, and findings from the MAPS model, as run for the Hawaii Renewable Portfolio Standards Study (RPS study) [59], and that Switch 2.0 has also been configured to model the same power system in order to support planning for the state’s 100% renewable portfolio standard [46].

By necessity, this study is deep rather than broad, comparing fine details of the behavior of these two models in one particular region, as reported in the RPS Study. We are not able to compare Switch to other proprietary models or locations because complete reports of inputs and assumptions are not available for other models or locations. We are also unable to compare Switch to MAPS for factors that were not reported in the RPS Study, because we do not have access to all of the MAPS outputs.

In the “Results and discussion” section, we compare the behavior of Switch and MAPS for all the power-production results presented in the RPS Study and discuss the implications of this comparison and possible sources of disagreement between the models. The “Methods” section and Additional file 1 describe how Switch was configured for the study. Assembling datasets and configuring Switch for this study required significant effort, and the data and model are now available via a public repository at https://github.com/switch-hawaii/ge_validation.

Methods

Switch normally uses mixed-integer optimization methods for unit commitment and dispatch. MAPS uses linear optimization methods for unit commitment, with heuristic rules to enforce integer decision variables. For the RPS Study [59], MAPS was specially configured to match the Hawaiian Electric Companies’ heuristic unit commitment rules. The RPS Study report included a substantial body of information about how MAPS was configured, and GE was generous in providing additional data and answers to inquiries. However, the goal of the RPS Study was to evaluate different pathways of renewable integration in Hawaii, rather than to document modeling methods or data, which are constantly evolving. Consequently, we inferred some operational details from the figures and results presented in the RPS Study, and in some cases made assumptions that may have differed from ones that MAPS used. The model configuration and inputs used for this study are described below. Switch 2.0 is available from [48]. The data and code used for this model intercomparison are available from [60].

Switch model setup

The Switch model is primarily used as a power system capacity expansion model (co-optimizing selection of new assets and operational decisions), but it uses a flexible, modular software framework that allows it to be customized and run as a production cost model (optimizing operational decisions only). For this paper, we run Switch exclusively in production-cost mode, to match the work done with MAPS.

For this intercomparison, we used the following standard Switch modules (documented in more detail in [3]):

  • Time sampling (we ran Switch with a single month-long time series for each month of 2020, with hourly time steps—12 separate models for each scenario—then aggregated the results);

  • Financial calculations (Switch minimizes costs on an NPV basis, we used a 0% discount rate to calculate simple costs without discounting);

  • Generator construction (we specified that the applicable capacity for each of the 17 RPS Study scenarios would be built before the study period and prevented construction of any new capacity during the study);

  • Generator commitment (this module normally optimizes unit commitment, but we used a custom module described below to force it to follow the heuristic rules used in the RPS Study);

  • Generator dispatch (optimizes generator operation based on fuel prices and variable operation and maintenance costs);

  • Operating reserve balancing areas (we set up a pooled reserve balancing area for Oahu and Maui; more details below);

  • Fuel cost calculations (calculates fuel used by each plant, considering startup requirements, operating level, and full- and part-load efficiency); and

  • Transmission construction and operation (we pre-specified the available transmission capacity and allowed operation to be optimized, using a flowgate formulation; more details below).

We also used the following custom modules for this study:

  • Kalaeloa (shared with other Switch-Hawaii modules; enforces plant-specific unit commitment and production rules for the Kalaeloa combined-cycle power plant; more details are below) and

  • Reserves (implements heuristic unit commitment and reserve rules similar to the Hawaiian Electric rules, as used in MAPS).

For this study, we created 12 monthly models for each of the 17 scenarios, resulting in 204 production cost models to solve. These were solved in about 10 min via parallel processing on the University of Hawaii high-performance computing system. Total compute time for the 17 scenarios on a single four-core desktop computer would be about 4 h. The runtime for MAPS was reported to be under 30 min (Stenclik D (2017) Personal communication 8/22/2017).

Thermal generator properties

Operating costs

Details for modeling operation of utility-owned thermal power plants were taken from [59]. For each plant, these include retirement status, operating mode, type of fuel, heat rate (efficiency) curves, operation and maintenance costs, forced outage rates, minimum up- and downtime constraints, and energy required to startup plants. We modeled independent waste-to-energy plants (H-Power and Honua) as having a take-or-pay contract following a fixed schedule, as reported separately by GE (Stenclik D (2016) Personal communication 9/28/2016), and we modeled the independent AES and Kalaeloa plants using representative heat-rate (efficiency) curves as described in [25]. For AES, we used a variable operation and maintenance (O&M) rate of $2 per megawatt hour, as reported in (Coffman M, Bernstein P, Wee S (2014) Cost implications of GHG regulation in Hawai‘i. University of Hawaii Economic Research Organization (UHERO), Honolulu, Hawaii http://www.uhero.hawaii.edu/assets/WP_2014-5.pdf). For Kalaeloa, we set the variable O&M to $8.59/MWh, which resulted in the same full-load operating cost as reported in Fig. 30 of the RPS Study [59].

Fixed operating schedules

We configured Switch to follow the same commitment and dispatch schedules as GE described for their model runs [26, 59] (Stenclik D (2016) Personal communication 9/28/2016, Stenclik D (2017) Personal communication 3/14/2017, Stenclik D (2017) Personal communication 3/22/2017). As a general rule, plants identified in the RPS Study as baseload or firm renewable (“firm RE”) were committed (turned on) at all times that they were not out for maintenance. Cycling and peaking plants were committed as needed, based on the day-ahead renewable energy forecast. Commitment for peaking plants was further adjusted based on real-time conditions. Firm RE plants also had fixed dispatch (production) schedules (Stenclik D (2016) Personal communication 9/28/2016). A few plants followed commitment and dispatch schedules that did not fit this general pattern; they are discussed in the supplementary document.

Operating modes for combined-cycle plants

Kalaeloa power plant

The Kalaeloa combined-cycle power plant consists of two combustion turbines and one steam turbine powered by waste heat from the combustion turbines. GE modeled these as three units: Kalaeloa 1 and 2 each consisted of one combustion turbine and half of the steam turbine, and Kalaeloa 3 represented additional peaking capacity available if operating in dual-train combined-cycle mode [25, 57] (Stenclik D (2017) Personal communication 8/22/2017, Stenclik D (2016) Personal communication 9/28/2016). GE also reported that Kalaeloa 3 could operate in quick-start mode if Kalaeloa 1 and 2 were producing at their rated power level. We configured Switch to match this logic, i.e., only allowing Kalaeloa 3 to produce power if Kalaeloa 1 and 2 were at maximum output.

Maalaea combined-cycle plants

We modeled each of the Maalaea combined-cycle power plants as two single-train combined-cycle generators (a total of four units). Each of these plants consists of two combustion turbines and one steam turbine. Based on GE’s reporting, we believed that MAPS modeled these plants as four single-train units. However, GE reported properties for each of these plants on an aggregate basis in the RPS Study report [26]. So we split these properties in such a way that each pair of units would perform the same as the original aggregated plant, if both units were dispatched in tandem.

Minimum load and part-load heat rates for peaking plants

We modeled peaking plants with no minimum load (meaning they can operate anywhere between 0 and 100% of their rated load), and with a single incremental heat rate for all operating levels, following information reported separately by GE (Stenclik D (2016) Personal communication 9/28/2016).

Generator maintenance and forced outages

The RPS Study [59] showed Hawaiian Electric’s maintenance schedules, but MAPS used different schedules to avoid interfering with normal operation and reserve margins each week (Stenclik D (2017) Personal communication 8/22/2017). We inferred the dates of full maintenance outages for most thermal power plants in Oahu by inspection of hourly production data for Oahu plants in scenarios 2 and 16, which GE provided separately (Stenclik D (2018) Personal communication 2/28/2018). We assumed that baseload plants were on maintenance or forced outage on all days when they produced zero power. We assumed cycling plants were out of service when they produced no power while lower-priority peaking plants produced some power.

We were not able to identify forced outages for peaking plants or Maui plants by this technique. For these plants, we applied the utility’s maintenance schedules shown in the RPS Study [59] and then added random 3-day outages until each plant’s forced outage rate was 2.5% higher than the level shown in Table 7 of the RPS Study [59]. The 2.5% adder was used because we found that outage rates for the Oahu baseload and cycling plants in MAPS were an average of 2.5% higher than the sum of the maintenance schedules and forced outage rates shown in the RPS Study [59].

This technique was also unable to identify partial outages at power plants (e.g., times when they could only run at 35% or 50% of normal output). By inspection of the hourly production data (Stenclik D (2018) Personal communication 2/28/2018), we noted that there were a number of times when partial outages occurred; however we were not able to identify these systematically, so we omitted them from the Switch modeling. This is likely to introduce a bias toward baseload production rather than cycling or peaking production in all scenarios. It may also introduce a bias toward Oahu baseload over Maui baseload in the gen-tie scenarios (simply because there is more Oahu baseload capacity).

Fuel costs

We used fuel costs reported for Oahu and Maui in the RPS Study [59].

Transmission network

MAPS models transmission using an AC power flow, with DC variations with each commitment and dispatch decision [53] (Stenclik D (2017) Personal communication 8/22/2017). Switch is normally run with a flowgate-based transmission model, or it can be run with experimental security-constrained AC power flow. Since no network information is available publicly for Oahu and Maui, we ran Switch in flowgate mode, with no congestion or losses within each island, and finite transmission capacity between islands (in grid-tie scenarios). We assumed that all power flows over the grid-tie line incurred losses of 3.8%, based on an analysis of total production values reported for grid-tie and non-grid-tie scenarios in the RPS Study.

Hourly loads and renewable power production

We used hourly time series of renewable production potential, day-ahead forecasts, and electricity loads for Oahu and Maui which were provided by GE (Stenclik D (2016) Personal communication 5/9/2016). Gen-tie wind potential was increased by 5% before applying 5% transmission losses, to achieve consistency with values in the RPS Study.

Spinning reserve targets and allocation

Power systems must keep extra generating capacity committed (turned on) at all times in order to compensate for unforeseeable variations in operating conditions. These reserves can be divided into two main “product” categories: contingency reserves, which can compensate for rare events such as loss of a large generator or load, and regulating or operating reserves, which compensate for routine events such as mis-forecast of loads or renewable power. Reserves can also be divided into “up” and “down” directions (available to increase or decrease production). In Hawaiian power systems, all reserves are usually provided by “spinning” power plants (online and synced to the grid).

Up reserves

We configured Switch to use the same pre-calculated contingency and regulating reserve targets as MAPS (Stenclik D (2016) Personal communication 5/9/2016). In the grid-tie scenarios, we divided the regulating reserve target between the two power systems proportional to their hourly load levels; we were not able to identify how MAPS divided this target. We assumed the inter-island grid-tie cable could provide firm power transfers but could not directly provide spinning reserves.

It is not clear from the RPS Study report which power plants were designated to provide up reserves. Based on analysis of several sources [58, 59] (Stenclik D (2017) Personal communication 3/22/2017, Stenclik D (2017) Personal communication 6/13/2017), we allowed all baseload and cycling units to provide up reserves. Details of our inferences are given in Additional file 1. Switch, like MAPS (Stenclik D (2017) Personal communication 6/13/2017), was configured to optimize production levels for individual plants to minimize production cost while respecting the overall reserve target.

In the grid-tie scenarios, we configured Switch to divide the regulating reserve target between the Oahu and Maui power systems proportional to their hourly load levels. It is likely that GE used a different method to divide this target, but we were unable to find any documentation of this.

Down reserves

“Down” reserves are provided by power plants that are producing power above their minimum stable or permitted level and are able to reduce production on short notice. We set a contingency down reserve target of 10% of hourly load for Oahu, no regulating down reserve target for Oahu, and no down reserve target of either type for Maui, based on analysis of the RPS Study report [59].

Based on analysis of several sources [25, 26, 59] (Stenclik D (2017) Personal communication 6/13/2017), we divided the down reserve target between renewable and thermal generators based on their available capacity, and then among individual generators proportional to their committed capacity, as discussed in Additional file 1.

Generator unit commitment

“Unit commitment” refers to the process of selecting which power plants will be online during a particular time period. This is different from “dispatch,” which is the decision about how much power to produce from each committed plant. Switch and MAPS normally optimize unit commitment directly in order to provide enough capacity for energy and reserves, while respecting minimum up- and downtime limits for power plants. However, Hawaiian Electric instead uses a priority queue to specify the order in which thermal power plants will be committed. For the RPS Study, MAPS was configured to perform a linearized optimization of unit commitment, subject to this ordering, with additional heuristics to ensure integer constraints are satisfied (i.e., units must be fully committed or not at all) (Stenclik D (2017) Personal communication 8/22/2017, Stenclik D (2017) Personal communication 3/14/2017). MAPS used two rounds of unit commitment, one based on the day-ahead forecast, and one at real-time, using real-time conditions. All available plants were scheduled in the day-ahead unit commitment, but then peaking plants could be turned on or off as needed in real time (Stenclik D (2017) Personal communication 3/22/2017). For this study, we configured Switch with a custom commitment algorithm that followed this general approach. In Additional file 1, we report the assumptions we made about the order of plants in the commitment queues and the rules that were followed by this commitment algorithm.

Generator dispatch

For this study, Switch used its standard dispatch methods to choose how much power and reserves to produce from each committed power plant, in order to minimize cost while satisfying the balancing area’s requirements for power and reserves and respecting constraints on the operation of individual plants (e.g., down-reserve quotas).

Results and discussion

The commercial GE MAPS production cost model is widely used for renewable energy integration studies, including the recent Hawaii Renewable Portfolio Standards Study (RPS Study) [59]. The goal of this model intercomparison was to test whether similar results could be obtained from the open-source Switch model when studying the same high-renewable power systems. Switch is primarily designed as a capacity expansion model, which means that it selects which assets to build in order to minimize costs while meeting policy objectives. Embedded within this are unit-commitment and dispatch algorithms that decide which plants to turn on each hour and how much power to provide from those plants. MAPS is a production-cost model, which means that it focuses on unit-commitment and dispatch, using pre-specified portfolios of power system assets. Consequently, this model intercomparison focuses only on production-cost analysis. However, this encompasses some of the most important interactions between renewable power, thermal power plants, and electricity demand, and these elements underpin Switch’s findings when run in capacity expansion mode.

We compared the results from running Switch 2.0 to MAPS for 17 scenarios of interconnection and renewable resource adoption on the islands of Oahu and Maui previously studied with MAPS in the Hawaii Renewable Portfolio Standards Study (RPS Study) [59]. These had various amounts of wind and solar generating capacity and various combinations of inter-island transmission cables. Renewable resources ranged from 18 to 55% of total energy demand. New transmission options included a “grid-tie cable” to enable bidirectional sharing of power between the Oahu and Maui power systems and/or a “gen-tie” cable to carry power from wind farms on Lanai to the Oahu power system, without connecting to Lanai’s local power system. Scenario 1 in the RPS Study considered the current power systems without significant changes. We configured Switch to match scenarios 2–18, which represented future power systems. These are summarized in Table S1 in Additional file 1.

Although the comparisons reported below use quantitative metrics where possible, the comparison is fundamentally qualitative, intended to help researchers identify the areas where the models may differ and attention must be given to produce satisfactory results. We used this approach because statistical significance is not applicable in this context since there is no random variation in the results. It is clear that the models agree closely, but researchers must make their own qualitative judgments of whether they are similar enough to meet their needs.

Annual power production from each class of generator

Figures 1 and 2 show the annual power production from each major type of generator, in each of the 17 scenarios. Different energy sources are stacked in each column, and MAPS and Switch results are compared in pairs for each scenario. The agreement is generally within 0.5% of total power production for all categories except for baseload production in scenarios the grid-tie-only scenarios, which differ by 1–2%. There are several patterns in the differences between MAPS and Switch (see Fig. 2):

  • In scenarios with independent separate island power systems (scenarios 2–9, 10, 13, 16), Switch uses 23–72 GWh more of baseload generation on Oahu than MAPS and correspondingly less cycling and peaking generation. This equates to 0.3–0.8% of total production. In these scenarios, Switch uses slightly more Maui peaking generation and slightly less Maui baseload generation than MAPS (7–8 GWh, corresponding to 0.1% of total production).

  • In the scenarios with a grid-tie cable between the two islands but no gen-tie wind (scenarios 11, 14, 17), Switch uses 120–160 GWh more baseload generation on Oahu than MAPS (1.4–1.9% of total production). It also uses slightly more Maui peaking generation. These are matched by roughly equal decreases in baseload generation on Maui and cycling and peaking generation on Oahu.

  • The pattern in the scenarios with gen-tie wind and a grid-tie cable (scenarios 12, 15, 18) is similar to the grid-tie-only scenarios (more Oahu baseload and Maui peaking, less of other thermal plants), but less pronounced. In these scenarios, Switch also curtails more Oahu wind than MAPS and accepts more Maui wind and solar, with a net decrease in renewable production of 0–35 GWh, 0.0–0.4% of total production.

Fig. 1
figure 1

Annual production from each source in scenarios 2–9, as calculated by MAPS and Switch. Upper, blue-bordered rectangles show Oahu generators; lower, red-bordered rectangles show Maui generators

Full size image
Fig. 2
figure 2

Annual production from each source in scenarios 10–18, as calculated by MAPS and Switch. Upper, blue-bordered rectangles show Oahu generators; lower, red-bordered rectangles show Maui generators

Full size image

The coefficient of determination (R2 value) between results from MAPS and Switch across all energy sources and scenarios is 0.9999 for Oahu and 0.9963 for Maui. The R2 value is calculated by computing the square of the correlation coefficient between two vectors. In this case, the first vector was created from all the Oahu or Maui values in the MAPS (left) columns of Figs. 1 and 2, and the second vector was created from the corresponding values for Switch (right columns). These R2 values indicate that Switch and MAPS agree on more than 99.5% of the variation between energy sources and scenarios on each island.

Table 1 shows R2 values across all scenarios for each individual energy source. These values were calculated by comparing vectors which each showed the amount of production from one energy source on one island in all 17 scenarios. These indicate how well the Switch and MAPS agree on how production from each individual type of generator varied across the 17 scenarios. The R2 value is above 99% for all the renewable power sources and is 69–100% for the thermal power plants. The lower values for the thermal plants appear to reflect differences in prioritization of the various thermal plants relative to each other, as shown in Figs. 1 and 2.

Table 1 R2 value (squared correlation coefficient) between total energy production in MAPS and Switch, across scenarios 2–17, for each power source and island
Full size table

Annual curtailment in each scenario

Figure 3 compares the rate of curtailment of renewable power between Switch and MAPS modeling. Curtailment occurs at times when available wind and solar power exceeds system load net of generator minimum loads and down-reserve requirements. There is one colored marker for each scenario as modeled by Switch and one black ring for the equivalent scenario in MAPS. The x values for each marker show the amount of wind and solar power that is potentially available in that scenario, found by summing the hourly potential reported by GE in (Stenclik D (2016) Personal communication 5/9/2016). The y values show the percentage of renewable power that was left unused due to curtailment in each scenario. These calculations include wind, distributed solar, and utility-scale solar.

Fig. 3
figure 3

Curtailment rate calculated by Switch and MAPS in scenarios 2–18

Full size image

The comparison in Fig. 3 mostly follows from the results discussed in the previous subsection. MAPS and Switch produce very similar results overall, with a median difference in curtailment of 0.13 percentage points and differences of less than 0.3 percentage points for all but three scenarios (5, 15, and 18). Overall, the R2 value between the two models is 0.973, indicating that Switch and MAPS agree on about 97% of the variation in curtailment between scenarios.

The biggest difference is in scenario 5, where Switch has 1.0% curtailment vs. 3.1% for MAPS, driven by curtailment on Oahu, which is 1.0% for Switch and 3.4% for MAPS. We can only investigate this difference indirectly, since we do not have access to the hourly outputs from MAPS for this scenario. The installed resources on Oahu in scenario 5 are identical to scenario 10, except that scenarios 5 has 300 MW of wind on Oahu and scenario 10 has 100 MW on Oahu and 200 MW on Lanai, connected directly to the Oahu system with a gen-tie cable. Lanai has more wind than Oahu, so the amount of wind power available to the Oahu power system in scenario 5 is generally slightly lower than in scenario 10: Oahu wind in scenario 10 equals or exceeds scenario 5 during 59% of the year, and average wind during each hour of the day is higher in scenario 10 than in scenario 5 for all 24 h of the day, when averaged over all days of the year. Since scenario 5 usually has less renewable power available than scenario 10, we would expect scenario 5 to have less curtailment than scenario 10. This is what we found with Switch, but does not match the result from MAPS. However, we are not able to explain the difference further with the data available.

In scenarios 12, 15, and 18, Switch’s curtailment was 0.2–0.7 percentage points higher than MAPS. These are scenarios with both an inter-island grid-tie cable and Lanai wind connected directly to Oahu via a gen-tie cable. These scenarios have the largest amount of wind power of all the scenarios considered (see Additional file 1: Table S1). The total amount of wind power available between midnight and 5 am is 457 GWh in scenarios 12 and 15 and 521 GWh in scenario 18, compared to 277–372 GWh in scenarios 10, 11, 13, 14, 16, and 17. This high renewable availability corresponds with higher curtailment during these hours, totaling 87 GWh in scenario 12, 90 GWh in scenario 15, and 137 GWh in scenario 18, vs. 21–54 GWh in scenarios 10, 11, 13, 14, 16, and 17. These nighttime curtailments are the main difference distinguishing scenarios 12, 15, and 18 from the other scenarios, and these curtailments are about four times larger than the differences between MAPS and Switch shown in Fig. 3. So the differences in overall curtailment likely stem from different treatment of factors that affect low-load operation, such as calculation of down-reserve targets, commitment rules, and operating rules for Maui’s Maalaea combined-cycle plants. (As noted in the “Methods” section and Additional file 1, we made assumptions for Switch in these areas that may have differed from those used in MAPS.) However, we would need more data from the MAPS model to pin down the areas of disagreement more precisely than this.

Hourly system operation

Figure 4 shows 1 week of hourly operation of the Oahu power system in a low-renewable scenario (#2), as reported for MAPS (upper plot, reproduced from the RPS Study [59]) and Switch (lower plot). Similar to the annual results, Switch uses less cycling generation (purple) and more baseload generation (blue/teal shades) than MAPS on the Saturday and Sunday. Specifically, Switch commits only one cycling plant from 11 am to 3:59 pm on the Saturday and 8 am to 3:59 pm on the Sunday, while MAPS commits two cycling plants during these times. In both cases, the cycling plant(s) run at minimum load (22.5 MW each) and primarily provide reserves. Switch also decommits some Kalaeloa capacity earlier than MAPS on the last evening.

Fig. 4
figure 4

Hourly power production in scenario 2 during the week of June 22–28, calculated by MAPS and Switch (plot begins on a Monday)

Full size image

We could not identify a reason for the discrepancies in the hourly profiles from the two models. It is possible they are caused by different treatments of minimum up-/down-times for power plants (on the Saturday, one cycling unit exactly meets both of these limits), or MAPS may have been configured to optimize commitment beyond the minimum needed for load and reserves (Switch was configured to commit only the minimum required capacity).

We also note that MAPS slightly reduces output from Kalaeloa during the times of lowest power demand on Wednesday and Thursday nights, and this effect is slightly weaker in the Switch modeling. This suggests Switch may have used a lower minimum-load or down-reserve requirement for the Kahe and Waiau baseload units than MAPS.

Figure 5 compares hourly results for a high-renewable scenario (#16). Again, the match is close overall, but Switch used slightly more baseload generation and less cycling generation than MAPS. For example, Switch decommits Kalaeloa 2 and 3 at midday on Tuesday, late morning on Wednesday, and late evening on Sunday, while MAPS keeps them running. Switch also runs less cycling capacity (purple) than MAPS on Monday afternoon, Friday afternoon, and at 6 pm on Saturday.

Fig. 5
figure 5

Hourly power production in scenario 16 during the week of June 22–28, calculated by MAPS and Switch with standard model settings (plot begins on a Monday)

Full size image

Generalizability

Other regions and models

As noted in the “Background” section, this study focused on replicating the detailed findings from MAPS, as reported in the RPS Study. MAPS itself was calibrated based on actual operational practices on Oahu and Maui [57, 58]. Due to the unavailability of public data for MAPS in other regions or for other proprietary models in any region, we have not been able to perform a broader comparison of Switch against other models or in other regions. However, if Switch and other models are both calibrated to reflect local operational practices (as done here), we would expect similar results between them. Consequently, we recommend that users “ground-truth” power system models (whether open-source or proprietary) against local practices before conducting studies in any region. We also encourage modelers to publish their data and operating rules to support more intercomparisons in the future.

Larger networks

The RPS Study modeled only two zones (Oahu and Maui). As discussed in the “Transmission network” section, due to a lack of public network data, we ran Switch in flowgate mode for this study, with no losses or limits on flow within each zone, and finite, dispatchable transfer capability between zones. Flowgates are a fairly accurate representation of this particular power system—two regions with adequate internal transmission capacity joined by a controllable HVDC line—and the two models agreed fairly well about system operation. However, additional future work is needed to judge whether the models perform similarly for power systems with many network buses, transmission congestion, and security constraints. We have not yet identified a multi-node commercial study with enough public data to perform this assessment.

Conclusion

The commercial GE MAPS production cost model is widely used for renewable energy integration studies, including the recent Hawaii Renewable Portfolio Standards Study (RPS Study) [59]. The goal of this model intercomparison was to test whether similar results could be obtained from the open-source Switch model when operated in production-cost mode and configured to study the same high-renewable power systems.

We found that when Switch was configured similarly to MAPS, it produced results that were very close to MAPS, at least for the 17 scenarios evaluated here. Although the models agree closely, the agreement is not exact. The largest differences between the models are in the selection among different thermal power plants to provide power each hour. As noted in the “Methods” section and Additional file 1, there are a number of areas where Switch’s configuration may have differed from MAPS, possibly contributing to these differences. These areas include generator outages, calculation of down-reserve targets, allocation of reserve targets between islands, commitment order for Oahu and Maui power plants, commitment rules, treatment of the inter-island cable during unit commitment and dispatch, operating rules for Maui’s Maalaea combined-cycle plants, and variable cost of the Kalaeloa plant.

We are not able to judge which of these are the largest contributors, because the complete MAPS inputs and model are not available to the public. However, based on the qualitative similarity in findings between the models, we conclude that it is possible to configure Switch to obtain substantially similar results to MAPS, at least for the types of analysis reviewed here.

This study focused on replicating the detailed findings from MAPS, as reported in the RPS Study. Due to the unavailability of public data for MAPS in other regions or for other proprietary models in any region, we were not able to perform a broader comparison of Switch against other models or in other regions, but hope to do this in future work.

For this study, we ran Switch in flowgate mode, with no losses or limits on flow within each zone, and finite, dispatchable transfer capability between zones. Flowgates are a fairly accurate representation of this particular power system, and the two models agreed fairly well about system operation here. However, additional future work is needed to judge whether the models perform similarly for power systems with many network buses, transmission congestion, and security constraints.

As a general guideline, we recommend that researchers take care to ensure that their model assumptions match local practices and “ground-truth” their power system models (whether open-source or proprietary) against previous work or operational experience when conducting studies in a new region. We also encourage modelers to publish their data and operating rules to support more intercomparisons in the future.