1 Introduction

With the rapid development of the world economy, the depletion of oil resources increases year by year. It is now difficult to find additional large oil fields. Therefore, the demand for exploiting the limited oil reserves efficiently and economically becomes increasingly significant and has attracted more global attention in recent years. To achieve this goal, an important technique proposed is closed-loop reservoir management. It consists of two steps: automatic history matching and reservoir production optimization.

Automatic history matching is a sequential model updating method, where the estimate of uncertain reservoir properties is updated continuously according to the production measurements available at the time. Reservoir production optimization is a complete or partial automation process for maximizing the development effect within the lifecycle of a reservoir by optimizing operational parameters. The main idea is to exploit the oil reserves as near to the desired optimum as possible. Both automatic history matching and reservoir production optimization are optimization problems as mentioned by researchers such as Brouwer and Jansen (2004), Sarma et al. (2005) and Wang et al. (2009). These problems can be solved by optimization theories.

Automatic history matching has been studied since the 1960s (Jahns 1966; Wasserman and Emanuel 1976; Yang and Watson 1988), but it is still a very difficult problem at present. The existing history matching methods can be broken into two categories. One category is based on gradient, including finite difference approximation of derivative, adjoint gradient-based methods, and gradient simulator methods. The other category is based on gradient-free optimization, such as simultaneous perturbation stochastic approximation, genetic algorithm, particle swarm optimization, and pattern search methods (PSMs). Oliver and Chen (2011) have summarized the recent progress on automatic history matching.

The origin of solving reservoir production problems using optimization theories can be traced back to Lee and Aronofsky (1958). They used a linear programming method to maximize the net present value of production for a homogeneous reservoir. Later, some other papers appeared in journals such as Operation Research, Management Science, and Journal of Petroleum Technology. However, most papers published before the 1980s did not pay enough attention to optimization algorithms, and successful applications were very rare (Aronofsky and Williams 1962; Wattenbarger 1970; McFarland et al. 1984). With the advances in optimization algorithms and computing power, research has increased greatly since the 1980s (Sequeira et al. 2002; Chacón et al. 2004; Barragán et al. 2005; Gunnerud and Foss 2010; Knudsen and Foss 2013; Tavallali et al. 2013).

In order to find out under which operational parameters at current reservoir conditions the oil production might be most efficient and profitable, automatic history matching and reservoir production optimization should be combined together. This combination forms a concept of closed-loop reservoir management, where the geological model will be updated once the production measurements are available and the operational parameters will be optimized based on the newly updated reservoir model. Representative papers on this concept include Brouwer and Jansen (2004), Jansen et al. (2005), Nævdal et al. (2006), and Bieker et al. (2007).

This paper gives a general review of research on automatic history matching and reservoir production optimization; analyzes the characteristics and application conditions of gradient-based algorithms, gradient-free algorithms, and artificial intelligence algorithms; and finally discusses the emphases and directions of future research on both automatic history matching and reservoir production optimization.

2 Problem descriptions

2.1 Automatic history matching

The estimate of unknown geological properties using production measurements is recognized as history matching. It is an ill-posed inverse problem with many unknown reservoir parameters that could be adjusted to obtain a match against a relatively smaller number of measurements. Traditionally, the unknown parameters are adjusted manually by trial and error. This method is time-consuming and often yields a reservoir numerical model which may be unrealistic or not consistent with geological properties. To address these problems, automatic history matching has been studied for several decades. As shown in Fig. 1, automatic history matching is an iterative procedure where the unknown reservoir parameters are adjusted automatically with an optimizer to match the observed production or pressure data. In fact, automatic history matching is an optimization problem and the most commonly used objective can be written as follows:

$$O(\varvec{m}) = \sum\limits_{k = 1}^{K} {\left[ {(\varvec{g}_{k} - \varvec{d}_{k} )^{T} (\varvec{C}_{{\varvec{D}}} )_{k}^{ - 1} (\varvec{g}_{k} - \varvec{d}_{k} ) + (\varvec{m}_{k} - \hat{\varvec{m}}_{k} )^{T} (\varvec{C}_{{\varvec{M}}} )_{k}^{ - 1} (\varvec{m}_{k} - \hat{\varvec{m}}_{k} )} \right]},$$
(1)

where the subscript k is the discrete time step; K is the total number of time steps; m is the vector of reservoir parameters to be estimated; d is the vector of observed historical data; g is the vector of corresponding data to be matched; \(\hat{\varvec{m}}\) is the prior model information; C D is the covariance matrix of measurement errors; C M is the covariance matrix of prior probability density function. These matrices determine the weight of individual terms in the objective function.

Fig. 1
figure 1

Flow chart for automatic history matching

Although the least-square error has been used successfully to match the observed production data, it does not work well when the seismic data are history matched at the same time. To address this problem, Tillier et al. (2013) presented an appropriate objective function for history matching of seismic attributes based on image segmentation and a modified Hausdorff metric. The objective function of history matching commonly has a complex shape and multiple local minima (Oliver and Chen 2011). This is mainly because unknown parameters are always much more numerous than available production measurements.

In order to carry out history matching in a lower space and reduce the necessity of an explicit regularization term in the objective function, various parameterization methods have been presented, including the zonation method, pilot point method, subspace method, spectral decomposition, discrete cosine transform, truncated singular value decomposition, and the multiscale method. Jacquard (1965) and Jahns (1966) applied the zonation method to reduce variables in their automatic history matching study. The gradzone method in Bissell et al. (1994) and the adaptive multiscale method in Grimstad et al. (2003, 2004) were variations of the original zonation method. Marsily et al. (1984) presented the pilot point method, in which variables were estimated only at the pilot points and others were calculated by the Kriging method. Abacioglu et al. (2001) adopted a subspace method based on segmentation of the objective function. Oliver (1996a, b) applied the parameterization method based on spectral decomposition of the prior covariance matrix to history match a two-dimensional (2D) permeability field. Jafarpour and McLaughlin (2008), Jafarpour et al. (2010) used the discrete cosine transform as a parameterization method in their history matching study. Tavakoli and Reynolds (2010, 2011) provided a theoretical basis for parameterization based on truncated singular value decomposition of the dimensionless sensitivity matrix. He et al. (2013) applied the orthogonal decomposition method to transform the high-dimensional states into a low-dimensional subspace. They also described geological models in reduced terms by the Karhunen–Loève expansion of the log-transmissibility field.

The parameterization methods should be performed carefully, or results may sometimes mislead us. For example, Oliver et al. (2008) showed a small number of variables may underestimate the uncertainty in automatic history matching problem. They adjusted a spatially varying porosity with a uniform permeability or adjusted a spatially varying permeability with a uniform porosity to history match a same set of well-test data. Both of the two history matching results are good enough, but the assumptions on uncertainty are definitely different.

More and more attention has been paid to the generation and history matching of geologically realistic non-Gaussian reservoir models. Researchers have presented several methods including Gaussian mixture models, truncated pluri-Gaussian method, level set method, discrete cosine transform, and other principal component analysis methods. Dovera and Rossa (2011 ) proposed expressions of conditional means, covariances, and weights for Gaussian mixture models, so that the ensemble Kalman filter algorithm (EnKF) became usable in this case. Liu and Oliver (2005a, b) combined EnKF with a truncated pluri-Gaussian method for history matching of reservoir facies. Agbalaka and Oliver (2008) extended this method to a three-dimensional (3D) reservoir case. Chang et al. (2010) proposed a methodology to combine a level set method with EnKF for history matching of facies distribution in a 2D reservoir model. Hu et al. (2013) introduced a new method to update complex facies models generated by multipoint simulation while preserving their geological and statistical consistency. Jafarpour and McLaughlin (2008) tested a discrete cosine transform method on two 2D, two-phase reservoir models. Other principal component analysis methods can also be used to deal with non-Gaussian reservoir models. However, their computational cost may be high.

During the development of automatic history matching, various optimization algorithms have been introduced and modified (Bissell et al. 1994; Lee and Seinfeld 1987; Gomez et al. 2001). Streamline-based techniques were also used to improve computational efficiency of history matching by researchers such as Agarwal and Blunt (2003), Cheng et al. (2004, 2005), and Gupta and King (2007). With streamline-based techniques, a reservoir simulation model was automatically decoupled into a series of one-dimensional models along streamlines, which could minimize numerical dispersion and the effects of grid generation while maintaining a sharp displacement front. Caers (2003) and Negrete et al. (2008) combined streamline simulators with a deformation method and EnKF, respectively, to carry out automatic history matching. These streamline-based history matching techniques inherit the shortcomings of streamline methods such as inability to model very complex physics at the same time. Therefore, they are not suitable to all history matching cases.

In order to characterize the uncertainty of unknown geological properties, researchers introduced a Bayesian framework, with which one can formally construct a posterior density function. The books of Tarantola (2005) and Oliver et al. (2008) provide a detailed description about Bayesian framework. Generally, Bayesian estimation depends on a prior Gaussian model. To address this limitation, Sarma et al. (2008b) transformed this problem into feature space using kernel principal component analysis. Other important research works involved Bayesian framework include Chu et al. (1995), He et al. (1997), Zhang et al. (2002, 2005), Liu and Oliver (2003), Oliver et al. (2008), and Emerick and Reynolds (2012).

Although numerous papers on automatic history matching have been published, most of them have concentrated on a limited type of estimated parameters such as permeability and porosity. They are impractical at present for industrial field application in which we have to calibrate a myriad of other discrete and continuous parameters, including fluid contacts, rock compressibility, and relative permeability. However, automatic history matching is a meaningful research direction and much more effort will be necessary in the future.

2.2 Reservoir production optimization

Production optimization aims at achieving the best development performance for a given reservoir by optimizing well controls. Figure 2 shows the schematic for this optimization process. In order to evaluate the performance of different development programs, various objectives have been proposed during the long research into production optimization. For example, Rosenwald and Green (1974) minimized the difference between the production-demand curve and the flow curve actually attained. Babayev (1975) provided minimum total cost per unit output. Lasdon et al. (1986) maximized the deliverability of a gas reservoir at a specified time, minimized the total gas withdrawal shortfall between the demand schedule and the amount of gas that can actually be delivered in each month, and they also optimized the weighted combinations of the above two objectives. When optimizing production strategies, one often encounters multiple local maxima. This phenomenon may be a good thing sometimes because it means that there are extra degrees of freedom in the optimization problem, which can be used to accomplish other optimization objectives. For instance, Van Essen et al. (2011) incorporated short-term goals into the life-cycle optimization problem and proposed a hierarchical production optimization structure with multiple objectives. Chen et al. (2012) also optimized both long-term and short-term net present value. As more and more oilfields enter the high water cut period, the production costs increase gradually. Therefore, the net present value is commonly selected as the objective function for production optimization. In terms of water flooding projects, it is defined as

Fig. 2
figure 2

Flow chart for reservoir production optimization

$$J(\varvec{u}) = \sum\limits_{n = 1}^{L} {\left[ {\sum\limits_{j = 1}^{{N_{\text{P}} }} {\left( {r_{\text{o}} q_{{{\text{o}},j}}^{n} - r_{\text{w}} q_{{{\text{w}},j}}^{n} } \right)} - \sum\limits_{i = 1}^{{N_{\text{I}} }} {r_{\text{wi}} q_{{{\text{wi}},i}}^{n} } } \right]} \frac{{\Delta t^{n} }}{{(1 + i_{\text{c}} )^{{t^{n} }} }},$$
(2)

where J is the net present value and is a function of the control vector u; L is the total number of simulation time steps; N P is the total number of producers; N I is the total number of injectors; r o is the oil price; r w is the produced water treatment cost; r wi is the water injection cost; \(q_{{{\text{o}},j}}^{n}\) and \(q_{{{\text{w}},j}}^{n}\) are the average oil and water production rates of the jth producer during the nth simulation time step, respectively; \(q_{{{\text{wi}},i}}^{n}\) is the average water injection rate of the ith water injection well during the nth simulation time step; i c is the annual discount rate; ∆t n is the length of the nth simulation time step; t n is the cumulative time up to the nth simulation time step in years.

According to the general form of optimal control problems, a mathematical model for reservoir production optimization can be written as

$$ {\text{max}} \,\,\,J(\varvec{u}) = \sum\limits_{n = 1}^{L} {\left[ {\sum\limits_{j = 1}^{{N_{\text{P}} }} {\left( {r_{\text{o}} q_{{{\text{o}},j}}^{n} - r_{\text{w}} q_{{{\text{w}},j}}^{n} } \right)} - \sum\limits_{i = 1}^{{N_{\text{I}} }} {r_{\text{wi}} q_{{{\text{wi}},i}}^{n} } } \right]} \frac{{\Delta t^{n} }}{{(1 + i_{\text{c}})^{{t^{n} }} }}$$
(3)
$$\varvec{Au} \le \varvec{b}$$
(4)
$$\varvec{u}^{\text{low}} \le \varvec{u} \le \varvec{u}^{\text{up}},$$
(5)

where Eq. (4) represents the linear or nonlinear constraints; Eq. (5) gives the boundary constraints.

Asheim (1988) maximized the net present value for waterflooding with multiple vertical injectors and a vertical producer by optimizing rate allocation based on the product of permeability and thickness. Brouwer and Jansen (2004) studied static and dynamic waterflooding optimization. For the static one, they kept inflow control valves constant during the displacement process until water breakthrough. For the dynamic one, they applied gradients calculated with an adjoint method to dynamically optimize the production performance and considered a simple constraint where the total injection was equal to the total production. In order to increase the displacement efficiency, Sudaryanto and Yortsos (2000) optimized the front shape of injected fluid by controlling injection rates. Results showed that the waterflooding optimization was a “bang–bang” control problem, where each control variable took either its minimum or maximum allowed values. Zandvliet et al. (2007) further investigated why and under what conditions waterflooding problems had optimal solutions under bang–bang control. They concluded that waterflooding optimization with simple boundary constraints sometimes had bang–bang optimal solutions, while problems with other general inequality or equality constraints would have a smooth optimal solution. Gao and Reynolds (2006) proposed a log-transformation method to deal with boundary constraints. Alhuthali et al. (2007) also achieved optimal waterflooding management using rate control.

Many pilot tests and commercial projects using enhanced oil recovery (EOR) methods have been performed during the past few decades to improve the development effect of waterflooding. The challenges of huge investment, high cost, and high risk promote research into production optimization for EOR methods. As early as in 1972, Gottfried (1972) proposed a nonlinear programming model for a cyclic steam injection process. He maximized the net present value by optimizing steam injection volume and cycle length. Ramirez et al. (1984) and Fathi and Ramirez (1984) tried to maximize oil production at the minimum injection costs based on the calculus of variations and Pontryagin’s weak minimum principle. They optimized development strategies for waterflooding, carbon dioxide (CO2) flooding, and surfactant flooding. Amit (1986) formulated a two-phase dynamic optimization model which incorporated the relationships between extraction rates, investment decisions, and cumulative oil recovery. Wackowski et al. (1992) applied rigorous decision analysis methodology to find the optimal development strategy for a 20 years CO2 flooding project. The control variables included CO2 recycle capacity, CO2 purchase contract, processing rate, water-to-gas ratio, and slug size. Wu (1996) found that chemical flooding performance was sensitive to operational parameters such as chemical slug size, concentration and adsorption, price of oil and chemicals, annual discount rate, and reservoir permeability. Results showed that the optimal design was a large slug injection of low concentration surfactant and polymer, followed by a small slug of subsequent polymer injection. One of the best examples about gas lift optimization has been achieved by McKie et al. (2001), in which gas lift injection rates, compressor settings, and field fuel consumption for more than 500 wells were optimized to maximize liquid production. Codas et al. (2012) integrate simplified well deliverability models, vertical lift performance relations, and flowing pressure behavior of the surface gathering system to develop a framework of integrated production optimization for complex oil fields.

The above review of production optimization showed that the exploitation method involved has extended from waterflooding to EOR methods, the control variables optimized have extended from simple to complex operational parameters, and that the optimization algorithms applied have extended from gradient-based to gradient-free methods. In addition, production optimization has improved from an open adjustment process to a closed-loop management workflow (Mochizuki et al. 2006), which will be discussed in detail in the next section.

2.3 Closed-loop reservoir management

As more and more production and pressure measurements become available, the reservoir model can be updated to achieve a better estimate of the unknown geological properties. Then the operational parameters should be optimized again based on the newly updated reservoir model. This cycle of model update and production optimization is repeated during the whole process of the reservoir development. This forms the concept of closed-loop reservoir management. The schematic of this closed-loop process is shown in Fig. 3.

Fig. 3
figure 3

Schematic of closed-loop reservoir management

Nævdal et al. (2006) applied the EnKF method to update the reservoir simulation model and then optimized operational parameters with an adjoint formulation in order to maximize the economic profits. Sarma et al. (2005) presented a closed-loop management approach for efficient real-time production optimization. Their approach consisted of three key elements: adjoint models for gradient calculations, polynomial chaos expansions for uncertainty propagation, Karhunen–Loeve expansions and Bayesian inversion theory for history matching. Results showed that their approach increased the net present value by 25 %. For a similar problem, Saputelli et al. (2005) proposed a model predictive control method. It was a class of computer control algorithms that explicitly used a plant model for online prediction of future behavior, and computation of appropriate control action subjected to various constraints through online optimization of a cost objective. The method addressed the overwhelming complexity of overall optimization problems by suggesting an oilfield operations hierarchy which entailed different time scales. Due to the rapid development of ensemble-based optimization algorithms, Chen et al. (2009a, b) provided a new closed-loop reservoir management method which integrated an ensemble-based optimization method with the EnKF algorithm. Jansen et al. (2008) and Jansen (2011) discussed an emerging technique to increase oil recovery. Their technique is an operational use of model-based optimization which requires a combination of long-term and short-term objectives through multi-level optimization strategies. Moridis et al. (2013) established a self-teaching expert system to increase oil production by improving flooding efficiency and reducing geological uncertainty.

When reviewing closed-loop reservoir management, it is necessary to present the Brugge test case which was prepared for SPE (Society of Petroleum Engineers) Applied Technology Workshop held in Brugge in June 2008. In the test case, well-log data, reservoir structure, 10 years’ production data, inverted time-lapse seismic data, and other information necessary are given by TNO (Netherlands Organisation for Applied Science Research) to estimate unknown parameters such as permeability, porosity, and net-to-gross thickness. After history-matched reservoir models were created, water flooding strategies for 20 producers and 10 injectors were optimized. Peters et al. (2010) have summarized in detail the results of the Brugge test case obtained by nine research groups. Briefly, Table 1 compares the reservoir simulators, optimization methods, and the net present values optimized in Year 10. In the table, the net present value was obtained using the optimal strategy of each participant in the Brugge test model. As can be seen, three participants who applied ensemble-based methods in history matching step achieved a similar highest net present value, although their optimization algorithms used in the production optimization step were different. This is mainly because ensemble-based optimization methods had two distinct advantages. First, the search direction was approximated through the correlations provided by ensemble members. Second, the objective function was the expectation of each ensemble member. Therefore, the ensemble-based optimization method was fairly robust with respect to the uncertainty of the estimated geological models.

Table 1 Research results of the Brugge test case

With the rapid increase in research interests and great improvements of optimization techniques, the original Brugge test case shows its weakness in the low frequency of the feedback loop. In order to keep the Brugge test case as a challenging problem for testing and comparing different techniques in closed-loop reservoir management, Peters et al. (2013) provided a lot of additional data including well constraints and production history from individual well completions for another 20 years, as well as the updated data of oil saturation and reservoir pressure.

3 Optimization methods

According to the techniques for determining search direction and step size, the optimization methods can be classified into three categories: gradient-based algorithms, gradient-free algorithms, and artificial intelligence algorithms. In fact, artificial intelligence algorithms are also independent of gradient information. This paper considers them as an independent category because almost all of them are inspired by intelligent behaviors in nature such as inference, designing, thinking, and learning.

3.1 Gradient-based algorithms

The calculation of derivatives or the Hessian matrix is the key to solving optimization problems using gradient-based algorithms. In terms of production optimization and history matching, researchers have applied various calculation methods including numerical perturbation, sensitivity equation, and adjoint method, as summarized in Table 2.

Table 2 Comparison of calculation methods for gradients and Hessian matrix

The roots of the adjoint method can be found in Jacquard (1965). Later, Carter et al. (1974) formulated their work in a better way using Frechet derivatives. He et al. (1997) further extended Carter’s work to three dimensions approximately. For single-phase flow problems, Chen et al. (1974) and Chavent et al. (1975) proposed a method which was regarded as what we call the adjoint method now. Li et al. (2003) presented the first formulation of the adjoint method for three-phase flow problems and pointed out that the coefficient matrix of the adjoint equations is simply the transpose of the Newton–Raphson Jacobian matrix used in a fully implicit reservoir simulator. Therefore, the derivation of the individual adjoint equation can be avoided by extracting and saving the Jacobian matrices. Rodrigues (2006) derived the adjoint equations in a much neater way, and provided a method for multiplication of a vector by the sensitivity matrix or its transpose. Now, the adjoint method has become one of the most efficient methods existing today to compute gradients for gradient-based algorithms.

Generally, gradient-based algorithms can be classified into two categories. One is the first-order methods, which only require the derivative information. For example, the steepest ascent algorithm and the conjugate gradient method have been widely used by many researchers such as Brouwer and Jansen (2004), Sarma et al. (2008a), and Wang et al. (2009). The other category is the second-order methods, which not only require the derivative information but also require the Hessian matrix. Representative methods include Gauss–Newton, Levenberg–Marquardt, sequential quadratic programming (Barnes et al. 2007), and the limited memory Broyden Fletcher Goldfarb Shanno method (LBFGS).

When using the Gauss–Newton method for automatic history matching problems, Wu et al. (1999) introduced an artificially high variance of measurement errors at early iterations to damp the changes in model parameters and thus avoid undershooting or overshooting. Tan and Kalogerakis (1991) pointed out that the standard Gauss–Newton and Levenberg–Marquardt methods require the computation of all sensitivity coefficients in order to formulate the Hessian matrix, which seems impossible in reality due to the large number of unknown parameters relative to limited available measurements. In order to eliminate this problem, the quasi-Newton method was introduced by researchers. This method only requires the gradient of the objective function which can be computed from a single adjoint solution as done in Zhang et al. (2002). In order to further improve the computational efficiency and robustness of the LBFGS method, Gao and Reynolds (2006) proposed a new line search strategy, rescaled the model parameters, and applied damping factors to the production data. They also noticed that the new line search strategy had to satisfy the strong Wolfe conditions at each iteration, or the convergence rate would decrease significantly.

The Karhunen–Loeve expansion can create a differentiable parameterization of the numerical model in terms of a small set of independent random variables and deterministic eigenfunctions. With this expansion, the gradient-based algorithms can be applied while honoring the two-point statistics of the geological models (Gavalas et al. 1976). In order to further extend the existing gradient-based history matching techniques to deal with complex geological models characterized by multiple-point geostatistics, Sarma et al. (2007) applied a kernel principal component analysis method to model permeability fields. This method can preserve arbitrary high-order statistics of random fields, and it is able to reproduce complex geology while retaining reasonable computational requirements.

Gradient-based algorithms are widely used in research on production optimization and history matching because of their high computational efficiency and fast convergence behavior. However, these algorithms require detailed knowledge of the numerical simulators. They can hardly be used without adjoint code and they are difficult to transform from one simulator to another.

3.2 Gradient-free algorithms

In order to make full use of the advantages of commercial reservoir simulators when conducting production optimization and automatic history matching, researchers have introduced many gradient-free algorithms including simultaneous perturbation stochastic approximation algorithm (SPSA), EnKF, PSM, new unconstrained optimization algorithm (NEWUOA), and quadratic interpolation model-based algorithm guided by approximated gradient (QIM-AG).

Spall (1998) proposed the SPSA method based on the Kiefer–Wolfowitz algorithm. This new method perturbs all unknown parameters stochastically and simultaneously to generate a search direction at each iteration. The expectation of stochastic SPSA gradients is true gradient and it is always a downhill direction. Therefore, the SPSA method is a stochastic version of the steepest descent algorithm. Based on the simultaneous perturbation idea, Spall further provided a second-order SPSA method, which estimates the Hessian matrix at each iteration. Later Bangerth et al. (2006) described an integer SPSA method and used this modified SPSA method to solve well placement optimization problems. To the best of our knowledge, this is the first time that SPSA has been used in optimal control problems. In terms of closed-loop reservoir management, Wang et al. (2009) discussed the application of SPSA in the step of production optimization. Gao et al. (2007) applied a modified SPSA method for automatic history matching. In their study, the approximate Hessian matrix was calculated using the inverse of the covariance matrix of the prior model. Results showed that the modified SPSA performed almost as well as the steepest descent method. Of course, gradient-based algorithms like LBFGS would be preferred when the gradient can be calculated. Otherwise, the SPSA method may be a good choice. Based on SPSA, Li and reynolds (2011) proposed a stochastic Gaussian search discretion algorithm for history matching problems, and this modified method was successfully used in the well-known PUNQ-S3 (Production forecasting with uncertainty quantification) test case. Zhou et al. (2013) integrate the finite difference method and the SPSA method to optimize polymer flooding in a heterogeneous reservoir. But currently the SPSA method has not been widely used for reservoir optimization problems.

The EnKF algorithm was first proposed by Evensen in 1994 as a Monte Carlo approximation of the Kalman filter in the ocean dynamics literature. This method obtains gradient information through correlations of ensemble members. Nævdal et al. (2002) applied EnKF to estimate near-well permeabilities. Gu and Oliver (2005) examined EnKF for combined parameter and state estimation in a standardized reservoir test case. Gao et al. (2006) found that EnKF and the randomized maximum likelihood method (RML) gave similar computational results. Reynolds et al. (2006) further presented their mathematical connections. They also showed that EnKF may be viewed as updating each ensemble member with a single Gauss–Newton iteration. Liu and Oliver (2004, 2005a, b) investigated a highly nonlinear problem of facies estimation using the EnKF method. When using the EnKF method, Lorentzen et al. (2005) discussed the choice of initial ensemble members, while Wen and Chen (2006) focused on the effect of ensemble size. Emerick and Reynolds (2012, 2013) incorporated automatic history matching in an integrated geo-modeling workflow using the ensemble smoother method. Although it is not long since the introduction of EnKF into petroleum engineering, this method is developing rapidly and shows huge potential for solving history matching problems because it can deal with the uncertainties.

However, the toy problem in Zafari and Reynolds (2007) showed that the standard EnKF cannot handle multimodal nonlinear problems. To address this limitation, Gu and Oliver (2007) applied an iterative EnKF to assimilate multiphase flow measurements. Li and Reynolds (2009) provided two iterative EnKF procedures, with which they obtained better history matching results than that with the standard EnKF method in two examples including the toy problem. Lorentzen and Nævdal (2011) and Wang and Li (2011) also introduced an iterative extension of EnKF in order to improve estimate results in cases where the relationship between the model and the observations is nonlinear. Agbalaka et al. (2013) proposed a two-stage ensemble-based technique to improve the performance of EnKF for history matching with multiple modes. Chen et al. (2009a, b) addressed non-Gaussian effects through a change in parameterization. They demonstrated the effectiveness of the combination of their methods with the traditional EnKF by history matching of multiphase flow in a heterogeneous reservoir. Supposing that both of the multimodal priors and posteriors can be approximated by Gaussian mixture models, Dovera and Rossa (2011) presented a modified EnKF method especially for multimodal systems. Heidari et al. (2013) combined EnKF with pilot points and gradual deformation to preserve second-order statistical properties, so the departure of constrained petrophysical properties from prior information could be greatly reduced.

Custsódio and Vicente (2007) provided a modified PSM guided by simplex derivatives (SID-PSM). This new method has two modifications over the original PSM algorithm. First, the predefined search directions are ranked using the simplex gradient during the poll step and the search directions closest to the simplex gradient are tried first. Second, when there are enough objective evaluations, a quadratic interpolation model is built in the search step and then it is minimized by a trust-region method. The SID-PSM algorithm has been introduced into reservoir engineering for optimizing the settlement and adsorption parameters of asphaltene in porous media. This method can converge to a global optimum using only the objective evaluations. However, its global convergence is very sensitive to the choice of initial values. Therefore, close attention should be paid when using SID-PSM for practical problems.

For unconstrained optimization problems without derivatives or a Hessian matrix available, Powell (2008) proposed a NEWUOA method. It is a quadratic model-based gradient-free trust-region algorithm based on quadratic interpolation. At least N u  + 2 (N u is the number of control variables) objective evaluations are needed to build the initial quadratic model before the first optimization procedure can be achieved. So it can hardly be used when the number of control variables is very large. In order to improve the computational efficiency of NEWUOA, Zhao et al. (2011) developed a QIM-AG method. This method requires a minimum of only one interpolation point to build a quadratic model at each iteration. It is similar to the quasi-Newton method and converges very fast.

Taking the net present value as objective function, Zhao et al. (2011) compared the performance of various gradient-free algorithms. As shown in Fig. 4, the quadratic interpolation model-based algorithm guided by ensemble-based gradient (QIM-EnOpt) obtained the highest net present value. The quadratic interpolation model-based algorithm guided by SPSA gradient (QIM-SPSA) obtained a similar net present value but converges much faster than QIM-EnOpt and EnOpt. The SPSA and NEWUOA methods achieved similar values, but the computational efficiency of NEWUOA was much lower at the beginning of the optimization process. In this test case, the particle swarm optimization method and SID-PSM algorithm performed much worse than the other algorithms.

Fig. 4
figure 4

Comparison of various gradient-free algorithms by Zhao et al. (2011)

3.3 Artificial intelligence algorithms

Artificial intelligence algorithms have been used to solve production optimization and history matching problems for a long time. The representative methods include the simulated annealing algorithm, genetic algorithm, artificial neural networks, particle swarm optimization method, and the tabu search method.

The simulated annealing algorithm was first introduced into reservoir engineering. Farmer (1992) applied this algorithm to generate rock models with two-point geostatistics properties. Qian (1993) introduced Markov random field theory into the basic simulated annealing algorithm and thus transformed it into a probabilistic uphill algorithm. When solving automatic history matching problems, Ouenes et al. (1993) applied a simulated annealing algorithm directly while Carter and Romero (2002) combined it with other techniques such as geostatistics, a pilot point method, and a genetic algorithm. The convergence of the simulated annealing algorithm is sensitive to the choice of initial temperature and reduction factor. If the reduction factor is too large, many extreme points will be missed. However, if the reduction factor is too small, the simulated annealing algorithm will converge very slowly.

The genetic algorithm was proposed by Holland (1975) based on the idea of population evolution. This method has been used as a global optimizer for automatic history matching problems by Tokuda et al. (2004). Compared with a simulated annealing algorithm, a genetic algorithm starts with a population of realizations rather than a single realization. However, there is no mathematical guarantee for a genetic algorithm to reach a global optimum. In spite of these drawbacks, genetic algorithms still have much appeal because they can be easily parallelized and connected to any existing reservoir simulators.

Artificial neutral networks are a rough approximation and simplified simulation of a biological neuron network system. Generally, the neutral networks are used to establish proxy models. For this purpose, a small number of simulations are run at first to build a nonlinear relationship between the objective and unknown variables. Then the relationship can be used for reservoir production optimization in order to reduce the computational cost (Elkamel 1998; Saputelli et al. 2002). Artificial neutral networks were also used directly as an optimizer by Ramgulam et al. (2007) in their studies of history matching. Although artificial neutral networks have been widely used, the procedures for network construction are still not established.

The particle swarm optimization method was proposed by Kennedy and Eberhart based on social behavior observed in nature. It has been successfully used in well placement optimization (Onwunalu and Durlofsky 2009), automatic history matching, and water flooding optimization. This method can find the global optimum with a high probability and can be connected to any numerical simulators.

In fact, artificial intelligence algorithms are stochastic gradient-free methods. These algorithms need much more numerical simulations than the deterministic methods to converge. In addition, the input parameters for implementing gradient-free algorithms are commonly determined by the trial and error method. Therefore, the gradient-free algorithms should be modified and integrated with other optimizers in practical applications.

Based on the discussions in above three sections, the main characteristics for different kinds of optimization algorithms are summarized in Table 3.

Table 3 Characteristics of different optimization algorithms

4 Future developments

In order to solve the growing demand–supply gap, many studies have been carried out in terms of closed-loop reservoir management. However, most of the studies are focused on academic research and they are far from large-scale field application. As for field application, oilfield engineers are mainly concerned about three issues: processing time, decision veracity, and operating convenience. In order to reduce the processing time, efficient optimization methods and parallel distributed systems should be paid more attention. It is important for increasing decision veracity to carry out more uncertainty analyses. Since most of the oilfield engineers are not familiar with the theoretical bases of closed-loop reservoir management, integrated software is necessary to improve their operating convenience.

4.1 Hybrid solvers

Various optimization methods have been introduced into reservoir development engineering during the past decades. Each algorithm has its own advantages as well as drawbacks. For example, the gradient-based algorithms have high computational efficiency and fast convergence rate, but the calculation of gradients requires detailed knowledge of the numerical simulators. They can hardly be used without adjoint code and they are difficult to transform from one simulator to another. The gradient-free algorithms can converge to global optimum with a high probability and they can make full use of the technical advantages of commercial reservoir simulators. However, the computational efficiency and convergence rate of gradient-free algorithms are not satisfactory. Therefore, we should combine different algorithms together to develop new hybrid solvers, which can incorporate the advantages of different optimization methods.

4.2 Parallel algorithms and distributed systems

As well known, numerical simulations are necessary in closed-loop reservoir management. However, for large-scale reservoir optimization problems, even a single numerical simulation requires several days and the total computational cost is unbearably expensive. To address this problem, it is necessary to develop parallel algorithms and distributed systems. The parallel algorithms should necessarily conserve the mathematical convergence behavior of the original algorithms. Traditionally, the distributed systems can be classified into two categories. The first category is homogeneous distributed system which is a cluster of many similar computers or work stations. The other category is heterogeneous distributed system which is constructed by a set of work stations or computers with different architectures. Currently, with the rapid development of multicore computers and graphic processing unit (GPU) technology, many parallel programming methods have been proposed on the basis of Open Multiprocessing (OpenMp) and compute unified device architecture (CUDA). For example, Wu et al. (2014) discussed a multilevel preconditioner in a new-generation reservoir simulator and its implementation on multicore computers using OpenMP. In order to conduct large-scale closed-loop reservoir management, parallel algorithms and distributed systems should be emphasized in the future studies.

4.3 Uncertainty analyses

It is well known that automatic history matching is an underdetermined inverse problem because the number of unknown parameters is much larger than the number of available production measurements. In other words, many different parameter estimates can provide satisfactory data fits. However, some of the estimates may grossly lead to erroneous predictions of future production behavior. Therefore, developing methods that can quantify the uncertainty of parameter estimates are necessary to the success of automatic history matching. For example, Barker et al. (2001) applied the Markov chain Monte Carlo method to quantify the uncertainty. When predicting permeability from well logs, Olatunji et al. (2011) adopted a type-2 fuzzy logic system which is good at handling uncertainties in measurements and data used to calibrate the parameters. Arnold et al. (2013) carried out a hierarchical benchmark case study for history matching, uncertainty quantification, and reservoir characterization. Since automatic history matching has become increasingly important in the field of reservoir description, more efforts should be paid to uncertainty analyses to reduce the decision risks in the future.

4.4 Integrated software

In order to exploit the limited oil reserves more efficiently and economically, the field application of closed-loop reservoir management becomes increasingly significant. However, most of the oilfield engineers do not know much about its theoretical basis. Therefore, it is necessary to develop integrated software which can be used conveniently as a black box. As far as we know, some commercial simulators have done much work in this field. For example, the Computer Modelling Group Ltd has developed a CMOST studio, which integrated history matching, optimization, sensitivity analysis, and uncertainty assessment tool. Some optimization methods such as the Latin hypercube design, brute force search, random search, and PSO have been available for history matching and optimization tasks in CMOST studio. The Schlumberger developed MEPO software to deal with production optimization, uncertainty assessment, and semi-automatic history matching. Although CMOST studio and MEPO software did much work, automatic closed-loop reservoir management has not yet been achieved, especially for large-scale field applications. Therefore, much more effort should be paid on the development of integrated software, which can help the engineers to manage oilfields easily and scientifically.

5 Conclusions

Closed-loop reservoir management consisting of automatic history matching and reservoir production optimization is an effective technique to improve the technical and economic effect of reservoir development. Both of the steps are complex optimization problems which can be solved by gradient-based algorithms, gradient-free algorithms, and artificial intelligence algorithms. The computational efficiency of gradient-based algorithms is the highest but they are difficult to transform from one simulator to another. The gradient-free and artificial intelligence algorithms are independent of numerical simulators but they need much more simulations to find the optimum. More research should be conducted on hybrid optimization methods, parallel algorithms and distributed systems, uncertainty analyses and integrated software in the future.