1 Introduction

It is well known that uncertainty and risk are inherent in geotechnical and geological engineering and that the quantification of these uncertainties is rational [1,2,3,4,5,6,7]. Traditional deterministic design methods accommodate uncertainties through empirical safety factors. However, these safety factors do not quantitatively measure the safety margin of design and do not account for the influence of different design variables and their uncertainties on overall system performance.

Reliability is the probability of an event occurring or the probability of a positive outcome and can be interpreted as a probability of failure. Calculating reliability gives a means of assessing the degree of uncertainty involved in geotechnical engineering calculations [6].

From the reliability theory, the chance of unsatisfactory performance can be expressed as \(P_{f} = 1 - \Phi \left( \beta \right)\) where Pf is failure probability, β is the reliability index, and \({\Phi}\) is cumulative distribution function (CDF) of a standard normal variable. Phoon and Ching [6] define the reliability index, β, as the number of standard deviations between the most likely value of factor of safety and the critical factor of safety of 1.0. By setting target the reliability index, the chance of unsatisfactory performance can be reduced. For geotechnical systems, the target reliability index is often in the range of 2.0–4.0 [8].

Reliability-based optimization (RBO) is a methodology that determines the best design solution according to certain predefined design criteria while explicitly considering the effects of uncertainty [9]. Uncertainties in geotechnical engineering come from loads, geotechnical properties and calculation models [10, 11]. RBO constitutes a powerful methodology for solving geotechnical-related design problems because it results in a geotechnical solution optimized according to predefined criteria (e.g. construction costs, benefits, etc.) while accounting for unavoidable uncertainties and satisfying geotechnical design requirements.

RBO is emerging as a design based on safety-under-uncertainty. A number of studies have been carried out for the Reliability-Based Optimization of geotechnical systems in recent past. Valdebenito and Schuëller [9] carried a detailed literature review on reliability-based optimization. Wang and Kulhawy [12] and Wang [13] used partial factors derived from reliability theory to optimize the design of shallow foundations. The partial factors needed calibration for their method. The inverse reliability approach has been used to demonstrate the optimum design of cantilever sheet pile walls [14] and anchored sheet pile walls [15]. Ching and Hsu [16] proposed a methodology to transform reliability limit-state constraints into deterministic limit-state constraints. Ching [17] presented an equivalence between reliability and factor of safety. Zhang et al. [8] presented an indirect method on the basis of the Mean First-Order Reliability Method (MFORM) to optimize the geotechnical system. Langford and Diederchs [18] developed a reliability-based design of a composite tunnel lining by combining a modified Rosenbluth Point Estimate Method (PEM), First Order Reliability Method (FORM), Monte Carlo Sampling method, and finite element analysis. Gong et al. [19] proposed and applied a fuzzy set-based Robust Geotechnical Design (RGD) method for the design of shield-driven tunnels. Zhao et al. [20] employed an Artificial Bee Colony (ABC) algorithm for reliability-based optimization retaining walls and spread footings. Zevgolis et al. [21] proposed a probabilistic geotechnical design optimization framework for large open pit excavations. Zhao et al. [22] employed least square support vector machine (LSSVM) and artificial bee colony (ABC) algorithm for reliability-based support optimization of rockbolt reinforcement around tunnels. Santos et al. [23] used FORM based ant colony optimization (ACO) algorithm for reliability-based design optimization of geosynthetic-reinforced soil walls. Ji et al. [24] used inverse FORM approach for reliability-based design in geotechnical engineering. Raviteja and Basha [25] presented a target reliability-based design optimization (TRBDO) approach of V-shaped anchor trenches for municipal solid waste (MSW) landfills.

Although considerable research work has been reported in the field of RBO of geotechnical systems, the application is still limited. One limitation has been the difficulty of direct coupling between reliability assessment and cost minimization. This paper suggests a FORM-based direct method for RBO of geotechnical systems. In the proposed method, RBO is implemented by coupling reliability assessment and cost minimization. RBO is proposed as a double-loop (or nested-loop) constrained optimization problem, with cost optimization of the geotechnical system through the outer loop repeatedly calling the inner reliability evaluation loop. In the outer loop, the objective is the minimization of cost function while the constraint is a target reliability index; whereas, in the inner loop, the objective is the minimization of reliability index while constraints are geotechnical design requirements. The proposed method is used to illustrate the optimal design of retaining wall. MATLAB’s constraint optimization function, fmincon [26], is used to implement this algorithm. The interior-point algorithm is used in the outer loop to search the design variables in global search space. The interior-point algorithm is used in the inner loop to calculate the reliability index.

2 Hasofer–Lind index and FORM algorithm

The stability of an engineering system, owing to the inherent uncertainty of input variables, is measured by the reliability index. The Hasofer-Lind index is widely used as a reliability index [27], the matrix formulation of which is,

$$\beta = {}_{x \in F}^{\hbox{min} } \sqrt {\left( {X - \mu } \right)^{T} C^{ - 1} \left( {X - \mu } \right)}$$
(1)

where X is a vector that represents a set of random variables xi, μ is a vector of mean values, C is the covariance matrix, and F is the failure domain. The superscripts “T” and “− 1” denote transpose and inverse respectively. The Hasofer-Lind index can be regarded as the shortest distance in directional standard deviation units from the mean value point of the random variables to the boundary of the limit state surface.

Low and Tang [28,29,30] presented an alternative formulation of the Hasofer-Lind index, Eq. 1, based on the perspective of an expanding ellipsoid in the original space of the basic random variables and expressed the index as

$$\beta = {}_{x \in F}^{\hbox{min} } \sqrt {\left[ {\frac{{x_{i} - \mu_{i} }}{{\sigma_{i} }}} \right]^{T} R^{ - 1} \left[ {\frac{{x_{i} - \mu_{i} }}{{\sigma_{i} }}} \right]}$$
(2)

where R is the correlation matrix, σi is the standard deviation of random variable xi.

For correlated non-normal parameters, Low and Tang [30] modified Eq. 2 as the following

$$\beta = {}_{x \in F}^{\hbox{min} } \sqrt {\left[ {\frac{{x_{i} - \mu_{i}^{N} }}{{\sigma_{i}^{N} }}} \right]^{T} R^{ - 1} \left[ {\frac{{x_{i} - \mu_{i}^{N} }}{{\sigma_{i}^{N} }}} \right]}$$
(3)

where \(\mu_{i}^{N}\) and \(\sigma_{i}^{N}\) are the equivalent normal mean and equivalent normal standard deviation, respectively, for random variable xi and can be calculated by the Rackwitz and Fiessler [31] transformation. The computation of β is regarded as that of finding the smallest equivalent hyperellipsoid (centered at the equivalent normal mean-value μN and with equivalent standard deviation σN) that is tangent to the limit state surface.

Low and Tang [32] presented an efficient alternative to Eq. 3 to calculate the reliability index β by varying dimensionless number ni as the following equation

$$\beta = {}_{x \in F}^{\hbox{min} } \sqrt {{\mathbf{n}}^{T} R^{ - 1} {\mathbf{n}}}$$
(4)

where n, termed as an equivalent standard normal vector, is a column vector of ni. Low and Tang [32] have shown that random variable xi can be obtained from ni by Eq. 5a for normal distribution and by Eq. 5b for the lognormal distribution.

$$x_{i} = \mu_{i} + n_{i} \sigma_{i}$$
(5a)
$$x_{i} = \exp \left[ {\lambda + \zeta n_{i} } \right],\quad \zeta = \sqrt {\ln \left[ {1 + \left( {\sigma_{i} /\mu_{i} } \right)^{2} } \right]} ,\quad \lambda = \ln \mu_{i} - 0.5\zeta^{2}$$
(5b)

The reliability index, in this study, is calculated using Eq. 4 as proposed by Low and Tang [32].

3 MATLAB constrained optimization function, fmincon

The constrained minimization [26] is the technique of finding a vector x that is the local minimum to a scalar function f(x) subject to constraints on the allowable x:

$$\mathop {\hbox{min} }\limits_{x} f(x)\quad {\text{such}}{\mkern 1mu} {\text{that}}\quad \left\{ {\begin{array}{*{20}l} {c(x) \le 0} \hfill \\ {c_{eq} (x) = 0} \hfill \\ {A \cdot x \le b} \hfill \\ {A_{eq} \cdot x = b_{eq} } \hfill \\ {lb \le x \le ub} \hfill \\ \end{array} } \right.$$
(6)

where b and beq are vectors, A and Aeq are matrices, c(x) and ceq(x) are functions that return vectors, lb and ub are lower and upper bounds, respectively and f(x) is a function that returns a scalar. In MATLAB’s Optimization toolbox, fmincon function is used for constrained optimization. MATLAB uses different nonlinear methods including Trust Region Reflective algorithm, Sequential Quadratic Programming (SQP) algorithm, and Interior Point algorithm.

4 Reliability-based optimization using a constrained optimization technique

In order to minimize the cost of a geotechnical system while satisfying a minimum value of the reliability index, termed as target reliability index βT, MATLAB constrained optimization function, fmincon, is used. It is a two-level (or double loop) optimization problem. The fmincon function is used in two loops; the outer loop to minimize the cost of the geotechnical system and the inner loop to minimize the reliability index β to a target reliability index value βΤ while satisfying geotechnical design requirements (Fig. 1).

Fig. 1
figure 1

Flowchart of reliability-based optimization algorithm

Full size image

Calculation sequence to minimize the cost function of the geotechnical system, referred to as an outer loop, is summarized as follows:

  • Step 1: Initialize design variables (e.g. dimension of the geotechnical system).

  • Step 2: Run the interior-point algorithm with MATLAB fmincon function. The objective is the minimization of the cost of the geotechnical system. The constraint is the target reliability index βΤ. The calculated β should be less than or equal to βΤ. For the calculation of β, the control is passed to the inner loop where the β is calculated (Eq. 4) and control is passed back.

  • Step 3: If the cost of the geotechnical system is minimized while satisfying the target beta constraint (i.e. β − βΤ ≤ 0), optimal design requirements are satisfied.

  • Step 4: If the cost of the geotechnical system is not minimum, design variables are modified and the process is repeated from step 2.

Calculation sequence to obtain the reliability index β, referred to as the inner loop, is summarized as follows:

  • Step 1: Initialize vector n.

  • Step 2: Run the interior-point algorithm with MATLAB fmincon function. The objective is the minimization of the reliability index β. The constraints are geotechnical design equations (e.g. factor of safety against overturning, sliding, bearing capacity, and eccentricity for a gravity retaining wall).

  • Step 3: Calculate β using Eq. 4. If the β is minimum while satisfying the geotechnical design equations, pass control to the outer loop.

  • Step 4: If the β is not minimum, vector n is modified and the process is repeated from Step 2.

The flowchart of the above-mentioned calculation sequence is shown in Fig. 1.

5 Illustrative example: retaining wall design

Gravity retaining wall design is adopted to illustrate the application RBO. Four failure modes are considered: (1) overturning failure, (2) sliding failure, (3) bearing capacity failure and, (4) eccentricity failure. Each failure mode is a design constraint. The goal is to obtain a design with minimum wall area (i.e. cost function) satisfying both target reliability requirements in addition to all design requirements. Gravity retaining wall design, adopted from the literature [8, 20], is based on Knappett and Craig [33]. A 6 m high concrete gravity retaining wall, as shown in Fig. 2 is considered.

Fig. 2
figure 2

Reliability-based design of gravity retaining wall (based on Zhang et al. [8])

Full size image

The active force Fa, based on Coulomb’s earth pressure theory [33] is given as

$$F_{a} = 0.5K_{a} \gamma_{s} H^{2}$$
(7)

where Ka is coulomb’s active earth pressure coefficient. Based on Fig. 2, the following geometric relationships exist:

$$\theta = a\tan (\frac{{b_{3} }}{H})$$
(8)
$$\alpha = \delta_{w} + \theta$$
(9)
$$B = b_{1} + b_{2} + b_{3}$$
(10)

The forces and moment arms involved in the overturning limit state are listed in Table 1.

Table 1 Forces and moment arms for rotational failure of retaining wall
Full size table

5.1 Limit state function for overturning failure

Based on the forces and moment arms shown in Table 1, the resisting moment MR and overturning moment MO are given in the following equations:

$$M_{R} = \mathop \sum \limits_{i = 1}^{4} W_{i} a_{i} + F_{V} a_{{F_{V} }}$$
(11)
$$M_{O} = F_{H} a_{{F_{H} }}$$
(12)

The limit state function for overturning failure is

$$g_{1} \left( {\mathbf{x}} \right) = M_{R} - M_{O}$$
(13)

5.2 Limit state function for sliding failure

Considering the force equilibrium in the vertical direction, the resultant base reaction RV is

$$R_{V} = \mathop \sum \limits_{i = 1}^{4} W_{i} + F_{V}$$
(14)

The limit state function for sliding failure is

$$g_{2} \left( {\mathbf{x}} \right) = R_{V} \tan \left( {\delta_{b} } \right) + Bc_{b} - F_{H}$$
(15)

5.3 Limit state functions of bearing capacity and eccentricity failure

Considering the moment equilibrium at the toe, the moment arm of the resultant base reaction is given by

$$a_{{R_{V} }} = \left( {M_{R} - M_{O} } \right)/R_{V}$$
(16)

The eccentricity of base reaction is given by

$$e = 0.5B - a_{{R_{V} }}$$
(17)

The maximum base pressure is given by

$$q_{\hbox{max} } = \frac{{R_{V} }}{B}\left( {1 + \frac{6e}{B}} \right)$$
(18)

The limit state functions for bearing capacity and eccentricity are given by Eqs. 19 and 20 respectively.

$$g_{3} \left( {\mathbf{x}} \right) = q_{u} - \frac{{R_{V} }}{B}\left( {1 + \frac{6e}{B}} \right)$$
(19)
$$g_{4} \left( {\mathbf{x}} \right) = 1 - \frac{6e}{B}$$
(20)

Limit state functions for overturning (Eq. 13), sliding (Eq. 15), bearing capacity (Eq. 19), and eccentricity (Eq. 20) are the geotechnical design requirements.

5.4 Uncertainties of input variables

Six uncertain variables are considered: the friction angle of backfill material (ϕf); the friction angle between the backfill material and the retaining wall (δw); the unit weight of backfill material (γf); the cohesion along the interface between the wall base and the founding soil (cb); the friction angle along the interface between the wall base and the founding soil (δb), and the maximum allowable bearing pressure of the foundation (qu). The distribution and statistics of these variables are adopted from [8] and summarized in Table 2. The correlation matrix assumed for these variables is also adopted from [8] and summarized in Table 3.

Table 2 Distribution and statistics of uncertain variables for retaining wall design
Full size table
Table 3 Correlation matrix of uncertain variables for retaining wall design
Full size table

5.5 Procedure for design optimization

Suppose the objective is to minimize the area of a retaining wall (i.e. the cost function) by changing the design variables d = {b1, b2, b3, h}. The target failure probability for retaining wall is in the range of 0.01–0.0001 [34, 35]. In this example, the target reliability index βT ≥ 3.0 is set for all failure modes which corresponds to the failure probability of 0.0013. The design space is considered as 0 < b1 < 3 m; 0.5 m < b2 < 3 m, 0 < b3 < 3 m, 0.4 m < h < 6 m. The reliability-based optimization problem can be written as

$$\hbox{min} \,C\left( {\mathbf{d}} \right) = b_{1} h + 0.5\left( {H - h} \right)b_{1} + b_{2} H + 0.5b_{3} H$$
(21)
$$\begin{aligned} & {\text{subject}}\,{\text{to}}\quad \beta_{i} \left( {\mathbf{d}} \right) \ge 3\quad i = 1, 2, 3, 4 \\ & 0 < b_{1} < 3 \\ & 0.5 < b_{2} < 3 \\ & 0 < b_{3} < 3 \\ & 0.5 < h < 6 \\ \end{aligned}$$

where C(d) is area of retaining wall and β1, β2, β3, β4 and are the reliability constraints for the limit states of overturning, sliding, bearing capacity and eccentricity respectively.

MATLAB’s constrained optimization function, fmincon, is used for the double loop optimization problem. In the outer loop, the objective is the minimization of the area of the retaining wall (Eq. 21) while the constraint is \(\beta_{i} \le \beta_{T}\). In the inner loop, the objective is the minimization of the reliability index βi (Eq. 4) while the constraints are corresponding limit state functions of overturning failure (Eq. 13), sliding failure (Eq. 15), bearing capacity failure (Eq. 19) and eccentricity (Eq. 20) respectively.

5.6 Results and discussion

The optimization process is started with initial design values as b1 = 1.6 m, b2 = 1 m, b3 = 0.6 m, and h = 1 m. The proposed method presented in Sect. 4 is implemented in MATLAB R2018a on a laptop equipped with an Intel Core i7-7500 CPU running at 2.70 GHz. The average computational time is less than 31 s. After running the optimization, the results summary is shown in Fig. 3 and Table 4. Four optimal scenarios listed in Table 4 are discussed below.

Fig. 3
figure 3

Calculated values of reliability constraints βi for optimal scenarios of overturning (OT), sliding (SL), bearing capacity (BC) and eccentricity (e) failures, respectively, for the target reliability constraint βT ≥ 3.0

Full size image
Table 4 Optimization results for target reliability constraint βT ≥ 3.0
Full size table

Overturning failure: Optimal 1 is the optimized solution when the design constraint is limit state function of overturning failure (Eq. 13) with the reliability constraint β1 minimized to the target reliability index value βT. Figure 4a shows the convergence of cost function and reliability constraint β1. From the calculated values of design variables (i.e. retaining wall dimensions), reliability constraints β2, β3 and β4 corresponding to limit state functions of sliding failure, bearing capacity failure and eccentricity, respectively are then calculated. Figure 4b shows the convergence of these reliability constraints when these are determined using Eq. 4. It is noted that only β1 corresponding to overturning failure is equal to βT whereas the rest of reliability constraints are less than βT. Thus optimal 1 does not satisfy all reliability constrains, and thus design requirements.

Fig. 4
figure 4

a Convergence of objective and reliability constraint β1 for overturning failure, b determination of other reliability constraints from the optimized solution

Full size image

Sliding failure: Optimal 2 is the optimized solution when the design constraint is limit state function of sliding failure (Eq. 15) with the reliability constraint β2 minimized to the target reliability βT. The convergence of cost function and reliability constraint β2 is shown in Fig. 5a. The rest of reliability constraints β1, β3 and β4 are calculated from the optimized solution. The convergence of β1, β3 and β4 are shown in Fig. 5b. Only β1 and β2 are above βT. Thus Optimal 2 does not satisfy bearing capacity and eccentricity requirements.

Fig. 5
figure 5

a Convergence of objective and reliability constraint β2 for sliding failure, b determination of other reliability constraints from the optimized solution

Full size image

Bearing capacity failure: Optimal 3 is the optimized solution when the design constraint is limit state function of bearing capacity failure (Eq. 19) with the reliability constraint β3 minimized to the target reliability βT. Figure 6a shows the convergence of cost function and reliability constraint β3. The rest of reliability constraints β1, β2 and β4 are calculated from the optimized solution as shown in Fig. 6b. Only β4 is below βT. Thus optimal 3 does not satisfy eccentricity requirement.

Fig. 6
figure 6

a Convergence of objective and reliability constraint β3 for bearing capacity failure, b determination of other reliability constraints from the optimized solution

Full size image

Eccentricity failure: Optimal 4 is the optimized solution when the design constraint is limit state function of eccentricity failure (Eq. 20) with the reliability constraint β4 minimized to the target reliability βT. Figure 7a shows the convergence of cost function and reliability constraint β4. Other reliability constraints β1, β2 and β3 are calculated from the optimized solution as shown in Fig. 7b. All reliability constraints are above βT.

Fig. 7
figure 7

a Convergence of objective and reliability constraint β4 for eccentricity failure, b determination of other reliability constraints from the optimized solution

Full size image

As seen in Fig. 3 and Table 4, only optimal 4 case satisfies all the reliability constraint requirements (i.e. All βi ≥ βT). Thus optimal 4 is the optimized solution satisfying the reliability requirement as well as the design requirements. The Convergence of design variables (i.e. retaining wall dimensions) for the optimized solution are shown in Fig. 8. The calculated values of the reliability constraints for initial and optimized design are summarized in Table 5. To check the accuracy of proposed method, the reliability constraints for the optimized solution are also calculated by Microsoft Excel’s built-in optimization routine SOLVER as proposed by Low and Tang [32]. GRG nonlinear solving method is used in Excel’s optimization routine. Figure 9 shows the Excel’s spreadsheet results. The reliability constraints calculated by Excel’s optimization retinue match with the present study results. Equivalent standard normal vectors, n, for all optimal scenarios are summarized in Table 6.

Fig. 8
figure 8

Convergence of design variables for optimal 4 scenario

Full size image
Table 5 Summary of optimization results for target reliability constraint βT ≥ 3.0
Full size table
Fig. 9
figure 9

Verification of Optimal 4 calculations for βi using EXCEL’s solver

Full size image
Table 6 Equivalent standard normal vector n for different optimal scenarios
Full size table

The results obtained from the present study are compared, in Table 7, with past studies [8, 20]. The results of the present study match precisely with those reported by Zhao et al. [20]. Zhao et al. [20] used Artificial Bee Colony (ABC) algorithm and Microsoft Excel’s SOLVER. Zhang et al. [8] used an indirect method by using a mapping function. The advantage of the proposed method, over others [8, 20], is its direct approach which couples reliability assessment and cost minimization. This study shows that MATLAB’s constrained optimization technique can be used in double-loop optimization problems encountered in geotechnical engineering.

Table 7 Comparison with past studies
Full size table

6 Conclusions

The findings of the study reported in this paper are summarized as follows:

  1. 1.

    A direct method is proposed for reliability-based optimization (RBO) by coupling reliability assessment and cost optimization. RBO is proposed as a double-loop (or nested-loop) constrained optimization problem, with cost optimization of the geotechnical system through the outer loop repeatedly calling the inner reliability evaluation loop.

  2. 2.

    MATLAB’s constrained optimization function, fmincon, is used to implement the algorithm. The interior-point algorithm is used in both loops.

  3. 3.

    The optimal design of gravity retaining wall is illustrated using the proposed method. The results of the optimal design match precisely with those reported earlier [8, 20]. The reliability constraints of the optimized solution are verified by calculating them using Microsoft Excel’s SOLVER.

  4. 4.

    The advantage of the proposed method is its direct approach which couples reliability assessment and cost minimization.