Abstract
In this paper, the main idea is to compute the robust regression model, derived by experimentation, in order to achieve a model with minimum effects of outliers and fixed variation among different experimental runs. Both outliers and nonequality of residual variation can affect the response surface parameter estimation. The common way to estimate the regression model coefficients is the ordinary least squares method. The weakness of this method is its sensitivity to outliers and specific residual behavior, so we pursue the modified robust method to solve this problem. Many papers have proposed different robust methods to decrease the effect of outliers, but trends in residual behaviors pose another important issue that should be taken into account. The trends in residuals can cause faulty estimations and thus faulty future decisions and outcomes, so in this paper, an iterative weighting method is used to modify both the outliers and the residuals that follow abnormal trends in variation, like descending or ascending trends, so they will have less effect on the coefficient estimation. Finally, a numerical example illustrates the proposed approach.
Introduction
In many cases, especially in experimental results, some of the data are wrong and should be treated as outliers. These points, which may occur because of operator reading faults and the like, may have a confusing effect on the total interpretation of the results. A common method of explaining and analyzing the results of experiments is by response surface design. This term is used for a regression equation that shows the whole behavior of the control variables, the nuisance factors, and the response or responses. We can use the estimated function to predict the response to changes in the values of specific controllable factors. After determining an experimental design and performing experiments, the next steps are generally statistical analysis and then the selection of values for the input variables so as to optimize the output. This can be done by fitting a regression model between the controllable factors and the response variables. Future interpretations are based on this regression model, so the exact model is very important and may affect the optimization stage. This model is generally constructed by the ordinary least squares (OLS) method. But basic OLS is very sensitive to outliers, and they may have an inordinate effect on the ultimate conclusion. So a robust method or a modified OLS should be used for decreasing the outliers’ sensitivity.
Our goal in this study is to decrease the destructive effect of outliers. In order to do so, at the first stage, the robust regression or modified regression model should be computed. Then the trend of residuals in response surface design is another aspect which should be considered. The trend behaviors among residuals, both descending and ascending trends, can cause faulty interpretations. However, there are some assumptions in estimating the regression coefficients that should not be violated. It seems that by decreasing the effects of some residuals in the coefficient estimation stage, the initial assumptions can be satisfied, and moreover, because this decreases the overall variability, the robustness of the model will be increased. The main purpose of this paper is to decrease the effects of outliers that violate the variance equality test of residuals. An example of such trends and abnormal behavior is shown in Figure 1.
As mentioned before, the OLS method is very sensitive to outliers. To diminish the effect of these points, some alternative methods of model fitting, such as least absolute deviations and other robust approaches that simplify the task of outlier identification by weighting the large residuals, are used instead of OLS. Response surfaces have been studied by many researchers, and many approaches have been proposed either to obtain efficient response surfaces or optimize the response surface by different models. Hejazi et al. (2010) proposed a novel approach based on goal programming to find the best combination of factors to optimize multiresponsemulticovariate surfaces by considering location and dispersion effects. Kazemzadeh et al. (2008) proposed a method to optimize multiresponse surfaces based on a goal programming method. Robust regression approaches have also been surveyed by many researchers. Huber (Bertsimas and Shioda 2007) proposed Mestimator methods to obtain robust regression. Morgenthaler and Schumacher (1999) discussed robust response surfaces in chemistry based on design of experiment. Because of the weakness of the previous methods in compensating for outliers, the redescending Mestimators (also named GMestimators) were proposed by Andrews et al. (1972), which are able to reject extreme outliers entirely. Hund et al. (2002) presented various methods of outlier detection and evaluated robustness tests with different experimental designs. Bickela and Frühwirthb (2006) compared different robust estimators with their applications. The M and GMestimators work by an iterative procedure. As a consequence, several authors (e.g., Cummins and Andrews 1995) have called these estimators as iteratively reweighted least squares, or IRLS methods. Ortiz et al. (2006) discussed some of the robust methods used for robust regression in analytical chemistry. To obtain a more efficient and yet robust method, Siegel (1982) proposed the repeated median estimator. Also another useful robust method is the least median squared (LMS) method proposed by Rousseeuw (1984). Massart et al. (1986) showed the advantages of its use in chemical analysis. The other useful method is the least trimmed squared (LTS) that was proposed by Rousseeuw and Leroy (1987). Nguyena and Welsch (2010) studied outlier detection and proposed a new least trimmed squares approximation. Both the LMS and the LTS are defined by minimizing a robust measure of the scatter of the residuals. Generalizing this, Rousseeuw and Yohai (1984) introduced Sestimators which are significantly more efficient than the previous estimators. A more recent suggestion is the constrained Mestimates, or CM, proposed by Mendes and Tyler (1995), which combines the good local properties of the Mestimates and the good global robustness properties of the Sestimates. A ‘partial’ version of the Mestimator based on the ‘fair’ ψ function and an appropriate weighting scheme was recently proposed by Serneels et al. (2005). The authors believed that the partial robust Mregression outperforms existing methods for robust partial least square regression. Bertsimas and Shioda (2007) presented mixed integer programming or MIP models for the classification and robust regression problems. Zioutas and Avramidis (2005) examined the effect of deleting outliers in the regression model obtained by mixed integer programming and compared the performance of this model with that of least squares, or LS, and LMS. Another new method in robust regression is the mixed linear model surveyed by Dornheim and Brazauskas (2011). (Pop and Sârbu 1996) proposed a new fuzzy regression algorithm to obtain robust models. Maronna et al. (2006) proposed many Mestimators using robust regression methods in both single response and multiple responses. Shahriari et al. (2011) proposed a novel twostep robust estimation of the process mean method based on Mestimator and their method is less sensitive to the presence of outliers. For better illustration of proposed method, the literature review has been classified in Table 1.
In this paper, a novel robust approach considering both outlier data and trends in residuals variations which do not violate the normality assumption is discussed.
This paper is organized as follows. The section ‘Robust estimation of the coefficients by iterative weighting methods’ presents the robust modification of the response surface by an iterative weighting procedure. The proposed method is defined in section ‘Robust estimation of coefficients by testing equality of variations in specified intervals’. To illustrate the proposed method, a numerical example is presented in section ‘Numerical example’. Finally, the last section is the ‘Conclusion’ of this paper.
Robust estimation of the coefficients by iterative weighting methods
To compensate for the effects of the outlier values, we can either remove the outlier data or modifying them by weighting the residuals. The first approach is not rational, so we choose to modify them in order to decrease the effect of outliers in the coefficient estimation stage. The proposed idea is as follows:
In this equation, μ_{ i } is a function defined by unknown coefficients (β_{ i }). For example, if μ_{1} = β_{1} + β_{2}x_{1} and x_{ i } are constants, the response y_{ i } can be obtained from the experimental results, and the regression model describes the relation between the variables and the expected values of the y_{ i }.
If all the measurements are good, then the OLS method provides a reasonable model and the coefficients are estimated by minimizing the following equation:
However, if the results appear abnormal, which may be a consequence of residual behavior in the experiments, the coefficients are determined by minimizing the following equation. The abnormality occurs when a residual behaves like an outlier:
The weights are not preassigned values because the quality of each y_{ i } is not known in advance. The reasonable values for the weights are based on the residuals defined by the following equation:
To make the estimator invariant with respect to the scale of the residuals, the r_{ i } is divided by ‘s,’ which is a robust estimation of the scale. The value of ‘s’ is often taken to be equal to 1.4826 MAD, where MAD is the median of the absolute deviations of the residuals from their median and 1.4826 is a bias adjustment for the standard deviation under the normal distribution.
The weights should be inversely proportional to the value of the residuals, {w}_{i}=\frac{c}{\left{r}_{i}\right}. In other words, the residuals with large values are weighted less, and this method produces better coefficient estimates. These weights can be chosen by a function such as the Huber weight function:
where c is a constant. The procedure is as follows: compute the first coefficients of the regression model, compute the residuals and weights, and then compute the new coefficients by the equation. This procedure can be repeated until a good solution is obtained, because the values of the coefficients and the values of the residuals and weights are different. This procedure is known as iterative weighting OLS. The procedure terminates when the change in the estimation from one iteration to the next is sufficiently small.
This iterative method is good for modifying outliers, but the trends in residual behavior are not considered. As illustrated before, another approach, in addition to taking outliers into account, is the equality of variation between residuals which is the main idea of the rest of this paper.
Robust estimation of coefficients by testing equality of variations in specified intervals
First, normality assumption is checked. If the normality assumption is violated, this robust approach based on Huber function cannot be applied. The method proposed in this part begins by dividing the experiment runs into a intervals to examine the hypothesis of equality in variations in these intervals. The equality test used in this paper is Bartlett’s test (Anderson and McLean 1974). The number of points in each interval can be chosen in the analysis stage. This stage satisfies one of the OLS hypotheses. If this parameter is small, the variation between points might be large and if the number of the points is large, the equality test of the variances may not be reliable, so this value should be determined rationally.
Bartlett’s test
Although residual plots are frequently used to diagnose inequality of variance, statistical tests have also been proposed. One widely used procedure is Bartlett’s test. The procedure involves computing a statistic whose sampling distribution is closely approximated by the chisquare distribution with a1 degrees of freedom, where the a random samples are from independent normal populations. This statistic is defined as
where
and {s}_{i}^{2} is the sample variance of the i th population.
The hypothesis of equality of variances is rejected if {x}_{0}^{2}>{x}_{\alpha ,\alpha 1}^{2}, where {x}_{\alpha ,\alpha 1}^{2} is the upper α percentiles of the chisquare distribution with a 1 degrees of freedom.
Proposed robust approach (robust equal variances iterative method)
The following steps are proposed as an iterative method to decrease the effect of trends in the residuals and improve the robust estimation of coefficients. The proposed model is based on OLS method which should be modified.
First of all, our goal is that the residuals that violate the hypothesis of equality in variances should have less effect on the estimates of the coefficients of the regression model, so we should consider modifying these points to have equal variances. Therefore, the residuals derived by experiments are divided into a intervals, and then the variances of each interval are calculated and denoted by {s}_{i}^{2}. The next step is to test the equality of variances with Bartlett’s test; if the result shows that the variances in a intervals do not have significant differences, this part of the procedure is stopped, whereas if the result shows that the variances in the a intervals differ significantly, the iterative weighting procedure is used to modify. As the next step, the critical q statistic in Bartlett’s test, for which the hypothesis will be rejected, is computed. The critical q for Bartlett’s test is denoted by q_{c}, and in our case, q > q_{c}. The {s}_{\left(1\right)}^{2} is the i th ranksorted variances of the intervals, in descending order. The {s}_{\left(i\right)}^{{2}^{\left(j\right)}} is the variances of the points in i th ranked interval in the j th. In the proposed approach, in higher and higher iterations approach, the residual variances approach equality. The maximum feasible variance of each intervals for which Bartlett’s test is fulfilled is denoted as {s}_{i}^{2}max.
Because the q value is greater than q_{ c } after its computation, it should be decreased iteratively until the both values are equal. To do this, we select the largest variance of all intervals. This value is decreased in each iteration. If the value of {s}_{\left(1\right)}^{2} equals {s}_{\left(2\right)}^{2} in this decreasing procedure, both {s}_{\left(1\right)}^{2} and {s}_{\left(2\right)}^{2} decrease in the next iteration. If the value q does not equal q_{c}, the values of the variances decrease again. If the values of {s}_{\left(1\right)}^{2} and {s}_{\left(2\right)}^{2} equal {s}_{\left(3\right)}^{2} in this decreasing procedure, all three variances decrease in the next iteration, and this procedure continues until q is equal to q_{c}. All maximum feasible variances of intervals that satisfy the hypothesis of Bartlett’s test are computed with this iterative procedure. Next, we consider two parallel lines l_{1}:y = +ω and l_{2}:y = ω, with slope zero and parallel to the xaxis. We decrease the ω values iteratively and compute the variances of points between these two lines. Until the variances of the points are equal to the maximum variance of intervals derived from the last step, this decreasing procedure continues. After that, the points outside these lines are weighted by function in (10).
The pseudo code of the proposed method illustrates this approach:

1.
Divide residuals into a intervals.

2.
Compute the variances of outliers in each interval, and denote these variances by {s}_{i}^{2}.

3.
Sort the variances of each interval in descending order and denote them by {s}_{\left(i\right)}^{2}.

4.
Calculate the critical value of Bartlett’s test in terms of confidence level and the number of intervals.

5.
Compute the q value by formula in (7).

6.
Compute the statistic of Bartlett’s test by formula in (6).

7.
Compute the q _{c} by considering the critical value of Bartlett’s test based on formula (6).

8.
Do while q ≥ q _{c},
\begin{array}{c}{s}_{\left(1\right)}^{{2}^{\left(j+1\right)}}={s}_{\left(1\right)}^{{2}^{\left(j\right)}}\delta \phantom{\rule{19em}{0ex}}\mathit{If}{s}_{\left(i\right)}^{{2}^{\left(j\right)}}>{s}_{\left(2\right)}^{{2}^{\left(j\right)}}\hfill \\ {s}_{\left(1\right)}^{{2}^{\left(j+1\right)}}={s}_{\left(1\right)}^{{2}^{\left(j+1\right)}}=\dots ={s}_{\left(f\right)}^{{2}^{\left(j+1\right)}}={s}_{\left(1\right)}^{{2}^{\left(j\right)}}\delta \phantom{\rule{5em}{0ex}}\mathit{If}{s}_{\left(1\right)}^{{2}^{\left(j\right)}}\le {s}_{\left(f\right)}^{{2}^{\left(j\right)}}f=2,3,\dots \hfill \\ j=j+1.\hfill \end{array} 
9.
Determine {s}_{i}^{2}max as the maximum feasible variances of each intervals to satisfy the hypothesis of equal variances.

10.
Consider two parallel lines, l _{1,} l _{2},
l_{1}:y = +ω and l_{2}:y = ω.

11.
Do while s _{ k } ^{2} < s _{ i } ^{2} max,
l_{1}:y = +m ω and l_{2}:y = m ω.

12.
Determine the points outside lines l _{1}and l _{2}.

13.
Determine the weight of the outside residuals by function below (formula 10).
\{\begin{array}{c}\hfill {w}_{i}=1\phantom{\rule{5.1em}{0ex}}\mathit{if}\phantom{\rule{0.12em}{0ex}}{l}_{2}<{r}_{i}<{l}_{i}\hfill \\ \hfill {w}_{i}=\frac{1}{\left{r}_{i}\right}\phantom{\rule{4em}{0ex}}f\phantom{\rule{0.12em}{0ex}}{r}_{i}>{l}_{1},{r}_{i}<{l}_{2}\hfill \end{array}(10)
After this step, by the robust iterative weighting method, the outliers are modified. After each iteration, the test of equality in variances is checked, and if the hypothesis is violated, the above mentioned method is applied. After the equality of variances is satisfied, the iterative weighting method continues, and this procedure continues as long as a minor change in the estimation occurs from each iteration to the next one. This procedure is called the robust equal variances iterative method procedure (REVIM).
The flowchart in Figure 2 illustrates this method.
Numerical example
This is a hypothetical numerical experiment. Suppose that we have an experiment containing one response variable and four explanatory control variables, each of which has three levels, and the objective of the study is to optimize the yield of a product. The data to be used are shown in Table 2. We want to explore the yield response surface by using a secondorder regression model. A BoxBehnken design with 27 treatments is used for this experiment. The blocking is used to diminish the effect of nuisance factors, and the blocks are assigned, for example, to 3 days.
The primary fitted response regression model is as follows:
The normality assumptions are checked in this example and the results are given in Figure 3. The P value obtained by normality test is 0.414. This value shows that the residuals follow normal distribution.
The analysis of variance (ANOVA) results are shown in Table 3.
Figure 4 shows the residuals of the model in the order of runs. As shown in the figure, the residuals have a rough trend, and if we divide the runs into 3 equal intervals, each containing 9 runs, the second interval has larger variance. This can be proved by Bartlett’s test.
The result of Bartlett’s test is illustrated in Figure 5 and Table 4.
In this case, we have three intervals, so a 1 = 2. If we consider the significance level 0.95, the critical statistic {{\displaystyle x}}_{0.05,2}^{2} is equal to 5.99 and the test statistic = 9.09 is greater than 5.99, then the hypothesis is rejected. Therefore, we want to compute the maximum standard deviation of each interval that satisfies the hypothesis of Bartlett’s test. By the proposed method, the maximum values of these standard variations are ordered as follows: s_{1} = 2.07, s_{2} = 4.95, s_{3} = 4.95. Based on these values, we can compute the limits, l_{1} = 8.7 and l_{1} = 8.75. Two points 5 and 16 are outside these limits (circled in the figure), so the weighting procedure is applied and new coefficients are calculated by the robust weighting method. Figure 6 shows the method graphically.
The residuals are computed by this procedure, and the hypothesis of equal variance in three intervals is satisfied. Bartlett’s test is applied to the residuals obtained from the proposed method, and the results are given in Figure 7.
The value of Bartlett’s test is 4.32, and the hypothesis of equal variances is not rejected. Figure 8 illustrates the result better.
Therefore, the iterative weighting method based on these coefficients to modify the effect of outliers by Huber function with c = 2 is applied. This process is applied in each iteration, and if the equality test is not satisfied, the modification is applied. The final robust coefficients and residuals are presented in Tables 5 and 6.
So the residuals in final iteration show that residuals which hinted to be an outlier in first iteration are really outliers, and the model estimation is more accurate and close to the model with no outlier data that we call it actual model. These results can be compared with the results obtained by the model with no outlier data. The comparison shows that the proposed model is more precise and accurate than the OLS method in estimation of the regression coefficients by considering unequal variation between residuals. The results are given in Table 7.
Conclusions
A robust estimation of the response surface is the primary goal of this paper. To this end, the proposed method is defined, instead of the common ordinary least squares method of estimating coefficients of the response surface, to decrease the effects of two main causes of the imprecise estimation of coefficients, outliers, and trends in residuals. As the effect of trends in residuals should be taken into account, the proposed method simultaneously modifies the effects of trends and outliers. For each iteration, an equality test of residual variances is performed, and after this hypothesis is satisfied, the outliers are modified. A goal for future research may be to examine the weighting; instead of computing the distance of the residuals from base line after plotting the responses in a normal probability plot (NPP), the weighting function may be proportional to the distance of the response from the NPP regression line.
Authors’ information
MB is an associate professor of industrial engineering and his research interests are multiple response optimization and facilities location problem. AM is a Ph.D. student of industrial engineering at Amirkabir University of Technology (Tehran Polytechnic).
References
Anderson VL, McLean RA: Design of experiments: a realistic approach. New York: Dekker; 1974.
Andrews DF, Bickel PJ, Hampel FR, Huber PJ, Rogers WH, Tukey JW: Robust estimates of location: survey and advances. Princeton: Princeton University Press; 1972.
Bertsimas D, Shioda R: Classification and regression via integer optimization. Oper Res 2007, 55: 252–271. 10.1287/opre.1060.0360
Bickela DR, Frühwirthb R: On a fast, robust estimator of the mode: comparisons to other robust estimators with applications. Comput Stat Data An 2006, 50: 3500–3530. 10.1016/j.csda.2005.07.011
Cummins DJ, Andrews CW: Iteratively reweighted partial least squares: a performance analysis by Monte Carlo simulation. J Chemometr 1995, 9: 489–507. 10.1002/cem.1180090607
Dornheim H, Brazauskas V: Robustefficient fitting of mixed linear models: methodology and theory. J Stat Plan Infer 2011, 141: 1422–1435. 10.1016/j.jspi.2010.10.016
Hejazi TH, Bashiri M, Noghondarian K, Atkinson AC: Multiresponse optimization with consideration of probabilistic covariates. Qual Reliab Eng Int 2010, 27: 437–449.
Huber PJ: Robust statistics. New York: Wiley; 1981.
Hund E, Massart DL, SmeyersVerbeke J: Robust regression and outlier detection in the evaluation of robustness tests with different experimental designs. Anal Chim Acta 2002, 463: 53–73. 10.1016/S00032670(02)003379
Kazemzadeh RB, Bashiri M, Atkinson AC, Noorossana R: A general framework for multiresponse optimization problems based on goal programming. Eur J Oper Res 2008, 189: 421–429. 10.1016/j.ejor.2007.05.030
Maronna RA, Martin RD, Yohai VJ: Robust statistics: theory and methods. New York: Wiley; 2006.
Massart DL, Kaufman L, Rousseeuw PJ, Leroy A: Least median of squares: a robust method for outlier and model error detection in regression and calibration. Anal Chim Acta 1986, 187: 171–179.
Mendes B, Tyler DE: Robust statistics, data analysis and computer intensive methods. New York: Springer; 1995.
Morgenthaler S, Schumacher MM: Robust analysis of a response surface design. Chemometr Intell Lab 1999, 47: 127–141. 10.1016/S01697439(98)001993
Nguyena TD, Welsch R: Outlier detection and least trimmed squares approximation using semidefinite programming. Comput Stat Data An 2010, 54: 3212–3226. 10.1016/j.csda.2009.09.037
Ortiz MC, Sarabia LA, Herrero A: Robust regression techniques. A useful alternative for detection of outlier data. Talanta 2006, 70: 499–512. 10.1016/j.talanta.2005.12.058
Pop HF, Sârbu C: A new fuzzy regression algorithm. Anal Chem 1996, 68: 771–778. 10.1021/ac950549u
Rousseeuw PJ: Least median of squares regression. J Am Stat Assoc 1984, 79: 871–880. 10.1080/01621459.1984.10477105
Rousseeuw PJ, Yohai VJ: Robust and nonlinear time series analysis. New York: Springer; 1984.
Rousseeuw PJ, Leroy AM: Robust regression and outlier detection. New York: Wiley; 1987.
Serneels S, Croux C, Filzmoser P, Van Espen PJ: Partial robust Mregression. Chemometr Intell Lab 2005, 79: 55–64. 10.1016/j.chemolab.2005.04.007
Shahriari H, Ahmadi O, Shokouhi AH: A twostep robust estimation of the process mean using Mestimator. J Appl Stat 2011, 38: 1289–1301. 10.1080/02664763.2010.498502
Siegel AF: Robust regression using repeated medians. Biometrika 1982, 69: 242–244. 10.1093/biomet/69.1.242
Zioutas G, Avramidis A: Deleting outliers in robust regression with mixed integer programming. Acta Math Appl Sin 2005, 21: 323–334.
Acknowledgments
Authors are thankful to the reviewers for their valuable comments.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
MB has worked on the literature and modeling, he also proposed response surface methodology. AM has formulated the robust concept and numerical example. Both authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Bashiri, M., Moslemi, A. The analysis of residuals variation and outliers to obtain robust response surface. J Ind Eng Int 9, 2 (2013). https://doi.org/10.1186/2251712X92
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/2251712X92
Keywords
 Coefficient estimation
 Response surface
 Ordinary least squares
 Outliers
 Robust model