Abstract
This paper presents a methodology, called ROBOUT, to identify outliers conditional on a highdimensional noisy information set. In particular, ROBOUT is able to identify observations with outlying conditional mean or variance when the dataset contains multivariate outliers in or besides the predictors, multicollinearity, and a large variable dimension compared to the sample size. ROBOUT entails a preprocessing step, a preliminary robust imputation procedure that prevents anomalous instances from corrupting predictor recovery, a selection stage of the statistically relevant predictors (through crossvalidated LASSOpenalized Huber loss regression), the estimation of a robust regression model based on the selected predictors (via MM regression), and a criterion to identify conditional outliers. We conduct a comprehensive simulation study in which the proposed algorithm is tested under a wide range of perturbation scenarios. The combination formed by LASSOpenalized Huber loss and MM regression turns out to be the best in terms of conditional outlier detection under the above described perturbed conditions, also compared to existing integrated methodologies like Sparse Least Trimmed Squares and Robust Least Angle Regression. Furthermore, the proposed methodology is applied to a granular supervisory banking dataset collected by the European Central Bank, in order to model the total assets of euro area banks.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Data quality is a fundamental prerequisite for any kind of quantitative analysis and the large datasets which are becoming increasingly available present specific challenges to the task of monitoring and ensuring data quality. One critical aspect of the data quality monitoring is outlier detection, i.e. the identification of values which are either obviously mistaken or seem to be unjustified from an empirical perspective. An outlier is defined by Hawkins (1980) as ‘an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism’. Similarly, Barnett and Lewis (1994) defines an outlier as ‘an observation (or subset of observations) which appears to be inconsistent with the remainder of that set of data’. Classical statistical methods, which underpin analytical tools supporting analysis of large datasets, are sensitive to the presence of outliers and, consequently, could reach distorted conclusions (Rousseeuw and Hubert 2018).
In this paper, we focus on outlier detection in highdimensional datasets, i.e., where the number of variables p (i.e. the dimension of the data space) is large, possibly larger than the number of observations n (i.e. the sample size). In this way, we also include socalled fat datasets, featuring \(p>n\). Such datasets arise in diverse fields, such as bioinformatics, economics, neuroscience, signal processing and others.
Our aim is to retrieve from such datasets any anomaly in a target variable y with respect to a set of K related variables (the predictors of y) that constitute a subset of the \(p \gg K\) variables of the dataset (the candidate predictors of y). The K predictors of y are exante unknown and need to be identified from the p variables, which are also usually affected by multicollinearity effects. In addition, we tolerate the presence in the K predictors of leverage outliers, a feature that typically inflates the predictive power of irrelevant predictors, as well as the presence of multivariate outliers in the remaining variables of the dataset.
If the conditional outliers in y are present in the same observation with anomalous instances of the related predictors, it has been shown that predictors are typically not identified correctly (see Khan et al. 2007). That is the reason why we develop a robust preliminary imputation procedure, that prevents such points (i.e. leverage points) from corrupting predictor recovery, thus restoring the identification of true predictors. Ensuring the recovery of true predictors and limiting as much as possible the inclusion of irrelevant predictors is the key to successfully apply a subsequent robust regression in a consistent way and to identify conditional outliers by means of the robustly estimated residual scale. Other confounding factors for predictor recovery are multicollinearity, a high number of variables p compared to the sample size n, and a small overall signaltonoise ratio.
The practical relevance of the problem as formulated above can be clarified by referring to the banking supervisory dataset that is used in this paper to show the behaviour of the proposed outlier detection method. A bank may present a particularly high value of e.g. total assets (the variable y in the above formulation), which may be spotted as an anomalous instance compared to the rest of banks. However, considering other related bank indicators, such as debt securities and derivatives, we may realize that the total asset value for the bank in question is perfectly in line with expectations. On the other hand, a bank with an average value of total assets may be judged as anomalous with respect to the predictors. This could be for example the case when for a specific observation (i.e. bank) a relatively moderate amount of assets corresponds to a disproportionally high amount of derivatives, a financial instrument that one would normally expect to be used extensively by the largest banks.
In general, whether an observation of variable y is considered to be an outlier does not depend solely on the distribution of y but also on the corresponding values of the most relevant predictors of y. This kind of outlier detection is referred to as conditional outlier detection (Hong and Hauskrecht 2015), because it focuses on the detection of improbable values in a target variable given (i.e. conditionally on) the observed values of a set of related variables.
We propose a new method, called ROBOUT, to solve the problem of conditional outlier detection in high dimensions. ROBOUT can accommodate diverse statistical properties of the examined dataset (such as multicollinearity, multivariate outliers in or out of the predictors and highdimensional information sets) while at the same time being computationally efficient. Our specific contributions comprise the separation of predictor recovery and final model estimation, which renders the identification of true predictors possible under above mentioned perturbations, the study and recovery of conditional outliers with perturbed residual variance (beyond the usual ones with perturbed mean), and the adoption of a preliminary imputation method, which is effective in preserving model recovery from anomalies in the potential predictors.
The paper is structured as follows. Section 2 formulates the problem and motivates its solution. Section 3 presents a literature review on highdimensional variable selection and robust regression methods. ROBOUT methodology is formally defined in Sect. 4, entailing a preprocessing, a robust preliminary imputation, a variable selection, a lowdimensional regression and a conditional outlier detection step. The wide simulation exercise that we undertake in Sect. 5 aims to test the performance of ROBOUT, placing emphasis on the versatility of the ensuing conditional outlier detection under several perturbed scenarios for different p/n ratios. Subsequently, we apply ROBOUT to a banking dataset collected by the European Central Bank in Sect. 6, and we provide concluding remarks in Sect. 7.
2 Motivating examples and first formalization
Figure 1 illustrates visually the problem that we aim to solve and the idea of the method that we propose. We consider the target variable y and two variables of the dataset \(x_1\) and \(x_2\) (in principle, the variables of the dataset could be many more than just two). \(x_1\) is a predictor of y while \(x_2\) is not, however this is not known exante. We assume that there is an outlier in variable y, namely point 4 which is an outlier of y conditional on predictor \(x_1\). Point \(4{'}\) shows the ‘correct’ position that 4 would have occupied if it followed the same data generating process as the other three points. The perturbation of point 4 affects the linear relationship between y and \(x_1\), as line \(l_0\) is shifted below to line \(l_1\), but also affects the relationship between y and \(x_2\), potentially leading to a spuriously statistically significant relationship.
Identifying the perturbation in point 4 requires to solve two tasks: first to identify that \(x_1\) is the relevant conditional variable and second to identify that 4 is an outlier of y conditional on \(x_1\). Since the presence of outliers in y may disrupt the identification of the statistical relationship between two related variables (in this case y and \(x_1\)) and may generate a misleading statistical relationship between unrelated variables (in this case y and \(x_2\)), in this paper we separate the two problems: we first run a robust variable selection step, with the aim to identify the relevant predictors, and we subsequently run a lowdimensional robust regression to identify conditional outliers. As will be argued in this paper, performing predictor recovery and the model estimation in consecutive steps has substantial advantages in terms of computational efficiency and effectiveness in detecting conditional outliers.
Let us formalize the described problem in probabilistic terms. In (Hong and Hauskrecht 2015), unconditional outliers are defined as instances which fall into a low probability density region of \(\textrm{f}(y)\), where \(\textrm{f}(y)\) is the unconditional probability density function of the scalar target variable y. Instead, conditional outliers are defined as instances of y which fall into a low probability density region of \(\textrm{f}(y\vert \textbf{x})=\textrm{f}(y,\textbf{x})/\textrm{f}(\textbf{x})\), where \(\textrm{f}(y\vert \textbf{x})\) is the conditional probability density function of the scalar target variable y given a vector of predictors \(\textbf{x}\), and the set of predictors \(\textbf{x}\) is a subset of a wider noisy information set \(\Omega \), which is given at the outset of the problem. In this paper, we assume that the expected value of \(\textrm{f}(y\vert \textbf{x})\) is a linear function of the variables in \(\textbf{x}\). Note that this does not constrain the nature of the prescribed relationships between y and the initial information set \(\Omega \) from which \(\textbf{x}\) is identified, because we can always include e.g. quadratic and exponential functions of specific variables in the vector \(\textbf{x}\).
For the sake of simplicity, let us suppose that for the single observation \(i\in \{1,\ldots ,n\}\) it holds that \(y_i \vert \textbf{x}_{D,i} \sim N(a_i+\textbf{x}_{D,i}'\mathbf {\beta },\sigma _i^2)\), where, for observation i, \(y_i\) is the specific value of y, \(a_i\) is the intercept term, \(\textbf{x}_{D,i}\) is the \(K \times 1\) vector of predictors, \(\mathbf {\beta }\) is the \(K \times 1\) vector of regression coefficients, and \(\sigma _i^2\) is the variance of \(y_i\) conditional on \(\textbf{x}_{D,i}\). We denote by D the set of predictor indices, such that \(\vert D \vert =K\) and \(D\subseteq \{1,\ldots ,p\}\), and by \({\bar{D}}\) the complementary set of D with respect to the set \(\{1,\ldots ,p\}\). The \(n \times K\) matrix \(\textbf{X}_D\) contains as columns the predictor variables, indexed by D, the \(n \times (pK)\) matrix \(\textbf{X}_{{\bar{D}}}\) contains as columns the nonpredictor variables, indexed by \({\bar{D}}\), and the \(n \times p\) complete matrix \(\textbf{X}=[\textbf{X}_{D} \vert \textbf{X}_{{\bar{D}}}]\) represents the available information set \(\Omega \).
We define a set of outlier indices O with \(O\subseteq \{1,\ldots ,n\}\) such that \(\vert O \vert =[\alpha n]\). The parameter \(\alpha \) represents the contamination rate, with \(\alpha \in [0,0.5]\). We assume, without loss of generality, that for any \(i \in {\bar{O}}\) it holds \(a_i = a\) and \(\sigma _i^2=\sigma ^2\), i.e. that all nonoutlying observations feature a constant conditional mean and variance.
Definition 1
A conditional outlier in mean is defined as any observation \(i{'}\) such that \(y_{i{'}}\) falls in a low probability density region of \(N(a+\textbf{x}_{D,i{'}}'\mathbf {\beta },\sigma ^2)\) because \(a_{i{'}} \gg a\) or \(a_{i{'}} \ll a\).
Definition 2
A conditional outlier in variance is defined as any observation \(i{'}\) such that \(y_{i{'}}\) falls in a low probability density region of \(N(a+\textbf{x}_{D,i{'}}'\mathbf {\beta },\sigma ^2)\) because \(\sigma _{i{'}}^2 \gg \sigma ^2\).
Definition 3
A leverage outlier is defined as any observation \(i{'}\) such that its vector of predictors \(\textbf{x}_{D,i{'}}\) is perturbed by replacing \(\textbf{x}_{D,i{'}k}\) with \(\textbf{x}_{D,i{'}k}+\textbf{m}\), \(\textbf{m} \in {\mathbb {R}}^{K}\), \(\vert m_k \vert \gg 0\), \(k=1,\ldots ,K\).
Definition 4
A rowwise outlier is defined as any observation \(i{'}\) such that a subset of its vector of nonpredictors, say, \(\textbf{x}_{{\tilde{D}},i{'}}\), with \({\tilde{D}} \subset {\bar{D}}\), is perturbed by replacing \(\textbf{x}_{{\tilde{D}},i{'}k{'}}\) with \(\textbf{x}_{{\tilde{D}},i{'}k{'}}+\textbf{m}\), \(\textbf{m} \in {\mathbb {R}}^{\vert {\tilde{D}} \vert }\), \(\vert m_{k{'}} \vert \gg 0\), \(k{'} \in {\tilde{D}}\).
In practice, Definitions 1 and 2 may reflect that the target variable y is affected by measurement errors, idiosyncratic events, unpredictable shocks etc., as well as structural differences in the data generating mechanism. Definitions 3 and 4 represent the same situations in the matrix of predictors \(\textbf{X}_D\) and in the matrix of nonpredictors \(\textbf{X}_{{\bar{D}}}\), respectively. Conditional outlier recovery is critical to spot hidden inconsistencies or frauds in y, and cannot be performed properly if the unknown predictors of y, contained in \(\textbf{X}_{D}\), are not identified.
Few integrated methods have been presented in the literature that recover simultaneously both the K predictors of the response variable y (out of the p candidate predictors) and the conditional outliers in y, such as Sparse Least Trimmed Squares (SPARSELTS, Alfons et al. 2013) and Robust Least Angle Regression (RLARS, Khan et al. 2007). To the best of our knowledge, their performance as conditional outlier detection methods in the presence of conditional outliers in variance in the sense of Definition 2 has not been explored yet. Another existing method, called SNCD (Semismooth Newton Coordinate Descent) algorithm (Yi and Huang 2017), provides a fast and reliable solution to the recovery of predictors, by minimizing a Huber or a Least Absolute Deviation (LAD) loss of the residuals penalized by an elastic net (Zou and Hastie 2005). However, SNCD has a significant drawback: it is not robust to leverage outliers, as shown in Alfons et al. (2013).
For this reason, we propose in Sect. 4.2.2 a preliminary imputation procedure which is robust to potentially disruptive outliers in \(\textbf{X}\) while being computationally efficient. In the following, we apply SNCD to the clean imputed dataset to identify the right predictors, on which a robust regression model is finally calculated to spot conditional outliers. The described method, which we call ROBOUT, is very efficient as regards computational cost, which is a direct function of the degree of perturbation in the dataset. ROBOUT improves existing methods in the literature by enhancing both predictor recovery and conditional outlier detection performance in high dimensions also under challenging conditions, when other methods may fail. This result is achieved by exploiting the large cross section to setup the preliminary imputation procedure, whose accuracy benefits from the presence of many pairs of highly correlated variables, thus turning the curse of dimensionality to a blessing.
3 State of the art
Large and highdimensional datasets appear very suitable for conditional outlier detection in the regression context (Rousseeuw and Leroy 1987). As Varian (2014) notes, the large size of the data requires automated techniques for the identification of subsets of variables which are statistically linked. In such datasets, the challenge is to identify the critical predictors and then to perform a conditional outlier detection for the variables of interest. To this end, we need to define both a variable selection and a robust regression procedure, in order to recover consistently both the true set of predictors and the regression coefficients. This type of outlier detection may spot outliers which could remain unnoticed if single variables are considered separately, thus taking advantage of the information content present in a large cross section.
The original idea of robust regression is based on the application of a weighting scheme to observations, with the aim to dampen the influence of outlying points on regression estimates, both in \(\textbf{y}\) (conditional outliers) and in \(\textbf{X}\) (leverage outliers). Two established methods for lowdimensional robust regression are:

least trimmed squares (LTS) estimation (Rousseeuw 1984), which identifies the \(100\times (1\alpha )\%\) most concentrated observations and estimates the regression coefficients on those via ordinary least squares;

MMestimation (Yohai 1987), which is a threestage procedure that minimizes a Tukey’s bisquare function of the residuals using a robust initialization of the coefficients in \(\mathbf {\beta }\) (obtained by Sregression) and of the residual scale \(\sigma \) (obtained by Mestimation).
In the \(p>n\) case, the mentioned traditional robust regression techniques no longer work, because they simply are weighted versions of the least squares method. Filzmoser and Nordhausen (2021) provides an exhaustive literature review on robust linear regression for highdimensional data and describes the different strategies that have been consequently proposed to perform robust regression in high dimensions.
There are few methods proposed in the literature that perform simultaneously variable selection and conditional outlier detection. A first group is based on LASSO regression (Tibshirani 1996). A robust LASSO regression method performing both variable selection and conditional outlier detection is SPARSELTS (Alfons et al. 2013), that is based on the simultaneous optimization of a trimmed least squares loss and a LASSO penalty. The coefficient estimates are derived by performing at each iteration a LASSO estimation on the \(100\times (1\alpha )\%\) observations with the smallest squared residuals. Another robust LASSO regression method is MMLASSO (Smucler and Yohai 2017). Building on the Sridge approach to highdimensional regression model estimation in Maronna (2011), MMLASSO combines (adaptive) MMregression and LASSO penalization to provide consistent identification of predictors and estimation of coefficients under fat tailed distributions.
A previous relevant solution of different nature is plugin RLARS (Khan et al. 2007), that is the robustified version of LARS (Efron et al. 2004) where the covariance matrix of the features is robustly estimated via an adjusted Winsorization step. RLARS manages to deal with outliers by rendering them uninfluential in the iterative computation of the covariances that are needed to retrieve predictors. Predictor selection is then performed by iteratively selecting the best angle to update coefficients, as in LARS, and the MM regression is used as final step.
A further approach to identify predictors and spot conditional outliers in high dimensions is by robustifying the elastic net regression. In this respect, a robust solution is ENETLTS (Kurnaz et al. 2018), that provides the trimmed version of the elastic net. In Freue et al. (2019), the penalized elastic net Sestimator PENSE and its refinement, the penalized elastic net Mestimator PENSEM, provide a consistent solution to conditional outlier detection, with the relevant advantage for PENSEM to provide the most precise \(\tau \)estimate of the residual scale.
Forward Search (see Atkinson et al. 2004) is a fascinating method to iteratively identify the subset of the least anomalous observations from a multivariate dataset, which are ranked according to their degree of outlyingness. This method has been scaled to very large samples (Riani et al. 2015) via a fast calculation of robust Mahalanobis distances. A Bayesian version of it has been presented in Atkinson et al. (2017).
The SNCD algorithm (Yi and Huang 2017), that minimizes simultaneously a Huber loss or a LAD loss of the residuals and the elastic net penalty by explicitly deriving the KarushKuhnTucker (KKT) conditions of the objective function, is instead not robust to leverage outliers, even if it is scalable to both large sample sizes and high dimensions, as its computational cost is O(pn).
Another related strand of the literature concerns the cellwise contaminated outliers: Rousseeuw and Bossche (2018) identifies deviating data cells, Bottmer et al. (2022) adapts the sparse shooting Sestimator of Öllerer et al. (2016) in the presence of cellwise outliers, Filzmoser et al. (2020) proposes cellwise robust M regression. Even if the performed task is similar with that of ROBOUT (sparse regression model identification and estimation in a perturbed context), the perturbations which we consider in the data matrix are somewhat different, as we consider the presence of multivariate outliers in the true predictors (similarly to SPARSELTS and RLARS) as well as beyond the true predictors, with the aim to extend as much as possible robustness to multivariate outliers. In addition, our method avoids running pairwise robust regressions in order to keep computational cost low and aiming to be fit for highdimensional data. We leave to future studies a formal comparison with the sparse shooting Sestimator and a modification of ROBOUT able to deal with randomly scattered individual outliers.^{Footnote 1}
4 ROBOUT: a comprehensive approach for conditional outlier detection
In this section, we describe ROBOUT procedure. We first present the data model behind its usage in Sect. 4.1 and we elaborate on its five steps in Sect. 4.2.
4.1 Model
Let us consider n numerical observations of one response variable y and p additional variables. We call the unknown set of conditional outlier indices O with \(\vert O \vert =[\alpha n]\) and \(\alpha \in [0,0.5]\), and \({\bar{O}}\) the index set of nonoutlying points. The response variable vector \(\textbf{y}\) may be expressed in terms of the following regression model
where \(\textbf{y}\) is the \(n \times 1\) vector of the response variable, \(\textbf{a}\) is the \(n \times 1\) vector of intercepts, \(\textbf{X}_D\) is the \(n \times K\) matrix of predictors (whose columns are indexed by D), \(\mathbf {\beta }\) is the \(K \times 1\) vector of regression coefficients and \(\epsilon \) is the \(n \times 1\) vector of residuals. The same regression model for the single observation \(i\in \{1,\ldots ,n\}\) can be written as
where \(a_i\) denotes the intercept and \(\textbf{x}_{D,i}\) denotes the \(K \times 1\) vector of predictors. We name \(\textbf{R}_\textbf{X}\) the \(p \times p\) covariance matrix of \(\textbf{X}\), and we identify the covariance matrix of the K predictors as \(\textbf{R}_{D}\), of the \(pK\) nonpredictors as \(\textbf{R}_{{\bar{D}}}\), of the variables in \({\tilde{D}}\) as \(\textbf{R}_{{\tilde{D}}}\).
For each nonoutlier index \(i \in {\bar{O}}\), we assume that \(a_i=a\), \(\epsilon _i \sim N(0,\sigma ^2)\), \(\textbf{x}_{D,i} \sim MVN(\textbf{0}_K,\textbf{R}_{D})\) and \(\textbf{x}_{{\bar{D}},i}\sim MVN(\textbf{0}_{pK},\textbf{R}_{{\bar{D}}})\), where \(\textbf{x}_{{\bar{D}},i}\) is the vector of nonpredictors (\({\bar{D}}\) stores the indices of nonpredictor variables). We distinguish in the outlier set O: the index sets of conditional outliers in y, \(O_{y}\), with size \([\alpha _{y}n]\); of leverage outliers in the predictors, \(O_{lev}\), with size \([\alpha _{lev}n]\); of rowwise outliers out of the predictors, \(O_{row}\), with size \([\alpha _{row}n]\). Note that the proportions of index sets \(\alpha _{y}\), \(\alpha _{lev}\), \(\alpha _{row}\) are allowed to vary, and we allow \(O_{y}\), \(O_{lev}\), and \(O_{row}\) to overlap.
Let us define for any \(x\in {\mathbb {R}}\) the sign operator \(\textrm{sgn}\), such that \(\textrm{sgn}(x)=1\), if \(x>0\), \(\textrm{sgn}(x)=0\), if \(x=0\), \(\textrm{sgn}(x)=1\), if \(x<0\). Then, for each outlier index \(i{'} \in O\), we fix \(a_{i{'}}=a\), and we generate the four types of outliers (Definitions 14) in the following way:

1)
conditional outliers in mean are generated consistently with Definition 1 as \(\epsilon _{i'} \sim N((m1)a,\sigma ^2)\), \(i'\in O_y\), \(m \in {\mathbb {R}}\), \(m>1\);

2)
conditional outliers in variance are generated consistently with Definition 2 as \(\epsilon _{i'} \sim N(0,m^2\sigma ^2)\), \(i'\in O_y\), \(m \in {\mathbb {R}}\), \(m>1\);

3)
the vector of predictors is generated consistently with Definition 3 as \(\textbf{x}_{D,i'}\sim MVN(\textbf{0}_K,\textbf{R}_{D})\), and leverage outliers are generated by replacing a posteriori \(x_{D,i'k} \) with the perturbed values \(x_{D,i'k}+(m+2)\times \textrm{sgn}(\textrm{Unif}[1,1])\), \(i'\in O_{lev}\), \(k=1,\ldots ,K\), \(m \in {\mathbb {R}}\), \(m>1\);

4)
the vector \(\textbf{x}_{{\tilde{D}},i'}\) is generated consistently with Definition 4 as \(\textbf{x}_{{\tilde{D}},i'}\sim MVN(\textbf{0}_K,\textbf{R}_{{\tilde{D}}})\), and rowwise outliers are generated by replacing a posteriori \(x_{{\tilde{D}},i'k'} \) with the perturbed values \(x_{{\tilde{D}},i'k'}+(m+2)\times \textrm{sgn}(\textrm{Unif}[1,1])\), \(i'\in O_{row}\), \(k'\in {\tilde{D}}\), \({\tilde{D}} \subset {\bar{D}}\), \(m \in {\mathbb {R}}\), \(m>1\).
Normality is assumed to ensure the validity of residual diagnostics and model inference (also see Sect. 4.2.1). The multiplier m is the outlyingness parameter, and the value \(m+2\) is chosen in 3) and 4) to ensure that nonrobust correlation estimates are impacted by those outliers, as well as the randomized sign on individual perturbations obtained by the uniform distribution \(\textrm{Unif}[1,1]\) contributes to ensure.
In all the above cases of outliers, the parameter \(m>1\) represents the degree of perturbation. Because more than one type of outliers can coexist within the given observation (row), there are in total nine outlier schemes for each \(i' \in O\):

(1–2)
“Only mean" or “Only variance" cases: only response variable outliers (in mean or variance) are present. The outliers \(i'\) are only generated in \(y_{i{'}}\) according to Definition 1 or 2.

(3–4)
“Meanleverage" and “Varianceleverage" cases: simultaneous presence of conditional outliers (in mean or in variance) and leverage outliers, where the outliers \(i'\) are generated in \(y_{i{'}}\) according to Definition 1 or 2, and in \(x_{i'k}\) according to Definition 3, \(k \in D\).

(5–6)
“Meanrow" and “Variancerow" cases: simultaneous presence of conditional outliers (in mean or in variance) and rowwise outliers, where the outliers \(i'\) are generated in \(y_{i{'}}\) according to Definition 1 or 2, and in \(x_{i'k'}\) according to Definition 4, \(k' \in {\tilde{D}}\).

(7–8)
“Meanleveragerow" and “Varianceleveragerow" cases: simultaneous presence of conditional outliers (in mean or in variance), leverage outliers and rowwise outliers, where the outliers \(i'\) are generated in \(y_{i{'}}\) according to Definition 1 or 2, in \(x_{i'k}\) according to Definition 3, \(k \in D\), and in \(x_{i'k'}\) according to Definition 4, \(k' \in {\tilde{D}}\).

(9)
"Leveragerow" case: simultaneous presence of leverage outliers and rowwise outliers, where the outliers \(i'\) are generated in \(x_{i'k}\) according to Definition 3, \(k \in D\), and in \(x_{i'k'}\) according to Definition 4, \(k' \in {\tilde{D}}\).
The technical derivations of expected perturbations in the expected value and in the variance of \(\epsilon _i\) are reported in supplement Sect. 1 for each outlier scheme. They allow us to define \(O_{all}=O_{y}\) for the cases where outliers in the response variable coexist with only one other type of outlier in the remaining variables (i.e., scenarios from 1 to 6), \(O_{all}=O_{y}+O_{lev}\) for cases where all three types of outliers coexist (scenarios 7–8), \(O_{all}=O_{lev}\) for the case where rowwise and leverage outliers occur without any outlier in the response variable (scenario 9). We then define \(\alpha =\vert O_{all} \vert \), where \(O_{all}\) is the set of actual conditional outliers to be recovered. Note that leverage outliers become conditional outliers under cases 7–9 (i.e., all “...leveragerow" cases), due to the contemporaneous presence of rowwise outliers and outliers in the predictors.
In the end, we can derive the expected overall SignaltoNoise Ratio \(SNR=\sqrt{ESS/RSS}\) of model (2) under the different cases, where ESS, the expected Explained Sum of Squares, is constant across cases and equal to \(ESS=n\mathbf {\beta }'\textbf{R}_{D} \mathbf {\beta }=n\sum _{k=1}^K \beta _k[\beta _k+\textbf{R}_{kk'}\sum _{k' \ne k}\beta _{k'}]\), and the formula for RSS, the expected Residual Sum of Squares, varies under each case (see Supplement Sect. 1). We highlight that ESS depends on the coefficient vector \(\mathbf {\beta }\) (signs and magnitudes), the covariance matrix of predictors \(\textbf{R}_{X}\), and the number of predictors K, while RSS may depend on the outlier proportion \(\alpha \), the outlyingness parameter m, the intercept a, the residual variance \(\sigma ^2\), the number of predictors K, the perturbations occurred in the predictors, and the coefficient vector \(\mathbf {\beta }\), according to the underlying outlier scheme.
4.2 Methodology
We present here our proposed conditional outlier detection method, ROBOUT, consisting of five robust steps, namely, preprocessing (Sect. 4.2.1), preliminary imputation (Sect. 4.2.2), variable selection (Sect. 4.2.3), lowdimensional regression (Sect. 4.2.4) and conditional outlier detection (Sect. 4.2.5). Approximate normality is needed both in the target variable and in potential predictors, to make a linear model meaningful and to rely on residual diagnostics tools. The preliminary imputation procedure is needed to prevent anomalous points in potential predictors from corrupting predictor recovery, which is then performed via SNCD, applied to the robustly imputed dataset. In the end, a lowdimensional regression method like MM, which is doubly robust with respect to y and \(\textbf{X}_{D}\), is applied to spot conditional outliers. Note that ROBOUT, unlike SPARSELTS and RLARS, separates the steps of variable selection and conditional outlier detection, because this leads to a better performance for predictor selection and conditional outlier identification, as it is shown by our simulation study in Sect. 5.
4.2.1 Preprocessing
A prescreening based on a robust correlation measure aimed at excluding one of two very correlated variables (say, more than 0.8 in absolute value) is first performed to detect nearly identities (see also Sect. 6) and to annihilate the impact of possibly masked bivariate outliers (see Sect. 4.2.2). For this purpose, we calculate robust pairwise correlations \({\hat{\varrho }}_{j{'}j{''}}\) for each pair \(j{'}j{''}\) of variables, \(j{'},j{''}=1,\ldots ,p\), \(j{'} \ne j{''}\). One can consider Spearman’s rho (Spearman 1904), Kendall’s tau (Kendall 1938), or Huber correlation (as employed in Khan et al. 2007), or other robust measures (see Raymaekers and Rousseeuw 2021).
If the model is intrinsically nonlinear, polynomial effects could be included in the data matrix if known. Otherwise, as suggested by Rousseeuw and Bossche (2018), one can operate a preprocessing of the single variables in \(\textbf{X}\) by applying a nonlinear transformation (like BoxCox, YeoJohnson, etc.) able to ensure at least the symmetry of predictors, in order to avoid the insurgence of nonlinear effects. The same holds a fortiori for the target variable \(\textbf{y}\), for which residual normality must hold in order to justify residual diagnostic analysis. In the empirical application of Sect. 6, applying the logarithmic transform is enough to obtain a normal y and a valid linear model. In general, the R function bestNormalize may be of great help.
If the variables in the dataset are intrinsically nonsymmetric, then different specific procedures apply to determine the relevant outlyingness cutoffs (see for instance Rousseeuw and Hubert 2018). We do not include this case in the present paper.
4.2.2 Preliminary imputation
Let us consider the \(n \times p\) matrix of potential predictors \(\textbf{X}\). First, we identify univariate outliers in that matrix. We calculate the median and the unbiased median absolute deviation of each variable. This means that, for each \(j=1,\ldots ,p\), we compute \(x_{med,j}=\textrm{med}(x_j)\) and \(x_{mad,j}=1.4826 \textrm{mad}(x_j)\).^{Footnote 2} Then, relying on \(x_{med,j}\) and \(x_{mad,j}\), we derive a robust zscore for each entry ij of \(\textbf{X}\): \(z_{ij}=\frac{x_{ij}x_{med,j}}{x_{mad,j}}\), \(i=1,\ldots ,n\), \(j=1,\ldots ,p\). In the end, we flag as outliers the entries that present an outlying \(z_{ij}\), i.e. we set \(w_{ij}=0\) if \(\vert z_{ij} \vert > \phi _z^{1}(1{\tilde{\delta }}/2)\), where \(\phi _z^{1}\) is the inverse standard normal distribution function and \({\tilde{\delta }}\) is the significance level of the outlyingness test, usually set in a first run as 0.01 or 0.05, and \(w_{ij}=1\) otherwise.
Second, we impute values to the previously identified univariate outliers, utilising the information present in the whole matrix of potential predictors. Then, for each entry ij, \(i=1,\ldots ,n\), \(j=1,\ldots ,p\), we apply Algorithm 1.
Let us explain Algorithm 1 in more detail. We first identify univariate outliers in each variable by robustified zscores with median and MAD. For any outlying cell (i, j), we use the robust correlation measure \({\hat{\varrho }}_{j{j{'}}}\), \(j{'}\ne j\), to obtain a ranking of the most correlated variables with \(x_{j}\). We consider the first one in the ranking such that \(w_{ij{'}}=1\), and we call it \({\tilde{j}}\). We derive the set of nearest neighbors \(I_{ij,MAD}\) to be used for imputation by including all the points \(i{'}\) presenting \(w_{i{'}{\tilde{j}}}=1\) with \(x_{i{'}{\tilde{j}}}\) within one MAD from \(x_{i{\tilde{j}}}\). At this stage, since bivariate outliers may impact on the predictor set choice, we calculate the Mahalanobis distances of any point \(i{'} \in I_{ij}^{MAD}\) by means of the vector of robustified scores \(\textbf{z}_{i{'},j{j{'}}}\) and the estimated covariance matrix \(\widehat{\textbf{R}}_{j{\tilde{j}}}=\left[ \begin{array}{cc} 1 &{} {\hat{\varrho }}_{j{j{'}}}\\ {\hat{\varrho }}_{j{j{'}}} &{} 1 \end{array} \right] \) and we identify the set of bivariate outliers \(I_{ij}^{BIV}\) with a significance level equal to \(\delta \), typically fixed to 0.01 or 0.05. In the end, we derive the set \(I_{ij}^{NB}=I_{ij}^{MAD} \backslash I_{ij}^{BIV}\) and we obtain the imputed value \(x^{*}_{ij}\) as the median value \(\textrm{med}(\textbf{X}_{I_{ij}^{NB}j})\).
Three relevant features of Algorithm 1 need to be stressed. First, the computational cost is a direct function of \(\sum _{i=1}^n\sum _{j=1}^p\mathbbm {1}(w_{ij}=0)\), that represents the degree of dataset perturbation. Second, Algorithm 1 exploits the multicollinearity structure of a large cross section, such that a large p and a rich multicollinearity structure are actually improving the imputation procedure systematically. Third, the procedure adapts the number of neighbors to the distribution of the closest related variable, by allowing all the points within one MAD in the neighbor set. The overall computational cost of the imputation step is thus proportional to \(O(pn\log n)\) as p and n diverge.
At this stage, it may still be objected that Algorithm 1 may miss a fraction of bivariate outliers, namely, all bivariate outliers \(\textbf{x}_{i,j'j''}\) such that both \(w_{ij'}=1\) and \(w_{ij''}=1\). Although this occurs very rarely in practice, it is indeed true, and the size of this set of masked bivariate outliers is actually enlarging if \(\textrm{sgn}(x_{ij'})=\textrm{sgn}(x_{ij''})\) and \(\varrho _{j'j''}\) is strongly negative, or if \(\textrm{sgn}(x_{ij'})\ne \textrm{sgn}(x_{ij''})\) and \(\varrho _{j'j''}\) is strongly positive.
The practical remedy for this potential drawback is to reapply Algorithm 1 with larger levels \({\tilde{\delta }}\) and \({\delta }\). In particular, it can be noted that setting \({\tilde{\delta }}=0.1\) is enough to obtain a bivariate outlyingness test level \(\delta =0.01\) when \(\textrm{sgn}(x_{ij'})=\textrm{sgn}(x_{ij''})\) and \(\varrho _{j'j''}=0.4125\) (or \(\textrm{sgn}(x_{ij'})\ne \textrm{sgn}(x_{ij''})\) and \(\varrho _{j'j''}=0.4125\)). This means that if the predictor set retrieved in Sect. 4.2.3 changes abruptly when \({\tilde{\delta }}\) is lower than a certain level, one should stop the descent of \({\tilde{\delta }}\) at the previous value. We suggest trying \({\tilde{\delta }}=\delta =0.1,0.05,0.01\) and verifying the stability of the recovered predictor set, as we do in Sect. 6. In the supplement, at the end of Sect. 6, we have devoted a specific experiment to this limit case.
Concerning potential outliers in more than two dimensions, we can note that their impact on predictor selection by LASSOpenalized robust regression (see Sect. 4.2.3) is actually negligible, because Algorithm 1 preserves the correct estimation of pairwise correlations, which is crucial to solve iteratively reweighted least squares problems like (3). For this reason, Algorithm 1 is effective for the purpose of preserving the true predictor recovery in the presence of outliers such as those of Definitions 3 and 4, under which SPARSELTS and RLARS break down (see Sect. 5). In the robust regression step (see Sect. 4.2.4), a doubly robust method like MM is employed, so that those points can eventually be recovered as conditional outliers in y.
Given that we allow for the presence of bivariate outliers, we should wonder if the three considered robust correlation measures, Spearman, Kendall and Huber, are sensitive to bivariate outliers and how much. This additional simulation study is reported in Supplement Sect. 4. We discover that, among the three measures, Kendall correlation is the most biased, followed by Spearman and Huber. At the same time, once fixed \(R_{ij}=\varrho ^{\vert ij \vert }\), even Kendall correlation correctly selects the most correlated variable almost always, unless \(\varrho \) is very small, like the other two measures. This means that, even using a correlation measure which is nonrobust to bivariate outliers, the identification of the closest variable is not affected. This is why in the simulation study of Sect. 5 we employ Kendall correlation, thus proving that the most biased correlation measure among the three still produces excellent results for ROBOUT, although the calculation of \(\chi \)score at step 5 may be biased. What is more, a large \(\varrho \) enlarges the set of possibly masked bivariate outliers, but at the same time ensures the selection of the closest variable via any correlation measure (although not robust to bivariate outliers) under any p/n ratio.
Algorithm 1 is also equipped with systematic protections against misimputation. In particular, step 4 selects the neighbor set as the one containing all points within one MAD in the closest variable, in order to prevent overfitting. Then, the minimum neighbor set size \(\zeta \) can further control at step 7 that the neighbor set is large enough to avoid overfitting. Throughout the paper we set \(\zeta =3\), but \(\zeta \) may also be selected by crossvalidation. A large p increases prediction informativeness (thus decreasing imputation bias), and a large n prevents overfitting (thus decreasing imputation variance). Increasing both p and n concurs to prevent any point to be imputed by the general median of \(x_j\), which occurs at step 3 if no close variables are available for imputation. That situation is very unlikely to occur, unless p and n are really small. For these reasons, our method is a large p large n one, where multicollinearity actually improves imputation accuracy, because it increases the probability to select the closest variable in step 2. We stress that Algorithm 1 analogously functions when applied to missing data entries.
Alternative methods include employing a correlation measure proved to be robust to any bivariate outlier, like the one in Raymaekers and Rousseeuw (2021), or computing a fast MCD as proposed in Riani et al. (2015). These two alternatives would not help anyway overcome the current limitations of ROBOUT, which require to have less than 50% univariate outliers per variable, or bivariate outliers per variable pair, and require that all points have enough variable values to exploit for a safe imputation, which means that the maximum number of variables involved in outliers per each row must be limited, and enough close points for imputation must be present (that is, we need approximately symmetric distributions). This is the reason why randomly scattered outliers in the style of Rousseeuw and Bossche (2018) and Bottmer et al. (2022) are not well managed by Algorithm 1. We leave to future studies to modify Algorithm 1 to overcome these limitations, in order to compare theoretically and empirically its performance to the shooting Sestimator proposed by Bottmer et al. (2022).
4.2.3 Variable selection
Starting with our target response variable \(y_i\), \(i=1,\ldots ,n\), we aim to consistently select the relevant set of its predictors from a large set of variables under the presence of conditional outliers in mean or variance. Consequently, we aim to identify a model such as (2) from the data. Importantly, the focus of this step is on the selection of predictors rather than on the estimated coefficients and residual scale. This is a core idea of ROBOUT, i.e. that a robust predictor selection step is distinguished from the robust regression step, as this renders the method more reliable in challenging situations compared to the existing methods that combine these two steps.
Predictor retrieval receives as input the dataset imputed by Algorithm 1 without further standardization or normalization.^{Footnote 3} Two options are considered for this step. The first is based on Yi and Huang (2017):
where \(\epsilon _i=y_i\beta _0\sum _{j=1}^p \beta _j x_{ij}^{*}\), \(i=1,\ldots ,n\), \(j=1,\ldots ,p\), \(\lambda \) is a penalization parameter, and
is the Huber weight function (Ronchetti and Huber 2009), where \(\theta \) is a tuning parameter. Henceforth, we call (3) SNCDH objective function.
SNCDH estimates an elasticnet penalized Huber loss regression, optimized by using the semismooth Newton coordinate descent algorithm presented in Yi and Huang (2017). SNCD has a computational cost proportional to O(pn), i.e. linear in both p and n. Weighting observations is precisely what renders the results robust in the face of conditional outliers in y, because it annihilates their influence.
The other robust alternative that we consider in the simulation study substitutes \(\rho _{H,\theta } (\epsilon _i)\) in (3) with the absolute loss \(\rho _L(\epsilon _i)\), where
We call that version the SNCDL objective function. SNCDL estimates an elasticnet penalized absolute loss regression, optimized in the same way.
The minimization of (3) with \(\rho _{H,\theta } (t)\) or \(\rho _{L} (t)\) is practically performed for a decreasing sequence of values \(\lambda =\lambda _t\), \(t=1,\ldots ,T\), such that \(\lambda _0=\lambda _{max}\) and \(\lambda _T=\lambda _{min}\), where \(\lambda _{max}\) returns no predictors, and \(\lambda _{min}\) returns \(K_{max}\) predictors (we set \(K_{max}=[p/10])\). Predictor selection is performed by the adaptive version of the strong rule of Tibshirani et al. (2012), proposed in Yi and Huang (2017). At each value of \(\lambda _t\), that rule exploits the coefficient estimates at \(\lambda _{t1}\). The optimal \(\lambda \) is then selected by crossvalidation, using as OutOfSample (OOS) metric the loss \(\rho _{H,\theta }(t)\) for SNCDH or the loss \(\rho _{L}(t)\) for SNCDL.
Following Friedman et al. (2010), two optimal values of \(\lambda \) can be selected. The value \(\lambda _{0SE}\) is the one returning the minimum OOS loss \(\rho ^{min}_{H,\theta }\) or \(\rho ^{min}_{L}\) along the sequence \(\lambda _t\). The value \(\lambda _{1SE}\) is the maximum value along the sequence \(\lambda _t\) such that the OOS metric \(\rho _{H,\theta }\) or \(\rho _{L}\) stands within one standard error by \(\rho ^{min}_{H,\theta }\) or \(\rho ^{min}_{L}\). In this application, we use \(\lambda _{min}=\lambda _{0SE}\), and then we apply a screening rule on the selected predictors by trimming the relative estimated coefficients \(\widehat{\mathbf {\beta }}_{min}\) as follows: for SNCDH, or for SNCDL (\(\delta \) is typically set to 0.01 or 0.05). The predictor indices corresponding to the nonnull entries of constitute the set \({\widehat{D}}\) of selected predictor indices, and the estimate of K is the size of \({\widehat{D}}\), \({\widehat{K}}=\vert {\widehat{D}} \vert \). This screening process is able to systematically avoid redundancy in the predictor set, which is typical of existing procedures such as SPARSELTS and RLARS, by killing irrelevant predictors while keeping the right predictors contained in D into \({\widehat{D}}\).
4.2.4 Robust regression
Once identified in the previous step the predictors of y in the matrix \(\textbf{X}_{{\widehat{D}}}\), we apply a robust lowdimensional regression method in order to robustly estimate coefficients, residuals, and residual scale. To this aim, we utilise MM regression (Yohai 1987). In the simulation study, we also test the performance of Least Trimmed Squares (LTS) (Rousseeuw 1984). Concerning computational cost, we know that both LTS and MM share a burden of O(n) operations, due to the use of FastLTS (Rousseeuw and Van Driessen 2006) for LTS, and of FastS (SalibianBarrera and Yohai 2006) for the initial scale in MM. Residual indipendence, homoscedasticity, and normality must be appropriately verified by residual diagnostics, and must hold in the recovered nonoutlier set. At the end of this step, we get the estimated \({\widehat{K}} \times 1\) vector of coefficients \(\widehat{\mathbf {\beta }}\), the estimated \(n \times 1\) vector of residuals \(\widehat{\mathbf {\epsilon }}\), and the residual scale estimate \({\widehat{\sigma }}\).
4.2.5 Outlier detection
As a last step, we recover the vector \({\widehat{O}}\) of conditional outlier indices as the set of all the points \(i \in \{1,\ldots ,n\}\) with robustly rescaled residuals \({\widehat{\epsilon }}_i/{\widehat{\sigma }}\) larger than \(\phi _z^{1}(1\delta /2)\) in absolute value, where \(\delta \) is typically set to 0.01 or 0.05 (see for instance Atkinson and Riani 2000; Rousseeuw and Van Driessen 2006; Rousseeuw and Bossche 2018; Alfons et al. 2013, or Khan et al. 2007).
5 A comparative simulation study
In this section, we conduct simulation experiments aiming both to compare the performance of ROBOUT to competitor methods but also to identify the optimal design of ROBOUT with respect to its constituent components. The competitor methods against which the ROBOUT versions are tested are SPARSELTS and RLARS.^{Footnote 4}
5.1 Parameter settings
We test the performance of the four variants of the ROBOUT methodology, distinguished on the basis of the variable selection and robust regression estimation options, as presented in Sect. 4.2. Specifically, in the first step, either the SNCDH or the SNCDL objective function can be used, while in the second step either LTS or MM can be used to estimate the regression equation. We call the ensuing four variants of ROBOUT as H+LTS, L+LTS, H+MM and L+MM, where H and L refer to SNCDH and SNCDL, respectively. We provide detailed information about the parameterisation of the various methods in the Sect. 2 of the supplementary material.
Both competitors are run with the default preprocessing step for all the potential predictors: unitnorm normalization for SPARSELTS, robust standardization for RLARS. The intercept is included for all estimations, and the levels \(\delta \) and \({\tilde{\delta }}\) are fixed to 0.01. Note that, according to the above settings, H+MM, H+LTS, and SPARSELTS present a breakdown point equal to 25%, L+MM, L+LTS, and RLARS present a breakdown point equal to 50%.
5.2 Data settings
To fully define the scenarios that we use in the simulation study, we complement the nine outlier schemes presented in Sect. 4 with specifications of the dataset dimensions and parameters.
The set of predictors D is randomly generated with \(K=3\). The intercept is \(a_i=10\) for all \(i=1,\ldots ,n\). The coefficients are generated in the following way: \(\beta _1,\beta _3 \sim U(15,20)\), \(\beta _2 \sim U(20,15)\), with residual variance \(\sigma ^2=1\). According to the definitions provided in Sect. 4.1, we generate the data from model (1) for \(m=1,5,9,13,17,23,29,37,45,55\), where m represents the outlyingness parameter (the larger it is, the more perturbed the data setting). In practice, we generate leverage outliers \(i{'}\) as \(\textbf{x}_{D,i{'}}\sim MVN(\textbf{m},\textbf{R}_D/m)\), where \(\textbf{m} \in {\mathbb {R}}^{K}\), \(\vert m_k \vert \gg 0\), \(k=1,\ldots ,K\), to obtain some (small) variability among outliers.
Each scenario is further defined by the relative size of the matrix (i.e., the p/n ratio). We examine four different settings for the dimensions and the ensuing p/n ratio of the dataset: “Very fat" (VF), with \(p=300\), \(n=60\) (i.e. \(p/n=5\), \(p\gg n\)); “Fat" (F), with \(p=200\), \(n=100\) (i.e. \(p/n=0.5\), \(p<n\)); “Tall" (T), with \(p=100\), \(n=200\) (i.e. \(p/n=2\), \(p>n\)); “Very tall" (VT), with \(p=60\), \(n=300\) (i.e. \(p/n=0.2\), \(p\ll n\)). As suggested by Alfons (2021), we set the \(p \times p\) covariance matrix \(\textbf{R}_\textbf{X}\) as \(\textbf{R}_{j{'}j{''}}=0.5^{\vert j{'}j{''} \vert }\), \(j{'},j{''}=1,\ldots ,p\), \({j{'} \ne j{''}}\).
The rationale for differentiated scenarios with respect to the p/n ratio is that relative dimensions affect the performance of the various methods that are tested, for example by impacting on the effectiveness of the selection of predictors.
The contamination rates \(\alpha _{y}\), \(\alpha _{lev}\), and \(\alpha _{row}\) and the number of variables in the set of perturbed predictors \({\tilde{D}}\), \(\vert {\tilde{D}} \vert \), vary according to the outlier scheme and the dimension setting. The general idea behind the choice of \(\alpha _{y}\), \(\alpha _{lev}\), \(\alpha _{row}\) and \(\vert {\tilde{D}} \vert \) is that a larger contamination rate is expected as n increases because more reasons for outlying behaviour may materialise (e.g., presence of subpopulations generated by a different statistical process). More details about the contamination rates and the signaltonoise ratios across scenarios are provided in the Supplementary material (Sect. 3).
Henceforth, we refer to scenarios by combining the name of the outlier scheme, e.g., “meanrow" and the dimensional setup, e.g., “tall" dataset. Each scenario is run 100 times for each value of m and the various performance metrics, presented in the next section, are calculated by averaging across these 100 replications.
5.3 Performance metrics
Performance metrics, as stated in the introductory section, pertain to three different dimensions: outlier detection, predictor recovery, and predictors’ coefficient estimation.
Let us define \(D_{k}\) as the kth predictor in the set D and \(\backslash \) as the set difference. To measure the performance with respect to predictor recovery, we calculate for each scenario, each value of parameter m, each outlier detection method, and each iteration \(r=1,\ldots ,100\), the set of recovered predictors \({\widehat{D}}^{(r)}\). \({\widehat{D}}^{(r)}\) could be in general either a subset, superset or nonoverlapping set with respect to D, ideally however the two sets should be identical. Consequently, we calculate the following metrics:

masked predictor rate, \(MPR_r={K}^{1}{\vert D \backslash {\widehat{D}}^{(r)} \vert }\), which is the false negative rate of recovered predictors;

swamped predictor rate, \(SPR_r={\vert {\widehat{D}}^{(r)}\vert }^{1}{\vert {\widehat{D}}^{(r)}\backslash D \vert }\), which is the false positive rate of recovered predictors;

recovered predictor ratio, \(RPR^{(r)}={K}^{1}{\vert {\widehat{D}}^{(r)} \vert }\), which is a measure of the propensity to recover irrelevant predictors;

true predictor rate, defined as \(TPR^{(r)}=\mathbbm {1}(MPR^{(r)}=0)\), which is the predictor set recovery rate;

true predictor rate for each predictor \(D_k\), \(k=1,\ldots ,K\), defined as \(TPR_{k}^{(r)}=\mathbbm {1} (D_{k}\in {\widehat{D}}^{(r)})\);

adjacent predictor rate, defined as \(APR^{(r)}=1\) if \({K}^{1} \sum _{k=1}^K \mathbbm {1}(D_{k}\in {\widehat{D}}^{(r)} \vert (D_{k}1)\in {\widehat{D}}^{(r)} \vert (D_{k}+1) \in {\widehat{D}}^{(r)})=1\), \(APR^{(r)}=0\) otherwise, which is a measure of the propensity to recover wrong models having strongly correlated predictors with the true ones.
To quantify the performance of coefficient estimation, we obtain estimated intercept \({\widehat{\beta }}_{0}^{(r)}\) and coefficient vector \(\mathbf {{\widehat{\beta }}}^{(r)}\), and we calculate:

for each coefficient \(\beta _k\), \(k=1,\ldots ,K\), the coefficient squared error, defined as \(CSE_{k}^{(r)}=({\widehat{\beta }}_{k}^{(r)}\beta _k)^2\) if \(TPR_{k}^{(r)}=1\), \(CSE_{k}^{(r)}=0\) if \(TPR_{k}^{(r)}=0\);

the intercept squared error, \(ISE^{(r)}=({\widehat{\beta }}_{0}^{(r)}\beta _0)^2\).
The two indicators are then averaged as follows:

the average root relative coefficient squared error, \(RCSE=K^{1}\sum _{k=1}^K\vert \beta _k \vert ^{1}\sqrt{(\sum _{r=1}^R TPR_{k}^{(r)})^{1}{\sum _{r=1}^R CSE_{k}^{(r)}}}\);

the average root relative intercept squared error, \(\textit{RISE}={\vert \beta _0 \vert }^{1}{\sqrt{{R}^{1}{\sum _{r=1}^R \textit{ISE}^{(r)}}}}\).
To assess performance with regard to conditional outlier detection, we derive for each iteration \(r=1,\ldots ,100\) the set of recovered outliers \({\widehat{O}}^{(r)}\). Then, we calculate the following performance metrics:

outlier rate ratio, defined as the ratio \(OR^{(r)}={[\alpha n]}^{1}{\vert {\widehat{O}}^{(r)}\vert }\);

zerooutlier indicator, defined as \(OZ^{(r)}=1\) if \(\vert {\widehat{O}}^{(r)}\vert =0\), 0 otherwise;

the masking rate \(MR^{(r)}\), defined as the proportion of masked outliers (i.e. false negatives) over the true number of outliers: \(MR^{(r)}={[\alpha n]}^{1}{\sum _{i\in O} \mathbbm {1}(i \not \in {\widehat{O}}^{(r)})}\);

the swamping rate \(SR^{(r)}\), defined as the proportion of swamped outliers (i.e. false positives) over the number of recovered outliers: \(SR^{(r)}={\vert {\widehat{O}}^{(r)}\vert }^{1}\sum _{i \in {\widehat{O}}^{(r)}} \mathbbm {1}(i \not \in {O})\);

the \(F_1\) score, defined as \(2\frac{PREC^{(r)} \times REC^{(r)}}{PREC^{(r)}+REC^{(r)}}\), where the precision \(PREC^{(r)}\) is equal to \(PREC^{(r)}=1MR^{(r)}\), and the recall \(REC^{(r)}\) is equal to \(REC^{(r)}=1SR^{(r)}\), which is an overall performance measure incorporating both masking and swamping effects.
The above measures are averaged across the 100 replications of each scenario and value of m for each method, and their standard deviation is calculated where appropriate.
5.4 Simulation results
Our simulations, presented in this and the next Sect. 5.5 and complemented by the supplementary material, show that the ROBOUT approach compares favourably to the competitors, as it is not plagued by the systematic predictor redundancy of RLARS and the systematic suboptimal estimation of regression coefficients of SPARSELTS. We also find that overall H+MM is the most reliable ROBOUT option because it is expected to recover predictors, estimate coefficients and identify outliers in a systematically better and more stable way than competitors. In other words, the optimal ROBOUT version utilises the MM regression as the final robust regression step, similarly to RLARS, while modifying RLARS predictor selection procedure to be robust to multivariate outliers and parsimonious.
In more detail, under all scenarios ROBOUT performs better than SPARSELTS and RLARS with respect to coefficient estimation (see Table 1). The competitor methods exhibit relatively high values of RCSE; SPARSELTS even exceeds \(60\%\), especially as p/n decreases, i.e., as the dataset becomes “taller", and as the outlier scheme becomes more complex, e.g., in the case of the outlier scheme 7. RLARS faces problems in avoiding the inclusion of irrelevant predictors, presenting RPR much larger than 1 (see Fig. 2 and also Sects. 6 and 7 in the supplementary material). The above patterns hold both for scenarios with conditional outliers in mean and in variance.
Concerning outlier detection, the performance of ROBOUT is clearly superior in the more complex outlier scheme “mean/varianceleveragerow" (schemes 7 and 8), see, e.g., Fig. 3. SPARSELTS is doing systematically worse than competitors when conditional outliers in variance are present (see the results for the outlier schemes 2, 4 and 6 in Table 2). RLARS faces challenges in recovering the true predictors and the same holds for SPARSELTS when \(p/n<1\) for “tall" and “very tall" datasets (see the supplementary material for the respective TPR scores).
We also observe a difference between the ROBOUT versions SNCDH and SNCDL, in that the former is slightly more parsimonious than the latter in predictor recovery for the “very fat" dataset as reflected in the RPR metric (see Sects. 6 and 7 in the supplementary material). Concerning outlier detection, a difference between MM and LTS emerges, that is, the LTS options of ROBOUT are systematically more erratic when the “very fat" dataset is used (see Table 2).
Furthermore, under scenarios featuring the outlier scheme “leveragerow" (case 9), SPARSELTS and RLARS do not perform well under the “very fat" case and their performance improves as the dataset becomes “taller". Finally, all ROBOUT options are more effective than competitors also in the absence of outliers.
We now report in detail the results for scenarios featuring outliers in mean (schemes 1, 3, 5 and 7), which provide a representative overview of the relative performance of the various methods. We refer to supplementary material for the results of all other scenarios.
5.5 Scenarios with outliers in mean
We first examine the cases where a maximum two types of outliers are present, one of them in being outliers in mean for the response variable. Specifically, this applies for the outlier schemes 1, 3 and 5. A representative sample of these findings is shown in Fig. 4, for the scenario with the “very fat" dataset and the “meanleverage" outlier scheme. In these scenarios and irrespective of the p/n dimensions, the average of TPR is around 1 for all methods, meaning that all methods perform satisfactorily with respect to predictor identification in these relatively simple scenarios. Concerning outlier detection, the \(F_1\) score of all methods is very close to 1, however we note that H+LTS and L+LTS are slightly more erratic than the competitors, reflected in their somewhat elevated standard deviation across iterations.
In general and especially as the p/n ratio increases, the performance of ROBOUT stands out compared to the competitor methods, which face challenges regarding predictor recovery and coefficient estimation. This is shown clearly in Fig. 4 where the “very fat" dataset is used. These challenges are caused by the relatively high dimensions of the potential predictors set. For example, SPARSELTS faces severe challenges as regards the coefficient estimation, performing systematically worse than other methods with respect to the RCSE and RISE indicators. The SPARSELTS estimated coefficients are structurally worse: on average, RISE is around \(4\%\) for SPARSELTS, while all the other methods stand below \(1\%\). Furthermore, RLARS performs worse than the other methods with regard to predictor recovery: the RPR indicator for RLARS is well beyond 1. Finally, for high values of m RLARS often crashes, due to multicollinearity issues.
In the less extreme case of a “fat" dataset, these challenges to the competitors are attenuated, however ROBOUT remains overall the preferred choice. The metrics for the scenario with a “fat" dataset and a “meanrow" outlier scheme are shown in Fig. 5. On the one hand, the RPR of SPARSELTS converges to 1, the instability of \(F_1\) score for H+LTS and L+LTS disappears, and the patterns of RCSE and RISE are much smoother. However, the RPR of RLARS keeps well beyond 1, and the relative standings of RCSE and RISE maintain the same pattern, with SPARSELTS standing around \(4\%\) and the other methods below \(1\%\). For “tall" and “very tall" datasets, the patterns remain qualitatively similar.
We now examine the more challenging scenarios featuring all three types of outliers, i.e., the “meanleveragerow" outlier scheme (case 7). In the “very fat" dataset case, predictor recovery is suffering more than under the simpler schemes 1, 3 and 5 examined previously (see Fig. 6). The ROBOUT versions SNCDH and SNCDL are consistently the best, in all aspects. First, both the two competitor methods underperform with respect to predictor recovery. For small values of m, SPARSELTS is not effective, exhibiting values of TPR below 0.8, even if the performance improves for larger m. RLARS performs even worse with values of the TPR persistently low, irrespective of the degree of perturbation.^{Footnote 5} Similarly, as regards the coefficient estimation, we observe that RLARS is standing around \(20\%\) for RCSE, while the performance of SPARSELTS improves as m increases, paralleling the Huber options of ROBOUT, which are the most stable. The competitor methods are also worse than ROBOUT with respect to the outlier detection, as can be seen clearly in the \(F_1\) score. Based on this metric, RLARS exhibits by far the worst performance, due to a masking rate which stabilizes around \(20\%\). SPARSELTS starts from a masking and swamping rate above 15% and its performance improves when m increases. The MMbased versions of ROBOUT are the best performing because the versions using the LTS option present a small but systematic masking rate (around \(5\%\)). Overall, it is found that the H+MM option of ROBOUT is the most stable and reliable method when predictor selection, coefficient estimation and outlier detection are jointly considered.
In the milder case where the outlier scheme “meanleveragerow" (case 7) is combined with a “fat" dataset (see Fig. 7), the pattern of TPR does not qualitatively change compared to the “very fat" dataset case. Concerning predictor recovery, the RPR of SPARSELTS converges to Huber and LAD, while RLARS is still clearly the worse performing. Coefficient estimation metrics RCSE and RISE show that coefficients are not recovered well by SPARSELTS and RLARS, while all ROBOUT options are doing almost perfectly. The \(F_1\) score of all methods is now fine, apart from SPARSELTS and RLARS when m is small.
When the outlier scheme “meanleveragerow" (case 7) is combined with a “tall" dataset (see Fig. 8), SPARSELTS breaks down. Its TPR metric drops toward zero at \(m=23\), because it systematically recovers a contiguous predictor. Its RPR attains values as high as 20. At \(m=23\), its RCSE approaches 1, its masking rate 0.4, its swamping rate 0.25, and its \(F_1\) score is as low as 0.7. The other methods do not behave differently compared to the “fat" dataset case.
Under the “very tall" dataset, also RLARS crashes (see Fig. 9). Its TPR becomes close to zero as m approaches 13, descending faster than SPARSELTS. RPR remains very high (above 10) for SPARSELTS, and high for RLARS (around 3). It follows that their estimated coefficients are completely inconsistent. Both methods also present a masking rate above 40%, which leads to a very poor \(F_1\) score across m. We consider also nooutlier scenarios, that is when \(m=1\) is combined with the outlier scheme “only mean" (case 1). The results for this case are presented in the Supplementary material Sect. 5. Finally, the results of scenarios with outliers in variance are also presented in the Supplementary material Sect. 3 and are consistent with the ranking of methods with regard to their performance as it has been derived based on the scenarios examined in this section.
6 A real banking supervisory data example
In this section, we apply the ROBOUT conditional outlier detection procedure to a real dataset that contains granular data on the activities of the largest euro area banks, both on the asset and the liability side of their balance sheet. These data are submitted by the euro area banks to the European Central Bank in the context of their supervisory reporting requirements. The dataset in question is compositional, i.e. it includes some parent categories, like ‘Debt’, and their subparent categories like ‘Debt versus other banks’, ‘Debt versus central government’, etc. In addition, the dataset is very sparse, as not all banks are engaged in all the activities spanned by the granular set of variables. The reference date of the data is end2014.
Since the original scale of the variables is in the order of billions (expressed in euros), we apply to each entry of the data matrix \(\textbf{X}\) a logarithmic transformation of the following kind:
where \(i=1,\ldots ,364\) banks and \(j=1,\ldots ,771\) variables. This preliminary step is needed to render the distribution shapes symmetrical. In this way, the few variables with negative values, representing cost or loss items, are measured in the same scale of other variables, without neglecting the sign information. Then, we apply a variable screening based on robust Spearman correlation: we derive the list of all the variable pairs with Spearman correlation exceeding 0.8, and we exclude from the dataset the variable with smaller index. At the end of this procedure, 360 variables survive.
Our final data matrix presents \(79.23\%\) of zero entries, \(18.86\%\) of positive entries, and \(2.08\%\) of negative entries. The retained variables show a mean Spearman correlation of 0.089 and a mean absolute Spearman correlation of 0.171. The Spearman correlation matrix displays the presence of a very rich multicollinearity structure, whose pattern matches the order of the dataset variables. A rich negative correlation is especially present among the variables related to loans and receivables.
We set the log of bank’s size as the target variable on which we would like to identify conditionally outlying observations. The distribution of the log of banks’ size shows that the lognormality assumption on total assets cannot be rejected.^{Footnote 6} The fact that the size follows a lognormal distribution is intuitive, given the high variance, due to the existence of both large and small banks, reinforced by the size dispersion of home countries in the sample e.g. with respect to their GDP, and the nonnegativity of the size variable. The mean and the median of the logsize are almost equal (23.04 vs 23.14), while the standard deviation is slightly larger than the rescaled MAD (2.14 vs 1.73).
We apply the four variants of ROBOUT method plus the competitor methods SPARSELTS and RLARS on the final data matrix to identify conditional outliers. The application of the ROBOUT versions that utilise SNCDH and SNCDL is conducted as follows. First, we employ Spearman correlation in Algorithm 1 and we set \({\tilde{\delta }}=\delta =0.01\). For the selection of the predictors we experiment with several values of \(K_{\max }\): \(\{12,15,18,21,24,27,30\}\). We note that the are two stability areas, one with \({\widehat{K}}=6\) and the other with \({\widehat{K}}=9\), and we observe small differences between the predictor sets returned by SNCDH and SNCDL. We select \(K_{\max }=24\) (with \({\widehat{K}}=9\)) because all the coefficients estimated by MM and LTS are statistically significant, considering both predictor sets as returned by SNCDH and SNCDL. The adjusted \(R^2\) of the MM and the LTS models are both slightly larger when the SNCDH predictor set is used, compared to the SNCDL set. Consequently, we select as the preferred predictor set that estimated by H+MM and H+LTS, which is the same.^{Footnote 7} The nine predictors identified by the Huber method are reported in Table 3.
As a robustness test, we repeat the same exercise but we set \({\tilde{\delta }}=\delta =0.05,0.1\). Although a larger \({\tilde{\delta }}\) leads to a systematically more redundant model, due to the variable selection procedure of Sect. 4.2.3, we observe that the predictors recovered with \(\delta ={\tilde{\delta }}=0.01\) always appear when \({\tilde{\delta }}=\delta =0.05,0.1\), which is a confirmation that the predictor set recovered with \({\tilde{\delta }}=0.01\) is valid.
In contrast, when RLARS is applied, a substantially larger number of predictors is returned under all values of \(K_{\max }\). Similarly when SPARSELTS is used, also when setting \(\alpha =0.5\) or \(\alpha =0.9\) instead of \(\alpha =0.75\).^{Footnote 8} These findings seem to be consistent with what is observed in the simulation study under the scenarios with the outlier scheme “meanleveragerow", i.e., where the predictor recovery rate for SPARSELTS and RLARS were found to be clearly lower than that of ROBOUT while simultaneously their swamping rates was higher.^{Footnote 9}
The predictor set that we use includes deposits (related to trading), loans and advances, debt securities and trading items such as interest rate derivatives. These are all fundamental elements of the banks’ activities, therefore it makes economic sense that they are chosen as predictors for banks’ total assets. In addition, these are variables that are expected to appear more frequently in the balance sheets of larger and more sophisticated institutions, therefore these variables can discriminate banks with respect to their size.
The application of the ROBOUT method starts with the step comprising the procedure described in Sect. 4.2.2, applied across the 360 potential predictors. It is found that approximately one third of the variables presents no flagged outlying cells, and 36 variables out of 360 present more than 6 flagged cells (out of 364).
Table 4 reports the results of the MM regression estimated on the predictors reported in Table 3. All estimated coefficients are positive and strongly significant. We observe that extremely significant predictors are V5V8, that reflect the extent of trading activities (on the liabilities side). The nine predictors are all positively correlated, with an average Spearman estimated correlation of 0.4252. The heat map of the Spearman predictor correlation matrix is reported in Fig. 10. It is remarkable that \(63.83\%\) of entries in the \(364 \times 9\) predictor matrix is zero, with 64 of 364 rows being completely null.
Starting from the retrieved predictors displayed in Table 3, MM and LTS provide very similar estimated coefficients, and recover the same outliers, irrespective of whether \(\delta =0.01\) or \(\delta =0.05\). With a level \(\delta =0.01\), the recovered outliers are eleven, of which seven negative and four positive.
In Fig. 11, two recovered outliers on the right side of the distribution present a very low value of logassets, while most of the remaining outliers lie within the range of intermediate values for the logassets. While the former two clearly are reporting errors due to a wrong scale (expressed in 1000s of euros instead of simply euros), the other outliers correspond to banks with a disproportionately large or small value of logassets compared to the values of the predictors, i.e., conditional outliers. As reported in Table 5, recovered outliers are values with a systematically larger presence of zeros in the predictors compared to nonoutliers, that is, they are banks with a more sparsely populated balance sheet. The presence of more zero elements has a bearing on the ability of predictors to explain the observed amount of logassets. Upon closer examination, the outliers are mainly idiosyncratic banks such as public financing institutions, branches of investment banks, or clearing houses, presenting an anomalous value of total assets with respect to the values recorded in the main predictors of total assets.
Figure 12 shows residual diagnostic plots for the MM residuals to check the validity of the estimated model. We can see that the estimated residuals respect the assumptions of independence, homoscedasticity and normality, with the only exception of the two aforementioned strongly negative outliers. Similar plots are observed for the LTS residuals, which are reported alongside the table of LTS coefficients in the Supplementary material (Sect. 8).
7 Conclusions
In this paper, we propose a new conditional outlier detection method, called ROBOUT. ROBOUT is versatile and flexible, as it is able to robustly spot conditional outliers under many different perturbed conditions and combinations of sample size n and dimension p. ROBOUT works efficiently on datasets with many observations and variables and it is able to robustly select the most relevant predictors of any target variable. Importantly, ROBOUT is effective when the ratio p/n is larger than 1, multicollinearity is present, and potential predictors are perturbed.
ROBOUT presents two options to select relevant predictors, based on LASSOpenalized Huber (SNCDH) or the Least Absolute Deviation (SNCDL) loss, and two options to estimate a robust regression, namely, the LTS and the MM methods. In a comprehensive simulation study, we have tested perturbation conditions including conditional outliers in mean or in variance in the response variable, multicollinearity and multivariate outliers in the potential predictors, and we have considered cases when the dimension is both (much) smaller and (much) larger than the sample size. The simulation study shows that when the ROBOUT performance is compared to that of the competitors such as RLARS and SPARSELTS, the option SNCDH+MM is overall the most resilient with respect to predictor recovery and conditional outlier detection across all tested scenarios.
We also test ROBOUT in a large granular banking dataset containing several balance sheet indicators for euro area banks. We are able to robustly model the logsize of euro area banks through a set of predictors that includes loans, deposits, securities, and trading assets, and to identify the banks presenting anomalous values in the total assets conditional on the identified predictors. The recovered outliers constitute a set of idiosyncratic banks in comparison to the textbook prototype of traditional bank, comprising public financing institutions, local branches of investment banks, and clearing houses. ROBOUT may thus be a useful tool for bank supervisors, who need to spot hidden anomalies from raw balance sheet data.
Data availibility
The dataset used in Sect. 5 is confidential, subject to institutional constraints. For these reasons, it may not be made publicly available. However, the paper is accompanied by two R functions, cond_out_gen and robout, which can regenerate all the simulated contamination rates of this paper and compute all the possible variants of ROBOUT procedure. We have no conflict of interest to disclosure of any kind.
Notes
Our approach distinguishes between a response variable and the conditional information set while (Rousseeuw and Bossche 2018) scans the whole original dataset for cellwise outliers. This difference however can be reconciled by running the proposed ROBOUT method consecutively defining each column of the dataset as the response variable.
In R, the function mad automatically calculates the rescaled MAD \(x_{mad,j}\).
The preprocessing operations of Sect. 4.2.1 prevent the variables to have tremendously different scale. If the variables are of different nature, a preliminary robust standardization is adviced.
We also experimented with additional competitor methods, namely MMLASSO, ENETLTS and PENSEM, on the same simulation scenarios, but the results were clearly worse compared to the rest of the methods we tried, therefore we do not present the results for simplicity. More, we did not implement (Bottmer et al. 2022) because of computational cost, difficult parameter tuning, and different underlying assumptions.
When the APR metric is considered, which considers as successes also the cases where the recovered predictors are contiguous to the true ones, RLARS improves but does not attain values above the range 0.70.8.
The pvalue of the JarqueBera test is 46.72%.
The LTS options of ROBOUT return multicollinearity warnings when \(K_{\max }=12,15,18\), because the MCD covariance matrix is singular, due to the high number of predictors.
When the two components of the vector \(nsamp=c(500,10)\), representing subset size and number, are lowered, SPARSELTS may not work, due to the rank deficiency of covariance matrix in the subsamples.
From a robustness perspective it can be noted that when we run the outlier methods inserting a duplicate row in the \(364 \times 361\) data matrix the predictor sets retrieved by ROBOUT do not change, while SPARSELTS and RLARS still return a massive number of predictors. This suggests that ROBOUT is robust to possible duplications of specific records.
References
Atkinson AC, Riani M (2000) Robust diagnostic regression analysis, vol 2. Springer, New York
Atkinson AC, Riani M, Cerioli A (2004) Exploring multivariate data with the forward search, vol 1. Springer, New York
Atkinson AC, Corbellini A, Riani M (2017) Robust Bayesian regression with the forward search: theory and data analysis. Test 26:869–886
Alfons A, Croux C, Gelper S (2013) Sparse least trimmed squares regression for analyzing highdimensional large data sets. Ann Appl Stat 7(1):226–48
Alfons A (2021) RobustHD: an R package for robust regression with highdimensional data. J Open Source Softw 6(67):3786
Barnett V, Lewis T (1994) Outliers in statistical data. Wiley series in probability and mathematical statistics. Wiley, New York
Bottmer L, Croux C, Wilms I (2022) Sparse regression for large data sets with outliers. Eur J Oper Res 297(2):782–794
Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–99
Filzmoser P, Höppner S, Ortner I, Serneels S, Verdonck T (2020) Cellwise robust M regression. Comput Stat Data Anal 147:106944
Filzmoser P, Nordhausen K (2021) Robust linear regression for highdimensional data: an overview. Wiley Interdiscip Rev 13(4):e1524
Freue GVC, Kepplinger D, SalibiánBarrera M, Smucler E (2019) Robust elastic net estimators for variable selection and identification of proteomic biomarkers. Ann Appl Stat 13(4):2065–2090
Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33(1):1–22
Hawkins DM (1980) Identification of outliers. Chapman and Hall, London
Hong C, Hauskrecht M (2015) Multivariate conditional anomaly detection and its clinical application. Proc AAAI Conf Artif Intell 29(1):4239–4240
Ronchetti EM, Huber PJ (2009) Robust statistics. Wiley, Hoboken
Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1/2):81–93
Khan JA, Van Aelst S, Zamar RH (2007) Robust linear model selection based on least angle regression. J Am Stat Assoc 102(480):1289–99
Kurnaz FS, Hoffmann I, Filzmoser P (2018) Robust and sparse estimation methods for highdimensional linear and logistic regression. Chemomet Intell Lab Syst 172:211–222
Maronna RA (2011) Robust ridge regression for highdimensional data. Technometrics 53(1):44–53
Öllerer V, Alfons A, Croux C (2016) The shooting Sestimator for robust regression. Comput Stat 31:829–844
Raymaekers J, Rousseeuw PJ (2021) Fast robust correlation for highdimensional data. Technometrics 63(2):184–198
Riani M, Perrotta D, Cerioli A (2015) The forward search for very large datasets. J Stat Softw 67:1–20
Rousseeuw PJ (1984) Least median of squares regression. J Am Stat Assoc 79(388):871–80
Rousseeuw PJ, Van Driessen K (2006) Computing LTS regression for large data sets. Data Min Know Discov 12(1):29–45
Rousseeuw PJ, Hubert M (2018) Anomaly detection by robust statistics. Wiley Interdiscip Rev 8(2):e1236
Rousseeuw PJ, Leroy AM (1987) Robust regression and outlier detection. WileyInterscience, New York
Rousseeuw PJ, Bossche WVD (2018) Detecting deviating data cells. Technometrics 60(2):135–145
SalibianBarrera M, Yohai VJ (2006) A fast algorithm for Sregression estimates. J Comput Graph Stat 15(2):414–27
Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101
Smucler E, Yohai VJ (2017) Robust and sparse estimators for linear regression models. Comput Stat Data Anal 111:116–130
Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B 58(1):267–88
Tibshirani R, Bien J, Friedman J, Hastie T, Simon N, Taylor J, Tibshirani RJ (2012) Strong rules for discarding predictors in lassotype problems. J R Stat Soc B 74(2):245–266
Varian HR (2014) Big data: new tricks for econometrics. J Econ Perspect 28(2):3–28
Yohai VJ (1987) High breakdownpoint and high efficiency robust estimates for regression. Ann Stat 15:642–656
Yi C, Huang J (2017) Semismooth newton coordinate descent algorithm for elasticnet penalized huber loss regression and quantile regression. J Comput Graph Stat 26(3):547–557
Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc B 67(2):301–320
Acknowledgements
We gratefully thank the Supervisory Statistics Division of the European Central Bank for granting the permission to conduct research using the supervisory data of Sect. 5, and one associate editor and two anonymous reviewers, who relevantly improved the paper by their insightful and constructive comments.
Funding
Open access funding provided by Alma Mater Studiorum  Università di Bologna within the CRUICARE Agreement.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Farnè, M., Vouldis, A. ROBOUT: a conditional outlier detection methodology for highdimensional data. Stat Papers 65, 2489–2525 (2024). https://doi.org/10.1007/s00362023014923
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362023014923