PolieDRO: a novel classification and regression framework with non-parametric data-driven regularization

Abstract

PolieDRO is a novel analytics framework for classification and regression that harnesses the power and flexibility of data-driven distributionally robust optimization (DRO) to circumvent the need for regularization hyperparameters. Recent literature shows that traditional machine learning methods such as SVM and (square-root) LASSO can be written as Wasserstein-based DRO problems. Inspired by those results we propose a hyperparameter-free ambiguity set that explores the polyhedral structure of data-driven convex hulls, generating computationally tractable regression and classification methods for any convex loss function. Numerical results based on 100 real-world databases and an extensive experiment with synthetically generated data show that our methods consistently outperform their traditional counterparts.

Appendices

Appendix 1: Problem reformulation

Consider the problem formulation in (4). Naturally, this is not a solvable problem in its current form. We then write the problem’s Lagrangian formulation:

$$\begin{aligned} \begin{array}{ll} {\mathcal {L}}(P, \kappa , \lambda ) =&{} \displaystyle \int _{{\mathcal {C}}_0} h(w;\beta ) dP \\ &{}\displaystyle - \sum _{i \in {\mathcal {F}}}\ \big [ \big ( \underline{p_i} - \int _{{\mathcal {C}}_0}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}}dP\big ) \lambda _i \big ]\\ &{}\displaystyle - \sum _{i \in {\mathcal {F}}}\ \big [ \big ( \int _{{\mathcal {C}}_0}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}}dP - \overline{p_i} \big ) \kappa _i \big ]\\ \end{array} \end{aligned}$$

(22)

Reorganizing the terms, we can then write:

$$\begin{aligned} \begin{array}{ll} {\mathcal {L}}(P, \kappa , \lambda ) =&{} \displaystyle \int _{{\mathcal {C}}_0}\big [ h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i)\big ] dP \\ &{}\displaystyle - \sum _{i \in {\mathcal {F}}}(\lambda _i \underline{p_i} - \kappa _i\overline{p_i}) \end{array} \end{aligned}$$

(23)

The Lagrange dual function $g(\lambda , \kappa )$ can then be written as:

$$\begin{aligned} \begin{array}{ll} g(\lambda , \kappa ) &{}= \displaystyle \sup _{P \in {\mathcal {P}}} {\mathcal {L}}(P, \kappa , \lambda )\\ &{}= \displaystyle \sup _{P \in {\mathcal {P}}} \bigg \{ \displaystyle \int _{{\mathcal {C}}_0}\big [ h(W;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i)\big ] dP\bigg \} \\ &{} \hspace{15mm} \displaystyle - \sum _{i \in {\mathcal {F}}}(\lambda _i \underline{p_i} - \kappa _i\overline{p_i}) \end{array} \end{aligned}$$

(24)

Notice that the term within parenthesis can be analyzed as:

$$\begin{aligned} \begin{array}{ll} \displaystyle \sup _{P \in {\mathcal {P}}} \bigg \{ \displaystyle \int _{{\mathcal {C}}_0}\big [ h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i)\big ] dP\bigg \} &{} \\ = {\left\{ \begin{array}{ll} 0, \hspace{10mm} \text {if } h(W;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i) \le 0\\ \infty , \hspace{9mm} \text {otherwise} \end{array}\right. }&\end{array} \end{aligned}$$

(25)

Since we are only interested in the finite cost case, the dual problem $\displaystyle \min _{\lambda \ge 0,\kappa \ge 0} g(\lambda ,\kappa )$ is given by:

$$\begin{aligned} \begin{array}{llc} \displaystyle \min _{\lambda ,\kappa }&{} \displaystyle \sum _{i \in {\mathcal {F}}} (\kappa _i \overline{p_i} - \lambda _i \underline{p_i}) \\ \text {s.t.} &{}\displaystyle h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i) \le 0,&{} \forall w \in {\mathcal {C}}_0\\ &{} \lambda _i \ge 0, &{} \forall i \in {\mathcal {F}}\\ &{} \kappa _i \ge 0, &{} \forall i \in {\mathcal {F}}\\ \end{array} \end{aligned}$$

(26)

Although we were able to remove the integrals by writing the dual version of the problem, the problem is still not tractable, as the first constraint implies an infinite amount of points to evaluate.

Proposition 2

Let ${\mathcal {R}}(w)$ be a constraint valid $\forall w \in {\mathcal {C}}_0$, and $\bigcup _{i \in {\mathcal {F}}}$ $\overline{{\mathcal {C}}}_i$ be a partition of ${\mathcal {C}}_0$. Then, the following are equivalent:

$$\begin{aligned} {\mathcal {R}}(w), \forall w \in {\mathcal {C}}_0 \iff {\mathcal {R}}(w), \forall w \in \overline{{\mathcal {C}}}_i, \forall i \in {\mathcal {F}} \end{aligned}$$

(27)

Based on Proposition 2, we can rewrite the constraint:

$$\begin{aligned} h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i) \le 0, \forall w \in {\mathcal {C}}_0 \end{aligned}$$

(28)

as

$$\begin{aligned} h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i) \le 0, \forall w \in \overline{{\mathcal {C}}}_i, \forall i \in {\mathcal {F}} \end{aligned}$$

(29)

Proposition 3

Consider the sets ${\mathcal {F}} = \{0, 1,\ldots , I\}$, $\{{\mathcal {V}}_i\}_{i \in {\mathcal {F}}}$ and $\{{\mathcal {C}}_i\}_{i \in {\mathcal {F}}}$ obtained from a procedure as defined in Algorithm 1. In addition, let $w' \in {\mathcal {C}}_0$ and $\bigcup _{i \in {\mathcal {F}}}$ $\overline{{\mathcal {C}}}_i$ a partition of ${\mathcal {C}}_0$. Therefore, $w' \in \overline{{\mathcal {C}}}_i$ for some $i \in {\mathcal {F}}$. In addition, we define the index sets of all supersets (antecedents of ${\mathcal {C}}_i$) by ${\mathcal {A}}(i)= \{i\} \cap \{ i' \in {\mathcal {F}}: {\mathcal {C}}_i \subsetneq {\mathcal {C}}_{i'}\}$ and the index sets of all subsets (descendants of ${\mathcal {C}}_i$) as ${\mathcal {D}}(i)= {\mathcal {F}} {\setminus } {\mathcal {A}}(i)$.

Thus we can write:

• 1. $w'\in \{{\mathcal {C}}_j\}_{j \in {\mathcal {A}}(i)}$, $w' \notin \{{\mathcal {C}}_j\}_{j \in {\mathcal {D}}(j)}$ for some $j \in {\mathcal {F}}$

• 2. $\displaystyle \sum _{i \in {\mathcal {F}}} {\mathbb {I}}_{\{w' \in {\mathcal {C}}_i\}} \displaystyle = \sum _{i' \in {\mathcal {A}}(i)} 1$

Based on Proposition 3, we can rewrite the constraint:

$$\begin{aligned} h(w;\beta ) - \sum _{i \in {\mathcal {F}}}{\mathbb {I}}_{\{w \in {\mathcal {C}}_i\}} (\kappa _i - \lambda _i) \le 0, \forall w \in \overline{{\mathcal {C}}}_i, \forall i \in {\mathcal {F}} \end{aligned}$$

(30)

as

$$\begin{aligned} h(w;\beta ) - \sum _{i' \in {\mathcal {A}}(i)} (\kappa _i - \lambda _i) \le 0, \forall w \in \overline{{\mathcal {C}}}_i, \forall i \in {\mathcal {F}} \end{aligned}$$

(31)

Since the constraint above is valid $\forall w \in \overline{{\mathcal {C}}}_i, \forall i \in {\mathcal {F}}$, we can write:

$$\begin{aligned} \underset{w \in \overline{{\mathcal {C}}}_i}{\min } \bigg \{h(w;\beta )\bigg \} - \sum _{i' \in {\mathcal {A}}(i)} (\kappa _i - \lambda _i) \le 0, \forall i \in {\mathcal {F}} \end{aligned}$$

(32)

Concluding the proof $\blacksquare$.

Appendix 2: The estimation of empirical probabilities

During the process of estimating the data-driven ambiguity set, we must obtain two results: (i) the polyhedral convex sets and (ii) their associated probability coverage intervals. In Sect. 2.2, we describe the data-driven procedure and present the approximation used to estimate the intervals. In this section, we present some results to endorse the approximation methodology usage.

Consider the outcome of the Algorithm 1 application and the coverage intervals associated to each convex hull ${\mathcal {C}}_i$. Ideally, the probability interval $[\underline{p}_i, \hspace{1mm} \overline{p}_i]$ should include the true probability of a random point obtained from the original data distribution to fall within the convex hull ${\mathcal {C}}_i$, considering the specified significance level $\alpha$. In this context, we refer to accuracy as the percentage of probability intervals $[\underline{p}_i, \hspace{1mm} \overline{p}_i]$ that includes the true probability $p_i$. In addition, another interesting property would be how well the approximation that the outside hull ${\mathcal {C}}_0$ replicates the true distribution support—which we refer to as coverage.

To assess such properties, we ran the following experiment considering a random variable X that follows a Multivariate Normal distribution as the data-generating process:

• 1. Let $X \sim N(\mu , \Sigma )$, where $\mu$ is the unit vector of dimension $d = 3$ and $\Sigma$ is the $d \times d$ identity matrix;

• 2. Let N be the sample size used to construct the ambiguity set;

• 3. For a given N, we generate a sample and apply Algorithm 1, obtaining the convex hulls $\{{\mathcal {C}}_i\}_{i = 0}^{{\mathcal {I}}}$ and probability intervals $\{[\underline{p}_i, \hspace{1mm} \overline{p}_i]\}_{i = 0}^{{\mathcal {I}}}$;

• 4. For each convex hull ${\mathcal {C}}_i$ we approximate the true probability coverage value $p_i$ considering the data generating process distribution by generating an extremely large sample size $S = 1.000.000$ and verify whether $p_i \in [\underline{p}_i, \hspace{1mm} \overline{p}_i]$. Such interval is calculated in step 3 using the sample with size N;

• 5. We calculate the experiment’s accuracy (percentage of estimated intervals that contain the true probability) and coverage (percentage of the distribution support within

We vary the sample size N from 10 to 10.000 and repeat the experiment 5.000 times for each value. We consider the significance level $\alpha = 10\%$. The average accuracy is presented in Fig. 10 and the average coverage is presented in Fig. 11.

Fig. 10

Average accuracy for each sample size

Fig. 11

Average coverage for each sample size

One can observe that as the sample size grows, both accuracy and coverage converge to the expected values considering the significance level of the experiment. Naturally, the quality of such approximation methodology grows as the sample size grows. However, we argue that for a relatively small sample size, one can obtain a decent approximation. In addition, our empirical results validate the method as the experiments in Sects. 4.2 and 4.3 show.

Appendix 3: The choice of significance level

To apply the PolieDRO framework in classification or regression models, one should define the value of the significance level $\alpha$. Such value can be interpreted as the flexibility of the probability coverage of each convex hull that defines the hyperspace of distributions considered, that is, the ambiguity set. The idea is that such convex hulls imply some structure that arises from the data. The added flexibility controls the degree to which the resulting ambiguity set considers possible distributions.

Those values should be considered statistical significance parameters, using typical values such as $\alpha = 10\%$, $\alpha = 5\%$, or $\alpha = 1\%$. We repeat the experiment considering the different values and display them in Tables 11, 12, 13, 14 and 15 for the real world data sets and in Tables 16, 17 and 18 for the synthetic data.

In Tables 11, 12, 13, 14 and 15, we have highlighted in bold the cases where the PolieDRO version outperformed its nominal counterpart for each value of $\alpha$. We have summarized the results in Table 19.

For the synthetic datasets, we followed the same criteria as in Sect. 4.3. We identified the highest number of wins (W), ties (T), or losses (L) for each experiment in Tables 16, 17 and 18, and provided a summary of the results in Table 20.

Our results indicate that the choice of the statistical parameter $\alpha$ has little impact on the study results. In most cases, it does not alter the performance of the PolieDRO models, and in the few cases where it does, the change is not substantial.

Table 11 Mean out of sample accuracy for different $\alpha$

Table 12 Mean out of sample accuracy for different $\alpha$

Table 13 Mean out of sample accuracy for different $\alpha$

Table 14 Average out of sample MSE for different $\alpha$

Table 15 Average out of sample MSE for different $\alpha$

Table 16 Experiments with synthetically generated data sets, $\alpha = 0.10$

Table 17 Experiments with synthetically generated data sets, $\alpha = 0.05$

Table 18 Experiments with synthetically generated data sets, $\alpha = 0.01$

Table 19 Pairwise performance of the PolieDRO models against their nominal counterparts using real-world data sets, for varying $\alpha$

Table 20 Pairwise performance of the PolieDRO models against their nominal counterparts using synthetic data sets, for varying $\alpha$