VolcanoML: speeding up end-to-end AutoML via scalable search space decomposition

Abstract

End-to-end AutoML has attracted intensive interests from both academia and industry which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VolcanoML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VolcanoML introduces and implements basic building blocks, which decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VolcanoML further supports a Volcano-style execution model—akin to the one supported by modern database systems—to execute the plan constructed. Our evaluation demonstrates that, not only does VolcanoML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn.

Appendix

In this section, we describe more details about the background, system design and implementations.

1.1 AutoML formulations and motivations

1.1.1 Formulations

Definition and notation. There are K candidate algorithms $\mathcal {A}=\{A^1, ..., A^K\}$. Each algorithm $A^i$ has a corresponding hyper-parameter space $\Lambda _i$. The algorithm $A^i$ with hyper-parameter configuration $\lambda$ and new feature set F is denoted by $A^i_{(\lambda ,F)}$. Given the dataset $D=\{D_{train}, D_{valid}\}$ of a learning problem, the AutoML problem is to find the joint algorithm, feature, and hyper-parameter configuration $A^{*}_{(\lambda ^{*},F^{*})}$ that minimizes the loss metric (e.g., the validation error on $D_{valid}$):

$$\begin{aligned} A^*_{(\lambda ^*, F^{*})} = {\text {argmin}}_{A^i\in \mathcal {A},\lambda \in \Lambda ^i,F\in \mathcal {F}^i} \mathcal {L}(A_{(\lambda , F)}^i; D), \end{aligned}$$

(14)

where $\mathcal {F}^i=Gen(A^i, D, \mathbf {op})$ is the feature space of $A^i$ that can be generated from the raw feature (data) set D, and $\mathbf {op}$ is the set of available FE operators.

Fig. 13

Validation error on pc4 when increasing the number of hyper-parameters in auto-sklearn given the same time budget

Challenge: ever-growing search space. Enriching the search space can lead to performance improvement since the enriched search space may bring better configurations. However, an ever-growing search space can significantly increase the complexity of searching for ML pipelines. Existing AutoML systems usually can only explore very limited configurations in a huge search space and thus suffer from the low-efficiency issue [25] that hampers the effectiveness of AutoML systems. In Fig. 13, we provide a brief example of auto-sklearn, one state-of-the-art system AutoML system. Its search algorithm cannot scale to a high-dimensional search space [25]. To alleviate this issue, in this paper, we focus on developing a scalable AutoML system.

1.1.2 Observations and motivations about AutoML

We now present several important observations that inspired the design of VolcanoML.

Observation 1. The search space can be partitioned according to ML algorithms. The entire search space is the union of the search spaces of individual algorithms, i.e., $\Omega =\{S^1, ..., S^K\}$, where $S^i$ is the joint space of features and hyper-parameters, i.e., $S^i=(\Lambda ^i \times \mathcal {F}^i)$.

Fig. 14

The performance distribution of ML pipelines constructed by 30 FE and HPO configurations on fri_c1 using Random Forest. For FE configurations, the performance increases from top to down; for HPO configurations, the performance increases from left to right (the deeper, the better)

Observation 2. The sub-space of algorithm $A^i$ can be very large, e.g., in auto-sklearn, $S^i$ usually includes more than 50 hyper-parameters. When exploring the search spaces via extensive experiments, we observe the following:

• If hyper-parameter configuration $\lambda _1$ performs better than $\lambda _2$, i.e., $\lambda _1 \le \lambda _2$, then it often holds that $(\lambda _1, F) \le (\lambda _2, F)$ for the joint configuration $(\lambda , F)$ with F fixed;

• If FE pipeline configuration $F_1$ performs better than $F_2$, i.e., $F_1\le F_2$, then it often holds that $(\lambda , F_1) \le (\lambda , F_2)$ for the joint configuration $(\lambda , F)$ with $\lambda$ fixed.

Figure 14 presents an example for these observations. This motivates us to solve the joint FE and HPO problem via alternating optimization. That is, we can alternate between optimizing FE and HPO, and we can fix the FE configuration (resp. HPO configuration) when optimizing for HPO (resp. FE). This alternating manner is indeed similar to how human experts solve the joint optimization problem manually. One obvious advantage of alternating optimization is that each time only a much smaller subspace ($\Lambda ^i$ or $\mathcal {F}^i$) needs to be optimized, instead of the joint space $S^i=(\Lambda ^i \times \mathcal {F}^i)$.

Observation 3. The sensitivity of ML algorithms to FE and HPO is often different. Taking Fig. 14 for example, compared to HPO, FE has a larger influence on the performance of ‘Random Forest’ on ‘fri_c1’; in this case, optimizing FE more frequently can bring more performance improvement.

Observation 4. The above observations motivate the use of meta-learning. We can learn (1) the algorithm performance across ML tasks and (2) the configuration selection of each ML algorithm across tasks. Such meta-knowledge obtained from historical tasks can greatly improve the efficiency of ML pipeline search.

Therefore, a scalable AutoML system should include two basic components: (1) an efficient framework that can navigate in a huge search space, and (2) a meta-learning module that can extract knowledge from previously ML tasks and apply it to new tasks.

1.2 VolcanoML components and implementations

1.2.1 Compenents and search space

Feature engineering. The feature engineering pipeline is shown in Fig. 2. It comprises four sequential stages: preprocessors (compulsory), scalers (5 possible operators), balancers (1 possible operators) and feature transformers (13 possible operstors). For each of the latter three stages, VolcanoML picks one operator and then execute the entire pipeline. Table 13 presents the details of each operator. The total number of hyper-parameters for FE is 52.

We follow the design of the search space for feature engineering in the existing AutoML systems, e.g., autosklearn and TPOT. It limits the search space for feature engineering by adopting a fixed pipeline including different stages, and each stage is equipped with an operation (featurizer) that is selected from a pool of featurizers. The pool of featurizers at each stage is relatively small, and Bayesian optimization can be used to choose the proper featurizers for each stage. When the pool of featurizers is very large, high-dimensional Bayesian optimization algorithms could work better. In many real-world cases, though this architecture is not good enough for feature engineering (effectiveness), there still remains space to explore to conduct feature engineering effectively and efficiently. To support real scenarios, VolcanoML provides API for user-defined feature engineering operators and we recommend users to add domain-specific feature engineering operators to the search space for better search performance. In addition, users can replace the original feature engineering part in VolcanoML with other iterative feature engineering methods easily.

ML algorithms. VolcanoML implements 11 algorithms for classification and 10 algorithms for regression, with a total of 50 and 49 hyper-parameters, respectively. The built-in algorithms include linear models, support vector machine, discriminant analysis, nearest neighbors, and ensembles. Table 12 presents the details.

Ensemble methods. Ensembles that combine predictions from multiple base models have been known to outperform individual models, often drastically reducing the variance of the final predictions [88]. VolcanoML provides four ensemble methods: bagging, blending, stacking, and ensemble selection [89]. During the search process, the top $N_{\text {top}}$ configurations for each algorithm are recorded and the corresponding models are stored. After the optimization budget exhausts, the saved models are treated as the base models for the ensemble method. We use ensemble selection as the default method and build an ensemble of size 50.

Table 12 Hyper-parameters of ML algorithms in VolcanoML. We distinguish categorical (cat) hyper-parameters from numerical (cont) ones. The numbers in the brackets are conditional hyper-parameters

Table 13 Hyper-parameters of FE operators in VolcanoML

1.2.2 Programming interface

Consider a tabular dataset of raw values in a CSV file, named train.csv, where the last column represents the label. We take a classification task as an example. With VolcanoML, only six lines of code are needed for searching and model evaluation.

By calling load_train and load_test, the data manager automatically identifies the type of each feature (continuous, discrete, or categorical), imputes missing values, and converts string-like features to one-hot vectors. By calling fit, VolcanoML splits the dataset into folds for training and validation, evaluates various configurations, and generates an ensemble from each individual configuration. For users who need to customize the search process, Classifier provides additional parameters to specify:

• time_limit controls the total runtime of the search process;

• include_algorithms specifies which algorithms are included (if not specified, all built-in algorithms are included);

• ensemble_method chooses which ensemble strategy to use;

• enable_meta determines whether to use meta-learning to accelerate the search process;

• metric specifies the metric used to evaluate the performance of each configuration.

Customized components. VolcanoML provides APIs to easily enrich the search space, such as the stage in FE pipeline, FE operators, and ML algorithms. The following is the syntax of defining customized components:

It is important to note that auto-sklearn does not support adding a new stage or updating the existing stages in the FE pipeline. In addition, auto-sklearn cannot add an operator for any stage (e.g., adding smote_balancer to the stage balancer), while VolcanoML supports this.

1.3 Experiment datasets

In our experiments, we splitted each dataset into five folds. Four are used for training and the remaining one is used for testing. The 60 OpenML datasets used are presented as follows (in the form of “dataset_name (OpenML id)”):

Classification datasets. kc1 (1067), quake (772), segment (36), ozone-level-8hr (1487), space_ga (737), sick (38), pollen (871), analcatdata_supreme (728), abalone (183), spambase (44), waveform(2) (979), phoneme (1489), page-blocks(2) (1021), optdigits(28), satimage (182), wind (847), delta_ailerons (803), puma8NH (816), kin8nm (807), puma32H (752), cpu_act (761), bank32nh (833), mc1 (1056), delta_elevators (819), jm1 (1053), pendigits (32), mammography (310), ailerons (734), eeg (1471), letter(2) (977), kropt (184), mv (881), fried (901), 2dplanes (727), electricity (151), a9a (A2), mnist_784 (554), higgs (23512), covertype (180).

Regression datasets. stock (223), socmob (541), Moneyball (41021), insurance (A1), weather_izmir (42369), us_crime (315), debutanizer (23516), space_ga (507), pollen (529), wind (503), bank8FM (572), bank32nh (558), kin8nm (189), puma8NH (225), cpu_act (573), puma32H (308), cpu_small (227), visualizing_soil (668), sulfur (23515), rainfall_bangladesh (41539).

Since the datasets insurance and a9a are not collected in OpenML, we use A1 and A2 as their OpenML ID instead.