Hierarchical Relevance Determination based on Information Criterion Minimization

This paper addresses the issue of hierarchical relevance determination (HRD), which boils down to determining all degrees of freedom in a supervised mixture distribution automatically. Such relevance determination is useful for wide range of machine learning applications. However, it is difficult to solve the HRD task because its objective function includes L0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{0}$$\end{document} norm such as the number of models in the mixture and the optimal features to use. Our contribution is twofold. As the main contribution, we formally defined the HRD problem and subsequently proposed a solution strategy, the SICM (sequential in formation criterion minimization) algorithm. The SICM algorithm enables us to continuously minimize an information criterion such as AIC or BIC, both of which includes the number of parameters, and therefore it enables us to determine all the degrees of freedom automatically. As another contribution, we realized a concrete implementation of the ideas, and tested this on actual data. Experiments using a hierarchical model which is constructed using the SICM algorithm have revealed that SICM has capability of constructing interpretable and highly accurate model.


Introduction
Recently, demand has been growing for the use of interpretable prediction models in many machine learning problems, whereas uninterpretable models with high prediction accuracy such as deep learning have been widely used. For example, when constructing a credit rating model, it is legally required that the model be made interpretable. Interpretable models include decision tree and regression models such as linear regression and logistic regression.
Such frequently used interpretable models are simple. Therefore, they are usually inferior to uninterpretable models in terms of prediction accuracy. One solution for prediction accuracy improvement is to construct an interpretable model by combining multiple interpretable models.
In addition, it is important to conduct relevance determination to construct an interpretable model. Relevance determination is useful to prune irrelevant features and to lead sparse models. Consequently, it is an important machinelearning tool.
Motivated by these backgrounds, we address the issue of relevance determination in supervised mixture distributions [1]. We designate this issue as hierarchical relevance determination (HRD). The HRD task is simultaneous optimization of the number of mixture components, model parameters of the individual components and latent variables for assigning observations to the components, and to select an optimal subset of the input variables used in the individual components. 1 As described below, the task is decomposed into a mixture model selection task and a variable selection task.
Mixture model selection task (model selection task hereinafter) is an estimation task for a mixture distribution, which is to optimize the number of mixture components, model parameters of the individual components and latent variables simultaneously. The expectation-maximization (EM) algorithm [2] has been used widely to estimate the parameters of mixture distributions. However, the number of components can not be optimized using the EM algorithm. Model comparison approaches are often used to determine the number of components automatically. A mixture with the minimum criterion value is selected as the best distribution from a set of candidate mixtures with different numbers of components. As a criterion, an information criterion such as AIC (Akaike information criterion) [3] and BIC (Bayesian information criterion) [4] has been usually used. AIC and BIC are model estimation measures, and they penalize mean training fit by the number of model parameters. Therefore, by choosing the minimum AIC/BIC model, we can determine the model complexity such as the number of components. The model comparison approaches are computationally expensive because we must estimate all candidate mixtures. We can avoid computational cost problems using Bayesian inference methods that directly minimize the upper bound of a negative marginal log-likelihood and can therefore estimate the number of components through a single mixture estimation. As the Bayesian inference methods, variational Bayesian inference (VB) [1,5], collapsed variational Bayesian inference (CVB) [6] and factorized asymptotic Bayesian inference (FAB) [7] have been proposed. The first, VB entails the shortcoming that the upper bound is loose because the latent variables and model parameters are assumed to be independent. In the case of CVB and FAB, the independence assumption is not required. Therefore the upper bounds of CVB and FAB are tighter than that of VB.
The variable selection task is an estimation task for a single distribution, which is to optimize the parameters and to select an optimal subset of input variables. Model comparison approaches such as the forward backward algorithm [8,9] have often been used for variable selection. As in the case of model selection, model comparison approaches are computationally expensive. Sparse estimation methods such as Lasso [10], SCAD [11] and least angle regression [12] have often been used for variable selection. The methods can solve the variable selection task because they can derive sparse parameters by minimizing the regularized objective functions. We can avoid the computational cost difficulties posed by model comparison approaches using sparse estimation methods, which directly minimize the regularized objective functions and therefore only require single estimation.
Then, how can the HRD task be solved? Apparently, we can readily construct a solution by simply combining the model selection and variable selection methods described above. However, simple combinations entail technical difficulties as discussed below. It is unrealistic to solve the HRD task using the model comparison approaches because the computational cost becomes too large. In the case of HRD, the number of candidate models becomes much larger than those in the cases of model selection and variable selection. It is also unrealistic to accomplish the task using sparse estimation approaches because it is not trivial how to solve the model selection task using a sparse estimation method. Using VB, we can solve the HRD task. However, VB presents the following two shortcomings. First, it is difficult to apply VB widely because update equations must be derived analytically. Second, it is required that the latent variables and the model parameters must be assumed as independent, although their dependence is fundamentally important for the true distributions. We can also solve the HRD task using the FAB-based method proposed in an earlier report of the literature [13]. The method solves the model selection task through continuous optimization (by variational inference) and the variable selection task by discrete optimization (using a model comparison approach).
Using these inference methods, hierarchical models have been developed and applied to many areas. Such models include Gaussian process mixtures [14], hidden Markov mixtures [15], gamma mixtures [16], and hierarchical multinomial-Dirichlet model [17].
As described in this paper, we propose the sequential information criterion minimization (SICM) algorithm, which is a method for relevance determination in mixture distributions. In addition to being a "consistent" framework for solving the HRD task, SICM has the following properties. First, the objective function of SICM is consistent. Therefore, for example, SICM is NOT a method which solves the model selection task by VB and solves the variable selection task by Lasso. Actually, SICM solves the HRD task as a minimization problem of an information criterion. Second, it is easy to derive the model parameter update equations used in SICM. In fact, SICM estimates the model parameters by iterating L 1 -regularized sparse estimation. It is, therefore, not necessary to derive the parameter update equations analytically as in the VB case. Third, SICM require no assumption of independence between the latent variables and the model parameters. Fourth, the optimization method used in SICM is consistent. Therefore, for example, SICM is NOT a method which solves the model selection task by variational inference (by continuous optimization) and solves the variable selection task by conducting model comparison (by discrete optimization). SICM minimizes the objective function continuously using variational inference and L 1 -regularized sparse estimation.
The key ideas of SICM are summarized as follows ("Overall flow of the SICM algorithm" presents details). First, we use an information criterion, AIC or BIC, as the objective function. SICM solves the HRD task by directly minimizing an information criterion. When minimizing BIC, SICM corresponds to the Bayesian inference methods such as VB and FAB because BIC is an approximate representation of a negative marginal log-likelihood. Second, in SICM, the difficulty of the L 0 term included in an information criterion is overcome by a concave continuous approximation.

SN Computer Science
The resulting optimization problem is solved using a method based on the majorization-minimization (MM) algorithm [18]. The MM algorithm minimizes an objective function approximately by minimizing an upper bound of the objective. Third, independence between the latent variables and the model parameters is not assumed because only the marginal distribution of the latent variables is used (that of the model parameters is not used) when variational inference is used in SICM.
We propose a method for constructing an interpretable hierarchical model by the application of SICM. The model is supervised and is represented as a combination of a decision tree and regression models. When conducting prediction, we assign a regression model to an observation by selection from a set of regression models in accordance with the decision tree. By SICM, the number of regression models used in the hierarchical model and subsets of input variables used in the individual regressions are determined automatically.
We demonstrate the utility of the SICM algorithm through experiments conducted using nine UCI datasets [19]. The results indicate that we can construct an interpretable prediction model with higher prediction accuracy than frequently used interpretable models such as decision tree and logistic regression. This is true because, in binary classification problems, the above-mentioned interpretable hierarchical model based on SICM (1) outperformed decision tree and logistic regression, (2) outperformed VB, and (3) performed comparably to support vector machine (SVM), which is representative of uninterpretable models having high prediction accuracy.
In summary, the main contributions of this study are the following.
1. We propose the SICM algorithm, which solves the HRD problem by continuously minimizing an information criterion: AIC or BIC. 2. As an SICM application, we propose an interpretable hierarchical model represented as a combination of decision tree and regression models. 3. Through binary classification experiments, we demonstrate that SICM is useful for supporting construction of an interpretable model with higher prediction performance than either decision tree or regression because, in the experiments, the earlier described interpretable hierarchical model based on SICM outperformed decision tree and regression and performed comparably to SVM, an uninterpretable model that exhibits high performance.
The remainder of this paper is organized as follows. The next section provides a problem setting of HRD. The following section explains the proposed SICM algorithm, which solves the HRD problem by minimizing information criteria continuously. The next section proposes a method for constructing an interpretable hierarchical model based on the SICM algorithm. The following section explains the experimentally obtained findings. The last section presents concluding remarks.

Problem Setting
This section presents a description of our problem settings. We consider the hierarchical relevance determination task for supervised mixture distributions, which we explain below.

Hierarchical Relevance Determination Problem
Presuming that we are given a dataset of N observations, each of which consists of D numerical input variables x and a target variable y: We regard supervised mixture distributions to be represented as where p, p k , K, Z, and k respectively represent a probability density function (pdf) of a mixture, a pdf of the kth mixture component, the number of components, latent variables explained below, and parameters of the kth mixture. Let us denote a set of latent variables as Z = {z nk } (n = 1, … , N, k = 1, … , K) , where Z is a set of binary variables representing the component assignments. If the nth observation (y n , x n ) is generated from the kth component distribution p k , then z nk = 1 , and z nk = 0 otherwise. Latent variables of one observation are mutually exclusive. Therefore, ∑ K k=1 z nk = 1 holds. We assume that the latent variables follow the following distribution: As described herein, we consider a task of hierarchical relevance determination (HRD) for mixture distributions. The HRD task is to optimize, simultaneously, the number of mixtures K, the parameters , the mixing coefficients and the latent variables Z, and to select an optimal subset of the input variables used in the individual components. The HRD task is to determine automatically all the degrees of freedom in the mixture distribution.
As discussed below, we solve the task by minimizing an information criterion. Therefore, when solving the task, we assume that the following two conditions hold. First, p k (y n |x n , k ) K k=1 satisfy the regularity conditions by which the Fisher information matrices of p k (y n |x n , k ) K k=1 are nonsingular around the maximum likelihood estimators. Second, the optimal assignment Z is unique.

Decomposition of the Hierarchical Relevance Determination Problem
We decompose the HRD problem into two: variable selection and model selection.

Variable Selection Problem
Next we consider the HRD problem where the number of components K is fixed to 1 and the pdf is represented as We define variable selection as the relevance determination problem in this case. The variable selection task is to optimize parameter and to select an optimal subset of the input variables simultaneously.

Model Selection Problem
Next, we consider the HRD problem for which all the input variables are used in the individual components. We define model selection as the relevance determination problem in this case. The model selection task is to optimize, simultaneously, the number of mixtures K, the parameters , the mixing coefficients , and the latent variables Z.

Proposed Method: SICM Algorithm
In this section, we propose the sequential information criterion minimization (SICM) algorithm, which solves the HRD task by continuously minimizing an information criterion of a supervised mixture distribution.
We derive the SICM algorithm for HRD by the following procedure. First, we propose an algorithm for solving the variable selection task by minimizing an information criterion continuously. Second, we propose an algorithm for solving the model selection task by continuously minimizing an information criterion. Third, we derive the SICM algorithm for solving the HRD problem by combining these two algorithms.

Overall Flow of the SICM Algorithm
We briefly review the SICM algorithm. We denote parameters included in the information criterion as . In the case of variable selection, = . In the case of model selection and HRD, = (Z, ).
The overall flow of the SICM algorithm is summarized as presented below.
1. We use the objective function E, which satisfies min E( ) = IC * , where IC * represents the minimum of an information criterion. We then formulate a minimization problem of the information criterion as a minimization problem of E. 2. We approximate the L 0 term of the parameters in the objective function by a function of L 1 norm, and numerically stabilize the problem as 3. We introduce new parameters 0 and derive an upper bound of the objective function: 4. By iteratively minimizing F, we estimate as where s represents an update step. The minimization method by which an upper bound of an objective function is iteratively minimized is called the majorizationminimization (MM) algorithm [18].

Objective Function
Next, we present a definition of the objective function of SICM. Let us denote as where h and v respectively represent a subset of the parameters used (selected) in a model and values of the "selected" parameters. In the case of regression models, h represents a subset of input variables which have non-zero regression coefficients, and v represents the values of the non-zero coefficients. An information criterion and its minimum are expressed as where f represents a model complexity, an L 0 term included in the information criterion, and v ML (h) represents a maximum likelihood estimator where the subset is fixed to h: Model complexity f is dependent on h and is independent of v. Therefore, f is expressed as From Eqs. (10) to (14), the minimum of the information criterion is expressed as As described in this paper, we solve the HRD problem by minimizing the information criterion of the mixture distribution. For this purpose, we use E in Eq. (17) below as the objective function to be minimized. We consider the following minimization problem: By solving the minimization problem in Eq. (16), we can minimize the information criterion for the following reasons. The objective function E is derived by replacing v ML of the information criterion with v. Therefore, E is not equal to the information criterion itself. However, the minimization problem of E coincides with that of the information criterion because the minimum E is equal to the minimum information criterion value IC * , as shown in Eq. (15).

SICM for Variable Selection
In this section, we propose a method for solving the variable selection problem. We consider the following information criterion minimization problem: We use AIC or BIC as the information criterion to be minimized. We set c IC = c AIC ( c IC = c BIC ) when we use AIC (BIC).
It is difficult to treat E( ) numerically because || || 0 is not always continuous with respect to . Therefore we use the following approximation: where i and respectively represent the ith component of and a user-defined small positive constant. The approximation in Eq. (23) is justified because ||s|| 0 and |s| |s|+ (s is a scalar) share the following properties: (1) they become 0 when s = 0 ; (2) they rapidly approach 1 when |s| increases; (3) they are convex upward functions with respect to |s|; and (4) they become nearly equal when approaches 0.
We introduce new parameters 0 , which have the same dimension as .
We propose an algorithm for minimizing F, i.e., for solving the variable selection task. The algorithm sequentially minimizes F. It therefore approximately minimizes the information criterion. Therefore, we designate it as the sequential information criterion minimization (SICM) algorithm.
We summarize the SICM algorithm for variable selection as Algorithm 1. The algorithm consists of the following two steps. First, F is minimized with respect to . As shown in Eq. (26), the problem in Eq. (27) is an L 1 -regularized maximum likelihood estimation problem with respect to . The L 1 regularization term has weights (| 0i |+ ) 2 corresponding to | i | like that of adaptive lasso [20]. It is important that, different from adaptive lasso, the weights be determined automatically based on the information criterion. Second, F is minimized with respect to 0 . The solution of this problem is 0 = because the equality in Eq. (24) holds if = 0 holds, as discussed above.

SICM for Model Selection
In this section, we propose a method for solving the model selection problem. We consider the following information criterion minimization problem: where D k represents the degrees of freedom of the k-th mixture component p k (y|x, k ) and is expressed as In the case of the model selection task, D k is a constant. We set g IC = g AIC ( g IC = g BIC ) when we minimize AIC (BIC). We derive an upper bound of the objective function E(Z, ) , as in the case of variable selection. We denote the minimum of E(Z, ) as E(Z * , * ) . A minimum is always less than or equal to an expectation. Consequently, the following inequality holds Therein, ⟨⋅⟩ p represents the expectation of ⋅ with respect to p. By applying Jensen's inequality (see Appendix A) to −log of Eq. (38), we obtain the following upper bound as

SN Computer Science
where q(Z) represents a pdf of Z.
As in the variable selection case, we use the following approximation of the L 0 term: where is a user-defined positive constant. Also, g(N k ) is a convex upward function of N k . Consequently, by introducing a pdf q(Z) , the following upper bound is derived: By combining inequalities (39) and (41), the following upper bound of the objective function E is derived: where we set g = g AIC and g � = g � AIC ( g = g BIC and g � = g � BIC ) when we minimize AIC (BIC).
Using the upper bound in Eq. (45), we define the model selection problem as the following minimization problem: We propose an algorithm for solving the problem in Eq. (53). We designate the algorithm as SICM algorithm because it minimizes the information criterion sequentially and approximately as in the variable selection case.
We summarize the SICM algorithm for model selection as Algorithm 2. The algorithm consists of the following four steps. First, we minimize F with respect to by solving the following maximum likelihood estimation problem: Second, F is minimized with respect to by maximum likelihood estimation: Third, F is minimized with respect to q(Z). We assume that q(Z) factorizes so that Then, using the variational inference method explained in Appendix B, q(Z) is optimized as (53) min , ,q,q F( , , q Z ,q Z ).
Fourth, F is minimized with respect to q(Z) . The solution of this problem is expressed as This is true because (1) q Z appears in F( , , q Z ,q Z ) with the form of ⟨G k (Z,q)⟩ q ; (2) G k (Z,q) is represented as Eq. (46); (3) Inequality (41) holds for g(N k ) , which is a convex upward function with respect to N k .

SICM for Hierarchical Relevance Determination
In this section, we propose the SICM algorithm for solving the HRD problem. We construct the algorithm based on the SICM algorithms described above, i.e., those for variable selection and model selection, which are summarized as Algorithms 1 and 2.
Let us start from the SICM for model selection. When we consider the HRD task, it is necessary to treat || k || 0 as variables, whereas they are constants in the model selection case. Therefore, when considering the HRD problem, it is necessary to make the following alterations to the SICM for model selection: By these changes, the upper bound to be minimized for solving the HRD problem is derived as

SN Computer Science
Under the alterations, the individual steps in Algorithm 2 are changed as follows. First, the estimation step of q is invariant except for the expression of g � (N k ) . It is necessary to replace D k included in g � (N k ) with || k || 0 . Second, the estimation step of q is invariant because (1) the optimal q is expressed as q = q , as discussed above, and (2) this result is independent of the expression of D k (independent of the described above alterations). Third, the estimation step of is invariant because the estimation of is only dependent on {⟨z nk ⟩ q } and is independent of D k . Fourth, the estimation step of k changes because ∑ K k=1 G k (Z,q, k ) becomes dependent on k . Because of the dependence, the estimation step of in the HRD case is expressed as Therefore k is estimated as presented below: where we set h = h AIC ( h = h BIC ) when we minimize AIC (BIC). If we make the following alterations, then the upper bound F in Eq. (26) in the variable selection case coincides with F k in Eq. (77) in the HRD case, except for their constant terms: Therefore we can estimate k using the SICM for variable selection (Algorithm 1) with the alterations in Eqs. (81) and (82). As a result, the problem for estimating k becomes a weighted maximum likelihood estimation with L 1 regularization. The minimization of F k is interpreted as a minimization of the information criterion, where the pdf is p k (y|x, k ) and the number of observations is N k because (1) r nk log p k (y n |x n , k ) is considered to be a weighted log-likelihood of N k observations because 0 ≤ r nk ≤ 1 and ∑ n r nk = N k ; (2) N k N k + c IC is approximately equal to c IC when N k is larger than , while we can adopt small and we need not estimate Summarizing the points presented above, the SICM algorithm can be constructed for solving the HRD problem in the following way. First, we replace Step 5 of Algorithm 2 (SICM for model selection) with Algorithm 1 (SICM for variable selection). Second, we make the alterations in Eqs. (81) and (82) to Algorithm 1. As an example, Appendix D summarizes the estimation step of k , which corresponds to Step 5 in Algorithm 2 (corresponds to Algorithm 1 with the alterations), for the HRD problem of a logistic regression mixture. In Algorithm 3, we summarize the overall flow of the SICM algorithm for solving the HRD problem. The objective function F in Algorithm 3 is represented as which is derived from Eqs.

Properties of SICM
In this section, we describe some properties of the SICM algorithm for HRD.

Sparsity
The SICM algorithm derives a sparse solution theoretically as discussed below. Automatically, SICM determines the number of mixture components. In fact, Eq. (65) in Algorithm 2 plays a fundamentally important role for the determination. By the effect of the term exp −g � (N k ) , a mixture component with a few observations is erased. Thereby, the number of the components is optimized because exp −g � (N k ) becomes close to 0 if N k , representing the number of observations of the k-th component, becomes small. Consequently, nk rapidly converges to 0 if N k becomes smaller. The parameters of the individual components are made sparse by SICM. As discussed in "SICM for Hierarchical Relevance Determination", SICM estimates the parameters of the individual mixtures by solving the L 1 -regularized maximum likelihood estimation problems. Therefore SICM derives sparse parameters of the individual components by the regularization effect of L 1 penalty, as in the case of sparse estimation methods such as Lasso.

Monotonicity
The SICM algorithm for HRD monotonically decreases the objective function, which corresponds to the upper bound of the information criterion to be minimized. We present a sketch of the proof below.
SICM for model selection monotonically decreases F in Eq. (45), the upper bound of the information criterion to be minimized. A sketch of the proof can be shown as where s represents an update step. Inequality (84) arises from Eqs. (90) and (92) in Appendix B, and Step 3 of Algorithm 2, which estimates the optimal q variationally by fixing q , and . Inequality (85) arises from Step 4 of Algorithm 2, which estimates the optimal q [as described immediately after Eq. (62)] by fixing q, and . Inequality (86) arises from Step 5 and Step 6, which conducts maximum likelihood estimations of and by fixing q and q.
In the case of SICM for HRD, which is a combination of SICM for model selection and SICM for variable selection, the maximum likelihood estimation of is replaced with L 1 -penalized sparse estimation of such as sparse logistic regression summarized in Appendix D. The sparse estimation decreases the upper bound expressed as the L 1 -regularized objective function. Therefore inequality (86) also holds in the HRD case.

Properties Related to Information Criterion Minimization
The SICM algorithm can avoid the singularity problem of mixture modeling described hereinafter. Both AIC and BIC are derived based on second-order expansions of their original objective functions and are derived by assuming the regularity condition. It is therefore not justifiable to apply the information criteria to singular models. However, a mixture model represented as p(y�x, ) = ∑ k k p k (y�x, k ) is singular. SICM avoids this difficulty for the following reasons. When constructing SICM, we started from a mixture represented as Eq. (2) and considered the objective function corresponding to its information criterion in Eq. (19). p(Y|X, Z, ) in Eq. (2) is regular because it is assumed that the conditions described in "Hierarchical Relevance Determination Problem" ("each mixture p k (y|x, ) is regular" and "the mixture assignment of each observation is unique") hold. Consequently, use of the information criterion corresponding to Eq. (19) is justified.
Actually, SICM with BIC minimization has asymptotic consistency. Because p(Y|X, Z, ) satisfies the regularity condition as described above, its Laplace approximation (BIC used in SICM) has asymptotic consistency. Actually, SICM with AIC does not have asymptotic consistency as AIC does not. Nevertheless, AIC is "consistent" in the sense that an estimated distribution asymptotically approaches the true distribution (not the true "model" in the case of BIC).
One benefit of SICM is that it can minimize either AIC or BIC, whereas Bayesian methods such as variational Bayesian inference correspond only to BIC minimization (actually, minimization of negative marginal log-likelihood). We expected, and we show in "Experiments", that SICM with AIC sometimes outperforms SICM with BIC in some cases.

Applicability to Unsupervised Learning
As described in this report, we have considered the task of relevance determination in supervised mixture distributions. However, the SICM algorithm is applicable to unsupervised mixtures by making the following alteration: Unsupervised mixtures include a mixture of normal distributions. For instance, by application of SICM to a normal mixture, we can estimate a mixture of sparse Gaussian graphical models.

Application to Interpretable Hierarchical Modeling
In this section, as an application of SICM, we propose a method for constructing an interpretable hierarchical model, which is constructed as a combination of interpretable models: a decision tree and regression models.
The overall flow of the model construction is summarized as presented below.
1. We estimate p(Y|X, Z, ) using the SICM algorithm. As the individual mixtures, we use interpretable regression models: linear regression models or logistic regression models. 2. We construct a decision tree for model assignment using the training dataset where {r nk } K k=1 ( r nk = ⟨z nk ⟩ q ) is a set of target variables and where x n is a set of input variables. The decision tree for predicting r ⋅k is estimated by solving a K-class classification problem. 3. Using the results of the two steps described above, we conduct prediction as follows: Here r k (x) represents the predicted value of r ⋅k estimated from a test observation x using the decision tree. Actually, r k (x) is considered p(k|x). A key point in constructing the model is the introduction of Step 2, the decision tree estimation. To conduct prediction, it is necessary to estimate the predicted value of r ⋅k from a test observation. However, the prediction mechanism of r ⋅k is not included in the SICM algorithm. Therefore, Step 2 is introduced to predict r ⋅k . Consequently, we adopt the two-step training process described above, which consists of (87) p k (y n |x n , k ) →p k (x n | k ).

Experimental Setting
Through the following experiments, we demonstrate the utility of the SICM algorithm using the hierarchical prediction model proposed in "Application to Interpretable Hierarchical Modeling".
We considered binary classification problems in the experiments. Therefore, we used a logistic regression as a mixture component of the hierarchical model. Appendix D summarizes the SICM algorithm for logistic regression mixtures. The maximum number of components, the constant for approximating the L 0 terms, and the threshold for the convergence condition were set to (K, , ) = (50, 1.0, 10 −6 ) . The maximum number of components K is equal to the initial number of components in the SICM algorithm. The number of components decreases as the SICM steps progress, but it never increases. Therefore it is necessary to set K to enough large number and we set K to 50. The approximation constant was not estimated from training datasets but was given by us as a fixed value.
We used nine datasets from the UCI repository [19]. We selected these datasets from the UCI repository because they are datasets for binary classification problems and consist of a binary target variable and numerical input variables. Some properties of the datasets are summarized in Table 1.
For training the models, we transformed categorical input variables into their one-hot encodings and standardized all input variables.
We observed the binary classification accuracies of the proposed models: SICM AIC and SICM BIC . Hereinafter, we denote the proposed hierarchical model estimated by minimizing AIC (BIC) as SICM AIC ( SICM BIC ). As a measure of the classification accuracy, we used the mean Area Under the Curve (AUC) of ROC curves estimated using five-fold cross validation.
In the experiments, we aimed to show the effectiveness of the SICM algorithm by demonstrating (a) we could construct accurate and interpretable models by solving the HRD problem, and (b) we could get better results using the SICM algorithm than using another existing inference method. In other words, (a) means that we could improve prediction accuracies of interpretable models by solving the HRD problem. In order to show (a), we compared SICMs ( SICM AIC and SICM BIC , the hierarchical models proposed in "Application to Interpretable Hierarchical Modeling", which are estimated using the SICM algorithm) to decision tree and logistic regression, which are frequently used as interpretable models. For this comparison, as our model, we selected the hierarchical prediction model proposed in "Application to Interpretable Hierarchical Modeling", which consisted of tree and logistic regression. Namely to compare to tree and logistic regression, we constructed the hierarchical model consisting of a tree and logistic regressions. For reference, we compared SICMs to support vector machine (SVM), which is a highly accurate uninterpretable model. We note that SICM is not proposed for constructing highly accurate uninterpretable models but proposed for improving prediction accuracies of interpretable models. Therefore, in the experiments, we did NOT aim to show SICMs outperformed SVM because SVM is uninterpretable. SVM was selected to observe how closely SICMs performs to an uninterpretable model. In order to show (b) SICMs derived better results than an existing inference method, we compared SICMs to VB. Here, VB represents the hierarchical prediction model which has the same structure as SICMs and its parameter inference method is replaced from the SICM algorithm to variational Bayesian inference (namely SICMs and VB are the same models estimated by the different methods). We selected VB for the following reasons. First, as described in "Introduction", our main aim and contribution are to construct a new inference method for solving the HRD task. Therefore, we compared the SICM algorithm to variational Bayesian inference by comparing the same models estimated by different inference methods. Variational Bayesian inference is one of the most popular method for relevance determination and is still a SOTA in this area. Second, in this paper, we do NOT aim to propose a new hierarchical model, and thus to select an optimal model family is out of scope. Therefore we selected VB having the same structure as SICMs, and did not select another latest hierarchical models belonging to different model families.
Let us give supplementary explanations to how to estimate the compared models. Logistic regression models were constructed by maximum likelihood estimation. Gaussian kernels were used in SVM. We optimized a decay parameter of a Gaussian kernel by conducting two-fold cross validation. As mentioned above, we used variational Bayesian inference for estimating VB.
The computational environment was as follows: Intel Xeon CPU ES-1650 3.50 GHz CPU and 64 GB memory on a Linux Ubuntu platform. We conducted the experiments using Python language. The implementations of SVM, logistic regression and decision tree (including the tree estimation part in VB and SICMs) were taken from scikit-learn [21], a public machine learning library in Python. We implemented the HRD part of VB (variational Bayesian inference of a logistic regression mixture) based on Refs. [1,22]. Table 2 presents the experimentally obtained results: the classification accuracies of six models. Results indicate the SICM algorithm's effectiveness for the following reasons.

Prediction Performance
By combining decision tree and logistic regression, we can construct a model with higher accuracy than that of any individual models. We can construct an interpretable and highly accurate model using the SICM algorithm. This statement is supported by the following results. First, combination models (hierarchical models), known as SICM AIC , SICM BIC and VB, outperformed decision tree and logistic regression. Second, the combination models performed comparably to SVM. The SICM algorithm which minimizes BIC is theoretically better than variational Bayesian inference because the independence between the latent variables and the parameters of the individual components is not assumed in SICM, whereas variational Bayesian inference requires the independence assumption. This statement is supported by the results that SICM BIC outperformed VB.
One benefit of SICM is that SICM can minimize either AIC or BIC, whereas Bayesian methods such as variational Bayesian inference only minimize the negative marginal log-likelihood. In fact, BIC is an approximate representation of the negative marginal log-likelihood. This statement is supported by results showing that SICM AIC performed comparably to SICM BIC from the viewpoint of their winloss record. Table 3 Model assignment rules of the proposed hierarchical model Each row represents a model assignment rule, which corresponds to a leaf of a decision tree and which is expressed as a product set of the listed conditions. For example, for the first rule in the figure, we use Model 1 for prediction if an observation simultaneously satisfies the following conditions: (1) bill statement in July ≤ − 0.5 ; (2) payment in August ≤ 4.0 ; and (3) payment delay in September ≤ 0.5 . In the rules, represent standard deviations of the respective input variables. The IDs of the assigned regression model (from Model 1 to Model 6) correspond to those in Table 4 Model Model  Rows and columns respectively represent an input variable and a logistic regression model. One logistic regression model in this table was used for prediction in accordance with the rules in Table 3. A coefficient is presented with red/white/blue color if its value is positive/exactly zero/negative. If a coefficient is positive/negative, then the default risk increases when the corresponding input variable increases/ decreases

Interpretability
Using the proposed method, we can construct an interpretable model because the proposed hierarchical model is represented as a combination of a decision tree and regression models, and has the sparse structure. In this section, we qualitatively describe the property of interpretability through an example of the proposed model. 2 As a related example, we used the hierarchical model trained on "Default of Credit Card Clients" dataset. An observation in the dataset represents a record of a credit card user, and consists of a binary target variable and 23 input variables. The target variable represents default on a payment in October. The input variables represent (1) amount of the given credit; (2) payment status recorded in April-September such as payment delay (month), amount of bill statement, and amount of payment; and (3) user profile such as age, gender, education, and marital status. We transformed the categorical input variables into their one-hot encodings. Tables 3 and 4 summarize the proposed hierarchical model trained on the "default of credit card clients" dataset based on the SICM algorithm. Table 3 represents the model assignment rules included in the decision tree. Table 4 shows regression coefficients of the individual logistic regression models. These results indicate that the proposed model has the following properties.
The SICM algorithm derives sparse solutions for the following reasons. First, as shown in Table 4, many coefficients were estimated as exactly zero. Second, the number of regression models diminished to six starting with K = 50.
The proposed model assigns a regression model to an observation based on its properties. For example, in the case of Table 3, (1) Model 1 corresponded to the "standard" clients (observations), who had low payment delay in September and low bill statement in July; (2) Model 2 corresponded to the clients who had larger bill statements in July than the standard clients had; (3) Model 3 corresponded to the clients who had more months of payment delay in September than the standard clients; and (4) Models 4, 5, and 6 corresponded to clients who had exceptionally large payments in August. Each regression model occupied an imbalanced number of observations. Three models (Models 4, 5 and 6) were used for the "exceptional" clients.
For each input variable, the signs of the regression coefficients may differ depending on the regression model. For example, in the case of Model 2, the regression coefficient of "age" was negative, which indicates that the default risk decreases as age increases. However, for Models 4 and 6, the coefficients were positive, which indicates that the default risk increases when age increases. Age inversely contributed to the default risk depending on whether the payment in August was exceptionally large ( > 4.0 ), or not. Such local sign inversion is regarded as one reason why the proposed hierarchical model is highly accurate.
Sign inversion does not always occur. For example, "the amount of the given credit" invariably had non-positive coefficients, which indicates that the default risk always decreases as the credit rating becomes better. This result is consistent with our intuition.

Summary
The SICM algorithm was proposed for solving the hierarchical relevance determination problem. The SICM algorithm minimizes an information criterion continuously and therefore enables us to determine the degrees of freedom in a mixture distribution automatically. A method for constructing an interpretable hierarchical model based on the SICM algorithm was also proposed. Experiment results obtained using the interpretable hierarchical model have demonstrated the utility of the SICM algorithm for the following reasons. First, the hierarchical model outperformed frequently used interpretable models (tree and logistic regression) in terms of prediction accuracy. Second, it was shown qualitatively that the hierarchical model derived interpretable results consistent with our intuition.
Future work includes the following two issues. One is a theoretical expansion of the interpretable hierarchical model. When constructing the model, the degrees of freedom in the decision tree for model assignment are not automatically determined. Therefore introduction of relevance determination mechanism to the decision tree estimation is a subject of future work. The other is an application of the proposed information criterion minimization method to those other than the HRD problem. Relevance determination based on the continuous minimization of the information criterion is widely applicable. A promising application is relevance determination in unsupervised distributions such as sparse estimation of Gaussian graphical model mixtures.

SN Computer Science
Equation (102) is derived because W ii > 0 holds. From Eq. (102), the following conditions for relations between u i and a i are derived: By combining Eqs. (101) and (103)-(105), the following solution is obtained: Equation (106) is an update equation for the ith component. By repeating the update step until convergence, the solution of the L 1 -regularized quadratic programming problem can be derived.
bound of ||w k || 0 in Eq. (24), the following upper bound of the objective function F k is obtained.
When deriving the upper bound, we introduced the additional parameters, w k0 and k . As discussed in "Proposed Method: SICM Algorithm", F k can be minimized approximately by minimizing its upper bound: F k (w k , w k0 , k ).
We erase k from F k (w k , w k0 , k ) in the following manner. When w k and k are given, the optimal w k0 , which minimizes F k (w k , w k0 , k ) , is expressed as In inequality (117), the equality holds if z = . Presuming z = z n = y n w † k x n and = kn in the inequality, the optimal k , which minimizes F k when w k is given, is expressed as From Eqs. (127) and (128), we select kn as and erase k from F k as shown below: (127) w k0 =w k .
(129) kn =y n w † k0 x n , As discussed in "Proposed Method: SICM Algorithm", we can approximately minimize F k and estimate the parameters of the kth mixture component by minimizing the upper bound F k in Eq. (131) with respect to w k and w k0 . We can minimize F k with respect to w k using the subgradient method explained in Appendix C. From the discussion presented above, the parameter estimation algorithm of the kth logistic regression component is summarized as Algorithm 4. The SICM algorithm for solving the HRD problem in the case of a logistic mixture is constructed by replacing Step 5 of Algorithm 2 with Algorithm 4 ( k = 1, 2, … , K).