Learning Interpretable Classification Rules with Boolean Compressed Sensing

Abstract

An important problem in the context of supervised machine learning is designing systems which are interpretable by humans. In domains such as law, medicine, and finance that deal with human lives, delegating the decision to a black-box machine-learning model carries significant operational risk, and often legal implications, thus requiring interpretable classifiers. Building on ideas from Boolean compressed sensing, we propose a rule-based classifier which explicitly balances accuracy versus interpretability in a principled optimization formulation. We represent the problem of learning conjunctive clauses or disjunctive clauses as an adaptation of a classical problem from statistics, Boolean group testing, and apply a novel linear programming (LP) relaxation to find solutions. We derive theoretical results for recovering sparse rules which parallel the conditions for exact recovery of sparse signals in the compressed sensing literature. This is an exciting development in interpretable learning where most prior work has focused on heuristic solutions. We also consider a more general class of rule-based classifiers, checklists and scorecards, learned using ideas from threshold group testing. We show competitive classification accuracy using the proposed approach on real-world data sets.

Appendices

Appendix 1: Dual Linear Program

We now derive the dual LP, which we use in Sect. 5. We start off by giving a reformulation of the LP in (10), i.e., we consider an LP with the same set of optimal solutions as the one in (10). First note that the upper bounds of 1 on the variables ξ _i are redundant.Let $(\bar{\mathbf{w}},\bar{\boldsymbol{\xi }})$ be a feasible solution of (10) without the upper bound constraints such that $\bar{\xi }_{i} > 1$ for some $i \in \mathcal{P}$. Reducing $\bar{\xi }_{i}$ to 1 yields a feasible solution (as $\mathbf{a}_{i}\bar{\mathbf{w}} +\bar{\xi } _{i} \geq 1$—the only inequality ξ _i participates in besides the bound constraints—is still satisfied). The new feasible solution has lower objective function value than before, as ξ _i has a positive coefficient in the objective function (which is to be minimized). One can similarly argue that in every optimal solution of (10) without the upper bound constraints, we have w _j  ≤ 1 (for j = 1, …, n). Finally, observe that we can substitute ξ _i for $i \in \mathcal{Z}$ in the objective function by a _i w because of the constraints a _i w = ξ _i for $i \in \mathcal{Z}$. We thus get the following LP equivalent to (10):

$$ \displaystyle\begin{array}{rcl} & & \min \quad \sum _{j=1}^{n}\left (\lambda +\|\mathbf{a}_{ \mathcal{Z}}^{j}\|_{ 1}\right )w_{j} +\sum _{ i=1}^{p}\xi _{ i} \\ & & \,\ \mathrm{s.t.}\quad 0 \leq w_{j},\,j = 1,\ldots,n \\ & & \qquad \quad 0 \leq \xi _{i},\,i = 1,\ldots,p \\ & & \qquad \quad \mathbf{A}_{\mathcal{P}}\mathbf{w} +\boldsymbol{\xi } _{\mathcal{P}}\geq \mathbf{1}. {}\end{array}$$

(19)

The optimal solutions and optimal objective values are the same as in (10).

Writing $\mathbf{A}_{\mathcal{P}}\mathbf{w} +\boldsymbol{\xi } _{\mathcal{P}}$ as $\mathbf{A}_{\mathcal{P}}\mathbf{w} + \mathbf{I}\boldsymbol{\xi }_{\mathcal{P}}$, where I is the p × p identity matrix, $\vert \vert \mathbf{a}_{\mathcal{Z}}^{j}\vert \vert _{1}$ as $\mathbf{1}^{T}\mathbf{a}_{\mathcal{Z}}^{j}$, and letting $\boldsymbol{\mu }$ be a row vector of p dual variables, one can see that the dual is:

$$\displaystyle\begin{array}{rcl} & & \max \quad \sum _{i=1}^{p}\mu _{ i} \\ & & \,\ \mathrm{s.t. }\quad 0 \leq \mu _{i} \leq 1,\,i = 1,\ldots,p \\ & & \qquad \quad \boldsymbol{\mu }^{T}\mathbf{A}_{ \mathcal{P}}\leq \lambda \mathbf{1}_{n} + \mathbf{1}^{T}\mathbf{A}_{ \mathcal{Z}}.{}\end{array}$$

(20)

Suppose $\boldsymbol{\bar{\mu }}$ is a feasible solution to (20). Then clearly $\sum _{i=1}^{p}\bar{\mu }_{i}$ yields a lower bound on the optimal solution value of (19).

Appendix 2: Derivation of Screening Tests

Let $\mathcal{S}(j)$ stand for the support of $\mathbf{a}_{\mathcal{P}}^{j}$. Furthermore, let $\mathcal{N}(j)$ stand for the support of $\mathbf{1} -\mathbf{a}_{\mathcal{P}}^{j}$, i.e, it is the set of indices from $\mathcal{P}$ such that the corresponding components of $\mathbf{a}_{\mathcal{P}}^{j}$ are zero.

Now consider the situation where we fix w ₁ (say) to 1. Let A′ stand for the submatrix of A consisting of the last n − 1 columns. Let w′ stand for the vector of variables w ₂, …, w _n . Then the constraints $\mathbf{A}_{\mathcal{P}}\mathbf{w} +\boldsymbol{\xi } _{\mathcal{P}}\geq \mathbf{1}$ in (19) become $\mathbf{A}_{\mathcal{P}}^{\prime}\mathbf{w}^{\prime} +\boldsymbol{\xi } _{\mathcal{P}}\geq \mathbf{1} -\mathbf{a}_{\mathcal{P}}^{1}$. Therefore, for all $i \in \mathcal{S}(1)$, the corresponding constraint is now $(\mathbf{A}_{\mathcal{P}}^{\prime})_{i}\mathbf{w}^{\prime} +\xi _{i} \geq 0$ which is a redundant constraint as $\mathbf{A}_{\mathcal{P}}^{\prime}\geq 0$ and w′, ξ _i  ≥ 0. The only remaining non-redundant constraints correspond to the indices in $\mathcal{N}(1)$. Then the value of (19) with w ₁ set to 1 becomes

$$\displaystyle{ \begin{array}{rl} \left (\lambda +\|\mathbf{a}_{\mathcal{Z}}^{1}\|_{1}\right ) +\min \quad &\sum _{j=2}^{n}\left (\lambda +\|\mathbf{a}_{\mathcal{Z}}^{j}\|_{1}\right )w_{j} +\sum _{i\in \mathcal{N}(1)}\xi _{i} \\ \mathrm{s.t.}\quad &0 \leq w_{j},\,j = 2,\ldots,n \\ &0 \leq \xi _{i},\,i \in \mathcal{N}(1) \\ &\mathbf{A}^{\prime}_{\mathcal{N}(1)}\mathbf{w}^{\prime} +\boldsymbol{\xi } _{\mathcal{N}(1)} \geq \mathbf{1}. \end{array} }$$

(21)

This LP clearly has the same form as the LP in (19). Furthermore, given any feasible solution $\boldsymbol{\bar{\mu }}$ of (20), $\boldsymbol{\bar{\mu }}_{\mathcal{N}(1)}$ defines a feasible dual solution of (21) as

$$\displaystyle\begin{array}{rcl} & \boldsymbol{\bar{\mu }}^{T}\mathbf{A}_{\mathcal{P}}\leq \lambda \mathbf{1}_{n} + \mathbf{1}^{T}\mathbf{A}_{\mathcal{Z}} & {}\\ & \Rightarrow \boldsymbol{\bar{\mu }}_{\mathcal{S}(1)}^{T}\mathbf{A}^{\prime}_{\mathcal{S}(1)} +\boldsymbol{\bar{\mu }}_{ \mathcal{N}(1)}^{T}\mathbf{A}^{\prime}_{\mathcal{N}(1)} \leq \lambda \mathbf{1}_{n-1} + \mathbf{1}^{T}\mathbf{A}_{\mathcal{Z}}^{\prime}& {}\\ & \Rightarrow \boldsymbol{\bar{\mu }}_{\mathcal{N}(1)}^{T}\mathbf{A}^{\prime}_{\mathcal{N}(1)} \leq \lambda \mathbf{1}_{n-1} + \mathbf{1}^{T}\mathbf{A}_{\mathcal{Z}}^{\prime}. & {}\\ \end{array}$$

Therefore $\sum _{i\in \mathcal{N}(n)}\bar{\mu }_{i}$ is a lower bound on the optimal solution value of the LP in (21) and therefore

$$\displaystyle{ \lambda +\vert \vert \mathbf{a}_{\mathcal{Z}}^{1}\vert \vert _{ 1} +\sum _{i\in \mathcal{N}(1)}\bar{\mu }_{i} }$$

(22)

is a lower bound on the optimal solution value of (19) with w ₁ set to 1. In particular, if $(\bar{\mathbf{w}},\bar{\boldsymbol{\xi }})$ is a feasible integral solution to (19) with objective function value $\lambda (\sum _{i=1}^{n}\bar{w}_{i}) +\sum _{ i=1}^{p}\bar{\xi }_{i}$, and if (22) is greater than this value, than no optimal integral solution of (19) can have w ₁ = 1. Therefore w ₁ = 0 in any optimal solution, and we can simply drop the column corresponding to w ₁ from the LP.

In order to use the screening results in this section we need to obtain a feasible primal and a feasible dual solution. Some useful heuristics to obtain such a pair are described in [12].

Appendix 3: Extending the Dual Solution for Row-Sampling

Suppose that $\hat{\boldsymbol{\mu }}^{p}$ is the optimal dual solution to the small LP in Sect. 5.3. Note that the number of variables in the dual for the large LP increases from p to $\bar{p}$ and the bound on the second constraint grows from $\lambda \mathbf{1}_{n} + \mathbf{1}^{T}\mathbf{A}_{\mathcal{Z}}$ to $\lambda \mathbf{1}_{n} + \mathbf{1}^{T}\bar{\mathbf{A}}_{\mathcal{Z}}$.

We use a greedy heuristic to extend $\hat{\boldsymbol{\mu }}^{p}$ to a feasible dual solution $\bar{\boldsymbol{\mu }}_{\bar{p}}$ of the large LP. We set $\bar{\mu }_{j} =\hat{\mu } _{j}$ for j = 1, . . , p. We extend the remaining entries $\bar{\mu }_{j}$ for $j = (p + 1),..,\bar{p}$ by setting a subset of its entries to 1 while satisfying the dual feasibility constraint. In other words the extension of $\boldsymbol{\bar{\mu }}$ corresponds to a subset $\mathcal{R}$ of the row indices $\{p + 1,\ldots,\bar{p}\}$ of $\bar{\mathbf{A}}_{\mathcal{P}}$ such that $\hat{\boldsymbol{\mu }}_{p}^{T}\mathbf{A}_{\mathcal{P}} +\sum _{i\in \mathcal{R}}(\bar{\mathbf{A}}_{\mathcal{P}})_{i} \leq \mathbf{1}^{T}\bar{\mathbf{A}}_{\mathcal{Z}}$. Having $\boldsymbol{\bar{\mu }}^{T}\mathbf{A}_{\mathcal{P}} \leq \mathbf{1}^{T}\mathbf{A}_{\mathcal{Z}}$ with $\boldsymbol{\bar{\mu }}$ extended by a binary vector implies that $\boldsymbol{\bar{\mu }}$ is feasible for (20). We initialize $\mathcal{R}$ to ∅ and then simply go through the unseen rows of $\bar{\mathbf{A}}_{\mathcal{P}}$ in some fixed order (increasing from p + 1 to $\bar{p}$), and for a row k, if

$$\displaystyle{\hat{\boldsymbol{\mu }}_{p}^{T}\mathbf{A}_{ \mathcal{P}} +\sum _{i\in \mathcal{R}}(\bar{\mathbf{A}}_{\mathcal{P}})_{i} + (\bar{\mathbf{A}}_{\mathcal{P}})_{k} \leq \mathbf{1}^{T}\bar{\mathbf{A}}_{ \mathcal{Z}},}$$

we set $\mathcal{R}$ to $\mathcal{R}\cup \{ k\}$. The heuristic (we call it H1) needs only a single pass through the matrix $\bar{\mathbf{A}}_{\mathcal{P}}$, and is thus very fast.

This heuristic, however, does not use the optimal solution $\hat{\mathbf{w}}^{m}$ in any way. Suppose $\hat{\mathbf{w}}^{m}$ were an optimal solution of the large LP. Then complementary slackness would imply that if $(\bar{\mathbf{A}}_{\mathcal{P}})_{i}\hat{\mathbf{w}}^{m} > 1$, then in any optimal dual solution $\boldsymbol{\mu },\mu _{i} = 0$. Thus, assuming $\hat{\mathbf{w}}^{m}$ is close to an optimal solution for the large LP, we modify heuristic H1 to obtain heuristic H2, by simply setting $\bar{\mu }_{i} = 0$ whenever $(\bar{\mathbf{A}}_{\mathcal{P}})_{i}\hat{\mathbf{w}}^{m} > 1$, while keeping the remaining steps unchanged.