1 Introduction

In many applications including homeland security and medical diagnosis, decision systems are composed of an ordered sequence of stages. Each stage is associated with a sensor or a physical sensing modality. Typically, a less informative sensor is cheap (or fast) while a more informative sensor is either expensive or requires more time to acquire a measurement. In practice, a measurement budget (or throughput constraint) does not allow all the modalities to be used simultaneously in making decisions. The goal in these scenarios is to attempt to classify examples with low cost sensors and limit the number of examples for which more expensive or time consuming informative sensor is required.

For example, in explosives detection, in the first stage, an infrared imager or a metal detector can be used with high throughput and low cost. A second stage could be the use of a slower, more expensive active millimeter wave (AMMW) scanner. The final third stage is a time consuming human inspection. In medical applications, first stages are typically non-invasive procedures (such as a physical exam) followed by more expensive tests (blood test, CT scan etc) and the final stages are invasive (surgical) procedures.

Many such examples share a common structure (see Fig. 1), and we list some of its salient aspects below:

  1. (A)

    Sensors & ordered stages: Each stage is associated with a new sensor measurement or a sensing modality. Multiple stages are an ordered sequence of sensors or sensor modalities with later stages corresponding to expensive or time-consuming measurements. In many situations, there is often some flexibility in choosing a sensing modality from a collection of possible modalities. In these cases, the optimal choice of sensing actions also becomes an issue. While our methodology can be modified to account for this more general setting, we primarily consider a fixed order of stages and sensing modalities in this paper. This is justified on account of the fact that many of the situations we have come across consist of a handful of sensors or sensing modalities. Consequently, for these situations, the problem of choosing sensor ordering is not justified since one could by brute force enumerate and optimize over the different possibilities.

  2. (B)

    Reject classifiers: Our sequential decision rules either attempt to fully classify an instance at each stage or “reject” the instance on to the next stage for more measurements in case of ambiguity. For example, in explosives detection, a decision rule in the first stage, based on IR scan, would attempt to detect whether or not a person is a threat and identify the explosive type/location in case of a threat. If the person is identified as a threat at the first stage it is unnecessary (and indeed dangerous—the explosive could be detonated) to seek more information. Similarly in medical diagnosis if a disease is diagnosed at an early stage, it makes sense to begin early treatment rather than waiting for more conclusive tests.

  3. (C)

    Information vs. computation: Note that our setup can only use the partial measurements acquired up to a stage in making a decision. In other methods, such as detection cascades (Viola and Jones 2004), the full measurement and therefore all the information is available to every stage. Therefore, any region in the feature space can be carved out with more complex regions in the measurement space, or equivalently complex features can be extracted but with higher costs. In contrast, we have only partial measurements (or information) and so any feature or classifier that we employ has to be agnostic to unavailable measurements at that stage.

Fig. 1
figure 1

Multi-stage system consists of K stages. Each stage is a binary classifier with a reject option. The system incurs a penalty of δ k+1 at kth stage if it rejects to seek more measurements. The kth classifier only sees the first k sensing modalities in making a decision

Full size image

The two stage example in Fig. 2 illustrates some of the advantages of our scheme over the alternative scheme that first acquires measurements from all the sensing modalities, which we refer to as the centralized classifier. A reject classifier utilizes the 2nd stage sensor only for a fraction of the data but achieves the same performance as the centralized classifier.

Fig. 2
figure 2

Advantage of a 2 stage classifier: 10 samples, binary (squares, circles). The red line is the optimal decision when using only 1st stage modality. The blue line is optimal if using both. The curve is classification error vs. samples rejected (cost) The red point corresponds to classifying everything at stage 1. The blue corresponds to rejecting everything and classifying using both modalities. The green is a partial reject strategy. The samples outside the green region are classified using only the first modality, and samples inside the region are rejected to stage 2 and are classified using both modalities. Note that blue and green have the same error, while the reject strategy (green) has to use 2nd stage sensor only for \({1 \over2}\) of examples, reducing the cost by a factor of 2 (Color figure online)

Full size image

Our approach is based on the so-called Prediction Time Cost Reduction approach (Kanani and Melville 2008). Specifically, we assume a set of training examples in which measurements from all the sensors or sensing modalities as well as the ground truth labels are available. Our goal is to derive sequential reject classifiers that reduces cost of measurement acquisition and error in the prediction (or testing) phase.

We show that this sequential reject classifier problem can be formulated as an instance of a partially observable Markov Decision Process (POMDP) (Kaelbling et al. 1998) when the class-specific probability models for the different sensor measurements are known. In this case the optimal sequential classifier can be cast as a solution to a Dynamic Program (DP). The DP solution is a sequence of stage-wise optimization problems, where each stage problem is a combination of the cost from the current stage and the cost-to-go function that is carried on from later stages.

Nevertheless, class probability models are typically unknown; our scenarios produce high-dimensional sensor data (such as images). Consequently, unlike some of the conventional approaches (Ji and Carin 2007), where probability models are first estimated to solve POMDPs, we have to adopt a non-parametric discriminative learning approach. We utilize the structure of the POMDP solution to empirically approximate the value of the cost-to-go function only at a discrete subset of the data-space. Next, instead of interpolating or parameterizing the cost-to-go function and learning it from data, we formulate an empirical discriminative objective that utilizes point-wise cost-to-go estimates evaluated on the training set and directly learn classifiers that minimize this objective. Using this decomposition, we formulate a novel multi-stage expected risk minimization (ERM) problem.

We solve this ERM problem at each stage by first factoring the cost function into classification and rejection decisions. When probability models are known, optimal strategies for a multi-class setting are given by the DP solution, but it is unclear how to mimic these strategies in the empirical setting. However, if we restrict ourselves to a binary classification setting then we can transform reject decisions into binary classification problems. Specifically, we show that the optimal reject classifier at each stage is a combination of two binary classifiers, one biased towards positive examples and the other biased towards negative examples. The disagreement region of the two then defines the reject region.

We then approximate this empirical risk with a global surrogates. We present an iterative solution and demonstrate local convergence properties. The solution is obtained in a boosting framework. We then extend well-known margin-based generalization bounds (Schapire et al. 1998) to this multi-stage setting. We tested our methods on synthetic, medical and explosives datasets. Our results demonstrate an advantage of multistage classifier: cost reduction without a significant sacrifice in accuracy.

1.1 Related work

1.1.1 Active feature acquisition (AFA)

The subject of this paper is not new and has been studied in the Machine Learning community as early as (MacKay 1992). Our work is closely related to the so called prediction time active feature acquisition (AFA) approach in the area of cost-sensitive learning. The goal there is to make sequential decisions of whether or not to acquire a new feature to improve prediction accuracy. A natural approach is to formalize a problem as an POMDP. Ji and Carin (2007) and Kapoor and Horvitz (2009) model the decision process and infer feature dependencies while taking acquisition costs into account. Sheng and Ling (2006), Bilgic and Getoor (2007), and Zubek and Dietterich (2002) study strategies for optimizing decision trees while minimizing acquisition costs. The construction is usually based on some purity metric such as entropy. Kanani and Melville (2008) propose a method that acquires an attribute if it increases an expected utility. However, all these methods require estimating a probability likelihood that a certain feature value occurs given the features collected so far. While surrogates based on classifiers or regressors can be employed to estimate likelihoods, this approach requires discrete, binary or quantized attributes. In contrast, our problem domain deals with high dimensional measurements (images consisting of million of pixels), so we develop a discriminative learning approach and formulate a multi-stage empirical risk optimization problem to reduce measurement costs and misclassification errors. At each stage, we solve the reject classification problem by factorizing the cost function into classification and rejection decisions. We then embed the rejection decision into a binary classification problem.

1.1.2 Single stage reject classifiers

Our paper is also closely related to the topic of reject classifiers, which has also been investigated. However, in the literature reject classifiers have been primarily considered in a single stage scenario. In the Bayesian framework, Chow (1970) introduced Chow’s rule for classification. It states that given an observation x and a reject cost δ and J classes, reject x if the maximum of the posteriors for each class is less than the reject cost: max k=1..J P(y=j|x)<δ. In the context of machine learning, the posterior distributions are not known, and a decision rule is estimated directly. One popular approach is to reject examples with a small margin. Specifically, in the context of support vector machine classifiers, Yuan and Casasent (2003), Bartlett and Wegkamp (2008), Rodríguez-Díaz and Castañón (2009), and Grandvalet et al. (2008) define a reject region to lie within a small distance (margin) to the separating hyperplane and embed this in the hinge loss of the SVM formulation. El-Yaniv and Wiener (2011) propose a reject criteria motivated by active learning but its implementation turns out to be computationally impractical. In contrast, we consider multiple stages of reject classifiers. We assume an error prone second stage which occurs in such fields as threat detection and medical imaging. In this scenario, rejecting in the margin is not always meaningful. Figure 3 illustrates that thresholding the margin to reject can lead to significant degradation. This usually happens when stage measurements are complimentary; then examples within a small margin of the 1st stage boundary may not be meaningful to reject. Multiple stages of margin based reject classifiers have been considered by Liu et al. (2008) using SVMs in image classification. The method does not take into account the cost of later stages and is similar to the myopic method that we compare in the Experiments section.

Fig. 3
figure 3

(a) Gaussian Mixture (binary). (b) Error rate vs reject rate on complementary measurements. 1st stage uses only dim 1. 2nd stage uses only dim 2. Myopic strategy (green) is thresholding the margin of the classifier, our method is global surrogate; Bayesian classifier (best performance). Thresholding the margin performs significantly worse than our method (Color figure online)

Full size image

1.1.3 Detection cascades

Our multi-stage sequential reject classifiers bears close resemblance to detection cascades. There is much literature on cascade design (see Zhang and Zhang 2010; Chen et al. 2012 and references therein) but most cascades roughly follow the set-up introduced by Viola and Jones (2004) to reduce computation cost during classification. At each stage in a cascade, there is a binary classifier with a very high detection rate and a mediocre false alarm rate. Each stage makes a partial decision; it either detects an instance as negative or passes it on to the next stage. Only the last stage in the cascade makes a full decision, namely, whether the example belongs to a positive or negative class.

There are several fundamental differences between detection cascades and the multi-stage reject classifiers (MSRC). A key difference is the system architecture. Detection cascades make partial binary decisions, delaying a positive decision until the final stage. In contrast, MSRCs can make full classification decisions at any stage. Conceptually, this distinction requires a fundamentally new approach; detection cascades work because their focus is on unbalanced problems with few positives and a large number of negatives; and so the goal at each stage is to admit large false positives with negligible missed detections. Consequently, each stage can be associated with a binary classification problem that is acutely sensitive to missed detections. In contrast, our scheme at each stage is a composite scheme composed of a classifier as well as a rejection decision. The rejection decision is itself a binary classification problem. In practice, MSRCs arise in important areas such as medical diagnosis and explosives detection as we argued in Sect. 1, item (B). As a performance metric detection cascades tradeoff missed detections at the final stage with average computation. MSRC’s tradeoff average misclassification errors against number of examples that reached later stages (i.e. required more sensors or sensing modalities). For these reasons it is difficult to directly compare algorithms developed for MSRCs to those developed for detection cascades. Nevertheless, our goals and resulting algorithms are similar to some of the issues that arise in cascade design (see Chen et al. 2012 and references therein), namely, perform a joint optimization for all the stages in a cascade given a cost structure for different features.

1.1.4 Other cost sensitive methods

Network intrusion detection systems (IDS) is an area where sequential decision systems have been explored (see Fan et al. 2000; Lee et al. 2002; Cordella and Sansone 2007). In IDS, features have different computation costs. For each cost level, a ruleset is learned. The goal is to use as many low cost rules as possible. In a related set-up, Fan et al. (2002) and Wang et al. (2003) consider a more general ensemble of base classifiers and explore how to minimize the ensemble size without sacrificing performance. In the test phase, for a sample, another classifier is added to the ensemble if the confidence of the current classification low. Here, similar to detection cascades, the goal is to reduce computation time. As we described in Sect. 1, item (C), the important distinction is that, in our setting, a decision is based only on the partial information acquired up to a stage. In a computation driven method, a stage (or base classifier) decides using a feature computed from the full measurement vector.

2 Problem statement

Let be distributed according to an unknown distribution . A data point has K features, x={x 1,x 2,…,x K }, and belongs to one of C classes indicated by its label y. A kth feature is extracted from a measurement acquired at kth stage. x k is allowed to be a vector. We define a truncated feature vector at kth stage: x k={x 1,x 2,…,x k }. Let be the space of the first k features such that .

The system has K stages, the order of the stages is fixed, and kth stage acquires a kth measurement. At each stage, k, there is a decision with a reject option, f k. It can either classify an example, , or delay the decision until the next stage, f k(x k)=r and incur a penalty of δ k+1. Here, r indicates the “reject” decision. f k has to make a decision using only the first k sensing modalities. The last stage K is terminal, a standard classifier. Define the system risk to be,

(1)

Here, R k is the cost of classifying at kth stage, and S k(x k)∈{0,1} is the binary state variable indicating whether x has been rejected up to kth stage.

If x is active and is misclassified, the penalty is 1.Footnote 1 If it is rejected then the system incurs a penalty of δ k+1, and the state variable for that example remains at 1.

(2)

2.1 Bayesian setting

In this section, we will digress from the discriminative setting and analyze the problem under the assumption that the underlying distribution is known. In doing so, we hope to discover some fundamental structure that will simplify our empirical risk formulation in the next section.

If is known the problem reduces to an POMDP, and the optimal strategy is to minimize the expected risk,

(3)

If we allow arbitrary decision functions then we can equivalently minimize conditional risk,

(4)

This problem—by appealing to dynamic programming—remarkably reduces to a single stage optimization problem for a modified risk function. To see this, we denote the cost-to-go,

and the modified risk functional,

and prove the following theorem,

Theorem 1

The optimal solution f 1,f 2,…,f K to the multi-stage risk in Eq. (4) decomposes to single stage optimization,

(5)

and the solution is:

(6)

Proof

To simplify our derivations, we assume uniform class prior probability: \(\mathrm{P}_{y} [ {y=\hat{y}} ]=\frac{1}{C}, \hat{y}=1, \ldots, C\). However, our results can be easily modified to account for a non-uniform prior. The expected conditional risk can be solved optimally by a dynamic program, where a DP recursion is,

(7)
(8)

Consider kth stage minimization, f k can take C+1 possible values {1,2,…C,r} and J k (x k,S k) can be recast as a conditional expected risk minimization,

(9)

Define,

$$\tilde{\delta}\bigl(x^k\bigr)=\delta^{k+1} + \mathbf{E}_{\mathbf{x}^{k+1} \ldots \mathbf{x}^K} \bigl[ {J_{k+1}\bigl(\mathbf{x}^{k+1},S^{k+1}=1 \bigr)} \bigm{|}{\mathbf{x}^k} \bigr] $$

and rewrite the conditional risk in (9),

(10)

Reject is the optimal decision if,

(11)

If reject is not the optimal strategy then a class is chosen to maximize the posterior probability:

(12)

which is exactly our claim. □

The main implication of this result is that if the cost-to-go function \(\tilde{\delta}^{k}(\mathbf{x}^{k})\) is known then the risk \(\tilde{R}_{k}(\cdot)\) is only a function of the current stage decision f k. Therefore, we can ignore all of the other stages and minimize a single stage risk. Effectively, we decomposed the multi-stage problem in Eq. (4) into a stage-wise optimization in Eq. (5).

Note that the modified risk functional, \(\tilde{R}_{k}\), is remarkably similar to R k except that the modified reject cost \(\tilde{\delta}^{k}(\mathbf{x}^{k})\) replaces the constant stage cost δ k. Also, consider the range for which δ k(x k) is meaningful. If we have C classes then a random guessing strategy would incur an average risk of \(1-\frac{1}{C}\). Therefore the risk for rejecting, \(\tilde{\delta}^{k}(\mathbf{x}^{k}) \leq1-\frac{1}{C}\) in order to be a meaningful option. The work in Chow (1970) contains a detailed analysis of single stage reject classifier in a Bayesian setting.

In the analysis of the POMDP, we allowed multiple classes because it is a natural extension of the binary case. However, each stage still has C+1 decisions, and it is unclear how to parameterizing such multi-class classifier with a reject option in an empirical setting. Parameterizing regular multi-class learning is a difficult problem in itself, and most existing techniques (Allwein et al. 2001) reduce the problem to a series of binary learning methods. In our setting, the reject option cannot be treated as an additional class since there is no ground truth labels for which examples should be rejected. So in forming the empirical risk problem, we restrict ourselves to the binary setting since it allows for an intuitive parametrization of a reject option which we describe in the next section. We leave the multi-class setting to be the subject of future research.

Reject classifier as two binary decisions

Consider a stage k classifier with a reject option from Theorem 1 in a binary classification setting, y∈{−1,+1}.

(13)

It is clear from the expression that we can express the decision regions in terms of two binary classifiers f n and f p . Observe that for a given reject cost \(\tilde{\delta}^{k}(\mathbf{x}^{k})\), the reject region is an intersection of two binary decision regions. To this end we further modify the risk function in terms of agreement and disagreement regions of the two classifiers, f n ,f p , namely,

(14)

Note that the above loss function is symmetric between f n and f p and so any optimal solution can be interchanged. Nevertheless, we claim:

Theorem 2

Suppose f n and f p are two binary classifiers that minimize \(\mathbf{E}[ L_{k}(\mathbf{x}^{k},y,f_{n},\allowbreak f_{p},\tilde{\delta}^{k}) \mid \mathbf{x}^{k} ]\) over all binary classifiers f n and f p . Then following resulting reject classifier:

(15)

is the minimizer for \(\mathbf{E}[ \tilde{R}_{k}(\mathbf{x}^{k},y,f,\tilde{\delta}^{k}) \mid \mathbf{x}^{k} ]\) in Theorem 1 and the kth stage minimizer in Eq. (3).

Proof

For a given x k and \(\tilde{\delta}(\mathbf{x}^{k})\),

By inspection, the decomposition in (15) is the optimal Bayesian classifier minimizing \(\mathbf{E}_{y} [ {\tilde{R}_{k}(\mathbf{x}^{k},y,f,\tilde{\delta}^{k})} \mid{ \mathbf{x}^{k}} ]\). □

We refer to Fig. 4 for an illustration. We can express the new loss compactly as follows:

(16)

Note that in arriving at this expression we have used: for binary variables a,b,c.

Fig. 4
figure 4

Optimal Reject Region can be expressed as the disagreement region of two binary classifiers (f n and f p )

Full size image

In summary, in this section, we derive the optimal POMDP solution and decouple a multi-stage risk to single stage optimization. Then, for the binary classification setting, we derive an optimal representation for a reject region classifier in terms of two biased binary decisions:

$$\min_{f^k} \mathbf{E}\bigl[ R(\mathbf{x},y, \ldots, f^k, \ldots \bigr] \rightarrow\min_{f^k} \mathbf{E}\bigl[ \tilde{R}_k \bigl(\mathbf{x}^k,y, f^k,\tilde{\delta}^k\bigr) \bigr] \rightarrow \min_{f_p^k,f_n^k} \mathbf{E}\bigl[ L_k\bigl( \mathbf{x}^k,y, f_p^k,f_n^k, \tilde{\delta}^k\bigr) \bigr] $$

2.2 Stage-wise empirical minimization

In this section, we assume that the probability model is no longer known and cannot be estimated due to high-dimensionality of the data. Instead, our task is to find multi-stage decision rules based on a given training set: (x 1,y 1),(x 2,y 2),…,(x N ,y N ). Here, we consider binary classification setting: y i ∈{+1,−1}.

We will take advantage of the stage-wise decomposition of the POMDP solution in Theorem 1 and parametrization of reject region in Theorem 2 to formulate an empirical version of the stage risk L k (⋅) in Eq. (16). However, this requires the knowledge of the cost-to-go, . Instead of trying to learn this complex function, we will define a point-wise empirical estimate of the cost-to-go on the training data:

$$\tilde{\delta}^k\bigl(\mathbf{x}^k_i\bigr) \to\tilde{\delta}^k_i,\quad i=1,2, \ldots, N $$

and use it to learn the decision boundaries directly.

Note that by definition, \(\tilde{\delta}^{k}(\mathbf{x}^{k}_{i})\) is a only function of f k+1,…,f K. So the cost-to-go estimate is conveniently defined by the recursion,

(17)

Now, we can form the empirical version of the risk in Eq. (5) and optimize for a solution at stage k over some family of functions, .

(18)

Observe that, as in standard setting, we need to constrain the class of decision rules here. This is because with no constraints the minimum risk is equal to zero and can be achieved in the first stage itself.

Note, our stage-wise decomposition significantly simplifies the ERM. The objective in Eq. (18) is only a function of \(f_{p}^{k},f_{n}^{k}\) given \(\tilde{\delta}^{k}_{i}\) and the state \(S^{k}_{i}\). To minimize an empirical version of a multi-stage risk in Eq. (3) is much more difficult due to stage interdependencies.

Given \(\delta^{k}_{i}\) and all the stages but the kth, we can solve (18) by iterating between \(f^{k}_{p}\) and \(f^{k}_{n}\). To solve for \(f^{k}_{p}\), we fix \(f^{k}_{n}\) and minimize a weighted error

(19)

We can solve for f n in the same fashion by fixing f p ,

(20)

To derive these expressions from (18), we used another identity for any binary variables a,b,c

(21)

Note the advantage of our parametrization from Theorem 2. We converted the problem from learning a complicated three region decision to learning two binary classifiers (f p ,f n ), where learning each of the binary classifiers reduces to solving a weighted binary classification problem. This is desirable since binary classification is a very well studied problem, and existing machine learning techniques can be utilized here, as we will demonstrate in the next section.

3 Algorithm

Minimizing the indicator loss is a hard problem. Instead, we take the usual ERM (empirical risk minimization; Friedman et al. 2001) approach and replace it with a surrogate. We introduce an algorithm in the boosting framework based on the analysis from the previous section. Boosting is just one of our many possible machine learning approaches that can be used to solve it. We use boosting because it is easy to implement and is known to have good performance.

Boosting is a way to combine simple classifiers to form a strong classifier. We are given a set of such weak classifiers . Note that the set of weak classifiers need not be finite. Also, denote as a subset of weak classifiers that operate only on the first k measurements of x. .

The strong classifier is the linear combination:

Boosting algorithm operates in rounds. Each round can be viewed as a coordinate descent step in the space of weak learners. In each round, a weak learner h j (x) (a descent direction) is selected to be added to the linear combination classifier, F(x). Then a weight q j (an optimal step size in that direction) is computed. This is repeated until termination criteria is reached. For details on boosting various losses please refers to (Rosset et al. 2004).

Global surrogate

In our algorithm, we use the sigmoid loss function \(\mathbf{C}(z)={1 \over1+\exp(z)} \) to approximate the indicator. Similar sigmoid based losses have been used in boosting before (Masnadi-Shirazi and Vasconcelos 2009). Each subproblem (19) reduces to boosting a weighted loss.

To solve for stage k, we keep the rest of the stages constant. To find \(f^{k}_{p}=\sum q_{j} h_{j}(\mathbf{x})\), we fix \(f_{n}^{k}\) and solve:

(22)

Note that the weights w i , state variables \(S^{k}_{i}\) and cost-to-go \(\tilde{\delta}^{k}_{i}\) are also expressed in terms of the C(z) instead of :

(23)

To solve for \(f_{n}^{k}\), we solve the same problem but keep \(f_{p}^{k}\) constant instead:

(24)

Note that the terms \(\tilde{\delta}_{i}^{k}\) and \(S_{i}^{k}\) do not depend on stage k and remain constant when solving for \(f_{p}^{k}\) and \(f_{n}^{k}\). For the ease of notation, we define a new term C r that indicates if x i is rejected at a kth stage. The term is close to one if \(f^{k}_{p}\) and \(f^{k}_{n}\) disagree (reject) and small if they agree.

$$\mathbf{C}_r\bigl(f^k_p,f^k_n, \mathbf{x}^k_i,y_i\bigr)= \mathbf{C} \bigl(y_i f^k_p(\mathbf{x}_i)\bigr)+ \mathbf{C}\bigl(y_i f^k_n(\mathbf{x}_i) \bigr) - 2 \mathbf{C}\bigl(y_i f^k_p( \mathbf{x}_i)\bigr) \mathbf{C}\bigl(y_i f^k_n( \mathbf{x}_i)\bigr) $$

The expressions for state variables and cost-to-go are now simplified.

(25)

The state variable remains greater than zero as long as x i is rejected at every stage. The expression for cost-to-go at kth stage is:

(26)

The last two terms are simply a surrogate for L k (⋅) from (16) in terms of C(⋅).

For the last stage (a standard binary classifier), we fix the first K−1 stages and solve:

(27)

Our algorithms performs cyclical optimization over the stages. To initialize \(f^{k}_{n},~f^{k}_{p}~\forall k\), we simply hard code \(f^{k}_{p}\) to classify any x as +1 and \(f^{k}_{n}\) as −1 so that all x’s are rejected to the last stage. Using these nominal classifiers, we compute \(S^{k}_{i}\) and \(\delta^{k}_{i}\) according to Eqs. (25) and (26), respectively.

At a stage k, for a fixed \(\delta^{k}_{i}\) and \(S^{k}_{i}\), we alternate among minimizing \(f^{k}_{p}\) and \(f^{k}_{n}\) according to Eqs. (22) and (24). In practice, we found that one iteration is sufficient.

Given a new estimate of stage k, we update \(\delta^{s}_{i}\) for s>k and \(S^{s}_{j}\) for s<k and then move on to optimizing another stage k′. Given an estimate for stage k′, we again update the state variables and cost-to-go for the rest of the system.

The stages are optimized in the following order. We start with the last stage and make our way backwards to the first stage. Then do a forward pass from 1st stage to last. These forward and back passes are repeated until convergence. See Algorithm 1.

Algorithm 1
figure 5

Global algorithm

Full size image

Our formulation allows us to form a surrogate for the entire risk in Eq. (1), not just for each subproblem. This enables us to prove the following theorem,

Theorem 3

Our global surrogate algorithm converges to a local minimum.

Proof

This is simply due to a fact that we are minimizing a global smooth cost function by coordinate descent over \(\mathbf{q}_{p}^{1}, \mathbf{q}^{1}_{n}, \mathbf{q}_{p}^{2}, \mathbf{q}^{2}_{n}, \ldots, \mathbf{q}^{K}\). Here, \(\mathbf{q}^{k}_{p}\) is the vector of weak learner weights parameterizing \(f^{k}_{p}\). For the derivation of three stage system global cost refer to Appendix B. □

However, since the global loss and the loss for each subproblem are non-convex programs, there is no global optimality guarantee. Theorem 3 ensures that our algorithm terminates.

Regularization to reduce overfitting

To reduce overtraining, we introduce a simple but effective regularization. For any loss C(z) and a parameter λ, we introduce a multiplicative term to the cost function: . Here, \(C'(z)=\frac{dC(z)}{dz}\). The term exp(λ|q|) limits how large a step size for a weak hypothesis can become. It also introduces a simple stopping criteria: abort if \({\sum_{i=1}^{n} C'(y_{i} f_{t}(x_{i})) y_{i} h_{t+1}(x_{i}) \over\sum_{i=1}^{n} \mathbf{C}(y_{i} f_{t}(x_{i}))} \leq\lambda\). This corresponds to a situation when no descent directions ( weak hypothesis h t+1 ) can be found to minimize the cost function.

4 Generalization error

Our system is composed of margin maximizing classifiers, therefore it is appropriate to derive generalization error bounds based on margins. It turns out that we can employ maximum margin generalization techniques from Schapire et al. (1998) to derive error bounds for a two stage version of the system. A two stage system consists of three boosted binary classifiers:

Theorem 4

Let be a distribution on , and let be a sample of m examples chosen independently at random according to , and a rejected subsample of size m r , . Assume that the base-classifier spaces and are finite, and let δ>0. Then with probability at least 1−δ over the random choice of the training set S, all boosted classifiers \(f^{1}_{n},f^{1}_{p},f^{2}\) satisfy the following bound for all θ 1>0 and θ 2>0:

(28)

Proof

The proof extends the approach in Schapire et al. (1998) to a two stage system. For complete details please refers to the appendix. □

The two stage system can be compactly expressed:

(29)

The system error is a sum of two terms: error at the 1st stage + error at the 2nd stage. Theorem 4 states the generalization error of F(x) is bounded by the empirical margin error over the training set S plus a term that is inversely proportional to the margins and the number of training samples at that stage. An interesting observation is that m r , number of samples that reaches the 2nd stage, depends on the reject classifier at the 1st stage. So if very few examples make it to the second stage then we do not have strong generalization.

5 Experiments

The goal is to demonstrate that a large fraction of data can be classified at an early stage using a cheap modality. In our experiments, we use four real life datasets with measurements arising from meaningful stages.

5.1 Related algorithms

We compare our algorithm to two methods:

Myopic

An absolute margin of a classifier is a measure of how confident a classifier is on an example. Examples with small margin have low confidence and should be rejected to the next stage to acquire more features. This approach is based on reject classification (Bartlett and Wegkamp 2008). We know from Theorem 1 that the optimal classifier is a threshold of the posterior. For each stage, we obtain a binary boosted classifier, f k(⋅), trained on all the data. We then threshold the margin of the classifier, |f k(x k)|. It is known that given an infinite amount of training data, boosting certain losses (sigmoid loss in our case) approaches the log likelihood ratio, \(f(\mathbf{x})={1 \over2} \log{ \mathrm{P}(y=1|\mathbf{x}) \over \mathrm{P}(y=-1|\mathbf{x})}\) (Masnadi-Shirazi and Vasconcelos 2009). So a reject region for a given threshold t k is defined: {x∣|f k(x)|≤t k }. This is a completely myopic approach as the rejection does not take into account performance of later stages. This method is very similar to TEFE (Liu et al. 2008) which also uses absolute margin as a measure for rejection. The difference is that our myopic strategy is a boosting classifier not an SVM as used in TEFE.

Expected utility/margin

An expected margin difference measures how a new attribute, if acquired, would be useful for an example. If this expected utility for an example is large then a new attribute should be acquired. This approach is based on the work by Kanani and Melville (2008). We train boosted binary classifiers on all the data for each stage: f k(x k). Given the measurement at the current stage x k, we compute an expected utility (change in normalized margin) of acquiring the next measurement x k+1:

An x k is rejected to the next stage if its utility U(x k)≥t k is greater than a threshold. Here, denotes the possible values that x k+1 can take. Note this approach requires estimating P(x k+1|x k),Footnote 2 therefore the (k+1)th measurement has to be discrete or distribution needs to be parametrized. Due to this limitation, we only compare this method on two datasets.

5.2 Simulations

Performance metric

A natural performance metric is the trade off between system error and measurement cost. Note, for utility and myopic methods, it is unclear how to set a thresholds t k for each stage given a measurement cost δ k . For this reason, we only compare them in a two stages system. More than two stages is not-practical because we would need to test every possible t k for every stage k.

In a two stage setting, since every example has to pass through the first stage, only the cost of the second stage, δ 2, affects the performance. The average measurement cost of the system is proportional to δ 1+ (the fraction of examples rejected to the second stage) ×δ 2. So knowing the exact cost of the second stage sensor, δ=δ 2, is not necessary. In our algorithm, we vary δ to generate a system error vs reject rate plot. For margin and utility, we sweep a threshold t k . System error is the sum of 1st stage and 2nd stage errors. Reject rate is the fraction of examples rejected to the 2nd stage and require additional measurements. Low reject rate (cost) corresponds to higher error rate as most of the data will be classified at the first stage using less informative measurements. High reject rate will have performance similar to a centralized classifier, as most examples will be classified at the 2nd stage.

Set up

In all our experiments, we use stumps as weak learners. A stump classifier h d,g,s ∈{+1,−1} is parametrized by a threshold g on dth dimension and a sign variable s∈{+1,−1}: h d,g,s (x)=s×sgn[x d −g]. We chose stumps for their simplicity, computation speed and relatively good performance. While more complicated weak learners, such as decision trees can be used, they would only change the absolute performance of our experiments. The entire curves would just move vertically up or down. Our goal is to demonstrate the advantage of a multi-stage classifier relative to the centralized system (a system that uses all the measurements for all examples).

For each dataset and experiment, we randomly split the data 50/50 for training and testing. The results are evaluated on a separate test set, and the simulations are averaged over 50 Monte-Carlo trials. The number of iterations for each boosting subproblem is set to T=50. In our global surrogate algorithm, the number of outer loop iterations is set to D=10. Refer to Table 1 for dataset descriptions and to Table 2 for summary of our experiments.

Table 1 Dataset descriptions
Full size table
Table 2 Performance illustration for different datasets (quantitate view of the curves). Datasets have 2 sensing modalities. Centralized denotes the test error obtained with all modalities. Last three columns denote performance for different approaches. Performance is measured by the average number of examples requiring 2nd stage to achieve error close to centralized. Utility approach does not work for last three datasets due to high-dimensionality issues. We note the significant gains of our approach over competing ones of many interesting datasets
Full size table

Discrete valued data experiments

To compare our method to the utility approach, we consider discrete data. The first dataset is a quantized (with 20 levels) Gaussian mixture synthetic data in two dimension. The 1st dimension is stage one; the 2nd dimension is stage two. The second dataset is Mammogram Mass from UCI Machine Learning Repository. It is used to predict the severity of a mammographic mass lesion (malicious or benign). It contains 3 attributes extracted from the CAD image and also an evaluation by a radiologist on a confidence scale in addition to the true biopsy results. The first stage are features extracted from the CAD image, and the second stage is the expert confidence rated on a discrete scale 1–5. Automatic analysis of the CAD image is cheaper than employing an opinion of a radiologist.

Simulations in Fig. 5 demonstrate that utility performs worse when compared to our approach. Our approach requires far fewer 2nd stage measurements to achieve an error rate close to the centralized strategy than the utility approach. This is possibly due to poor probability estimates in limited data setting.

Fig. 5
figure 6

Comparison of Global to Utility on (a) quantized two Gaussian clusters and (b) mammogram dataset. Reject Rate vs System Error. Reject Rate is the fraction of examples with measurements from both stages. Our approach outperforms Utility possibly because we do not need to estimate probability likelihoods

Full size image

Continuous valued data experiments

We compare our global method to the myopic method on three datasets. The Pima Indians Diabetes Dataset (UCI MLR) consists of 8 measurements. Since the stages are not specified in this dataset, we group measurements with similar costs into separate modalities. 6 of the measurements are inexpensive to acquire and consist of simple tests such as body mass index, age, pedigree. These we designate as the first stage. The other two measurements constitute the second stage and require more expensive procedures.

The polyp dataset consists of hyper-spectral measurements of colon polyps collected during colonoscopies (Rodríguez-Díaz and Castañón 2009). The attribute is a measured intensity at 126 equally spaced frequencies. Finer resolution requires higher photon count which is proportional to acquisition time. For a first stage, we use a coarse measurement downsampled to only 12 frequency bins. The second stage is the full resolution frequency response. Using the course measurements is cheaper than acquiring the full resolution.

The threat dataset contains images taken of people wearing various explosives devices. The imaging is done in three modalities: infrared (IR), passive millimeter wave (PMMW), and active millimeter (AMMW). All the images are registered. We extract many patches from the images and use them as our training data. A patch carries a binary label, it either contains a threat or is clean. IR and PMMW are the fastest modalities but also less informative. AMMW requires raster scanning a person and is slow but also the most useful.

In Fig. 6, global performs better than margin in most cases. On threat data, margin appears to be doing just marginally worse than global, however, we get only a few points on the curve with reject rates less than 50 %. Due to the heuristic nature of margin, we cannot construct a multistage classifier with an arbitrary reject rate.

Fig. 6
figure 7

Three datasets are evaluated: pima, polyps and threat. Reject Rate vs Error Rate for a varying reject cost δ. Reject Rate is the fraction of examples with measurements from both stages. Global and Myopic are compared. Global (our approach) has a better performance over all while Myopic does better in some situations

Full size image

The goal is to reach the performance of a centralized classifier (100 % reject rate) while utilizing the 2nd stage sensor only for a small fraction of examples. Overall, the results demonstrate the benefit of multi-stage classification: rejection rate can be set to less than 50 % with only small sacrifices in performance. For the mammogram data, this implies that for half of the patients a diagnoses can be made solely by an automatic analysis of a CAD image without an expensive opinion of a radiologist. For the Pima data, similar error can be achieved without an expensive medical procedures. For the polyps dataset, a fast low resolution measurement is enough to classify a large fraction of patience. In the threat dataset, IR and PMMW are sufficient to decide whether or not a threat is present for the majority of instances without requiring a person to go through a slower AMMW scanner.

Unbalanced false positive and false negative penalties

In medical diagnosis and threat detection, the penalty of false positives and false negatives is not equal. We can easily adapt our algorithm to account for such setting. Empirical Risk in (18) can be modified to include a penalty of w p for a Type I error and w n for a Type II error. The experiment in Fig. 7 demonstrates our global algorithms in such scenario. For each reject cost δ, we compute an ROC curve. This allows to select an operating point of the system with a desired false alarm or detection rate. We also compute a corresponding average reject rate for each value of δ. So the highest reject rate corresponds to the best performance but also to the highest acquisition cost incurred by the system. Note that very good performance can be achieved by requesting only 50 % of instances to be measured at the second stage.

Fig. 7
figure 8

Two Stage ROC using the global surrogate method. Each ROC curve corresponds to a different value of reject cost δ. The inset displays average reject rate for δ’s. Note, the red ROC corresponds to the centralized system (100 % reject rate). Very good performance can be achieved by requesting only 50 % of instances to be measured at the second stage (Color figure online)

Full size image

Three stages

Lastly, we demonstrate a three stage system, we apply our algorithm to three stages of threat dataset. Note for margin it is unclear how to generalize it to a multistage scenario and there is no way to define reject costs for different stages. We set the first stage to be IR, second PMMW and AMMW as third. There is no cost for acquiring IR. We vary the costs for the PMMW (2nd) stage, δ 1, and AMMW (3rd), δ 2, to generate an error map (color in Fig. 8). A point on the map corresponds to a performance of a particular multistage classification strategy. The vertical axis is the fraction of examples for which only IR and PMMW measurements are used in making a decision. The horizontal axis is the fraction of examples for which all three modalities are used. For example, a red point in the figure, {.4,.15,.195}, correspond to a system where 40 % of examples use IR and PMMW, 15 % use only IR and the rest of data (45 %) use all the modalities. And this strategy achieves a system error rate of 19.5 %. Note that the support lies below the diagonal. This is because the sum or reject rates has to be less than one. Results demonstrate some interesting observations. While best performance (about 19 %) is achieved when all the modalities are used for every example, we can move along the vertical lines and allow a fraction to be classified by IR and PMMW, avoiding AMMW all together. This strategy achieves performance comparable to a centralized system, (IR+PMMW+AMMW).

Fig. 8
figure 9

Three Stage System. The color maps error. A point on the map corresponds to a performance of a particular multistage classification strategy. The vertical axis is the fraction of examples for which only IR and PMMW measurements are used in making a decision. The horizontal axis is the fraction of examples for which all three modalities are used. An example red point in the figure, {.4,.15,.195}, correspond to a system where 40 % of examples use IR and PMMW, 15 % use only IR and the rest of data (45 %) use all the modalities. And this strategy achieves a system error rate of 19.5 % (Color figure online)

Full size image

6 Conclusion

In this paper, we propose a general framework for a sequential decision system in a non-parametric setting. Starting from basic principles, we derive the Bayesian optimal solution. Then, to simplify the problem, we parameterize a classifier at each stage in terms of two binary decisions. We formulate an ERM problem and optimize it by alternatively minimizing one stage at a time. Remarkably, all subproblems turn out to be weighed binary error minimizations. We introduce a practical boosting algorithm that minimizes a global surrogate of the empirical risk and test it on several datasets. Results show the advantage of our formulation to more heuristic approaches. Overall, our experiments demonstrate how multi-stage classifiers can achieve good performance by acquiring full measurements only for a fraction of samples.