Abstract
In many classification systems, sensing modalities have different acquisition costs. It is often unnecessary to use every modality to classify a majority of examples. We study a multistage system in a prediction time cost reduction setting, where the full data is available for training, but for a test example, measurements in a new modality can be acquired at each stage for an additional cost. We seek decision rules to reduce the average measurement acquisition cost. We formulate an empirical risk minimization problem (ERM) for a multistage reject classifier, wherein the stage k classifier either classifies a sample using only the measurements acquired so far or rejects it to the next stage where more attributes can be acquired for a cost. If we restrict ourselves to binary classification setting then, to solve the ERM problem, we show that the optimal reject classifier at each stage is a combination of two binary classifiers, one biased towards positive examples and the other biased towards negative examples. We use this parameterization to construct stagebystage global surrogate risk, develop an iterative algorithm in the boosting framework and present convergence and generalization results. We test our work on synthetic, medical and explosives detection datasets. Our results demonstrate that substantial cost reduction without a significant sacrifice in accuracy is achievable.
Introduction
In many applications including homeland security and medical diagnosis, decision systems are composed of an ordered sequence of stages. Each stage is associated with a sensor or a physical sensing modality. Typically, a less informative sensor is cheap (or fast) while a more informative sensor is either expensive or requires more time to acquire a measurement. In practice, a measurement budget (or throughput constraint) does not allow all the modalities to be used simultaneously in making decisions. The goal in these scenarios is to attempt to classify examples with low cost sensors and limit the number of examples for which more expensive or time consuming informative sensor is required.
For example, in explosives detection, in the first stage, an infrared imager or a metal detector can be used with high throughput and low cost. A second stage could be the use of a slower, more expensive active millimeter wave (AMMW) scanner. The final third stage is a time consuming human inspection. In medical applications, first stages are typically noninvasive procedures (such as a physical exam) followed by more expensive tests (blood test, CT scan etc) and the final stages are invasive (surgical) procedures.
Many such examples share a common structure (see Fig. 1), and we list some of its salient aspects below:

(A)
Sensors & ordered stages: Each stage is associated with a new sensor measurement or a sensing modality. Multiple stages are an ordered sequence of sensors or sensor modalities with later stages corresponding to expensive or timeconsuming measurements. In many situations, there is often some flexibility in choosing a sensing modality from a collection of possible modalities. In these cases, the optimal choice of sensing actions also becomes an issue. While our methodology can be modified to account for this more general setting, we primarily consider a fixed order of stages and sensing modalities in this paper. This is justified on account of the fact that many of the situations we have come across consist of a handful of sensors or sensing modalities. Consequently, for these situations, the problem of choosing sensor ordering is not justified since one could by brute force enumerate and optimize over the different possibilities.

(B)
Reject classifiers: Our sequential decision rules either attempt to fully classify an instance at each stage or “reject” the instance on to the next stage for more measurements in case of ambiguity. For example, in explosives detection, a decision rule in the first stage, based on IR scan, would attempt to detect whether or not a person is a threat and identify the explosive type/location in case of a threat. If the person is identified as a threat at the first stage it is unnecessary (and indeed dangerous—the explosive could be detonated) to seek more information. Similarly in medical diagnosis if a disease is diagnosed at an early stage, it makes sense to begin early treatment rather than waiting for more conclusive tests.

(C)
Information vs. computation: Note that our setup can only use the partial measurements acquired up to a stage in making a decision. In other methods, such as detection cascades (Viola and Jones 2004), the full measurement and therefore all the information is available to every stage. Therefore, any region in the feature space can be carved out with more complex regions in the measurement space, or equivalently complex features can be extracted but with higher costs. In contrast, we have only partial measurements (or information) and so any feature or classifier that we employ has to be agnostic to unavailable measurements at that stage.
The two stage example in Fig. 2 illustrates some of the advantages of our scheme over the alternative scheme that first acquires measurements from all the sensing modalities, which we refer to as the centralized classifier. A reject classifier utilizes the 2nd stage sensor only for a fraction of the data but achieves the same performance as the centralized classifier.
Our approach is based on the socalled Prediction Time Cost Reduction approach (Kanani and Melville 2008). Specifically, we assume a set of training examples in which measurements from all the sensors or sensing modalities as well as the ground truth labels are available. Our goal is to derive sequential reject classifiers that reduces cost of measurement acquisition and error in the prediction (or testing) phase.
We show that this sequential reject classifier problem can be formulated as an instance of a partially observable Markov Decision Process (POMDP) (Kaelbling et al. 1998) when the classspecific probability models for the different sensor measurements are known. In this case the optimal sequential classifier can be cast as a solution to a Dynamic Program (DP). The DP solution is a sequence of stagewise optimization problems, where each stage problem is a combination of the cost from the current stage and the costtogo function that is carried on from later stages.
Nevertheless, class probability models are typically unknown; our scenarios produce highdimensional sensor data (such as images). Consequently, unlike some of the conventional approaches (Ji and Carin 2007), where probability models are first estimated to solve POMDPs, we have to adopt a nonparametric discriminative learning approach. We utilize the structure of the POMDP solution to empirically approximate the value of the costtogo function only at a discrete subset of the dataspace. Next, instead of interpolating or parameterizing the costtogo function and learning it from data, we formulate an empirical discriminative objective that utilizes pointwise costtogo estimates evaluated on the training set and directly learn classifiers that minimize this objective. Using this decomposition, we formulate a novel multistage expected risk minimization (ERM) problem.
We solve this ERM problem at each stage by first factoring the cost function into classification and rejection decisions. When probability models are known, optimal strategies for a multiclass setting are given by the DP solution, but it is unclear how to mimic these strategies in the empirical setting. However, if we restrict ourselves to a binary classification setting then we can transform reject decisions into binary classification problems. Specifically, we show that the optimal reject classifier at each stage is a combination of two binary classifiers, one biased towards positive examples and the other biased towards negative examples. The disagreement region of the two then defines the reject region.
We then approximate this empirical risk with a global surrogates. We present an iterative solution and demonstrate local convergence properties. The solution is obtained in a boosting framework. We then extend wellknown marginbased generalization bounds (Schapire et al. 1998) to this multistage setting. We tested our methods on synthetic, medical and explosives datasets. Our results demonstrate an advantage of multistage classifier: cost reduction without a significant sacrifice in accuracy.
Related work
Active feature acquisition (AFA)
The subject of this paper is not new and has been studied in the Machine Learning community as early as (MacKay 1992). Our work is closely related to the so called prediction time active feature acquisition (AFA) approach in the area of costsensitive learning. The goal there is to make sequential decisions of whether or not to acquire a new feature to improve prediction accuracy. A natural approach is to formalize a problem as an POMDP. Ji and Carin (2007) and Kapoor and Horvitz (2009) model the decision process and infer feature dependencies while taking acquisition costs into account. Sheng and Ling (2006), Bilgic and Getoor (2007), and Zubek and Dietterich (2002) study strategies for optimizing decision trees while minimizing acquisition costs. The construction is usually based on some purity metric such as entropy. Kanani and Melville (2008) propose a method that acquires an attribute if it increases an expected utility. However, all these methods require estimating a probability likelihood that a certain feature value occurs given the features collected so far. While surrogates based on classifiers or regressors can be employed to estimate likelihoods, this approach requires discrete, binary or quantized attributes. In contrast, our problem domain deals with high dimensional measurements (images consisting of million of pixels), so we develop a discriminative learning approach and formulate a multistage empirical risk optimization problem to reduce measurement costs and misclassification errors. At each stage, we solve the reject classification problem by factorizing the cost function into classification and rejection decisions. We then embed the rejection decision into a binary classification problem.
Single stage reject classifiers
Our paper is also closely related to the topic of reject classifiers, which has also been investigated. However, in the literature reject classifiers have been primarily considered in a single stage scenario. In the Bayesian framework, Chow (1970) introduced Chow’s rule for classification. It states that given an observation x and a reject cost δ and J classes, reject x if the maximum of the posteriors for each class is less than the reject cost: max_{ k=1..J }P(y=jx)<δ. In the context of machine learning, the posterior distributions are not known, and a decision rule is estimated directly. One popular approach is to reject examples with a small margin. Specifically, in the context of support vector machine classifiers, Yuan and Casasent (2003), Bartlett and Wegkamp (2008), RodríguezDíaz and Castañón (2009), and Grandvalet et al. (2008) define a reject region to lie within a small distance (margin) to the separating hyperplane and embed this in the hinge loss of the SVM formulation. ElYaniv and Wiener (2011) propose a reject criteria motivated by active learning but its implementation turns out to be computationally impractical. In contrast, we consider multiple stages of reject classifiers. We assume an error prone second stage which occurs in such fields as threat detection and medical imaging. In this scenario, rejecting in the margin is not always meaningful. Figure 3 illustrates that thresholding the margin to reject can lead to significant degradation. This usually happens when stage measurements are complimentary; then examples within a small margin of the 1st stage boundary may not be meaningful to reject. Multiple stages of margin based reject classifiers have been considered by Liu et al. (2008) using SVMs in image classification. The method does not take into account the cost of later stages and is similar to the myopic method that we compare in the Experiments section.
Detection cascades
Our multistage sequential reject classifiers bears close resemblance to detection cascades. There is much literature on cascade design (see Zhang and Zhang 2010; Chen et al. 2012 and references therein) but most cascades roughly follow the setup introduced by Viola and Jones (2004) to reduce computation cost during classification. At each stage in a cascade, there is a binary classifier with a very high detection rate and a mediocre false alarm rate. Each stage makes a partial decision; it either detects an instance as negative or passes it on to the next stage. Only the last stage in the cascade makes a full decision, namely, whether the example belongs to a positive or negative class.
There are several fundamental differences between detection cascades and the multistage reject classifiers (MSRC). A key difference is the system architecture. Detection cascades make partial binary decisions, delaying a positive decision until the final stage. In contrast, MSRCs can make full classification decisions at any stage. Conceptually, this distinction requires a fundamentally new approach; detection cascades work because their focus is on unbalanced problems with few positives and a large number of negatives; and so the goal at each stage is to admit large false positives with negligible missed detections. Consequently, each stage can be associated with a binary classification problem that is acutely sensitive to missed detections. In contrast, our scheme at each stage is a composite scheme composed of a classifier as well as a rejection decision. The rejection decision is itself a binary classification problem. In practice, MSRCs arise in important areas such as medical diagnosis and explosives detection as we argued in Sect. 1, item (B). As a performance metric detection cascades tradeoff missed detections at the final stage with average computation. MSRC’s tradeoff average misclassification errors against number of examples that reached later stages (i.e. required more sensors or sensing modalities). For these reasons it is difficult to directly compare algorithms developed for MSRCs to those developed for detection cascades. Nevertheless, our goals and resulting algorithms are similar to some of the issues that arise in cascade design (see Chen et al. 2012 and references therein), namely, perform a joint optimization for all the stages in a cascade given a cost structure for different features.
Other cost sensitive methods
Network intrusion detection systems (IDS) is an area where sequential decision systems have been explored (see Fan et al. 2000; Lee et al. 2002; Cordella and Sansone 2007). In IDS, features have different computation costs. For each cost level, a ruleset is learned. The goal is to use as many low cost rules as possible. In a related setup, Fan et al. (2002) and Wang et al. (2003) consider a more general ensemble of base classifiers and explore how to minimize the ensemble size without sacrificing performance. In the test phase, for a sample, another classifier is added to the ensemble if the confidence of the current classification low. Here, similar to detection cascades, the goal is to reduce computation time. As we described in Sect. 1, item (C), the important distinction is that, in our setting, a decision is based only on the partial information acquired up to a stage. In a computation driven method, a stage (or base classifier) decides using a feature computed from the full measurement vector.
Problem statement
Let be distributed according to an unknown distribution . A data point has K features, x={x _{1},x _{2},…,x _{ K }}, and belongs to one of C classes indicated by its label y. A kth feature is extracted from a measurement acquired at kth stage. x _{ k } is allowed to be a vector. We define a truncated feature vector at kth stage: x ^{k}={x _{1},x _{2},…,x _{ k }}. Let be the space of the first k features such that .
The system has K stages, the order of the stages is fixed, and kth stage acquires a kth measurement. At each stage, k, there is a decision with a reject option, f ^{k}. It can either classify an example, , or delay the decision until the next stage, f ^{k}(x ^{k})=r and incur a penalty of δ ^{k+1}. Here, r indicates the “reject” decision. f ^{k} has to make a decision using only the first k sensing modalities. The last stage K is terminal, a standard classifier. Define the system risk to be,
Here, R _{ k } is the cost of classifying at kth stage, and S ^{k}(x ^{k})∈{0,1} is the binary state variable indicating whether x has been rejected up to kth stage.
If x is active and is misclassified, the penalty is 1.^{Footnote 1} If it is rejected then the system incurs a penalty of δ ^{k+1}, and the state variable for that example remains at 1.
Bayesian setting
In this section, we will digress from the discriminative setting and analyze the problem under the assumption that the underlying distribution is known. In doing so, we hope to discover some fundamental structure that will simplify our empirical risk formulation in the next section.
If is known the problem reduces to an POMDP, and the optimal strategy is to minimize the expected risk,
If we allow arbitrary decision functions then we can equivalently minimize conditional risk,
This problem—by appealing to dynamic programming—remarkably reduces to a single stage optimization problem for a modified risk function. To see this, we denote the costtogo,
and the modified risk functional,
and prove the following theorem,
Theorem 1
The optimal solution f ^{1},f ^{2},…,f ^{K} to the multistage risk in Eq. (4) decomposes to single stage optimization,
and the solution is:
Proof
To simplify our derivations, we assume uniform class prior probability: \(\mathrm{P}_{y} [ {y=\hat{y}} ]=\frac{1}{C}, \hat{y}=1, \ldots, C\). However, our results can be easily modified to account for a nonuniform prior. The expected conditional risk can be solved optimally by a dynamic program, where a DP recursion is,
Consider kth stage minimization, f ^{k} can take C+1 possible values {1,2,…C,r} and J _{ k }(x ^{k},S ^{k}) can be recast as a conditional expected risk minimization,
Define,
and rewrite the conditional risk in (9),
Reject is the optimal decision if,
If reject is not the optimal strategy then a class is chosen to maximize the posterior probability:
which is exactly our claim. □
The main implication of this result is that if the costtogo function \(\tilde{\delta}^{k}(\mathbf{x}^{k})\) is known then the risk \(\tilde{R}_{k}(\cdot)\) is only a function of the current stage decision f ^{k}. Therefore, we can ignore all of the other stages and minimize a single stage risk. Effectively, we decomposed the multistage problem in Eq. (4) into a stagewise optimization in Eq. (5).
Note that the modified risk functional, \(\tilde{R}_{k}\), is remarkably similar to R _{ k } except that the modified reject cost \(\tilde{\delta}^{k}(\mathbf{x}^{k})\) replaces the constant stage cost δ ^{k}. Also, consider the range for which δ ^{k}(x ^{k}) is meaningful. If we have C classes then a random guessing strategy would incur an average risk of \(1\frac{1}{C}\). Therefore the risk for rejecting, \(\tilde{\delta}^{k}(\mathbf{x}^{k}) \leq1\frac{1}{C}\) in order to be a meaningful option. The work in Chow (1970) contains a detailed analysis of single stage reject classifier in a Bayesian setting.
In the analysis of the POMDP, we allowed multiple classes because it is a natural extension of the binary case. However, each stage still has C+1 decisions, and it is unclear how to parameterizing such multiclass classifier with a reject option in an empirical setting. Parameterizing regular multiclass learning is a difficult problem in itself, and most existing techniques (Allwein et al. 2001) reduce the problem to a series of binary learning methods. In our setting, the reject option cannot be treated as an additional class since there is no ground truth labels for which examples should be rejected. So in forming the empirical risk problem, we restrict ourselves to the binary setting since it allows for an intuitive parametrization of a reject option which we describe in the next section. We leave the multiclass setting to be the subject of future research.
Reject classifier as two binary decisions
Consider a stage k classifier with a reject option from Theorem 1 in a binary classification setting, y∈{−1,+1}.
It is clear from the expression that we can express the decision regions in terms of two binary classifiers f _{ n } and f _{ p }. Observe that for a given reject cost \(\tilde{\delta}^{k}(\mathbf{x}^{k})\), the reject region is an intersection of two binary decision regions. To this end we further modify the risk function in terms of agreement and disagreement regions of the two classifiers, f _{ n },f _{ p }, namely,
Note that the above loss function is symmetric between f _{ n } and f _{ p } and so any optimal solution can be interchanged. Nevertheless, we claim:
Theorem 2
Suppose f _{ n } and f _{ p } are two binary classifiers that minimize \(\mathbf{E}[ L_{k}(\mathbf{x}^{k},y,f_{n},\allowbreak f_{p},\tilde{\delta}^{k}) \mid \mathbf{x}^{k} ]\) over all binary classifiers f _{ n } and f _{ p }. Then following resulting reject classifier:
is the minimizer for \(\mathbf{E}[ \tilde{R}_{k}(\mathbf{x}^{k},y,f,\tilde{\delta}^{k}) \mid \mathbf{x}^{k} ]\) in Theorem 1 and the kth stage minimizer in Eq. (3).
Proof
For a given x ^{k} and \(\tilde{\delta}(\mathbf{x}^{k})\),
By inspection, the decomposition in (15) is the optimal Bayesian classifier minimizing \(\mathbf{E}_{y} [ {\tilde{R}_{k}(\mathbf{x}^{k},y,f,\tilde{\delta}^{k})} \mid{ \mathbf{x}^{k}} ]\). □
We refer to Fig. 4 for an illustration. We can express the new loss compactly as follows:
Note that in arriving at this expression we have used: for binary variables a,b,c.
In summary, in this section, we derive the optimal POMDP solution and decouple a multistage risk to single stage optimization. Then, for the binary classification setting, we derive an optimal representation for a reject region classifier in terms of two biased binary decisions:
Stagewise empirical minimization
In this section, we assume that the probability model is no longer known and cannot be estimated due to highdimensionality of the data. Instead, our task is to find multistage decision rules based on a given training set: (x _{1},y _{1}),(x _{2},y _{2}),…,(x _{ N },y _{ N }). Here, we consider binary classification setting: y _{ i }∈{+1,−1}.
We will take advantage of the stagewise decomposition of the POMDP solution in Theorem 1 and parametrization of reject region in Theorem 2 to formulate an empirical version of the stage risk L _{ k }(⋅) in Eq. (16). However, this requires the knowledge of the costtogo, . Instead of trying to learn this complex function, we will define a pointwise empirical estimate of the costtogo on the training data:
and use it to learn the decision boundaries directly.
Note that by definition, \(\tilde{\delta}^{k}(\mathbf{x}^{k}_{i})\) is a only function of f ^{k+1},…,f ^{K}. So the costtogo estimate is conveniently defined by the recursion,
Now, we can form the empirical version of the risk in Eq. (5) and optimize for a solution at stage k over some family of functions, .
Observe that, as in standard setting, we need to constrain the class of decision rules here. This is because with no constraints the minimum risk is equal to zero and can be achieved in the first stage itself.
Note, our stagewise decomposition significantly simplifies the ERM. The objective in Eq. (18) is only a function of \(f_{p}^{k},f_{n}^{k}\) given \(\tilde{\delta}^{k}_{i}\) and the state \(S^{k}_{i}\). To minimize an empirical version of a multistage risk in Eq. (3) is much more difficult due to stage interdependencies.
Given \(\delta^{k}_{i}\) and all the stages but the kth, we can solve (18) by iterating between \(f^{k}_{p}\) and \(f^{k}_{n}\). To solve for \(f^{k}_{p}\), we fix \(f^{k}_{n}\) and minimize a weighted error
We can solve for f _{ n } in the same fashion by fixing f _{ p },
To derive these expressions from (18), we used another identity for any binary variables a,b,c
Note the advantage of our parametrization from Theorem 2. We converted the problem from learning a complicated three region decision to learning two binary classifiers (f _{ p },f _{ n }), where learning each of the binary classifiers reduces to solving a weighted binary classification problem. This is desirable since binary classification is a very well studied problem, and existing machine learning techniques can be utilized here, as we will demonstrate in the next section.
Algorithm
Minimizing the indicator loss is a hard problem. Instead, we take the usual ERM (empirical risk minimization; Friedman et al. 2001) approach and replace it with a surrogate. We introduce an algorithm in the boosting framework based on the analysis from the previous section. Boosting is just one of our many possible machine learning approaches that can be used to solve it. We use boosting because it is easy to implement and is known to have good performance.
Boosting is a way to combine simple classifiers to form a strong classifier. We are given a set of such weak classifiers . Note that the set of weak classifiers need not be finite. Also, denote as a subset of weak classifiers that operate only on the first k measurements of x. .
The strong classifier is the linear combination:
Boosting algorithm operates in rounds. Each round can be viewed as a coordinate descent step in the space of weak learners. In each round, a weak learner h _{ j }(x) (a descent direction) is selected to be added to the linear combination classifier, F(x). Then a weight q _{ j } (an optimal step size in that direction) is computed. This is repeated until termination criteria is reached. For details on boosting various losses please refers to (Rosset et al. 2004).
Global surrogate
In our algorithm, we use the sigmoid loss function \(\mathbf{C}(z)={1 \over1+\exp(z)} \) to approximate the indicator. Similar sigmoid based losses have been used in boosting before (MasnadiShirazi and Vasconcelos 2009). Each subproblem (19) reduces to boosting a weighted loss.
To solve for stage k, we keep the rest of the stages constant. To find \(f^{k}_{p}=\sum q_{j} h_{j}(\mathbf{x})\), we fix \(f_{n}^{k}\) and solve:
Note that the weights w _{ i }, state variables \(S^{k}_{i}\) and costtogo \(\tilde{\delta}^{k}_{i}\) are also expressed in terms of the C(z) instead of :
To solve for \(f_{n}^{k}\), we solve the same problem but keep \(f_{p}^{k}\) constant instead:
Note that the terms \(\tilde{\delta}_{i}^{k}\) and \(S_{i}^{k}\) do not depend on stage k and remain constant when solving for \(f_{p}^{k}\) and \(f_{n}^{k}\). For the ease of notation, we define a new term C _{ r } that indicates if x _{ i } is rejected at a kth stage. The term is close to one if \(f^{k}_{p}\) and \(f^{k}_{n}\) disagree (reject) and small if they agree.
The expressions for state variables and costtogo are now simplified.
The state variable remains greater than zero as long as x _{ i } is rejected at every stage. The expression for costtogo at kth stage is:
The last two terms are simply a surrogate for L _{ k }(⋅) from (16) in terms of C(⋅).
For the last stage (a standard binary classifier), we fix the first K−1 stages and solve:
Our algorithms performs cyclical optimization over the stages. To initialize \(f^{k}_{n},~f^{k}_{p}~\forall k\), we simply hard code \(f^{k}_{p}\) to classify any x as +1 and \(f^{k}_{n}\) as −1 so that all x’s are rejected to the last stage. Using these nominal classifiers, we compute \(S^{k}_{i}\) and \(\delta^{k}_{i}\) according to Eqs. (25) and (26), respectively.
At a stage k, for a fixed \(\delta^{k}_{i}\) and \(S^{k}_{i}\), we alternate among minimizing \(f^{k}_{p}\) and \(f^{k}_{n}\) according to Eqs. (22) and (24). In practice, we found that one iteration is sufficient.
Given a new estimate of stage k, we update \(\delta^{s}_{i}\) for s>k and \(S^{s}_{j}\) for s<k and then move on to optimizing another stage k′. Given an estimate for stage k′, we again update the state variables and costtogo for the rest of the system.
The stages are optimized in the following order. We start with the last stage and make our way backwards to the first stage. Then do a forward pass from 1st stage to last. These forward and back passes are repeated until convergence. See Algorithm 1.
Our formulation allows us to form a surrogate for the entire risk in Eq. (1), not just for each subproblem. This enables us to prove the following theorem,
Theorem 3
Our global surrogate algorithm converges to a local minimum.
Proof
This is simply due to a fact that we are minimizing a global smooth cost function by coordinate descent over \(\mathbf{q}_{p}^{1}, \mathbf{q}^{1}_{n}, \mathbf{q}_{p}^{2}, \mathbf{q}^{2}_{n}, \ldots, \mathbf{q}^{K}\). Here, \(\mathbf{q}^{k}_{p}\) is the vector of weak learner weights parameterizing \(f^{k}_{p}\). For the derivation of three stage system global cost refer to Appendix B. □
However, since the global loss and the loss for each subproblem are nonconvex programs, there is no global optimality guarantee. Theorem 3 ensures that our algorithm terminates.
Regularization to reduce overfitting
To reduce overtraining, we introduce a simple but effective regularization. For any loss C(z) and a parameter λ, we introduce a multiplicative term to the cost function: . Here, \(C'(z)=\frac{dC(z)}{dz}\). The term exp(λq) limits how large a step size for a weak hypothesis can become. It also introduces a simple stopping criteria: abort if \({\sum_{i=1}^{n} C'(y_{i} f_{t}(x_{i})) y_{i} h_{t+1}(x_{i}) \over\sum_{i=1}^{n} \mathbf{C}(y_{i} f_{t}(x_{i}))} \leq\lambda\). This corresponds to a situation when no descent directions ( weak hypothesis h _{ t+1} ) can be found to minimize the cost function.
Generalization error
Our system is composed of margin maximizing classifiers, therefore it is appropriate to derive generalization error bounds based on margins. It turns out that we can employ maximum margin generalization techniques from Schapire et al. (1998) to derive error bounds for a two stage version of the system. A two stage system consists of three boosted binary classifiers:
Theorem 4
Let be a distribution on , and let be a sample of m examples chosen independently at random according to , and a rejected subsample of size m _{ r }, . Assume that the baseclassifier spaces and are finite, and let δ>0. Then with probability at least 1−δ over the random choice of the training set S, all boosted classifiers \(f^{1}_{n},f^{1}_{p},f^{2}\) satisfy the following bound for all θ _{1}>0 and θ _{2}>0:
Proof
The proof extends the approach in Schapire et al. (1998) to a two stage system. For complete details please refers to the appendix. □
The two stage system can be compactly expressed:
The system error is a sum of two terms: error at the 1st stage + error at the 2nd stage. Theorem 4 states the generalization error of F(x) is bounded by the empirical margin error over the training set S plus a term that is inversely proportional to the margins and the number of training samples at that stage. An interesting observation is that m _{ r }, number of samples that reaches the 2nd stage, depends on the reject classifier at the 1st stage. So if very few examples make it to the second stage then we do not have strong generalization.
Experiments
The goal is to demonstrate that a large fraction of data can be classified at an early stage using a cheap modality. In our experiments, we use four real life datasets with measurements arising from meaningful stages.
Related algorithms
We compare our algorithm to two methods:
Myopic
An absolute margin of a classifier is a measure of how confident a classifier is on an example. Examples with small margin have low confidence and should be rejected to the next stage to acquire more features. This approach is based on reject classification (Bartlett and Wegkamp 2008). We know from Theorem 1 that the optimal classifier is a threshold of the posterior. For each stage, we obtain a binary boosted classifier, f ^{k}(⋅), trained on all the data. We then threshold the margin of the classifier, f ^{k}(x ^{k}). It is known that given an infinite amount of training data, boosting certain losses (sigmoid loss in our case) approaches the log likelihood ratio, \(f(\mathbf{x})={1 \over2} \log{ \mathrm{P}(y=1\mathbf{x}) \over \mathrm{P}(y=1\mathbf{x})}\) (MasnadiShirazi and Vasconcelos 2009). So a reject region for a given threshold t _{ k } is defined: {x∣f ^{k}(x)≤t _{ k }}. This is a completely myopic approach as the rejection does not take into account performance of later stages. This method is very similar to TEFE (Liu et al. 2008) which also uses absolute margin as a measure for rejection. The difference is that our myopic strategy is a boosting classifier not an SVM as used in TEFE.
Expected utility/margin
An expected margin difference measures how a new attribute, if acquired, would be useful for an example. If this expected utility for an example is large then a new attribute should be acquired. This approach is based on the work by Kanani and Melville (2008). We train boosted binary classifiers on all the data for each stage: f ^{k}(x ^{k}). Given the measurement at the current stage x ^{k}, we compute an expected utility (change in normalized margin) of acquiring the next measurement x _{ k+1}:
An x ^{k} is rejected to the next stage if its utility U(x ^{k})≥t _{ k } is greater than a threshold. Here, denotes the possible values that x _{ k+1} can take. Note this approach requires estimating P(x _{ k+1}x ^{k}),^{Footnote 2} therefore the (k+1)th measurement has to be discrete or distribution needs to be parametrized. Due to this limitation, we only compare this method on two datasets.
Simulations
Performance metric
A natural performance metric is the trade off between system error and measurement cost. Note, for utility and myopic methods, it is unclear how to set a thresholds t _{ k } for each stage given a measurement cost δ _{ k }. For this reason, we only compare them in a two stages system. More than two stages is notpractical because we would need to test every possible t _{ k } for every stage k.
In a two stage setting, since every example has to pass through the first stage, only the cost of the second stage, δ _{2}, affects the performance. The average measurement cost of the system is proportional to δ _{1}+ (the fraction of examples rejected to the second stage) ×δ _{2}. So knowing the exact cost of the second stage sensor, δ=δ _{2}, is not necessary. In our algorithm, we vary δ to generate a system error vs reject rate plot. For margin and utility, we sweep a threshold t _{ k }. System error is the sum of 1st stage and 2nd stage errors. Reject rate is the fraction of examples rejected to the 2nd stage and require additional measurements. Low reject rate (cost) corresponds to higher error rate as most of the data will be classified at the first stage using less informative measurements. High reject rate will have performance similar to a centralized classifier, as most examples will be classified at the 2nd stage.
Set up
In all our experiments, we use stumps as weak learners. A stump classifier h _{ d,g,s }∈{+1,−1} is parametrized by a threshold g on dth dimension and a sign variable s∈{+1,−1}: h _{ d,g,s }(x)=s×sgn[x _{ d }−g]. We chose stumps for their simplicity, computation speed and relatively good performance. While more complicated weak learners, such as decision trees can be used, they would only change the absolute performance of our experiments. The entire curves would just move vertically up or down. Our goal is to demonstrate the advantage of a multistage classifier relative to the centralized system (a system that uses all the measurements for all examples).
For each dataset and experiment, we randomly split the data 50/50 for training and testing. The results are evaluated on a separate test set, and the simulations are averaged over 50 MonteCarlo trials. The number of iterations for each boosting subproblem is set to T=50. In our global surrogate algorithm, the number of outer loop iterations is set to D=10. Refer to Table 1 for dataset descriptions and to Table 2 for summary of our experiments.
Discrete valued data experiments
To compare our method to the utility approach, we consider discrete data. The first dataset is a quantized (with 20 levels) Gaussian mixture synthetic data in two dimension. The 1st dimension is stage one; the 2nd dimension is stage two. The second dataset is Mammogram Mass from UCI Machine Learning Repository. It is used to predict the severity of a mammographic mass lesion (malicious or benign). It contains 3 attributes extracted from the CAD image and also an evaluation by a radiologist on a confidence scale in addition to the true biopsy results. The first stage are features extracted from the CAD image, and the second stage is the expert confidence rated on a discrete scale 1–5. Automatic analysis of the CAD image is cheaper than employing an opinion of a radiologist.
Simulations in Fig. 5 demonstrate that utility performs worse when compared to our approach. Our approach requires far fewer 2nd stage measurements to achieve an error rate close to the centralized strategy than the utility approach. This is possibly due to poor probability estimates in limited data setting.
Continuous valued data experiments
We compare our global method to the myopic method on three datasets. The Pima Indians Diabetes Dataset (UCI MLR) consists of 8 measurements. Since the stages are not specified in this dataset, we group measurements with similar costs into separate modalities. 6 of the measurements are inexpensive to acquire and consist of simple tests such as body mass index, age, pedigree. These we designate as the first stage. The other two measurements constitute the second stage and require more expensive procedures.
The polyp dataset consists of hyperspectral measurements of colon polyps collected during colonoscopies (RodríguezDíaz and Castañón 2009). The attribute is a measured intensity at 126 equally spaced frequencies. Finer resolution requires higher photon count which is proportional to acquisition time. For a first stage, we use a coarse measurement downsampled to only 12 frequency bins. The second stage is the full resolution frequency response. Using the course measurements is cheaper than acquiring the full resolution.
The threat dataset contains images taken of people wearing various explosives devices. The imaging is done in three modalities: infrared (IR), passive millimeter wave (PMMW), and active millimeter (AMMW). All the images are registered. We extract many patches from the images and use them as our training data. A patch carries a binary label, it either contains a threat or is clean. IR and PMMW are the fastest modalities but also less informative. AMMW requires raster scanning a person and is slow but also the most useful.
In Fig. 6, global performs better than margin in most cases. On threat data, margin appears to be doing just marginally worse than global, however, we get only a few points on the curve with reject rates less than 50 %. Due to the heuristic nature of margin, we cannot construct a multistage classifier with an arbitrary reject rate.
The goal is to reach the performance of a centralized classifier (100 % reject rate) while utilizing the 2nd stage sensor only for a small fraction of examples. Overall, the results demonstrate the benefit of multistage classification: rejection rate can be set to less than 50 % with only small sacrifices in performance. For the mammogram data, this implies that for half of the patients a diagnoses can be made solely by an automatic analysis of a CAD image without an expensive opinion of a radiologist. For the Pima data, similar error can be achieved without an expensive medical procedures. For the polyps dataset, a fast low resolution measurement is enough to classify a large fraction of patience. In the threat dataset, IR and PMMW are sufficient to decide whether or not a threat is present for the majority of instances without requiring a person to go through a slower AMMW scanner.
Unbalanced false positive and false negative penalties
In medical diagnosis and threat detection, the penalty of false positives and false negatives is not equal. We can easily adapt our algorithm to account for such setting. Empirical Risk in (18) can be modified to include a penalty of w _{ p } for a Type I error and w _{ n } for a Type II error. The experiment in Fig. 7 demonstrates our global algorithms in such scenario. For each reject cost δ, we compute an ROC curve. This allows to select an operating point of the system with a desired false alarm or detection rate. We also compute a corresponding average reject rate for each value of δ. So the highest reject rate corresponds to the best performance but also to the highest acquisition cost incurred by the system. Note that very good performance can be achieved by requesting only 50 % of instances to be measured at the second stage.
Three stages
Lastly, we demonstrate a three stage system, we apply our algorithm to three stages of threat dataset. Note for margin it is unclear how to generalize it to a multistage scenario and there is no way to define reject costs for different stages. We set the first stage to be IR, second PMMW and AMMW as third. There is no cost for acquiring IR. We vary the costs for the PMMW (2nd) stage, δ _{1}, and AMMW (3rd), δ _{2}, to generate an error map (color in Fig. 8). A point on the map corresponds to a performance of a particular multistage classification strategy. The vertical axis is the fraction of examples for which only IR and PMMW measurements are used in making a decision. The horizontal axis is the fraction of examples for which all three modalities are used. For example, a red point in the figure, {.4,.15,.195}, correspond to a system where 40 % of examples use IR and PMMW, 15 % use only IR and the rest of data (45 %) use all the modalities. And this strategy achieves a system error rate of 19.5 %. Note that the support lies below the diagonal. This is because the sum or reject rates has to be less than one. Results demonstrate some interesting observations. While best performance (about 19 %) is achieved when all the modalities are used for every example, we can move along the vertical lines and allow a fraction to be classified by IR and PMMW, avoiding AMMW all together. This strategy achieves performance comparable to a centralized system, (IR+PMMW+AMMW).
Conclusion
In this paper, we propose a general framework for a sequential decision system in a nonparametric setting. Starting from basic principles, we derive the Bayesian optimal solution. Then, to simplify the problem, we parameterize a classifier at each stage in terms of two binary decisions. We formulate an ERM problem and optimize it by alternatively minimizing one stage at a time. Remarkably, all subproblems turn out to be weighed binary error minimizations. We introduce a practical boosting algorithm that minimizes a global surrogate of the empirical risk and test it on several datasets. Results show the advantage of our formulation to more heuristic approaches. Overall, our experiments demonstrate how multistage classifiers can achieve good performance by acquiring full measurements only for a fraction of samples.
Notes
To simplify our discussion, we consider equal error penalties. However, our approach can be easily extended to unbalanced error penalties as we will demonstrate in the experiments section.
While there are many different ways to estimate a probability likelihood we used a Gaussian mixture due to its computational efficiency. The number of mixture components is equal to the number of discrete values that x _{2} can take from an alphabet . The conditional P(x _{1}∣x _{2}=j) is a Gaussian whose parameters are learned from the training set. Using Bayes rule, \(\mathrm{P}(x_{2} \mid x_{1})=\frac{\mathrm{P}(x_{1} \mid x_{2})}{ \sum_{x' \in X2} \mathrm{P}(x_{1} \mid x_{2}=x')}\).
References
Allwein, E. L., Schapire, R. E., & Singer, Y. (2001). Reducing multiclass to binary: a unifying approach for margin classifiers. Journal of Machine Learning Research, 1, 113–141.
Bartlett, P. L., & Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. The Journal of Machine Learning Research, 9, 1823–1840.
Bilgic, M., & Getoor, L. (2007). Voila: efficient featurevalue acquisition for classification. In AAAI conference on artificial intelligence.
Chen, M., Xu, Z., Weinberger, K. Q., Chapelle, O., & Kedem, D. (2012). Classifier cascade: tradeoff between accuracy and feature evaluation cost. In International conference on artificial intelligence and statistics.
Chow, C. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1), 41–46. doi:10.1109/TIT.1970.1054406.
Cordella, L. P., & Sansone, C. (2007). A multistage classification system for detecting intrusions in computer networks. Pattern Analysis & Applications, 10(2), 83–100.
ElYaniv, R., & Wiener, Y. (2011). Agnostic selective classification. In Advances in neural information processing systems.
Fan, W., Chu, F., Wang, H., & Yu, P. S. (2002). Pruning and dynamic scheduling of costsensitive ensembles. In AAAI conference on artificial intelligence.
Fan, W., Lee, W., Stolfo, S. J., & Miller, M. (2000). A multiple model costsensitive approach for intrusion detection. In European conference on machine learning.
Friedman, J., Hastie, T., & Tibshirani, R. (2001). Springer series in statistics: Vol. 1. The elements of statistical learning. Berlin: Springer.
Grandvalet, Y., Rakotomamonjy, A., Keshet, J., & Canu, S. (2008). Support vector machines with a reject option. In Advances in neural information processing systems.
Ji, S., & Carin, L. (2007). Costsensitive feature acquisition and classification. Pattern Recognition, 40(5), 1474–1485.
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1), 99–134.
Kanani, P., & Melville, P. (2008). Predictiontime active featurevalue acquisition for costeffective customer targeting. In Advances in neural information processing systems.
Kapoor, A., & Horvitz, E. (2009). Breaking boundaries: active information acquisition across learning and diagnosis. In Advances in neural information processing systems.
Lee, W., Fan, W., Miller, M., Stolfo, S. J., & Zadok, E. (2002). Toward costsensitive modeling for intrusion detection and response. Journal of Computer Security, 10(1), 5–22.
Liu, L. P., Yu, Y., Jiang, Y., & Zhou, Z. H. (2008). TEFE: a timeefficient approach to feature extraction. In International conference on data mining.
MacKay, D. J. (1992). Informationbased objective functions for active data selection. Neural Computation, 4(4), 590–604.
MasnadiShirazi, H., & Vasconcelos, N. (2009). On the design of loss functions for classification: theory, robustness to outliers, and savageboost. In Advances in neural information processing systems.
RodríguezDíaz, E., & Castañón, D. A. (2009). Support vector machine classifiers for sequential decision problems. In IEEE conference on decision and control.
Rosset, S., Zhu, J., & Hastie, T. (2004). Boosting as a regularized path to a maximum margin classifier. The Journal of Machine Learning Research, 5, 941–973.
Schapire, R. E., Freund, Y., Bartlett, P., & Lee, W. S. (1998). Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5), 1651–1686.
Sheng, V. S., & Ling, C. X. (2006). Feature value acquisition in testing: a sequential batch test algorithm. In International conference on machine learning (pp. 809–816).
Trapeznikov, K., Saligrama, V., & Castañon, D. A. (2012). Multistage classifier design. In Asian conference on machine learning.
Viola, P., & Jones, M. J. (2004). Robust realtime face detection. International Journal of Computer Vision, 57(2), 137–154.
Wang, H., Fan, W., Yu, P. S., & Han, J. (2003). Mining conceptdrifting data streams using ensemble classifiers. In Knowledge discovery and data mining.
Yuan, C., & Casasent, D. (2003). A novel support vector classifier with better rejection performance. In Computer vision and pattern recognition.
Zhang, C., & Zhang, Z. (2010). A survey of recent advances in face detection (Microsoft research technical report).
Zubek, V. B., & Dietterich, T. G. (2002). Pruning improves heuristic search for costsensitive learning. In International conference on machine learning.
Acknowledgements
This work is partially supported by the U.S. DHS Award 2008ST061ED000, NSF Grant 0932114 and NGA Grant HM15820910037.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editors: ZhiHua Zhou, Wee Sun Lee, Steven Hoi, Wray Buntine, and Hiroshi Motoda.
Appendices
Appendix A: Proof of Theorem 4
The proof will closely follow the proof of Theorem 1 in Schapire et al. (1998). We have to bound two terms:
A.1 First term
Let us bound the first term. Define to be the set of unweighted averages over N elements from ,
Any weighed classifier f=∑_{ h } q _{ h } h(x) can be approximated by drawing an element from by choosing h _{1},…,h _{ N } with prob. q _{ h }.
We can express our first term as a sum of probabilities of disjoint events.
Further, we can write,
The inequality holds for any g _{ p },g _{ n }. We take the expected value of the right hand side wrt to the distribution
The last term inside the expectation is the probability that an average of N Bernoulli random variables is larger than its expectation, we use a concentration result from Eq. (4) in Theorem 1 of Schapire et al. (1998).
To bound the first we use the result from Eq. (5) in Theorem 1 of Schapire et al. (1998). if we set , with probability at least 1−δ _{ N },
for any choice of θ and every distribution . Here, P_{ S }[ ] is probability taken with respect to a randomly drawn sample of size m from .
By the same argument as in inequality (32),
The expressions inside the expectation can be bounded using the same Chernoff bound result from (34),
By setting δ _{ N }=δ/(N(N+1)), and combining the terms,
By setting, ,
A.2 Second term
Here we will bound the second term, Define a new distribution:
Rewrite:
Note that is an iid sample from . Using Theorem 1 in Schapire et al. (1998),
Collecting the two terms produces the desired result.
Appendix B: Derivation of a global risk for a three stage system
Consider a three stage system. Define some terms:
Risk for three stages:
Plug in all the terms:
Minimize over \(f_{p}^{1},f^{1}_{n}\) and keep \(f_{p}^{2},f_{n}^{2},f^{3}\) constant. We can rearrange the terms to get:
Minimize over \(f^{2}_{p},f^{2}_{n}\) and keep \(f_{p}^{1},f^{1}_{n},f^{3}\) constant:
Minimize over f _{3} and keep \(f_{p}^{1},f^{1}_{n},f_{p}^{2},f_{n}^{2}\) constant:
Rights and permissions
About this article
Cite this article
Trapeznikov, K., Saligrama, V. & Castañón, D. Multistage classifier design. Mach Learn 92, 479–502 (2013). https://doi.org/10.1007/s1099401353494
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s1099401353494
Keywords
 Multistage classification
 Sequential decision
 Boosting
 Cost sensitive learning