Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice

Necessity and sufficiency are the building blocks of all successful explanations. Yet despite their importance, these notions have been conceptually underdeveloped and inconsistently applied in explainable artificial intelligence (XAI), a fast-growing research area that is so far lacking in firm theoretical foundations. Building on work in logic, probability, and causality, we establish the central role of necessity and sufficiency in XAI, unifying seemingly disparate methods in a single formal framework. We provide a sound and complete algorithm for computing explanatory factors with respect to a given context, and demonstrate its flexibility and competitive performance against state of the art alternatives on various tasks.


INTRODUCTION
Machine learning algorithms are increasingly used in a variety of high-stakes domains, from credit scoring to medical diagnosis.However, many such methods are opaque, in that humans cannot understand the reasoning behind particular predictions.Post-hoc, model-agnostic local explanation tools (e.g., feature attributions, rule lists, and counterfactuals) are at the forefront of a fast-growing area of research variously referred to as interpretable machine learning or explainable artificial intelligence (XAI).
Many authors have pointed out the inconsistencies between popular XAI tools, raising questions as to which method is more reliable in particular cases [Mothilal et al., 2020a;Ramon et al., 2020;Fernández-Loría et al., 2020].Theoretical foundations have proven elusive in this area, perhaps due to the perceived subjectivity inherent to notions such as "intelligible" and "relevant" [Watson and Floridi, 2020].Practitioners often seek refuge in the axiomatic guarantees of Shapley values, which have become the de facto stan-Figure 1: We describe minimal sufficient factors (here, sets of features) for a given input (top row), with the aim of preserving or flipping the original prediction.We report a sufficiency score for each set and a cumulative necessity score for all sets, indicating the proportion of paths towards the outcome that are covered by the explanation.Feature colors indicate source of feature values (input or reference).dard in many XAI applications, due in no small part to their attractive theoretical properties [Bhatt et al., 2020].However, ambiguities regarding the underlying assumptions of the method [Kumar et al., 2020] and the recent proliferation of mutually incompatible implementations [Sundararajan and Najmi, 2019;Merrick and Taly, 2020] have complicated this picture.Despite the abundance of alternative XAI tools [Molnar, 2021], a dearth of theory persists.This has led some to conclude that the goals of XAI are underspecified [Lipton, 2018], and even that post-hoc methods do more harm than good [Rudin, 2019].
We argue that this lacuna at the heart of XAI should be filled by a return to fundamentals -specifically, to necessity and sufficiency.As the building blocks of all successful explanations, these dual concepts deserve a privileged position in the theory and practice of XAI.Following a review of related work (Sect.2), we operationalize this insight with a unified framework (Sect.3) that reveals unexpected affinities between various XAI tools and probabilities of causation (Sect. 4).We proceed to implement a novel procedure for computing model explanations that improves upon the state of the art in various quantitative and qualitative comparisons (Sect.5).Following a brief discussion (Sect.6), we conclude with a summary and directions for future work (Sect.7).
We make three main contributions.(1) We present a formal framework for XAI that unifies several popular approaches, including feature attributions, rule lists, and counterfactuals.(2) We introduce novel measures of necessity and sufficiency that can be computed for any feature subset.The method enables users to incorporate domain knowledge, search various subspaces, and select a utility-maximizing explanation.(3) We present a sound and complete algorithm for identifying explanatory factors, and illustrate its performance on a range of tasks.

NECESSITY AND SUFFICIENCY
Necessity and sufficiency have a long philosophical tradition [Mackie, 1965;Lewis, 1973;Halpern and Pearl, 2005b], spanning logical, probabilistic, and causal variants.In propositional logic, we say that x is a sufficient condition for y iff x → y, and x is a necessary condition for y iff y → x.So stated, necessity and sufficiency are logically converse.However, by the law of contraposition, both definitions admit alternative formulations, whereby sufficiency may be rewritten as ¬y → ¬x and necessity as ¬x → ¬y.By pairing the original definition of sufficiency with the latter definition of necessity (and vice versa), we find that the two concepts are also logically inverse.
These formulae suggest probabilistic relaxations, measuring x's sufficiency for y by P (y|x) and x's necessity for y by P (x|y).Because there is no probabilistic law of contraposition, these quantities are generally uninformative w.r.t.P (¬x|¬y) and P (¬y|¬x), which may be of independent interest.Thus, while necessity is both the converse and inverse of sufficiency in propositional logic, the two formulations come apart in probability calculus.We revisit the distinction between probabilistic conversion and inversion in Rmk. 1 and Sect.4. These definitions struggle to track our intuitions when we consider causal explanations [Pearl, 2000;Tian and Pearl, 2000].It may make sense to say in logic that if x is a necessary condition for y, then y is a sufficient condition for x; it does not follow that if x is a necessary cause of y, then y is a sufficient cause of x.We may amend both concepts using counterfactual probabilities -e.g., the probability that Alice would still have a headache if she had not taken an aspirin, given that she does not have a headache and did take an aspirin.Let P (y x |x , y ) denote such a quantity, to be read as "the probability that Y would equal y under an intervention that sets X to x, given that we observe X = x and Y = y ."Then, according to Pearl [2000, Ch. 9], the probability that x is a sufficient cause of y is given by suf(x, y) := P (y x |x , y ), and the probability that x is a necessary cause of y is given by nec(x, y) := P (y x |x, y).Analysis becomes more difficult in higher dimensions, where variables may interact to block or unblock causal pathways.VanderWeele and Robins [2008] analyze sufficient causal interactions in the potential outcomes framework, refining notions of synergism without monotonicity constraints.In a subsequent paper, VanderWeele and Richardson [2012] study the irreducibility and singularity of interactions in sufficient-component cause models.Halpern [2016] devotes an entire monograph to the subject, providing various criteria to distinguish between subtly different notions of "actual causality", as well as "but-for" (similar to necessary) and sufficient causes.These authors generally limit their analyses to Boolean systems with convenient structural properties, e.g.conditional ignorability and the stable unit treatment value assumption [Imbens and Rubin, 2015].Operationalizing their theories in a practical method without such restrictions is one of our primary contributions.
Necessity and sufficiency have begun to receive explicit attention in the XAI literature.Ribeiro et al. [2018a] propose a bandit procedure for identifying a minimal set of Boolean conditions that entails a predictive outcome (more on this in Sect.4).Dhurandhar et al. [2018] propose an autoencoder for learning pertinent negatives and positives, i.e. features whose presence or absence is decisive for a given label, while Zhang et al. [2018] develop a technique for generating symbolic corrections to alter model outputs.Both methods are optimized for neural networks, unlike the modelagnostic approach we develop here.
Another strand of research in this area is rooted in logic programming.Several authors have sought to reframe XAI as either a SAT [Ignatiev et al., 2019;Narodytska et al., 2019] or a set cover problem [Lakkaraju et al., 2019;Grover et al., 2019], typically deriving approximate solutions on a prespecified subspace to ensure computability in polynomial time.We adopt a different strategy that prioritizes completeness over efficiency, an approach we show to be feasible in moderate dimensions (see Sect. 6 for a discussion).[2016]'s definitions of necessity and sufficiency to critique popular XAI tools, proposing a new feature attribution measure with some purported advantages.Their method relies on the strong assumption that predictors are mutually independent.Galhotra et al. [2021] adapt Pearl [2000]'s probabilities of causation for XAI under a more inclusive range of data generating processes.They derive analytic bounds on multidimensional extensions of nec and suf, as well as an algorithm for point identification when graphical structure permits.Oddly, they claim that non-causal applications of necessity and sufficiency are somehow "incorrect and mislead-ing" (p. 2), a normative judgment that is inconsistent with many common uses of these concepts.

Mothilal et al. [2020a] build on Halpern
Rather than insisting on any particular interpretation of necessity and sufficiency, we propose a general framework that admits logical, probabilistic, and causal interpretations as special cases.Whereas previous works evaluate individual predictors, we focus on feature subsets, allowing us to detect and quantify interaction effects.Our formal results clarify the relationship between existing XAI methods and probabilities of causation, while our empirical results demonstrate their applicability to a wide array of tasks and datasets.

A UNIFYING FRAMEWORK
We propose a unifying framework that highlights the role of necessity and sufficiency in XAI.Its constituent elements are described below.
Target function.Post-hoc explainability methods assume access to a target function f : X → Y, i.e. the model whose prediction(s) we seek to explain.For simplicity, we restrict attention to the binary setting, with Y ∈ {0, 1}.Multi-class extensions are straightforward, while continuous outcomes may be accommodated via discretization.Though this inevitably involves some information loss, we follow authors in the contrastivist tradition in arguing that, even for continuous outcomes, explanations always involve a juxtaposition (perhaps implicit) of "fact and foil" [Lipton, 1990].For instance, a loan applicant is probably less interested in knowing why her credit score is precisely y than she is in discovering why it is below some threshold (say, 700).Of course, binary outcomes can approximate continuous values with arbitrary precision over repeated trials.
Context.The context D is a probability distribution over which we quantify sufficiency and necessity.Contexts may be constructed in various ways but always consist of at least some input (point or space) and reference (point or space).For instance, we may want to compare x i with all other samples, or else just those perturbed along one or two axes, perhaps based on some conditioning event(s).
In addition to predictors and outcomes, we optionally include information exogenous to f .For instance, if any events were conditioned upon to generate a given reference sample, this information may be recorded among a set of auxiliary variables W .Other examples of potential auxiliaries include metadata or engineered features such as those learned via neural embeddings.This augmentation allows us to evaluate the necessity and sufficiency of factors beyond those found in X. Contextual data take the form Z = (X, W ) ∼ D. The distribution may or may not encode dependencies between (elements of) X and (elements of) W .We extend the target function to augmented inputs by defining f (z) := f (x).
Factors.Factors pick out the properties whose necessity and sufficiency we wish to quantify.Formally, a factor c : Z → {0, 1} indicates whether its argument satisfies some criteria with respect to predictors or auxiliaries.For instance, if x is an input to a credit lending model, and w contains information about the subspace from which data were sampled, then a factor could be c(z) = 1[x[gender = "female"] ∧ w[do(income > $50k)]], i.e. checking if z is female and drawn from a context in which an intervention fixes income at greater than $50k.We use the term "factor" as opposed to "condition" or "cause" to suggest an inclusive set of criteria that may apply to predictors x and/or auxiliaries w.Such criteria are always observational w.r.t.z but may be interventional or counterfactual w.r.t.x.We assume a finite space of factors C. Partial order.When multiple factors pass a given necessity or sufficiency threshold, users will tend to prefer some over others.For instance, factors with fewer conditions are often preferable to those with more, all else being equal; factors that change a variable by one unit as opposed to two are preferable, and so on.Rather than formalize this preference in terms of a distance metric, which unnecessarily constrains the solution space, we treat the partial ordering as primitive and require only that it be complete and transitive.This covers not just distance-based measures but also more idiosyncratic orderings that are unique to individual agents.Ordinal preferences may be represented by cardinal utility functions under reasonable assumptions (see, e.g., [von Neumann and Morgenstern, 1944]).
We are now ready to formally specify our framework.
Definition 1 (Basis).A basis for computing necessary and sufficient factors for model predictions is a tuple B = f, D, C, , where f is a target function, D is a context, C is a set of factors, and is a partial ordering on C.

EXPLANATORY MEASURES
For some fixed basis B = f, D, C, , we define the following measures of sufficiency and necessity, with probability taken over D.
Definition 2 (Probability of Sufficiency).The probability that c is a sufficient factor for outcome y is given by: P S(c, y) The probability that factor set C = {c 1 , . . ., c k } is sufficient for y is given by: P S(C, y) Definition 3 (Probability of Necessity).The probability that c is a necessary factor for outcome y is given by: The probability that factor set C = {c 1 , . . ., c k } is necessary for y is given by: Remark 1.These probabilities can be likened to the "precision" (positive predictive value) and "recall" (true positive rate) of a (hypothetical) classifier that predicts whether f (z) = y based on whether c(z) = 1.By examining the confusion matrix of this classifier, one can define other related quantities, e.g. the true negative rate P (c(z) = 0|f (z) = y) and the negative predictive value P (f (z) = y|c(z) = 0), which are contrapositive transformations of our proposed measures.We can recover these values exactly via P S(1 − c, 1 − y) and P N (1 − c, 1 − y), respectively.When necessity and sufficiency are defined as probabilistic inversions (rather than conversions), such transformations are impossible.

MINIMAL SUFFICIENT FACTORS
We introduce Local Explanations via Necessity and Sufficiency (LENS), a procedure for computing explanatory factors with respect to a given basis B and threshold parameter τ (see Alg. 1).First, we calculate a factor's probability of sufficiency (see probSuff) by drawing n samples from D and taking the maximum likelihood estimate P S(c, y).Next, we sort the space of factors w.r.t. in search of those that are τ -minimal.
Definition 4 (τ -minimality).We say that c is τ -minimal iff (i) P S(c, y) ≥ τ and (ii) there exists no factor c such that P S(c , y) ≥ τ and c ≺ c.
Since a factor is necessary to the extent that it covers all possible pathways towards a given outcome, our next step is to span the τ -minimal factors and compute their cumulative P N (see probNec).As a minimal factor c stands for all c such that c c , in reporting probability of necessity, we expand C to its upward closure.
Thms. 1 and 2 state that this procedure is optimal in a sense that depends on whether we assume access to oracle or sample estimates of P S (see Appendix A for all proofs).Theorem 1.With oracle estimates P S(c, y) for all c ∈ C, Alg. 1 is sound and complete.That is, for any C returned by Alg. 1 and all c ∈ C, c is τ -minimal iff c ∈ C.
Population proportions may be obtained if data fully saturate the space D, a plausible prospect for categorical variables of low to moderate dimensionality.Otherwise, proportions will need to be estimated.
Theorem 2. With sample estimates P S(c, y) for all c ∈ C, Alg. 1 is uniformly most powerful.That is, Alg. 1 identifies the most τ -minimal factors of any method with fixed type I error α.
Multiple testing adjustments can easily be accommodated, in which case modified optimality criteria apply [Storey, 2007].
Remark 2. We take it that the main quantity of interest in most applications is sufficiency, be it for the original or alternative outcome, and therefore define τ -minimality w.r.t.sufficient (rather than necessary) factors.However, necessity serves an important role in tuning τ , as there is an inherent trade-off between the parameters.More factors are excluded at higher values of τ , thereby inducing lower cumulative P N ; more factors are included at lower values of τ , thereby inducing higher cumulative P N .See Appendix B.
return n(c&y) / n(c) 8: function probNec(C, y, upward_closure_flag) 9: if upward_closure_flag then 10: return n(C&y) / n(y) return cands, cum_pn Explanatory measures can be shown to play a central role in many seemingly unrelated XAI tools, albeit under different assumptions about the basis tuple B. In this section, we relate our framework to a number of existing methods.
Feature attributions.Several popular feature attribution algorithms are based on Shapley values [Shapley, 1953], which decompose the predictions of any target function as a sum of weights over d input features: where φ 0 represents a baseline expectation and φ j the weight assigned to X j at point \S such that we may rewrite any x i as a pair of subvectors, (x S i , x R i ).Payoffs are given by: although this introduces some ambiguity regarding the reference distribution for X R (more on this below).The Shapley value φ j is then j's average marginal contribution to all subsets that exclude it: It can be shown that this is the unique solution to the attribution problem that satisfies certain desirable properties, including efficiency, linearity, sensitivity, and symmetry.
Reformulating this in our framework, we find that the value function v is a sufficiency measure.To see this, let each z ∼ D be a sample in which a random subset of variables S are held at their original values, while remaining features R are drawn from a fixed distribution D(•|S).1 Proposition 1.Let c S (z) = 1 iff x ⊆ z was constructed by holding x S fixed and sampling X R according to D(•|S).
Then v(S) = P S(c S , y).
Thus, the Shapley value φ j measures X j 's average marginal increase to the sufficiency of a random feature subset.The advantage of our method is that, by focusing on particular subsets instead of weighting them all equally, we disregard irrelevant permutations and home in on just those that meet a τ -minimality criterion.Kumar et al. [2020] and Popescu, 2008;Letham et al., 2015], more recent efforts have prioritized local explanation tasks [Lakkaraju et al., 2019;Sokol and Flach, 2020].
We focus in particular on the Anchors algorithm [Ribeiro et al., 2018a], which learns a set of Boolean conditions A (the eponymous "anchors") such that A(x i ) = 1 and The lhs of Eq. 4 is termed the precision, prec(A), and probability is taken over a synthetic distribution in which the conditions in A hold while other features are perturbed.Once τ is fixed, the goal is to maximize coverage, formally defined as E[A(x) = 1], i.e. the proportion of datapoints to which the anchor applies.
The formal similarities between Eq. 4 and Def. 2 are immediately apparent, and the authors themselves acknowledge that Anchors are intended to provide "sufficient conditions" for model predictions.
While Anchors outputs just a single explanation, our method generates a ranked list of candidates, thereby offering a more comprehensive view of model behavior.Moreover, our necessity measure adds a mode of explanatory information entirely lacking in Anchors.
Counterfactuals.Counterfactual explanations identify one or several nearest neighbors with different outcomes, e.g.all datapoints x within an -ball of x i such that labels f (x) and f (x i ) differ (for classification) or f (x) > f (x i ) + δ (for regression). 2The optimization problem is: where CF(x i ) denotes a counterfactual space such that f (x i ) = f (x) and cost is a user-supplied cost function, typically equated with some distance measure.[Wachter et al., 2018] recommend using generative adversarial networks to solve Eq. 5, while others have proposed alternatives designed to ensure that counterfactuals are coherent and actionable [Ustun et al., 2019;Karimi et al., 2020a;Wexler et al., 2020].As with Shapley values, the variation in these proposals is reducible to the choice of context D.
For counterfactuals, we rewrite the objective as a search for minimal perturbations sufficient to flip an outcome.
Proposition 3. Let cost be a function representing , and let c be some factor spanning reference values.Then the counterfactual recourse objective is: where τ denotes a decision threshold.Counterfactual outputs will then be any z ∼ D such that c * (z) = 1.
Probabilities of causation.Our framework can describe Pearl [2000]'s aforementioned probabilities of causation, however in this case D must be constructed with care.
Proposition 4. Consider the bivariate Boolean setting, as in Sect. 2. We have two counterfactual distributions: an input space I, in which we observe x, y but intervene to set X = x ; and a reference space R, in which we observe x , y but intervene to set X = x.Let D denote a uniform mixture over both spaces, and let auxiliary variable W tag each sample with a label indicating whether it comes from the original (W = 1) or contrastive (W = 0) counterfactual space.Define c(z) = w.Then we have suf(x, y) = P S(c, y) and nec(x, y) = P S(1 − c, y ).
In other words, we regard Pearl's notion of necessity as sufficiency of the negated factor for the alternative outcome.By contrast, Pearl [2000] has no analogue for our probability of necessity.This is true of any measure that defines sufficiency and necessity via inverse, rather than converse probabilities.While conditioning on the same variable(s) for both measures may have some intuitive appeal, it comes at a cost to expressive power.Whereas our framework can recover all four explanatory measures, corresponding to the classical definitions and their contrapositive forms, definitions that merely negate instead of transpose the antecedent and consequent are limited to just two.
Remark 3. We have assumed that factors and outcomes are Boolean throughout.Our results can be extended to continuous versions of either or both variables, so long as c(Z) |= Y | Z.This conditional independence holds whenever W |= Y | X, which is true by construction since f (z) := f (x).However, we defend the Boolean assumption on the grounds that it is well motivated by contrastivist epistemologies [Kahneman and Miller, 1986;Lipton, 1990;Blaauw, 2013] and not especially restrictive, given that partitions of arbitrary complexity may be defined over Z and Y .

EXPERIMENTS
In this section, we demonstrate the use of LENS on a variety of tasks and compare results with popular XAI tools, using the basis configurations detailed in Table 1.A comprehensive discussion of experimental design, including datasets and pre-processing pipelines, is left to Appendix C. Code for reproducing all results is available at https: //github.com/limorigu/LENS.
Contexts.We consider a range of contexts D in our experiments.For the input-to-reference (I2R) setting, we replace input values with reference values for feature subsets S; for the reference-to-input (R2I) setting, we replace reference values with input values.We use R2I for examining sufficiency/necessity of the original model prediction, and I2R for examining sufficiency/necessity of a contrastive model prediction.We sample from the empirical data in all experiments, except in Sect.5.3, where we assume access to a structural causal model (SCM).
Partial Orderings.We consider two types of partial orderings in our experiments.The first, subset , evaluates subset relationships.For instance, if c(z) = 1[x[gender = "female"]] and c (z) = 1[x[gender = "female" ∧ age ≥ 40]], then we say that c subset c .The second, c cost c := c subset c ∧ cost(c) ≤ cost(c ), adds the additional constraint that c has cost no greater than c .The cost function could be arbitrary.Here, we consider distance measures over either the entire state space or just the intervention targets corresponding to c.

FEATURE ATTRIBUTIONS
Feature attributions are often used to identify the top-k most important features for a given model outcome [Barocas et al., 2020].However, we argue that these feature sets may not be explanatory with respect to a given prediction.To show this, we compute R2I and I2R sufficiency -i.e., P S(c, y) and P S(1 − c, 1 − y), respectively -for the top-k most influential features (k ∈ [1, 9]) as identified by SHAP [Lundberg and Lee, 2017] and LENS.Fig. 2 shows results from the R2I setting for German credit [Dua and Graff, 2017] and SpamAssassin datasets [SpamAssassin, 2006] goal is to answer questions of the form, "What are words with/without which our model would output the original/opposite prediction for an input sentence?"For this experiment, we train an LSTM network on the IMDB dataset for sentiment analysis [Maas et al., 2011].If the model mislabels a sample, we investigate further; if it does not, we inspect the most explanatory factors to learn more about model behavior.For the purpose of this example, we only inspect sentences of length 10 or shorter.We provide two examples below and compare with Anchors (see Table 2).
Consider our first example: READ BOOK FORGET MOVIE is a sentence we would expect to receive a negative prediction, but our model classifies it as positive.Since we are investigating a positive prediction, our reference space is conditioned on a negative label.For this model, the classic UNK token receives a positive prediction.Thus we opt for an alternative, PLATE.Performing interventions on all possible combinations of words with our token, we find the conjunction of READ, FORGET, and MOVIE is a sufficient factor for a positive prediction (R2I).We also find that changing any of READ, FORGET, or MOVIE to PLATE would result in a negative prediction (I2R).Anchors, on the other hand, perturbs the data stochastically (see Appendix C), suggesting the conjunction READ AND BOOK.Next, we investigate the sentence: YOU BETTER CHOOSE PAUL VERHOEVEN EVEN WATCHED.Since the label here is negative, we use the UNK token.We find that this prediction is brittle -a change of almost any word would be sufficient to flip the outcome.Anchors, on the other hand, reports a conjunction including most words in the sentence.Taking the R2I view, we still find a more concise explanation: CHOOSE or EVEN would be enough to attain a negative prediction.These brief examples illustrate how LENS may be used to find brittle predictions across samples, search for similarities between Anchors comparison.Anchors also includes a tabular variant, against which we compare LENS's performance in terms of R2I sufficiency.We present the results of this comparison in Fig. 3, and include additional comparisons in Appendix C. We sample 100 inputs from the German dataset, and query both methods with τ = 0.9 using the classifier from Sect.5.1.Anchors satisfies a PAC bound controlled by parameter δ.At the default value δ = 0.1, Anchors fails to meet the τ threshold on 14% of samples; LENS meets it on 100% of samples.This result accords with Thm. 1, and vividly demonstrates the benefits of our optimality guarantee.Note that we also go beyond Anchors in providing multiple explanations instead of just a single output, as well as a cumulative probability measure with no analogue in their algorithm.

COUNTERFACTUALS
Adversarial examples: spam emails.R2I sufficiency answers questions of the form, "What would be sufficient for the model to predict y?".This is particularly valuable in cases with unfavorable outcomes y .Inspired by adversarial interpretability approaches [Ribeiro et al., 2018b;Lakkaraju and Bastani, 2020], we train an MLP classifier on the SpamAssassins dataset and search for minimal factors sufficient to relabel a sample of spam emails as nonspam.Our examples follow some patterns common to spam emails: received from unusual email addresses, includes sus-  picious keywords such as ENLARGEMENT or ADVERTISE-MENT in the subject line, etc.We identify minimal changes that will flip labels to non-spam with high probability.Options include altering the incoming email address to more common domains, and changing the subject or first sentences (see Table 3).These results can improve understanding of both a model's behavior and a dataset's properties.
Diverse counterfactuals.Our explanatory measures can also be used to secure algorithmic recourse.For this experiment, we benchmark against DiCE [Mothilal et al., 2020b], which aims to provide diverse recourse options for any underlying prediction model.We illustrate the differences between our respective approaches on the Adult dataset [Kochavi and Becker, 1996], using an MLP and following the procedure from the original DiCE paper.
According to DiCE, a diverse set of counterfactuals is one that differs in values assigned to features, and can thus produce a counterfactual set that includes different interventions on the same variables (e.g., CF1: age = 91, occupation = "retired"; CF2: age = 44, occupation = "teacher").Instead, we look at diversity of counterfactuals in terms of intervention targets, i.e. features changed (in this case, from input to reference values) and their effects.We present minimal cost interventions that would lead to recourse for each feature set but we summarize the set of paths to recourse via subsets of features changed.Thus, DiCE provides answers of the form "Because you are not 91 and retired" or "Because you are not 44 and a teacher"; we answer "Because of your age and occupation", and present the lowest cost intervention on these features sufficient to flip the prediction.
With this intuition in mind, we compare outputs given by DiCE and LENS for various inputs.For simplicity, we let all features vary independently.We consider two metrics for comparison: (a) the mean cost of proposed factors, and (b) the number of minimally valid candidates proposed, where a Causal vs. non-causal recourse.When a user relies on XAI methods to plan interventions on real-world systems, causal relationships between predictors cannot be ignored.
In the following example, we consider the DAG in Fig. 5, intended to represent dependencies in the German credit dataset.For illustrative purposes, we assume access to the structural equations of this data generating process.(There are various ways to extend our approach using only partial causal knowledge as input [Karimi et al., 2020b;Heskes et al., 2020].)We construct D by sampling from the SCM under a series of different possible interventions.Table 4 describes an example of how using our framework with augmented causal knowledge can lead to different recourse options.Computing explanations under the assumption of feature independence results in factors that span a large part of the DAG depicted in Fig. 5.However, encoding structural relationships in D, we find that LENS assigns high explanatory value to nodes that appear early in the topological ordering.This is because intervening on a single root factor may result in various downstream changes once effects are fully propagated.

DISCUSSION
Our results, both theoretical and empirical, rely on access to the relevant context D and the complete enumeration of all feature subsets.Neither may be feasible in practice.When elements of Z are estimated, as is the case with the generative methods sometimes used in XAI, modeling errors could lead to suboptimal explanations.For high-dimensional settings such as image classification, LENS cannot be naïvely applied without substantial data pre-processing.The first issue is extremely general.No method is immune to model misspecification, and attempts to recreate a data generating process must always be handled with care.Empirical sampling, which we rely on above, is a reasonable choice when data are fairly abundant and representative.However, generative models may be necessary to correct for known biases or sample from low-density regions of the feature space.This comes with a host of challenges that no XAI algorithm alone can easily resolve.The second issue -that a complete enumeration of all variable subsets is often impractical -we consider to be a feature, not a bug.Complex explanations that cite many contributing factors pose cognitive as well as computational challenges.In an influential review of XAI, Miller [2019] finds near unanimous consensus among philosophers and social scientists that, "all things being equal, simpler explanations -those that cite fewer causes... are better explanations" (p.25).Even if we could list all τ -minimal factors for some very large value of d, it is not clear that such explanations would be helpful to humans, who famously struggle to hold more than seven objects in short-term memory at any given time [Miller, 1955].That is why many popular XAI tools include some sparsity constraint to encourage simpler outputs.
Rather than throw out some or most of our low-level features, we prefer to consider a higher level of abstraction, where explanations are more meaningful to end users.For instance, in our SpamAssassins experiments, we started with a pure text example, which can be represented via high-dimensional vectors (e.g., word embeddings).However, we represent the data with just a few intelligible components: From and To email addresses, Subject, etc.In other words, we create a more abstract object and consider each segment as a potential intervention target, i.e. a candidate factor.This effectively compresses a high-dimensional dataset into a 10-dimensional abstraction.Similar strategies could be used in many cases, either through domain knowledge or data-driven clustering and dimensionality reduction techniques [Chalupka et al., 2017;Beckers et al., 2019;Locatello et al., 2019].In general, if data cannot be represented by a reasonably low-dimensional, intelligible abstraction, then post-hoc XAI methods are unlikely to be of much help.

CONCLUSION
We have presented a unified framework for XAI that foregrounds necessity and sufficiency, which we argue are the fundamental building blocks of all successful explanations.We defined simple measures of both, and showed how they undergird various XAI methods.Our formulation, which relies on converse rather than inverse probabilities, is uniquely flexible and expressive.It covers all four basic explanatory measures -i.e., the classical definitions and their contrapositive transformations -and unambiguously accommodates logical, probabilistic, and/or causal interpretations, depending on how one constructs the basis tuple B. We illustrated illuminating connections between our measures and existing proposals in XAI, as well as Pearl [2000]'s probabilities of causation.We introduced a sound and complete algorithm for identifying minimally sufficient factors, and demonstrated our method on a range of tasks and datasets.Our approach prioritizes completeness over efficiency, suitable for settings of moderate dimensionality.Future research will explore more scalable approximations, model-specific variants optimized for, e.g., convolutional neural networks, and developing a graphical user interface.

A PROOFS
A.1 THEOREMS A.1.1 Proof of Theorem 1 Theorem.With oracle estimates P S(c, y) for all c ∈ C, Alg. 1 is sound and complete.
Proof.Soundness and completeness follow directly from the specification of (P1) C and (P2) in the algorithm's input B, along with (P3) access to oracle estimates P S(c, y) for all c ∈ C. Recall that the partial ordering must be complete and transitive, as noted in Sect.3.
Assume that Alg. 1 generates a false positive, i.e. outputs some c that is not τ -minimal.Then by Def. 4, either the algorithm failed to properly evaluate P S(c, y), thereby violating (P3); or failed to identify some c such that (i) P S(c , y) ≥ τ and (ii) c ≺ c. (i) is impossible by (P3), and (ii) is impossible by (P2).Thus there can be no false positives.
Assume that Alg. 1 generates a false negative, i.e. fails to output some c that is in fact τ -minimal.By (P1), this c cannot exist outside the finite set C. Therefore there must be some c ∈ C for which either the algorithm failed to properly evaluate P S(c, y), thereby violating (P3); or wrongly identified some c such that (i) P S(c , y) ≥ τ and (ii) c ≺ c.Once again, (i) is impossible by (P3), and (ii) is impossible by (P2).Thus there can be no false negatives.
A.1.2Proof of Theorem 2 Theorem.With sample estimates P S(c, y) for all c ∈ C, Alg. 1 is uniformly most powerful.
Proof.A testing procedure is uniformly most powerful (UMP) if it attains the lowest type II error β of all tests with fixed type I error α.Let Θ 0 , Θ 1 denote a partition of the parameter space into null and alternative regions, respectively.The goal in frequentist inference is to test the null hypothesis H 0 : θ ∈ Θ 0 against the alternative H 1 : θ ∈ Θ 1 for some parameter θ.Let ψ(X) be a testing procedure of the form 1[T (X) ≥ c α ], where X is a finite sample, T (X) is a test statistic, and c α is the critical value.This latter parameter defines a rejection region such that test statistics integrate to α under H 0 .We say that ψ(X) is UMP iff, for any other test ψ (X) such that we have where E θ∈Θ1 [ψ(X)] denotes the power of the test to detect the true θ, 1 − β ψ (θ).The UMP-optimality of Alg. 1 follows from the UMP-optimality of the binomial test (see [Lehmann and Romano, 2005, Ch. 3]), which is used to decide between H 0 : P S(c, y) < τ and H 1 : P S(c, y) ≥ τ on the basis of observed proportions P S(c, y), estimated from n samples for all c ∈ C. The proof now takes the same structure as that of Thm. 1, with (P3) replaced by (P3 ): access to UMP estimates of P S(c, y).False positives are no longer impossible but bounded at level α; false negatives are no longer impossible but occur with frequency β.Because no procedure can find more τ -minimal factors for any fixed α, Alg. 1 is UMP.

A.2 PROPOSITIONS
A.2.1 Proof of Proposition 1 Proposition.Let c S (z) = 1 iff x ⊆ z was constructed by holding x S fixed and sampling X R according to D(•|S).
Then v(S) = P S(c S , y).
As noted in the text, D(x|S) may be defined in a variety of ways (e.g., via marginal, conditional, or interventional distributions).For any given choice, let c S (z) = 1 iff x is constructed by holding x S i fixed and sampling X R according to D(x|S).Since we assume binary Y (or binarized, as discussed in Sect.3), we can rewrite Eq. 2 as a probability: where x i denotes the input point.Since conditional sampling is equivalent to conditioning after sampling, this value function is equivalent to P S(c S , y) by Def. 2.
The proof for this proposition is essentially identical, except in this case our conditioning event is A given by the lhs of Eq. 3, is defined over a conditional distribution D(x|A).
Since conditional sampling is equivalent to conditioning after sampling, this probability reduces to P S(c A , y).

A.2.3 Proof of Proposition 3
Proposition.Let cost be a function representing , and let c be some factor spanning reference values.Then the counterfactual recourse objective is: where τ denotes a decision threshold.Counterfactual outputs will then be any z ∼ D such that c * (z) = 1.
There are two closely related ways of expressing the counterfactual objective: as a search for optimal points, or optimal actions.We start with the latter interpretation, reframing actions as factors.We are only interested in solutions that flip the original outcome, and so we constrain the search to factors that meet an I2R sufficiency threshold, P S(c, 1 − y) ≥ τ .Then the optimal action is attained by whatever factor (i) meets the sufficiency criterion and (ii) minimizes cost.Call this factor c * .The optimal point is then any z such that c * (z) = 1.

A.2.4 Proof of Proposition 4
Proposition.Consider the bivariate Boolean setting, as in Sect. 2. We have two counterfactual distributions: an input space I, in which we observe x, y but intervene to set X = x ; and a reference space R, in which we observe x , y but intervene to set X = x.Let D denote a uniform mixture over both spaces, and let auxiliary variable W tag each sample with a label indicating whether it comes from the original (W = 1) or contrastive (W = 0) counterfactual space.Define c(z) = w.Then we have suf(x, y) = P S(c, y) and nec(x, y) = P S(1 − c, y ).
We may rewrite the former as P R (y), where the reference space R denotes a counterfactual distribution conditioned on x , y , do(x).Similarly, we may rewrite the latter as P I (y ), where the input space I denotes a counterfactual distribution conditioned on x, y, do(x ).Our context D is a uniform mixture over both spaces.
The key point here is that the auxiliary variable W indicates whether samples are drawn from I or R. Thus conditioning on different values of W allows us to toggle between probabilities over the two spaces.Therefore, for c(z) = w, we have suf(x, y) = P S(c, y) and nec(x, y) = P S(1 − c, y ).

B ADDITIONAL DISCUSSIONS OF METHOD B.1 τ -MINIMALITY AND NECESSITY
As a follow up to Remark 2 in Sect.3.2, we expand here upon the relationship between τ and cumulative probabilities of necessity, which is similar to a precision-recall curve quantifying and qualifying errors in classification tasks.In this case, as we lower τ , we allow more factors to be taken into account, thus covering more pathways towards a desired outcome in a cumulative sense.We provide an example of such a precision-recall curve in Fig. 6, using an R2I view of the German credit dataset.Different levels of cumulative necessity may be warranted for different tasks, depending on how important it is to survey multiple paths towards an outcome.Users can therefore adjust τ to accommodate desired levels of cumulative P N over successive calls to LENS.C ADDITIONAL DISCUSSIONS OF EXPERIMENTAL RESULTS

C.1 DATA PRE-PROCESSING AND MODEL TRAINING
German Credit Risk.We first download the dataset from Kaggle, 3 which is a slight modification of the UCI version [Dua and Graff, 2017].We follow the pre-processing steps from a Kaggle tutorial. 4In particular, we map the categorical string variables in the dataset (Savings, Checking, Sex, Housing, Purpose and the outcome Risk) to numeric encodings, and mean-impute values missing values for Savings and Checking.We then train an Extra-Tree classifier [Geurts et al., 2006] using scikit-learn, with random state 0 and max depth 15.All other hyperparameters are left to their default values.The model achieves a 71% accuracy.
German Credit Risk -Causal.We assume a partial ordering over the features in the dataset, as described in Fig. 5.We use this DAG to fit a structural causal model (SCM) based on the original data.In particular, we fit linear regressions for every continuous variable and a random forest classifier for every categorical variable.When sampling from D, we let variables remain at their original values unless either (a) they are directly intervened on, or (b) one of their ancestors was intervened on.In the latter case, changes are propagated via the structural equations.We add stochasticity via Gaussian noise for continuous outcomes, with variance given by each model's residual mean squared error.
For categorical variables, we perform multinomial sampling over predicted class probabilities.We use the same f model as for the non-causal German credit risk description above.
SpamAssassins.The original spam assassins dataset comes in the form of raw, multi-sentence emails captured on the Apache SpamAssassins project, 2003-2015. 5We segmented the emails to the following "features": From is the sender; To is the recipient; Subject is the email's subject line; Urls records any URLs found in the body; Emails denotes any email addresses found in the body; First Sentence, Second Sentence, Penult Sentence, and Last Sentence refer to the first, second, penultimate, and final sentences of the email, respectively.We use the original outcome label from the dataset (indicated by which folder the different emails were saved to).Once we obtain a dataset in the form above, we continue to pre-process by lower-casing all characters, only 3 See https://www.kaggle.com/kabure/german-credit-data-with-risk?select=german_ credit_data.csv.
keeping words or digits, clearing most punctuation (except for '-' and '_'), and removing stopwords based on nltk's provided list [Bird et al., 2009].Finally, we convert all clean strings to their mean 50-dim GloVe vector representation [Pennington et al., 2014].We train a standard MLP classifier using scikit-learn, with random state 1, max iteration 300, and all other hyperparameters set to their default values. 6This model attains an accuracy of 98.3%.
IMDB.We follow the pre-processing and modeling steps taken in a standard tutorial on LSTM training for sentiment prediction with the IMDB dataset. 7The CSV is included in the repository named above, and can be additionally downloaded from Kaggle or ai.standford. 8In particular, these include removal of HTML-tags, non-alphabetical characters, and stopwords based on the the list provided in the ntlk package, as well as changing all alphabetical characters to lower-case.We then train a standard LSTM model, with 32 as the embedding dimension and 64 as the dimensionality of the output space of the LSTM layer, and an additional dense layer with output size 1.We use the sigmoid activation function, binary cross-entropy loss, and optimize with Adam [Kingma and Ba, 2015].All other hyperparameters are set to their default values as specified by Keras. 9The model achieves an accuracy of 87.03%.
Adult Income.We obtain the adult income dataset via DiCE's implementation 10 and followed Haojun Zhu's preprocessing steps. 11For our recourse comparison, we use a pretrained MLP model provided by the authors of DiCE, which is a single layer, non-linear model trained with Ten-sorFlow and stored in their repository as 'adult.h5'.

C.2 TASKS
Comparison with attributions.For completeness, we also include here comparison of cumulative attribution scores per cardinality with probabilities of sufficiency for the I2R view (see Fig. 7).
Sentiment sensitivity analysis.We identify sentences in the original IMDB dataset that are up to 10 words long.Out of those, for the first example we only look at wrongly predicted sentences to identify a suitable example.For the other

Figure 2 :
Figure 2: Comparison of top k features ranked by SHAP against the best performing LENS subset of size k in terms of P S(c, y).German results are over 50 inputs; SpamAssassins results are over 25 inputs.

Figure 3 :
Figure 3: We compare P S(c, y) against precision scores attained by the output of LENS and Anchors for examples from German.We repeat the experiment for 100 inputs, and each time consider the single example generated by Anchors against the mean P S(c, y) among LENS's candidates.Dotted line indicates τ = 0.9.errors, or test for model reliance on sensitive attributes (e.g., gender pronouns).

Figure 4 :
Figure 4: A comparison of mean cost of outputs by LENS and DiCE for 50 inputs sampled from the Adult dataset.

Figure 6 :
Figure 6: An example curve exemplifying the relationship between τ and cumulative probability necessity attained by selected τ -minimal factors.
observe that, "since there is no standard procedure for converting Shapley values into a statement about a model's behavior, developers rely on their own mental model of what the values represent" (p.8).By contrast, necessary and sufficient factors are more transparent and informative, offering a direct path to what Shapley values indirectly summarize.

Table 1 :
. Our Overview of experimental settings by basis configuration.
cost Counterfactuals: Recourse, causal vs. non-causal German Extra-Trees I2R causal Full interventions cost method attains higher P S for all cardinalities.We repeat the experiment over 50 inputs, plotting means and 95% confidence intervals for all k.Results indicate that our ranking procedure delivers more informative explanations than SHAP at any fixed degree of sparsity.Results from the I2R setting are in Appendix C. 5.2 RULE LISTS Sentiment sensitivity analysis.Next, we use LENS to study model weaknesses by considering minimal factors with high R2I and I2R sufficiency in text models.Our

Table 2 :
Example prediction given by an LSTM model trained on the IMDB dataset.We compare τ -minimal factors identified by LENS (as individual words), based on P S(c, y) and P S(1 − c, 1 − y), and compare to output by Anchors.choosepaulverhoeven even watched' correctly predicted negative[choose, better, even, you, paul, verhoeven]0.95choose, even better, choose, paul, even

Table 3 :
(Top) A selection of emails from SpamAssassins, correctly identified as spam by an MLP.The goal is to find minimal perturbations that result in non-spam predictions.(Bottom) Minimal subsets of feature-value assignments that achieve non-spam predictions with respect to the emails above.

Table 4 :
Recourse example comparing causal and non-causal (i.e., feature independent) D. We sample a single input example with a negative prediction, and 100 references with the opposite outcome.For I2R causal we propagate the effects of interventions through a user-provided SCM.