This section begins with definitions and notation. We then present two real-world data issues: missing values and selection bias, explaining how they arise and defining them using our notation.
Definitions and notation
Let \({\mathbf {R}} = \{{\mathbf {r}}_1, {\mathbf {r}}_2, \ldots , {\mathbf {r}}_n\}\) be the training set containing n individual observations and \({\mathbf {r}}_i\) represents the \(i{\mathrm{th}}\) observation in the sample. Each observation contains one binary outcome, one categorical sensitive attribute and multiple categorical non-sensitive features. Let \({\mathbf {X}}\) = \(\{ {\mathbf {x}}^1, {\mathbf {x}}^2, \ldots , {\mathbf {x}}^p\}\) be the set of p non-sensitive categorical features such as education level and state of residence that are used as features for prediction, and let \({\mathbf {x}}^j = \{x_1^j, x_2^j \ldots x_n^j \}\) be a set of categorical feature values for the \(j{\mathrm{th}}\) non-sensitive feature in \({\mathbf {X}}\). It can be the case that \({\mathbf {X}}\) contains missing values. We define missing values as an additional feature value, where \(x_i^j = \emptyset \) if the feature value for the \(j{\mathrm{th}}\) non-sensitive feature at the \(i{\mathrm{th}}\) observation is missing. Let \({\mathbf {M}} = \{ {\mathbf {m}}^1, {\mathbf {m}}^2, \ldots , {\mathbf {m}}^p\}\) be a set of binary missing value indicators for the non-sensitive features \({\mathbf {X}}\) and \({\mathbf {m}}^j = \{m_1^j, m_2^j \ldots m_n^j \}\) be the set of binary missing value indicators for the \(j{\mathrm{th}}\) non-sensitive feature in \({\mathbf {X}}\). We say that \(m_i^j = 1\) if \(x_i^j\) is missing, and \(m_i^j = 0\) if \(x_i^j\) is not missing for observations \(i \in \{1,2,\ldots ,n\}\) and non-sensitive features \(j \in \{1,2,\ldots ,p\}\).
Let S = \(\{s_1, s_2, \ldots , s_n\}\) be a binary sensitive attribute that is constructed from either a binary variable, e.g., gender, or a categorical variable, e.g., race, where \(s_i\) is the sensitive attribute value for the \(i{\mathrm{th}}\) observation and S \(\notin \) X. If the sensitive attribute is categorical, it is converted to a binary attribute with two values, privileged group and unprivileged group. For example, our sample may have race as the sensitive attribute and {White, Black, Hispanic, Asian} \(\in \) S. We convert these values into a binary attribute. We treat one or more races as the privileged group, and all the other races as the unprivileged group (1 vs. the rest). Hypothetically, we may treat White and Asian as the privileged group and Black and Hispanic as the unprivileged group in a classifier deciding on loan approvals. There is no universal rule on categorization—it is task-specific.Footnote 2 For the \(i{\mathrm{th}}\) observation, we say \(s_i\) = 0 if the observation is in the unprivileged group and \(s_i\) = 1 if the observation is in the privileged group. Finally, in this work we make the assumption that a sensitive attribute does not contain any missing values. By doing this, we can better understand the impact of missing values on outcomes as it relates to the sensitive attribute without adding a confounding factor.
In this work, our classification task is binary. Our goal is to predict labels in Y using a set of non-sensitive features. Let Y = \(\{ y_1, y_2, \ldots , y_n\}\) and for the \(i{\mathrm{th}}\) observation, \(y_i \in \{+,-\}\) with \(+\) be a favorable outcome such as getting approved for a loan and − be an unfavorable outcome. Similarly, let \({\hat{Y}}\) represent the predicted outcome with \({\hat{Y}} = \{{\hat{y}}_1, {\hat{y}}_2, \ldots , {\hat{y}}_n\}\) and \({\hat{y}}_i \in \{+,-\}\) is the predicted label of the \(i{\mathrm{th}}\) observation.
Fairness background
Bias is prejudice in favor of or against something or someone, usually in a way that is unfair and unfairness is different treatment of people based on a sensitive attribute.
Formal definition of fairness From a legal perspective, many anti-discrimination laws define unfairness using disparate treatment and disparate impact to show unfair treatment of people based on a sensitive attribute. Disparate treatment is intentional discrimination based on a sensitive attribute, whereas disparate impact is unintentional discrimination [3].
Researchers define fairness using two concepts, demographic parity and equalized odds [51]. Demographic parity requires that the predicted label be independent of the sensitive attribute. More formally, \(P ( {\hat{Y}} = 1 | S = 0) = P ({\hat{Y}} =1 | S = 1)\). Equalized odds requires that the prediction label and the sensitive attribute are independent conditional on the true class. That is, \(P ({\hat{Y}} = 1 | S = 0,Y = y) = P ({\hat{Y}} = 1 | S = 1,Y = y)\), \(\forall y \in \{-,+\}\) [28]. It has been shown that except in trivial cases, any practically useful classifier satisfying equalized odds cannot be discriminatory [25].
Measurements of fairness A number of researchers have proposed different metrics for quantifying fairness. Feldman et al. and Zafar et al. [23, 53] propose disparate impact (“p%-rule”) to measure fairness. This metric is closest to the legal definition of fairness and is often used in anti-discrimination law to quantify fairness and discrimination. Chouldechova et al. [17] propose group conditioned fairness measures, including grouped false positive rate (s-FPR) and grouped false negative rate (s-FNR). These metrics are closely connected to the notion of equalized odds.
In this paper, we use p%-rule and error rate balance to quantify fairness of classifiers. The p%-rule is defined as:
$$\begin{aligned} \qquad \qquad \text {min } \left( \frac{P({\hat{Y}} = + | S=1)}{P({\hat{Y}} = + | S=0)},\frac{P({\hat{Y}} = + | S=0)}{P({\hat{Y}} = + | S=1)}\right) \end{aligned}$$
The higher the value, the fairer the classifier is. Generally, if the p%-rule is greater than 80%, or 0.8, the classifier is considered to be non-discriminatory [6]. Consider the example of a loan approval. Suppose gender is the sensitive attribute and getting approved for a loan is the positive outcome. The p%-rule measures the ratio between the probability of getting approved if the customer is female versus male. If the approval rate for male applicants is significantly higher than female applications, we can say that this classifier is discriminatory based on gender.
Error rate balance is defined as balancing the false positive rate and false negative rate across all sensitive groups. In particular, the goal is to achieve: \(P({\hat{Y}} = -|S=0, Y=y) = P({\hat{Y}} = -|S=1, Y=y), \forall y\in \{-,+\}\). When \(y=+\), the constraint equalizes the false negative rate (FNR) across two sensitive groups. When \(y=-\), the constraint equalizes the false positive rate (FPR). For a fair classifier, the error rate difference should be small across all sensitive groups. Using our loan approval example, the false negative rate (FNR) represents the how often an applicant is qualified for the loan, but the classifier decides that applicant is not qualified. The false positive rate (FPR) is the rate represents the rate at which an applicant is not qualified for a loan, but the classifier decides the applicant is qualified. The goal of error rate balancing is to make sure that the FPR and the FNR are similar and small.
Data issue: missing values
Missing values are very common in quantitative research especially in survey research [1]. King and colleagues show that approximately half of the respondents of political science surveys do not answer all of the questions [38]. Missing values occur at two different levels, the unit level and the item level [19]. A unit level missing value occurs when there is no information collected from a respondent and information about the respondent does not appear in the training set. If the non-response rate is different across sensitive groups, then unit level missing values can be viewed as a type of selection bias. An item level missing value occurs when a respondent does not answer all of the survey questions, and the incomplete information is represented as missing values in the training set. In this paper, when we discuss missing values, we are discussing item level missing values. When we discuss selection bias, we are considering unit level missing values. The missing value problem can be described based on different characteristics. We consider the following characteristics since they are prevalent in the literature: the proportion of missing values, the missing value mechanisms, and the patterns of missing values [19].
Proportion of missing values The proportion of missing values can affect the quality of the statistical analysis and prediction output. Schafer and colleagues [48] and Bennett and colleagues [5] suggest that statistical analysis is likely to be biased if the percentage of missing values is more than 5% to 10%. Let U = \(\{u_1, u_2, \ldots , u_n\}\) be a binary variable indicating whether an observation contains one or more missing values. Recall \({\mathbf {M}}\) is the set of binary missing value indicators in \({\mathbf {X}}\) and \({\mathbf {m}}^j = \{m_1^j, m_2^j \ldots m_n^j \}\) is the set of binary missing value indicators for the \(j{\mathrm{th}}\) non-sensitive feature in \({\mathbf {X}}\). We say that \(m_i^j = 1\) if \(x_i^j\) is missing, and \(m_i^j = 0\) if \(x_i^j\) is not missing. For the \(i{\mathrm{th}}\) observation, \(u_i\) = 0 if there are no missing values for any of the variables in this observation, i.e., \(m_i^1 = m_i^2= \cdots = m_i^p =0\), and \(u_i\) = 1 if the equality does not hold. We define the proportion of missing values in a data set as follows: \(\lambda = \frac{\sum \nolimits _{i=1}^{n}u_i}{n}\), where n is the number of observations.
Missing value mechanisms To understand different missing value mechanisms, we partition the training set \({\mathbf {R}}\) into two subsets: the observed data, \({\mathbf {R}}_{obs}\), and the missing data, \({\mathbf {R}}_{mis}\), where \({\mathbf {R}} = ({\mathbf {R}}_{obs}, {\mathbf {R}}_{mis})\). Previous work has defined three classes of generative mechanisms for missing values: missing at random (MAR), missing completely at random (MCAR), and missing not at random (MNAR) [47]. With MAR, the probabilities of missing values given the data set \({\mathbf {R}}\) depend only on observed data (\({\mathbf {R}}_{obs}\)) [19]:
$$\begin{aligned} \qquad \qquad P({\mathbf {M}}|R) = P({\mathbf {M}}|{\mathbf {R}}_{obs},{\mathbf {R}}_{mis}) =P({\mathbf {M}}|{\mathbf {R}}_{obs}) \end{aligned}$$
For example, suppose we are given a training data set with one binary outcome, one binary sensitive attribute and three categorical non-sensitive attributes. Suppose education level is a categorical non-sensitive feature containing missing values. If the probabilities of missing values in the education level depend only on observed data, i.e., the sensitive attribute and the other two non-sensitive features, then those missing values are under MAR. With MAR, the proportion of missing values can vary across different categories of sensitive values as long as the probability of the missing values in the non-sensitive categorical feature (education level) depends only on observed data.
MCAR is a special case of MAR in which the probabilities of missing values depend neither on the observed data (\({\mathbf {R}}_{obs}\)) nor the missing data (\({\mathbf {R}}_{mis}\)) [19]. With MCAR, missing values are completely randomly distributed:
$$\begin{aligned} \qquad \qquad P({\mathbf {M}}|{\mathbf {R}}) = P({\mathbf {M}}|{\mathbf {R}}_{obs},{\mathbf {R}}_{mis}) =P({\mathbf {M}}) \end{aligned}$$
The last missing value mechanism is MNAR. It occurs when the probability of a missing value depends on both the observed values and the missing values themselves [19]:
$$\begin{aligned} \qquad \qquad P({\mathbf {M}}|R) = P({\mathbf {M}}|{\mathbf {R}}_{obs},{\mathbf {R}}_{mis}) \ne P({\mathbf {M}}|{\mathbf {R}}_{obs}) \end{aligned}$$
Using our education example, suppose the missing values in education level depend on the education level of the respondents. For example, respondents with a lower education level are less likely to disclose their information than respondents with a higher education level. We would say that those missing values are under MNAR.
With MAR and MNAR, there are multiple methods for filling missing values, the most popular of which is imputation [40]. There are no established methods for filling in missing data under MNAR, but Jakobsen and colleagues show that it is possible to fill missing data under MNAR with sensitivity analyses [30]. It can also be difficult to determine the missing value mechanism for a data set. It is possible to use Little’s multivariate test [41] to determine the MCAR mechanism. However, the MAR and MNAR mechanisms cannot be distinguished using observed data [15, 29].
Patterns of missing values Missing values can also be characterized based on the types of patterns they form. There are three patterns of missing values identified in the literature: univariate (multivariate), monotone, and arbitrary [48]. The univariate missing value pattern occurs if only one feature in the data contains missing values.Footnote 3 The monotone missing value pattern occurs when \({\mathbf {x}}^j\) is missing, and \({\mathbf {x}}^{j+1}\), \({\mathbf {x}}^{j+2}, \ldots {\mathbf {x}}^{p}\) are also missing. It typically arises during longitudinal studies if respondents drop out during the study period. Finally, the arbitrary missing value pattern is the most general pattern in which any set of variables may be missing from any observation in the data. For example, there may exist missing values in the outcome variable Y and/or in the sensitive attribute S. In this paper, we focus on the impact of missing values within non-sensitive features. Therefore, we do not consider missing values in Y and S, but instead only consider missing values in non-sensitive features \({\mathbf {X}}\) that have a univariate missing value pattern.
Data issue: selection bias
Selection bias occurs if observations from some groups in the sample are oversampled and others are undersampled. In this case, some groups are over represented and others are under represented. If we build a classifier using the biased data, the model would be biased toward some groups of users. Kamiran et al. explain that if a data set is unbiased, i.e., sensitive attribute S and outcome Y are statically independent, we have [32]:
$$\begin{aligned}&P_{exp}(S = s, Y = y) = P(S = s) \times P(Y=y), \\&s \in \{0,1\}, y \in \{-,+\} \end{aligned}$$
\(P_{exp} (s,y)\) is the probability of the sample distribution given statistical independence.
Table 1 Definition and examples of different types of selection bias Selection bias can be caused by many types of sampling bias such as representation bias, population bias, and non-response bias. Table 1 summarizes each type of selection bias.
To quantify the severity of selection bias, we define bias, \(\beta \) as follows:
$$\begin{aligned}&\beta = \sum \limits _{s\in S}\sum \limits _{y\in Y} \frac{\mid \left| \mathbf{R }(S = s,Y = y)| - |E(\mathbf{R }(S = s,Y = y))\right| \mid }{n} \\&\forall S = \{0,1\}, Y = \{-,+\} \end{aligned}$$
where \(|\mathbf{R }(s,y)|\) is actual number of observations with sensitive attribute \(S = s\) and outcome \(Y=y\) and \(|E(\mathbf{R }(s,y))|\) is the expected number of observations given statistical independence, \(|E(\mathbf{R }(s,y))| = P_{exp}(S=s,Y=y)*n\). We take the absolute value of the difference between the expected number of observations and the actual number of observations in each group. With this metric, \(\beta \) = 0 if there is no selection bias and 1 if the data are extremely biased, i.e., the dataset contains only observations from one sensitive group and the individuals in the group have the same outcome.