Estimating Group Properties in Online Social Networks with a Classifier

Abstract

We consider the problem of obtaining unbiased estimates of group properties in social networks when using a classifier for node labels. Inference for this problem is complicated by two factors: the network is not known and must be crawled, and even high-performance classifiers provide biased estimates of group proportions. We propose and evaluate AdjustedWalk for addressing this problem. This is a three step procedure which entails: (1) walking the graph starting from an arbitrary node; (2) learning a classifier on the nodes in the walk; and (3) applying a post-hoc adjustment to classification labels. The walk step provides the information necessary to make inferences over the nodes and edges, while the adjustment step corrects for classifier bias in estimating group proportions. This process provides de-biased estimates at the cost of additional variance. We evaluate AdjustedWalk on four tasks: the proportion of nodes belonging to a minority group, the proportion of the minority group among high degree nodes, the proportion of within-group edges, and Coleman’s homophily index. Simulated and empirical graphs show that this procedure performs well compared to optimal baselines in a variety of circumstances, while indicating that variance increases can be large for low-recall classifiers.

The authors thank members of the Social Dynamics Laboratory and anonymous reviewers for their helpful suggestions. The authors were supported while this research was conducted by grants from the U.S. National Science Foundation (SES 1357488), the National Research Foundation of Korea (NRF-2016S1A3A2925033), the Minerva Initiative (FA9550-15-1-0162), and DARPA (NGS2). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

7. Appendix

1.1 7.1 Variance of Corrected Estimates

Consider the simple case of classifying group proportions with two groups. We obtain a sample from the population with true proportions $\hat{p}$ and estimated proportions $\hat{m}$. Multiplying out Eq. 2 gives expressions for for the estimated $\hat{p}_a$ and $\hat{p}_b$,

$$\begin{aligned} \hat{p}_a = \frac{\hat{m}_a c_{\hat{b} \mid b} - \hat{m}_b c_{\hat{a} \mid b}}{\det (C)}, \quad \hat{p}_b = \frac{\hat{m}_b c_{\hat{a} \mid a} - \hat{m}_a c_{\hat{b} \mid a}}{\det (C)}. \end{aligned}$$

(6)

The variance of the mean is given by,

$$\begin{aligned} {{\mathrm{Var}}}(E[\hat{p}_a])&= {{\mathrm{Var}}}(\frac{E[\hat{m}_a] - c_{\hat{a} \mid b}}{\det (C)}) \end{aligned}$$

(7)

$$\begin{aligned}&= \frac{1}{\det (C)^2}{{\mathrm{Var}}}(E[\hat{m}_a]), \end{aligned}$$

(8)

where we use the assumption that C is constant to pull it out of the variance expression.

When there is no classification error, $\det (C) = 1$, and when the classifier guesses randomly (.5 in every cell), $\det (C) = 0$ and the variance is undefined. $\det ^2(C)$ provides a clear quantification of the variance increase we expect for group proportions. For instance, if $C = [0.8, 0.2; 0.2, 0.8]$ with $\det ^2(C) = 0.6$, we expect a variance increase of $1/0.6^2 = 2.78$. If classifier performance improves to $C = [0.9, 0.1; 0.1, 0.9]$, the variance increase is $1/0.8^2 = 1.56$.

${{\mathrm{Var}}}(E[\hat{m}_a])$ comes from the random walking procedure itself and is generally not known in closed form. Two methods for closed-form variance have been proposed [16, 38]. The Volz-Heckathorn [38] estimator is biased but provides reasonable estimates in practice. The Goel and Salganik [16] variance estimator relies on knowing the homophily of the network. Bootstrap resampling methods based on creating “synthetic chains” from the estimated transition matrix between groups have also often been used [19].

Simulations of various RWRW estimators [12] show that factors such as a non-equilibrium seed selection, group homophily, and number of waves from each seed affect both the bias and variance of RWRW estimates. Generally, one long chain provides the best results, rather than many shorter chains. It is easier to sample from lower homophily networks, and equilibrium seed selection (proportional to degree) is useful if one must use relatively short chains. Otherwise, if chains may be long, a burn-in period can be used to simulate equilibrium seed selection.

1.2 7.2 Correcting Visibility

While RWRW gives the mean of g over the population of nodes, the distribution of g is often an object of interest. For instance, if we wish to estimate the proportion of minority group members in the top 20% of the degree distribution, we need to estimate the joint distribution of $(g(i), d_i)$ and take nodes in the top 20% of the distribution of $d_i$.

Fortunately, importance resampling [26, 34] based on the data obtained during an RWRW walk provides a method to do this. If we know node i with degree $d_i$ is sampled with probability $\pi (i)$ and we want to sample it with probability $1{\slash }N$ (a uniform distribution over the nodes), then we construct an importance weight using the ratio of desired over actual distributions

$$\begin{aligned} \frac{1/N}{\pi (i)} = \frac{D}{N d_i} = \frac{\bar{d}}{d_i} = w_i. \end{aligned}$$

(9)

$w_i$ provides a resampling weight for node i. We then normalize $w_i / \sum _j w_j$ and resample data $(f(i), d_i)$ according to this probability to approximate draws from the desired distribution $1{\slash }N$.

An importance resample produces a distribution of $(d_i, g(i))$ which mirrors the distribution in the population. We then sort the resampled nodes by degree $d_i$ and take the proportion in the top 20% of degree where $g(i) = b$, or where i is a member of the minority group. In the case with no classification error, this procedure produces an unbiased estimate of the fraction of minority group members in the top 20% of the degree distribution.

With classification error, we need to add an additional step to correct the importance resample. Call $\hat{m}_b^{\mathcal {I}}(20)$ the measured proportion of group b in the top $20\%$ of the degree distribution in importance resample $\mathcal {I}$. Likewise, there is a vector that contains measures for all groups $\mathbf {\hat{m}}^{\mathcal {I}}(20)$. Then we can use a procedure similar to Eq. 2 to correct the importance resample proportions:

$$\begin{aligned} \mathbf {\hat{p}}^{\mathcal {I}}(20) = C^{-1}\mathbf {\hat{m}}^{\mathcal {I}}(20). \end{aligned}$$

(10)

To see when $\mathbf {\hat{p}}^{\mathcal {I}}(20)$ is unbiased, repeat the reasoning for estimating the population proportion $\mathbf {\hat{p}}$ above. This shows that $\mathbf {\hat{p}}^{\mathcal {I}}(20)$ is unbiased when the importance resample provides an unbiased estimate of $\mathbf {m}^{\mathcal {I}}(20)$. A similar argument applies for the variance, and the determinant of C may be used to estimate the increase in variance.

1.3 7.3 Correcting Edge Proportions

Correcting estimates of ties between groups presents a more substantial challenge than correcting group proportions. Akin to C, there is a dyadic misclassification matrix M which maps $\mathbf {s}$ to $\mathbf {t}$,

$$\begin{aligned} M \mathbf {s} = \mathbf {t}, \end{aligned}$$

(11)

where

$$\begin{aligned} M = \begin{bmatrix} c_{\hat{a} \mid a}^2&c_{\hat{a} \mid a} c_{\hat{a} \mid b}&c_{\hat{a} \mid b}^2\\ 2 * c_{\hat{a} \mid a} c_{\hat{b} \mid a}&c_{\hat{a} \mid a} c_{\hat{b} \mid b} + c_{\hat{a} \mid b} c_{\hat{b} \mid a}&2 * c_{\hat{a} \mid b} c_{\hat{b} \mid b} \\ c_{\hat{b} \mid a}^2&c_{\hat{b} \mid a} c_{\hat{b} \mid b}&c_{\hat{b} \mid b}^2 \end{bmatrix}, \end{aligned}$$

which implies that we can use a technique similar to Eq. 2 at the dyad level

$$\begin{aligned} \mathbf {s} = M^{-1} \mathbf {t}. \end{aligned}$$

(12)

In practice, we obtain a sample $\mathbf {\hat{t}}$ rather than $\mathbf {t}$ for the entire graph, which is then used to estimate true edge proportions $\mathbf {\hat{s}}$. $\mathbf {\hat{s}}$ is unbiased when the sampling method employed produces unbiased estimates of $\mathbf {t}$. If $B = M^{-1}$, the expectation for $\hat{s}_a$ is given by

$$\begin{aligned} E[\hat{s}_{aa}] = b_{00}E[\hat{t}_{aa}] + b_{01}E[\hat{t}_{ab}] + b_{02} E[\hat{t}_{bb}]. \end{aligned}$$

(13)

As in the node case, we can expect the variance of $E[\hat{s}_{aa}]$ and $E[\hat{s}_a]$ to increase when applying classification bias correction. Simulations below indicate that variance inflation for $E[\hat{s}_a]$ is larger than for $E[\hat{p}_a]$. Note that $\hat{s}_a = 2\hat{s}_{aa} / (2\hat{s}_{aa} + \hat{s}_{ab})$ is unbiased under the same conditions as $\hat{s}_{aa}$.

If $B = M^{-1}$, then the variance for $\hat{s}_{aa}$ is

$$\begin{aligned} {{\mathrm{Var}}}(E[\hat{s}_{aa}]) = b^2_{00}{{\mathrm{Var}}}(E[\hat{t}_{aa}]) + b^2_{01}{{\mathrm{Var}}}(E[\hat{t}_{ab}]) + b^2_{02} {{\mathrm{Var}}}(E[\hat{t}_{bb}]). \end{aligned}$$

(14)