Privacy-preserving worker allocation in crowdsourcing

Abstract

Crowdsourcing has been a prevalent way to obtain answers for tasks that need human intelligence. In general, a crowdsourcing platform is responsible for allocating workers to each received task, with high-quality workers in priority. However, the allocation results can in turn yield knowledge about workers’ quality. For example, those unallocated workers are supposed to be less-qualified. They can be upset if such information is known by the public, which is an invasion of their privacy. To alleviate such concerns, we study the privacy-preserving worker allocation problem in this paper, aiming to properly allocate the workers while protecting their privacy. We propose worker allocation methods with the property of differential privacy, which proceed by first computing weights for each potential allocation and then sampling according to the weights. The Markov Chain Monte Carlo-based method is shown in our experiments to improve over the trivial random allocation method by 18.9% in terms of worker quality on synthetic data. On the real data, it realizes differential privacy with less than 20% loss on quality even when $\epsilon = \frac{1}{3}$.

Appendices

A Proof of Lemma 1

Proof

We show that $g(\cdot )$ can be constructed to meet (i) and (ii), respectively.

• (i) We construct a bipartite graph with $\varPhi '$ and $\varPhi$ as the left and right sides, respectively. In addition, each $\phi \in \varPhi$ has $\lfloor \frac{|W| - (k -1 )}{k} \rfloor$ copies in its side. Each copy of $\phi$ is equally linked to all $\phi ' \in \varPhi '$ such that $|\phi \setminus \phi ' | = 1$. An example is given in Fig. 10. We show that there is a matching covering all $\phi \in \varPhi$, which then directly infers the feasibility of constructing $g(\cdot )$ meeting (i). Hall’s Theorem [25] states that for a bipartite graph, there is a matching covering $\varPhi$, if and only if

  $$\begin{aligned} \forall X \subseteq {\varPhi }, \quad |X| \le |Nbor(X)|\,, \end{aligned}$$

  where Nbor(X) contains all the neighbor nodes of X in $\varPhi '$. Given any $X \subseteq \varPhi$, we construct $X^*$ on top of X by including all copies of any $\phi \in X$ such that each $\phi \in X$ has $\lfloor \frac{|W| - (k -1 )}{k} \rfloor$ copies in $X^*$. Obviously we have $Nbor(X) = Nbor(X^*)$. There are $\frac{|X^*|}{\lfloor \frac{|W| - (k -1 )}{k} \rfloor }$ distinct $\phi$’s in $X^*$, each of which has $|W| - (k -1 )$ neighbors in $\varPhi '$. Different $\phi \in X^*$ can be incident to a same node in $Nbor(X^*)$. However, each $\phi '\in Nbor(X^*)$ has at most k neighbors in $X^*$. As a result,

  $$\begin{aligned} |Nbor(X^*)| \ge \frac{|X^*|}{\lfloor \frac{|W| - (k -1 )}{k} \rfloor } * (|W|-(k-1)) * \frac{1}{k} \ge |X^*|. \end{aligned}$$

  Then $|Nbor(X)| = |Nbor(X^*)| \ge |X^*| \ge |X|$.

• (ii) We still construct a bi-graph with $\varPhi$ and $\varPhi '$, but this time each $\phi \in \varPhi$ has $\lceil \frac{|W| - (k -1) }{k} \rceil$ copies. An example is given in Fig. 11. We show that there is a matching covering all $\phi ' \in \varPhi '$, which then directly infers the feasibility of constructing $g(\cdot )$ meeting (ii). With Hall’s Theorem, we need to show that

  $$\begin{aligned} \forall X \subseteq {\varPhi '}, \quad |X| \le |Nbor(X)|\,. \end{aligned}$$

  Given any $X \subseteq \varPhi '$, each $\phi ' \in X$ has $k * \lceil \frac{|W| - (k -1) }{k} \rceil$ neighbors in $\varPhi$. Each $\phi \in Nbor(X)$ has at most $|W| -(k - 1)$ neighbors in X. As a result,

  $$\begin{aligned} |Nbor(X)|\ge & {} |X| * k * \lceil \frac{|W| - (k -1) }{k} \rceil \\&* \frac{1}{|W| -(k - 1)} \ge |X|\,. \end{aligned}$$

  $\square$

Fig. 10

An example bigraph for proving (i) of Lemma 1, where $W = \{a,b,c,d,e,f\}$, $k = 2$, $\varPhi = W$, $\varPhi ' = \{ab, ac,..., ef\}$, and $\lfloor \frac{|W| - (k -1 )}{k} \rfloor = 2$. Each element in $\varPhi$ has 2 copies in the downside of the bigraph

Fig. 11

An example bigraph for proving (ii) of Lemma 1, where $W = \{a,b,c,d,e,f\}$, $k = 2$, $\varPhi = W$, $\varPhi ' = \{ab, ac,..., ef\}$, and $\lceil \frac{|W| - (k -1 )}{k} \rceil = 3$. Each element in $\varPhi$ has 3 copies in the downside of the bigraph

B Proof of Theorem 3

Proof

Given two neighboring worker pools $W_1$ and $W_2$ such that $W_1 \setminus W_2 = w_1$, and a query Q, let $P_1(A_h)$ and $P_2(A_h)$ denote the probability of getting the output $A_h$ from using Q over $W_1$ and $W_2$, respectively. Let $\varPhi _1$ and $\varPhi _2$ denote all the length-B subsets of $W_1$ and $W_2$, respectively. Then, we need to bound $\frac{P_1(A_h)}{P_2(A_h) }$ and $\frac{P_2(A_h)}{P_1(A_h) }$ for any $A _h\in \varPhi =\varPhi _1 \cup \varPhi _2$.

Let us first consider $A _h$ such that $w_1 \notin A _h$. Then, according to Algorithm 1,

$$\begin{aligned} P_1(A _h) = \frac{e^{\epsilon f(A _h)} }{\sum \limits _{A \in \varPhi _1} e^{\epsilon f(A) }} \,. \end{aligned}$$

$P_2(A _h)$ can be deducted similarly. Since $\varPhi _2 \subset \varPhi _1$, we have $\frac{P_1(A _h)}{P_2(A _h) } <\frac{P_2(A _h)}{P_1(A _h) }$, and thus focus on $\frac{P_2(A _h)}{P_1(A _h) }$.

$$\begin{aligned} \frac{P_2(A _h)}{P_1(A _h) }= & {} \frac{\sum \limits _{A \in \varPhi _2 } e^{\epsilon f(A)} + \sum \limits _{A \in \varPhi _1 \setminus \varPhi _2 } e^{\epsilon f(A) } }{ \sum \limits _{A \in \varPhi _2} e^{\epsilon f(A)} } \\= & {} 1 + \frac{\sum \limits _{A \in \varPhi _1 \setminus \varPhi _2 } e^{\epsilon f(A)} }{\sum \limits _{A \in \varPhi _2 } e^{\epsilon f(A) } } \\ \end{aligned}$$

Let $\varDelta \varPhi = \varPhi _1 \setminus \varPhi _2$. Each answer $A_p \in \varDelta \varPhi$ can be represented as a combination as $(\tilde{A},w_1)$, where $\tilde{A}$ is a size-(B-1) worker subset of $W_2$, i.e., $\tilde{A}\subset W_2 \wedge |\tilde{A}| = B - 1$. Given $\tilde{A}$, for any $A_p \in \varDelta \varPhi$ and any $A_q \in \varPhi _2$, such that $\tilde{A} \subset A_p$ and $\tilde{A} \subset A_q$, we have $f(A_p) \le f(A_q) + 1$. Note that $\tilde{A}$ and $A_q$ are size-$(B-1)$ and -B subsets of $W_2$. According to Lemma 1,we can construct an injection $g: \varPhi _2 \rightarrow \varDelta \varPhi$, such that for each $A_q \in \varPhi _2$, we have $g(A_q) \setminus A_q = w_1$ and each $A_p \in \varDelta \varPhi$ is reached via $g(\cdot )$ by at least $\lfloor {\frac{|W_2| - (B-1)}{B}} \rfloor$ times. Let $\alpha = \lfloor {\frac{|W_2| - (B-1)}{B}} \rfloor$, and we have

$$\begin{aligned} \frac{\sum \limits _{A \in \varDelta \varPhi } e^{\epsilon f(A)} }{\sum \limits _{A \in \varPhi _2 } e^{\epsilon f(A) } }\le & {} \frac{\frac{1}{\alpha } \sum \limits _{A \in \varPhi _2 } e^{\epsilon f(g(A)) } }{\sum \limits _{A \in \varPhi _2 } e^{\epsilon (f(A)) )} } \\\le & {} \frac{\frac{1}{\alpha } \sum \limits _{A \in \varPhi _2 } e^{\epsilon (f(A) + 1) } }{\sum \limits _{A \in \varPhi _2 } e^{\epsilon (f(A)) )} } \\= & {} \frac{e^\epsilon }{\alpha } \,. \end{aligned}$$

Then,

$$\begin{aligned} \frac{P_2(A _h)}{P_1(A _h) } \le 1 + \frac{e^\epsilon }{\alpha } \le e^{\epsilon + \ln (\frac{1}{\alpha } + 1)} \,. \end{aligned}$$

(5)

We then consider the other case when $w_1 \in A _h$, which suggests that $A _h \notin \varPhi _2$. Obviously we have $P_2(A _h) = 0$. For any $t \in \{1...B\}$ and $\varPhi _1^t = \{A\in \varPhi _1 \wedge |A \cap A_h |= t\}$, we have $|\varPhi _1^t| = \left( {\begin{array}{c}B\\ t\end{array}}\right) \left( {\begin{array}{c}|W_1| - B\\ B - t\end{array}}\right)$ and $\forall A \in \varPhi _1^t\,,~ f(A) \ge f(A_h) - (B - t)$. Then,

$$\begin{aligned} \begin{aligned} P_1(A _h)&= \frac{e^{\epsilon f(A_h)}}{\sum \limits _{t \in \{0...B\}}\sum \limits _{A \in \varPhi _1^t} e^{\epsilon f(A)} } \\&\le \frac{e^{\epsilon f(A_h) } }{\sum \limits _{t \in \{0...B\}} \left( {\begin{array}{c}B\\ t\end{array}}\right) \left( {\begin{array}{c}|W_1| - B\\ B - t\end{array}}\right) e^{\epsilon \left( f(A_h) - (B - t)\right) }} \\&\le \frac{ 1 }{ \sum \limits _{t \in \{0...B\}} \left( {\begin{array}{c}B\\ t\end{array}}\right) \left( {\begin{array}{c}|W_1| - B\\ B - t\end{array}}\right) e^{\epsilon ( t - B ) }} \,. \end{aligned} \end{aligned}$$

(6)

Combining Equation (5) and (6), we can obtain for any $A_h$,

$$\begin{aligned} \begin{aligned} P_1(A _h) \le ~&e^{\epsilon + \ln (\frac{1}{\alpha } + 1) }* P_2(A _h) + \\&\frac{ 1 }{ \sum \limits _{t \in \{0...B\}} \left( {\begin{array}{c}B\\ t\end{array}}\right) \left( {\begin{array}{c}|W_1| - B\\ B - t\end{array}}\right) e^{\epsilon ( t - B ) }} \,, \\ P_2(A _h) \le ~&e^{\epsilon + \ln (\frac{1}{\alpha } + 1)} * P_1(A _h) + \\&\frac{ 1 }{ \sum \limits _{t \in \{0...B\}} \left( {\begin{array}{c}B\\ t\end{array}}\right) \left( {\begin{array}{c}|W_1| - B\\ B - t\end{array}}\right) e^{\epsilon ( t - B ) }} \,. \end{aligned} \end{aligned}$$

Finally, since $|W_1| - |W_2| = 1$, we have ${{\mathcal {O}}}(|W_1|) = {{\mathcal {O}}}(|W_2|)$. We simply use the same ‘W’ in the theorem. $\square$

C Proof of Theorem 4

Proof

Case $\mathbf {B \le \frac{W+1}{2}}$. Let us consider the worker $\hat{w}$ who has the largest reliability. We consider all the worker subsets of length-B including and excluding $\hat{w}$, respectively, denoted by ${{\mathcal {W}}}^+ = \{W^+\}$ and ${{\mathcal {W}}^-} = \{W^-\}$. Then, we have

$$\begin{aligned} |{{\mathcal {W}}}^+| = \left( {\begin{array}{c}|W| - 1\\ B-1\end{array}}\right) \text {~~and~~} |{{\mathcal {W}}}^-| = \left( {\begin{array}{c}|W| - 1\\ B\end{array}}\right) \,. \end{aligned}$$

According to Lemma 1 (making $W\leftarrow W \setminus \hat{w}$), we can construct an injection $g: {{\mathcal {W}}^-} \rightarrow {{\mathcal {W}}}^+$ such that $g(W^-) \setminus W^-= \hat{w}$, and each $W^+ \in {{\mathcal {W}}}^+$ is mapped at most $\lceil \frac{|W| - B}{B} \rceil$ = $\lceil \frac{|W|}{B} \rceil - 1$ times.

Obviously we have $\forall W^-, ~~ f(W^-) \le f( g(W^-))$. Then, the probability that a worker allocation in ${{\mathcal {W}}}^-$ is selected by Algorithm 1 is

$$\begin{aligned} \begin{aligned}&Prob(\widehat{A} \in {{\mathcal {W}}}^-) \\ =&\frac{\sum \limits _{W^-\in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)}}{\sum \limits _{W^+\in { {{\mathcal {W}}}^+}} e^{\epsilon f(W^+)} + \sum \limits _{W^-\in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)}}\\ \le&\frac{\sum \limits _{W^- \in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)}}{ \frac{1}{\lceil \frac{|W|}{B} \rceil - 1}\sum \limits _{W^- \in {{\mathcal {W}}}^-} e^{\epsilon f(g(W^-)) } + \sum \limits _{W^-\in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)} } \\ \le&\frac{\sum \limits _{W^- \in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)}}{\frac{1}{\lceil \frac{|W|}{B} \rceil - 1 } \sum \limits _{W^- \in {{\mathcal {W}}}^-} e^{\epsilon f(W^-) } + \sum \limits _{W^-\in {{\mathcal {W}}}^-} e^{\epsilon f(W^-)} } \\ =&1 - \frac{1}{\lceil \frac{|W|}{B} \rceil } \,. \end{aligned} \end{aligned}$$

The probability that $\widehat{A}$ excludes $\hat{w}$ is at most $1 - \frac{1}{\lceil \frac{|W|}{B} \rceil }$, conditional on which we can also infer that $\widehat{A}$ excludes the worker with second largest reliability is $1 - \frac{1}{\lceil \frac{|W|- 1}{B - 1} \rceil }$ using the same analysis as above by replacing W and B with $W \setminus \hat{w}$ and $B-1$, respectively. In fact, let $\hat{w}_k$ denote the worker with the k-th largest reliability for $1 \le k \le B$, we have

$$\begin{aligned} Prob(\hat{w}_{k}\notin \widehat{A} |\hat{w}_1...\hat{w}_{k-1} \notin \widehat{A} ) \le 1 - \frac{1}{\lceil \frac{|W| - (k - 1)}{B - (k - 1)} \rceil } \,. \end{aligned}$$

Therefore, $Prob(\hat{w}_1...\hat{w}_{b} \notin \widehat{A} ) \le \prod \limits _{ k \in \{0...b-1\} }( 1 - \frac{1}{\lceil \frac{|W| - k}{B - k } \rceil } )$.

Case $\mathbf {B > \frac{W+1}{2}}$. Let $B' = |W| - B$. The algorithm opts to select the top-$B'$ workers, with the reversed reliabilities. Considering the original top reliable worker $\hat{w}$, w.r.t. the regulated reliabilities, it now has the smallest $r'$. Similarly denoting the worker subsets including and excluding $\hat{w}$ as ${{\mathcal {W}}}^+$ and ${{\mathcal {W}}}^-$, respectively, then missing $\hat{w}$ in the final allocation $W \setminus \widehat{A}$ is equivalent to selecting $\hat{w}$ in $\widehat{A}$. We have

$$\begin{aligned} \frac{|{{\mathcal {W}}}^+|}{|{{\mathcal {W}}}^-|} = \frac{B'}{|W| - B'} = \frac{|W| - B}{B} \,. \end{aligned}$$

Similarly, we can construct an injection $g: {{\mathcal {W}}^+} \rightarrow {{\mathcal {W}}}^-$ such that $W^+ \setminus g(W^+) = \hat{w}$, with each $W^- \in {{\mathcal {W}}}^-$ mapped at most $\lceil \frac{|W| - B}{B} \rceil$ times. In addition, since $\hat{w}$ has the smallest $r'$, we have $\forall W^+, ~~ f'(W^+) \le f'( g(W^+))$. Then we can deduct the bound for $Prob(\widehat{A} \in {{\mathcal {W}}}^+)$ in the same way as deducting $Prob(\widehat{A} \in {{\mathcal {W}}}^-)$ for the case $B \le \frac{W+1}{2}$, with $f(\cdot )$ replaced by $f'(\cdot )$. The subsequent deductions for $Prob(\hat{w}_1...\hat{w}_{b} \in \widehat{A} )$ follow the previous case as well. $\square$