A Simple D ²-Sampling Based PTAS for k-Means and Other Clustering Problems

Abstract

Given a set of points $P \subset\mathbb{R}^{d}$, the k-means clustering problem is to find a set of k centers $C = \{ c_{1},\ldots,c_{k}\}, c_{i} \in\mathbb{R}^{d}$, such that the objective function ∑ _x∈P e(x,C)², where e(x,C) denotes the Euclidean distance between x and the closest center in C, is minimized. This is one of the most prominent objective functions that has been studied with respect to clustering.

D ²-sampling (Arthur and Vassilvitskii, Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA’07, pp. 1027–1035, SIAM, Philadelphia, 2007) is a simple non-uniform sampling technique for choosing points from a set of points. It works as follows: given a set of points $P \subset\mathbb{R}^{d}$, the first point is chosen uniformly at random from P. Subsequently, a point from P is chosen as the next sample with probability proportional to the square of the distance of this point to the nearest previously sampled point.

D ²-sampling has been shown to have nice properties with respect to the k-means clustering problem. Arthur and Vassilvitskii (Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA’07, pp. 1027–1035, SIAM, Philadelphia, 2007) show that k points chosen as centers from P using D ²-sampling give an O(logk) approximation in expectation. Ailon et al. (NIPS, pp. 10–18, 2009) and Aggarwal et al. (Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pp. 15–28, Springer, Berlin, 2009) extended results of Arthur and Vassilvitskii (Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA’07, pp. 1027–1035, SIAM, Philadelphia, 2007) to show that O(k) points chosen as centers using D ²-sampling give an O(1) approximation to the k-means objective function with high probability. In this paper, we further demonstrate the power of D ²-sampling by giving a simple randomized (1+ϵ)-approximation algorithm that uses the D ²-sampling in its core.

Appendix: A Sampling Lemma

Here is argue that the approximation analysis of Algorithm 1 in Sect. 4 can be carried over to Algorithm 2. The following sampling lemma will be useful for this.

Lemma 14

(Sampling Lemma)

Let M≤N be positive integers and 0<α≤1. Let P be a finite set of elements. Consider multisets of size N comprising of elements from P. Let $\mathcal{S}$ denote a set of such multisets. Let $\mathcal{D}$ be an arbitrary distribution on elements of $\mathcal{S}$ and sampling an element from this distribution is denoted by $S \leftarrow\mathcal{D}$. Consider a fixed ordering of elements of P that defines ordering of elements in multisets of $\mathcal{S}$ and a fixed ordering of subsets of size M of any set of size N so that $\forall i \in\{1, \ldots, \binom{N}{M}\}$, the ith subset of a multiset $S \in\mathcal {S}$ is clearly defined. Let $f: \mathcal{S} \times\{1, \ldots, \binom{N}{M}\} \rightarrow\{0, 1\}$ be any function denoting some relation between S and its ith subset. Let T be any subset of $\{1, \ldots, \binom{N}{M}\}$. Consider the following two randomized procedures:

If Pr[A outputs 1]≥α, then Pr[B outputs 1]≥1−e ^−α.

Proof

Let n=|T|. Let the probability of sampling S from $\mathcal{S}$ be denoted by $\mathcal{D}(S)$. Consider a complete bipartite graph where the nodes on the left are members of the set $\mathcal{S}$ and nodes on the right are element in T. We label an edge (S,i) red if f(S,i)=1 and black otherwise. Edge (S,i) is given a weight of $\frac{\mathcal{D}(S)}{n}$ so that the sum of weight of all edges in this complete bipartite graph is 1. We first observe that the sum of weight of red edges is at least α/n. This is indeed the case since $\sum_{S \in\mathcal{S}, \exists i\ f(S, i)=1} \mathcal{D}(S) = \mathbf{Pr}[A \mbox{ outputs } 1]\geq \alpha$. Now let w _i denote the sum of weight of red edges incident on the node i on the right. The probability that in iteration i, B does not output 1 is (1−n⋅w _i ). So, the probability that B outputs 0 is at most

$$\prod_{i=1}^{n} (1 - n\cdot w_i) \leq e^{-n \cdot\sum_{i=1}^{n} w_i} = e^{-n \cdot{\scriptsize\mbox{(sum of weight of red edges)}}} = e^{-n \cdot (\alpha/n)} = e^{-\alpha} $$

So, Pr[B outputs 1]≥1−e ^−α. □

The following argument shows that the analysis of Algorithm 1 extends to Algorithm 2. In each step of Algorithm 1, we sample S with D ² sampling and then iterate over all subsets of S of size M. Suppose the probability that there is a subset i such that the ith subset of S is good (in the sense that its centroid is close to an optimal center) is at least 3/4. We indeed show this in Sect. 4. On the other hand, in Algorithm 2, we iterate over all subsets i and in the iteration corresponding to subset i, we sample a multiset S with D ² sampling. We succeed in finding a good center if there is a subset i such that the ith subset is good for S sampled in iteration i. We note that if the success probability in the first randomized procedure is large, then the probability of success in the second procedure is also large. For this, consider the Sampling Lemma given above. The above two randomized procedures correspond to procedures A and B in Lemma 14. Using this lemma, we get that the success probability in any iteration of step 1(b) in Algorithm 2 is at least (1−e ^−3/4)≥1/2. So, the probability that Algorithm 2 succeeds in finding a set of k good centers is at least 1/2^k. Since we repeat 2^k times, we get that Algorithm 2 succeeds with high probability.