SEEDS: Superpixels Extracted Via Energy-Driven Sampling

Abstract

Superpixel algorithms aim to over-segment the image by grouping pixels that belong to the same object. Many state-of-the-art superpixel algorithms rely on minimizing objective functions to enforce color homogeneity. The optimization is accomplished by sophisticated methods that progressively build the superpixels, typically by adding cuts or growing superpixels. As a result, they are computationally too expensive for real-time applications. We introduce a new approach based on a simple hill-climbing optimization. Starting from an initial superpixel partitioning, it continuously refines the superpixels by modifying the boundaries. We define a robust and fast to evaluate energy function, based on enforcing color similarity between the boundaries and the superpixel color histogram. In a series of experiments, we show that we achieve an excellent compromise between accuracy and efficiency. We are able to achieve a performance comparable to the state-of-the-art, but in real-time on a single Intel i7 CPU at 2.8 GHz.

Appendix: Evaluating Pixel-Level and Block-Level Movements

In this section we give more detail of both observations used to speed up the evaluation of the pixel-level and block-level movements.

1.1 Color Distribution Term

Recall that ${\mathcal {A}}^l_k$ is the set of pixels that are candidates to be moved from the superpixel ${\mathcal {A}}_k$ to ${\mathcal {A}}_n$.

Observation 1

Let the sizes of ${\mathcal {A}}_k$ and ${\mathcal {A}}_n$ be similar, and ${\mathcal {A}}^l_k$ much smaller, i.e. $|{\mathcal {A}}_k|\approx |{\mathcal {A}}_n| \gg |{\mathcal {A}}_k^l|$. If the histogram of ${\mathcal {A}}^l_k$ is concentrated in a single bin, then

$$\begin{aligned} \mathbf{int}(c_{{\mathcal {A}}_n},c_{{\mathcal {A}}_k^l}) \ge \mathbf{int}(c_{{\mathcal {A}}_k \backslash {\mathcal {A}}_k^l}, c_{{\mathcal {A}}_k^l}) \Longleftrightarrow H(s) \ge H(s_t). \end{aligned}$$

(18)

Recall that the color term of the energy function is:

$$\begin{aligned} H(s) = \sum _k \sum _{\{\mathcal {H}_j\}} \left( \frac{1}{|{\mathcal {A}}_k|} \sum _{i \in {\mathcal {A}}_k} \delta (I(i) \in \mathcal {H}_j) \right) ^2, \end{aligned}$$

(19)

in which we simply merged Eqs. (6) and (7). We write $H(s) \ge H(s_t)$ taking into account that $s$ and $s_t$ only differ in ${\mathcal {A}}^l_k$, and the assumption of the Observation on the size of the superpixels, i.e. $|{\mathcal {A}}_k|\approx |{\mathcal {A}}_n| \gg |{\mathcal {A}}_k^l|$. Thus, the expression does not take into account the color at superpixels different from $k$ and $n$, and we can get rid of the normalization of the histograms due to the assumption. Then, the evaluation becomes,

$$\begin{aligned}&H(s) \ge H(s_t) \Longleftrightarrow \nonumber \\&\quad \sum _{\{\mathcal {H}_j\}} \left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}_j) + \sum _{i \in {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}_j) \right) ^2 + \nonumber \\&\quad + \sum _{\{\mathcal {H}_j\}} \left( \sum _{i \in {\mathcal {A}}_k\backslash {\mathcal {A}}_k^l} \delta (I(i)\in \mathcal {H}_j) \right) ^2 \ge \nonumber \\&\quad \ge \sum _{\{\mathcal {H}_j\}} \left( \sum _{i \in {\mathcal {A}}_k\backslash {\mathcal {A}}_k^l} \delta (I(i)\in \mathcal {H}_j) \!+\! \sum _{i \in {\mathcal {A}}_k^l} \delta (I(i)\in \mathcal {H}_j )\right) ^2 \!+\! \nonumber \\&\quad +\sum _{\{\mathcal {H}_j\}} \left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}_j )\right) ^2 . \end{aligned}$$

(20)

The second assumption of the Observation is that ${\mathcal {A}}^l_k$ is concentrated in a single bin. Let $\mathcal {H}^*$ be the color in which $A^l_k$ is concentrated. Then, the evaluation in Eq. (20) becomes

$$\begin{aligned}&\left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}^\star ) + \sum _{i \in {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}^\star ) \right) ^2 +\nonumber \\&\quad +\sum _{\{\mathcal {H}_j\} \backslash \mathcal {H}^\star } \left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}_j) \right) ^2 +\nonumber \\&\quad +\sum _{\{\mathcal {H}_j\} } \left( \sum _{i \in {\mathcal {A}}_k \backslash {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}_j) \right) ^2 \ge \nonumber \\&\ge \left( \sum _{i \in {\mathcal {A}}_k \backslash {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}^\star ) + \sum _{i \in {\mathcal {A}}_k^l} \delta (I(i)\in \mathcal {H}^\star )\right) ^2 +\nonumber \\&\quad + \sum _{\{\mathcal {H}_j\} \backslash \mathcal {H}^\star } \left( \sum _{i \in {\mathcal {A}}_k \backslash {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}_j) \right) ^2+ \nonumber \\&\quad + \sum _{\{\mathcal {H}_j\}} \left( \sum _{i\in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}_j)\right) ^2. \end{aligned}$$

(21)

Then, note the following simple equality:

$$\begin{aligned}&\left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}^\star ) + \sum _{i \in {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}^\star )\right) ^2 =\end{aligned}$$

(22)

$$\begin{aligned}&\left( \sum _{i \in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}^\star )\right) ^2 + \left( \sum _{i\in {\mathcal {A}}_k^l} \delta (I(i)\in \mathcal {H}^\star )\right) ^2 + \nonumber \\&\quad +2 \left( \sum _{i\in {\mathcal {A}}_n} \delta (I(i) \in \mathcal {H}^\star ) \right) \left( \sum _{i\in {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}^\star )\right) , \end{aligned}$$

(23)

and we introduce it to the evaluation in Eq. (21). Reordering the terms, and canceling the same terms in both sides of the inequality, Eq. (21) becomes:

$$\begin{aligned}&H(s) \ge H(s_t) \Longleftrightarrow \end{aligned}$$

(24)

$$\begin{aligned}&\sum _{i\in {\mathcal {A}}_n}\delta (I(i)\in \mathcal {H}^\star ) \ge \sum _{i\in {\mathcal {A}}_k\backslash {\mathcal {A}}_k^l}\delta (I(i)\in \mathcal {H}^\star ). \end{aligned}$$

(25)

Now, we develop the intersection distances in the Observation to arrive to Eq. (24). We use the following expression:

$$\begin{aligned}&\mathbf{int}(c_{{\mathcal {A}}_n},c_{{\mathcal {A}}_k^l}) =\nonumber \\&\quad \sum _{\{\mathcal {H}_j\}} \min \left( \frac{1}{|{\mathcal {A}}_n|} \sum _{i\in {\mathcal {A}}_n}\delta (I(i)\in \mathcal {H}_j),\frac{1}{|{\mathcal {A}}_k^l|} \sum _{i\in {\mathcal {A}}_k^l} \delta (I(i) \in \mathcal {H}_j) \right) , \end{aligned}$$

(26)

and since we assumed that the histogram of ${\mathcal {A}}_k^l$ is concentrated in one bin, the expression becomes

$$\begin{aligned} \mathbf{int}(c_{{\mathcal {A}}_n},c_{{\mathcal {A}}_k^l}) = \frac{1}{|{\mathcal {A}}_n|}\sum _{i\in {\mathcal {A}}_n}\delta (I(i)\in \mathcal {H}^\star ). \end{aligned}$$

(27)

Finally, we use this expression and the assumption of $|{\mathcal {A}}_k|\approx |{\mathcal {A}}_n|$,and we obtain Eq. (24):

$$\begin{aligned}&\mathbf{int}(c_{{\mathcal {A}}_n},c_{{\mathcal {A}}_k^l}) \ge \mathbf{int}(c_{{\mathcal {A}}_k\backslash {\mathcal {A}}_k^l},c_{{\mathcal {A}}_k^l}) \Longleftrightarrow \end{aligned}$$

(28)

$$\begin{aligned}&\sum _{i\in {\mathcal {A}}_n} \delta (I(i)\in \mathcal {H}^\star ) \ge \sum _{i \in {\mathcal {A}}_k\backslash {\mathcal {A}}_k^l}\delta (I(i)\in \mathcal {H}^\star ) \Longleftrightarrow \end{aligned}$$

(29)

$$\begin{aligned}&H(s)\ge H(s_t) \end{aligned}$$

(30)

1.2 Boundary Prior Term

Observation 2

Let $\{b_{\mathcal {N}_i}(k)\}$ be the histograms of the superpixel labeling computed at the partitioning $s_t$ [(see Eq. (8)]. ${\mathcal {A}}_k^l$ is a pixel, and $\mathcal {K}_{{\mathcal {A}}_k^l}$ the set of pixels whose patch intersects with that pixel, i.e. $\mathcal {K}_{{\mathcal {A}}_k^l} = \{i : {\mathcal {A}}_k^l \in \mathcal {N}_i \}$. If the hill-climbing proposes moving a pixel ${\mathcal {A}}_k^l$ from superpixel $k$ to superpixel $n$, then

$$\begin{aligned} \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}}(b_{\mathcal {N}_i}(n)+1) \ge \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} b_{\mathcal {N}_i}(k) \Longleftrightarrow G(s) \ge G(s_t). \end{aligned}$$

(31)

Recall that $G(s)$ is:

$$\begin{aligned} G(s)=\sum _i \sum _k \left( \frac{1}{Z} \sum _{j \in \mathcal {N}_i} \delta (j \in {\mathcal {A}}_k)\right) ^2, \end{aligned}$$

(32)

where we merged Eqs. (8) and (9). We write $G(s)\ge G(s_t)$ taking into account that $s$ and $s_t$ only differ in ${\mathcal {A}}_k^l$, which is a single pixel, and it becomes

$$\begin{aligned}&G(s) \ge G(s_t) \Longleftrightarrow \nonumber \\&\sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} \left( \left( \frac{1}{Z}((-1)+\sum _{j\in \mathcal {N}_i}\delta (j \in {\mathcal {A}}_k)) \right) ^2\right. + \nonumber \\&\quad +\left. \left( \frac{1}{Z}(1+\sum _{j\in \mathcal {N}_i}\delta (j\in {\mathcal {A}}_n))\!\right) ^2\right) \ge \nonumber \\&\sum _{i \in \mathcal {K}_{{\mathcal {A}}_k^l}} \left( \!\left( \frac{1}{Z} \sum _{j\in \mathcal {N}_i} \delta (j\in {\mathcal {A}}_k) \!\right) ^2 \!\!+\! \left( \frac{1}{Z} \sum _{j\in \mathcal {N}_i}\delta (j\in {\mathcal {A}}_n)\right) ^2 \right) . \end{aligned}$$

(33)

Then, we develop the squares, and cancel the repeated terms in the inequality as well as $Z$:

$$\begin{aligned}&G(s) \ge G(s_t) \Longleftrightarrow \nonumber \\&\sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} \left( 1-2 \sum _{j\in \mathcal {N}_i}\delta (j\in {\mathcal {A}}_k) \right) + \nonumber \\&\quad +\left( 1+2\sum _{j\in \mathcal {N}_i}\delta (j\in {\mathcal {A}}_n)\right) \ge 0. \end{aligned}$$

(34)

Finally, we reorder the terms and obtain the inequality in the Observation:

$$\begin{aligned}&G(s) \ge G(s_t) \Longleftrightarrow \nonumber \\&\quad \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} \left( \!1\!+\!\sum _{j\in \mathcal {N}_i}\delta (j\in {\mathcal {A}}_n)\right) \!\ge \! \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}}\! \left( \sum _{j\in \mathcal {N}_i} \delta (j\in {\mathcal {A}}_k) \right) \nonumber \\&\quad \Longleftrightarrow \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} (b_{\mathcal {N}_i}(n)+1) \ge \sum _{i\in \mathcal {K}_{{\mathcal {A}}_k^l}} b_{\mathcal {N}_i}(k). \end{aligned}$$

(35)