PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

Abstract

Tremendous breakthroughs have been developed in Semi-Supervised Semantic Segmentation (S4) through contrastive learning. However, due to limited annotations, the guidance on unlabeled images is generated by the model itself, which inevitably exists noise and disturbs the unsupervised training process. To address this issue, we propose a robust contrastive-based S4 framework, termed the Probabilistic Representation Contrastive Learning (PRCL) framework to enhance the robustness of the unsupervised training process. We model the pixel-wise representation as Probabilistic Representations (PR) via multivariate Gaussian distribution and tune the contribution of the ambiguous representations to tolerate the risk of inaccurate guidance in contrastive learning. Furthermore, we introduce Global Distribution Prototypes (GDP) by gathering all PRs throughout the whole training process. Since the GDP contains the information of all representations with the same class, it is robust from the instant noise in representations and bears the intra-class variance of representations. In addition, we generate Virtual Negatives (VNs) based on GDP to involve the contrastive learning process. Extensive experiments on two public benchmarks demonstrate the superiority of our PRCL framework.

Proof of Equation 4

Generally, we regard the prototype as the posterior distribution after the $n\textrm{th}$ observations of representations $\{z_1, z_2, \dots , z_n\}$. Meanwhile, we assume that all the observations are conditionally independent, the distribution prototype can be derived as $p(\varvec{\rho }|\varvec{z}_1, \varvec{z}_2,..., \varvec{z}_{n})$. Without loss of generality, we only consider a one-dimensional case here. It is easy to extend the proof to all dimensions since each dimension of the feature is supposed to be independent. We assume that the distribution prototype $p(\varvec{\rho }|\varvec{z}_1, \varvec{z}_2,..., \varvec{z}_{n})$ is with $\hat{\mu }_n$ and $\hat{\sigma }^2_n$ as mean and variance, respectively. Now we need to add a new representation as the observation to obtain a new prototype $p(\varvec{\rho }|\varvec{z}_1, \varvec{z}_2,..., \varvec{z}_{n+1})$, if we take log on this prototype, we have:

(17)

where "const" means the constant which is irrelevant to the prototype $\varvec{\rho }$ and

(18)

(19)

$\varvec{\sigma }_0$ is the $\varvec{\sigma }$ of the first representation. At the beginning of the training process, the representation is unreasonable, so we consider that the reliability is quite low and $\varvec{\sigma }_0\rightarrow \infty$ (corresponds to an extremely large value in experiments). We have

(20)

(21)

Above is the process of obtaining $\rho _{n+1}$ through $\rho _{n}$. Next, we briefly give the solution of obtaining $\rho _{n+1}$ using $n+1$ representations.

(22)

where $\alpha =\frac{\prod _{i=1}^n p(\varvec{\rho }_i)}{p(\varvec{z}_1, \varvec{z}_2,..., \varvec{z}_{n})}$ and

(23)

(24)

Because $\varvec{\sigma }_0\rightarrow \infty$, we have

(25)

(26)