Adaptive Discriminative Regularization for Visual Classification

Abstract

How to improve discriminative feature learning is central in classification. Existing works address this problem by explicitly increasing inter-class separability and intra-class compactness by constructing positive and negative pairs for contrastive learning or posing tighter class separating margins. These methods do not exploit the similarity between different classes as they adhere to independent identical distributions assumption in data. In this paper, we embrace the real-world data distribution setting in that some classes share semantic overlaps due to their similar appearances or concepts. Regarding this hypothesis, we propose a novel regularization to improve discriminative learning. We first calibrate the estimated highest likelihood of one sample based on its semantically neighboring classes, then encourage the overall likelihood predictions to be deterministic by imposing an adaptive exponential penalty. As the gradient of the proposed method is roughly proportional to the uncertainty of the predicted likelihoods, we name it adaptive discriminative regularization (ADR), trained along with a standard cross entropy loss in classification. Extensive experiments demonstrate that it can yield consistent and non-trivial performance improvements in a variety of visual classification tasks (over 10 benchmarks). Furthermore, we find it is robust to long-tailed and noisy label data distribution. Its flexible design enables its compatibility with mainstream classification architectures and losses.

Appendix

Our adaptive discriminative regularization loss for one sample can be written as

$$\begin{aligned} \begin{aligned} \mathcal{L}_d({{\tilde{y}}}_i)&= \prod _{j=1}^{\tau } \frac{1}{\sqrt{2\pi \varphi _{i}}} \text {exp} \left\{ -\frac{1}{2\varphi _{i}}{{\hat{y}}}^2_{ij}\right\} . \\&= \prod _{j=1}^{\tau } \mathcal{F}_j({{\hat{y}}}_{ij}), \\ \end{aligned} \end{aligned}$$

(10)

where $\varphi _{i}$ is a function of ${{\tilde{y}}}_{i}$, ${{\hat{y}}}_{i}$ is generated by a non-linear function $\text {TopK}({\tilde{y}}_{i})$. The function $\mathcal{F}_j({{\hat{y}}}_{ij})$ in Eq. (10) can be denoted as

$$\begin{aligned} \begin{aligned} \mathcal{F}_j({{\hat{y}}}_{ij})&= \mathcal{H}({{\hat{y}}}_{ij})\mathcal{E}({{\hat{y}}}_{ij}), \end{aligned} \end{aligned}$$

(11)

where $\mathcal{H}({{\hat{y}}}_{ij})$ is called the base measure function “$\frac{1}{\sqrt{2\pi \varphi _{i}}}$", $\mathcal{E}({{\hat{y}}}_{ij})$ is named the exponential term “$\text {exp}\{-\frac{1}{2\varphi _{i}}{{\hat{y}}}^2_{ij}\}$".

In the backward propagation, $\frac{\partial \mathcal{L}_d({\tilde{y}}_i)}{\partial {{\tilde{y}}}_{i}}$ can be calculated with

$$\begin{aligned} \begin{aligned} \frac{\partial \mathcal{L}_d({{\tilde{y}}}_i)}{\partial {{\tilde{y}}}_{i}}&= \sum _{j=1}^{\tau } \left[ \mathcal{F}'_j({{\hat{y}}}_{ij}) \prod _{m\ne j}^{\tau } \mathcal{F}_m({{\hat{y}}}_{ij}) \right] , \end{aligned} \end{aligned}$$

(12)

The derivative function $\mathcal{F}'_j({{\hat{y}}}_{ij})$ in Eq. (12) can be computed with

$$\begin{aligned} \begin{aligned} \mathcal{F}'_{j}({{\hat{y}}}_{ij})&= \mathcal{H}'({{\hat{y}}}_{ij})\mathcal{E}({{\hat{y}}}_{ij}) + \mathcal{E}'({{\hat{y}}}_{ij})\mathcal{H}({{\hat{y}}}_{ij}), \end{aligned} \end{aligned}$$

(13)

In Eq. 13, $\mathcal{H}'({{\hat{y}}}_{ij})$ and $\mathcal{E}'({{\hat{y}}}_{ij})$ can be calculated by

$$\begin{aligned} \begin{aligned} \frac{\partial \mathcal{H}({{\hat{y}}}_{ij})}{\partial {{\hat{y}}}_{ij}}&= \frac{1}{\sqrt{2\pi }}\left( -\frac{1}{2}\varphi _{i}^{-\frac{3}{2}}\right) \varphi _{ij}' \\&= -\frac{\varphi _{ij}'}{2\varphi _{i}} \mathcal{H}({{\hat{y}}}_{ij}),\\ \frac{\partial \mathcal{E}({{\hat{y}}}_{ij})}{\partial {{\hat{y}}}_{ij}}&= \left[ \frac{-{{\hat{y}}}_{ij}^2}{2\varphi _{i}} \right] '\mathcal{E}({{\hat{y}}}_{ij}) \\&= \left[ \frac{{{\hat{y}}}_{ij}^2\varphi _{ij}'-2{{\hat{y}}}_{ij}\varphi _{i}}{2\varphi _{i}^2}\right] \mathcal{E}({{\hat{y}}}_{ij}). \end{aligned} \end{aligned}$$

(14)

Putting Eq. 14 into Eq. (13), $\mathcal{F}'_{j}({{\hat{y}}}_{ij})$ can be rewritten as

$$\begin{aligned} \begin{aligned} \mathcal{F}'_{j}({{\hat{y}}}_{ij})&= \left[ \frac{{{\hat{y}}}_{ij}^2\varphi _{ij}'-2{{\hat{y}}}_{ij}\varphi _{i}}{2\varphi _{i}^2} - \frac{\varphi _{ij}'}{2\varphi _{i}} \right] \mathcal{F}_{j}({{\hat{y}}}_{ij}) \\&= \left[ \frac{{{\hat{y}}}_{ij}^2\varphi _{ij}'-2{{\hat{y}}}_{ij}\varphi _{i} - \varphi _{i}\varphi _{ij}'}{2\varphi _{i}^2} \right] \mathcal{F}_{j}({{\hat{y}}}_{ij}), \end{aligned} \end{aligned}$$

(15)

Then, putting Eq. (15) into Eq. (12), $\frac{\partial \mathcal{L}_d({{\tilde{y}}}_i)}{\partial {{\tilde{y}}}_{i}}$ can be rewritten as

$$\begin{aligned} \begin{aligned} \frac{\partial \mathcal{L}_d({{\tilde{y}}}_i)}{\partial {{\tilde{y}}}_{i}}&= \sum _{j=1}^{\tau } \left[ \mathcal{F}'_j({{\hat{y}}}_{ij}) \prod _{m\ne j}^{\tau } \mathcal{F}_m({{\hat{y}}}_{ij}) \right] \\&= \sum _{j=1}^{\tau } \left[ \left( \frac{{{\hat{y}}}_{ij}^2\varphi _{ij}'-2{{\hat{y}}}_{ij}\varphi _{i} - \varphi _{i}\varphi _{ij}'}{2\varphi _{i}^2} \right) \prod _{j=1}^{\tau }\mathcal{F}_m({{\hat{y}}}_{ij}) \right] \\&= \sum _{j=1}^{\tau } \left[ \left( \frac{{{\hat{y}}}_{ij}^2\varphi _{ij}'-2{{\hat{y}}}_{ij}\varphi _{i} - \varphi _{i}\varphi _{ij}'}{2\varphi _{i}^2} \right) \mathcal{L}_d({{\tilde{y}}}_i)\right] , \end{aligned}\nonumber \\ \end{aligned}$$

(16)

where $\varphi _{i}'$ is the partial derivative function $\varphi _{i}$ with respect to ${{\hat{y}}}_{ij}$. We refer to Sen et al. (2005) and Guariglia (2021), $\varphi _{i}'$ can be computed with

$$\begin{aligned} \begin{aligned} \frac{\partial \varphi _{i}}{\partial {{\hat{y}}}_{ij}}&= - \frac{\varphi _{i}+\text {log}({{\hat{y}}}_{ij})}{1-{{\hat{y}}}_{ij}}. \end{aligned} \end{aligned}$$

(17)

We also give the derivative function of entropy $\mathcal{L}'_e(p)$ for binary classification. It can be calculated by

$$\begin{aligned} \begin{aligned} \frac{\partial \mathcal{L}_e(p)}{\partial p}&= -\left[ log(p)-log(1-p)\right] \\&= log\left( \frac{1-p}{p}\right) . \end{aligned} \end{aligned}$$

(18)