Learning domain invariant representations of heterogeneous image data

Abstract

Supervised deep learning requires a huge amount of reference data, which is often difficult and expensive to obtain. Domain adaptation helps with this problem—labelled data from one dataset should help in learning on another unlabelled or scarcely labelled dataset. In remote sensing, where a variety of sensors produce images of different modalities and with different numbers of channels, it would be very beneficial to develop heterogeneous domain adaptation methods that are able to work between domains that come from different input spaces. However, this challenging problem is rarely addressed, the majority of existing heterogeneous domain adaptation work does not use raw image-data, or they rely on translation from one domain to the other, therefore ignoring domain-invariant feature extraction approaches. This article proposes novel approaches for heterogeneous image domain adaptation for both the semi-supervised and unsupervised settings. These are based on extracting domain invariant features using deep adversarial learning. For the unsupervised domain adaptation case, the impact of pseudo-labelling is also investigated. We evaluate on two heterogeneous remote sensing datasets, one being RGB, and the other multispectral, for the task of land-cover patch classification, and also on a standard computer vision benchmark of RGB-depth map object classification. The results show that the proposed domain invariant approach consistently outperforms the competing methods based on image-to-image/feature translation, in both remote sensing and in a standard computer vision problem.

Appendices

Appendix A: Implementation details

1.1 A.1 Remote sensing

Unlike WDGRL, whose components are fully connected neural networks, SS-HIDA and UPL-HIDA are convolutional architectures (see Fig. 3 for details). The feature extractor for RESISC45 consists of two convolutional layers with 16 and 32 filters respectively. Each conv. layer is followed by $4\times 4$ max-pooling. The feature extractor for EuroSAT is the same, except that it has $2\times 2$ max-pooling after every conv. layer. The shared invariant feature extractor has two convolutional layers with 32 and 64 filters respectively, and one fully-connected (FC) layer of 100 nodes. All of the kernels have size $5\times 5$. The class discriminator has one FC layer with softmax activation. The domain critic (DC) is identical to that in WDGRL—it has an FC layer with 100 nodes followed by an FC layer with 1 node.

In each training step, the DC is trained for 10 iterations with a learning rate of $10^{-3}$, the DC is then frozen and the rest of the model is trained for 1 iteration with a learning rate of $10^{-4}$. The DC loss’ weight $\lambda$ is 0.1. The Adam optimiser is used.

The input data is standardised per channel so that each channel has mean 0 and standard deviation 1. The following augmentation transformations are used: flipping with a probability of 0.45, rotation with a probability of 0.75 for $90^{\circ }$, $180^{\circ }$, or $270^{\circ }$, changing contrast with the probability of 0.33 by multiplying the values of the pixels with the coefficient ranging between 0.5 and 1.5, changing brightness with the probability of 0.33 by adding the coefficient ranging between $-0.3$ and 0.3 scaled by the mean of pixel values per channel before standardisation, blurring with the probability of 0.33 with Gaussian filter with $\sigma$ parameter values ranging from 1.5 to 1.8, and finally adding Gaussian noise with mean 0 and standard deviation between 10 and 15 with the probability of 0.33. The batch size is 32, and in each iteration, half of the training batch (16) comes from the source, and the other half from the target domain. The model is trained for 40 epochs.

1.2 A.2 RGB-depth

At the first phase of training, the convolutional part of the network is frozen, and only the FC layers are trained with the learning rate $10^{-4}$. Then the whole network is fine-tuned with a smaller learning rate of $10^{-5}$. We present the results of two variants of our unsupervised method:

• UPL-HIDA—uses the CycleGAN (ResNet generator) to obtain pseudo-labels (trained for 50 epochs) with a source classifer initialised with a VGG-16 backbone (FC layers trained for 40 epochs, then finetuned for 40 additional epochs), the most confident 10 images per class (where available) are used for pseudo-labelling; UPL-HIDA (VGG-16 backbone) FC layers are trained for 1 epoch, then the whole network is finetuned for 5 epochs.

• U-HIDA—(VGG-16 backbone) FC layers trained for 40 epochs, then finetuned for 30 epochs.

The domain critic has a similar architecture to ADDA’s domain classifier: a fully connected NN having 3 layers (1024 nodes, 2048 nodes, and 1 node). The rest of the training process is as described in the RS experiments.

The data is preprocessed as is required for the pretrained VGG network—all the patches are resized to $224 \times 224$ pixels, and all the channels are zero-centered without scaling. Data augmentation is not used here to be fair in comparison to ADDA which also does not use it. The batch size used is 256, 128 samples per domain. Note that VGG can be easily replaced with any other architecture, e.g. ResNet.

Appendix B: Remote sensing unsupervised ablation study

In order to asses the impact of using different thresholds for pseudo-labelling, we show the results of UPL-HIDA on different amounts of pseudo-labelled target data, ranging from $100\%$ (whole dataset) to $1.25\%$ (the most confident 5 images per class are pseudo-labelled). The results without using any pseudo-labels ($0\%$) are also included. The performance of CycleGAN for HDA is given as a horizontal line. The comparison is shown in Fig. 13. These results are obtained using the RGB version of EuroSAT. As CycleGAN for HDA does not translate very well between RGB and multispectral data, the potential of UPL-HIDA using pseudo-labels given by CycleGAN is better seen on RGB-only data. N.B. These results are not comparable to those above as the ablation study is performed on the validation set of the target domain (rather than the test set).

Fig. 13

Accuracy of the unsupervised pseudo-labelled solution with varying thresholds for choosing the most confident pseudo labels. Horizontal dashed line shows the performance of CycleGAN for HDA. The numbers are expressed in percentages of labelled images

When RESISC45 is source and EuroSAT is target domain, starting with $1.25\%$, the accuracy grows as more pseudo-labelled data is added, outperforming CycleGAN for HDA from $2.5\%$, and reaching its peak with a threshold of $12.5\%$. Afterwards, additional pseudo-labels become less reliable and harm performance, but the accuracy remains higher than that of CycleGAN for HDA.

The situation is not as clear when adapting in the opposite direction, when EuroSAT is source and RESISC45 is target. As stated before, the RESISC45 dataset is more difficult to solve than EuroSAT, so the quality of pseudo-labels given by CycleGAN is not very high to begin with. Regardless, the model without using any pseudo-labels ($0\%$) already outperforms CycleGAN for HDA, and remains higher in all the cases. As can be seen, better performances could have been obtained using threshold values different than $12.5\%$ (notably $2.5\%$), however optimising this parameter would require using a certain amount of additionally labelled target data, so it was not done for these experiments.