Abstract
This paper presents Contrastive Transformer (CT), a contrastive learning scheme using the innate transformer patches. CT enables existing contrastive learning techniques, often used for image classification, to benefit dense downstream prediction tasks such as semantic segmentation. The scheme performs supervised patch-level contrastive learning, selecting the patches based on the ground truth mask, subsequently used for hard-negative and hard-positive sampling. The scheme applies to all patch-based vision-transformer architectures, is easy to implement, and introduces minimal additional memory footprint. Additionally, the scheme removes the need for huge batch sizes, as each patch is treated as an image.
We apply and test CT for the case of aerial image segmentation, known for low-resolution data, large class imbalance, and similar semantic classes. We perform extensive experiments to show the efficacy of the CT scheme on the ISPRS Potsdam aerial image segmentation dataset. Additionally, we show the generalizability of our scheme by applying it to multiple inherently different transformer architectures. Ultimately, the results show a consistent increase in mean Intersection-over-Union (IoU) across all classes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alonso, I., Sabater, A., Ferstl, D., Montesano, L., Murillo, A.C.: Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank (2021)
Audebert, N., Le Saux, B., Lefèvre, S.: Beyond RGB: very high resolution urban remote sensing with multimodal deep networks. ISPRS J. Photogramm. Remote. Sens. 140, 20–32 (2018)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. ArXiv (2020)
Dao, S.D., Zhao, E., Phung, D., Cai, J.: Multi-label image classification with contrastive learning. ArXiv (2021)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: Proceedings of the 9th International Conference on Learning Representations (ICLR), pp. 1–21 (2021)
Duman Keles, F., Mahesakya Wijewardena, P., Hegde, C., Agrawal, S., Orabona, F.: On the computational complexity of self-attention (2023)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning (2020)
Huang, L., et al.: A two-stage contrastive learning framework for imbalanced aerial scene recognition. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing May 2022, pp. 3518–3522 (2022)
isprs.org. ISPRS Potsdam Dataset
Khoshboresh-Masouleh, M., Alidoost, F., Arefi, H.: Multiscale building segmentation based on deep learning for remote sensing RGB images from different sensors. J. Appl. Remote Sens. 14(03), 1 (2020)
Li, K., et al.: UniFormer: unifying convolution and self-attention for visual recognition (2022)
Li, Q., et al.: Instance segmentation of buildings using keypoints. In: IGARSS 2020–2020 IEEE International Geoscience and Remote Sensing Symposium, pp. 1452–1455 (2020)
Li, W., Zhao, W., Zhong, H., He, C., Lin, D.: Joint semantic-geometric learning for polygonal building segmentation. In: 35th AAAI Conference on Artificial Intelligence, AAAI 2021, vol. 3A, pp. 1958–1965 (2021)
Liu, Q., Kampffmeyer, M., Jenssen, R., Salberg, A.B.: Dense dilated convolutions merging network for land cover classification. IEEE Trans. Geosci. Remote Sens. 58(9), 6309–6320 (2020)
Liu, S., Zhi, S., Johns, E., Davison, A.J.: Bootstrapping semantic segmentation with regional contrast (2021)
Liu, Y., Fan, B., Wang, L., Bai, J., Xiang, S., Pan, C.: Semantic labeling in very high resolution images via a self-cascaded convolutional neural network. ISPRS J. Photogramm. Remote Sens. 145, 78–95 (2018)
Liu, Z., et al.: Swin transformer V2: scaling up capacity and resolution (2022)
Matei, B.C., Sawhney, H.S., Samarasekera, S., Kim, J., Kumar, R.: Building segmentation for densely built urban regions using aerial LIDAR data. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition (2008)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shi, Y., Li, Q., Zhu, X.X.: Building segmentation through a gated graph convolutional neural network with deep structured feature embedding. ISPRS J. Photogramm. Remote. Sens. 159, 184–197 (2019)
Tang, M., Georgiou, K., Qi, H., Champion, C., Bosch, M.: Semantic segmentation in aerial imagery using multi-level contrastive learning with local consistency. In: Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023, pp. 3787–3796 (2023)
Oord, A.V.D., Li, Y., Vinyals, O.: DeepMind representation learning with contrastive predictive coding (2018)
Wang, L., Li, R., Duan, C., Zhang, C., Meng, X., Fang, S.: A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images. IEEE Geosci. Remote Sens. Lett. 19 (2022)
Wang, L., Li, R., Zhang, C., Fang, S., Duan, C., Meng, X., Atkinson, P.M.: UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery. ISPRS J. Photogramm. Remote. Sens. 190, 196–214 (2022)
Wang, W., Zhou, T., Yu, F., Dai, J., Konukoglu, E., Van Gool, L.: Exploring cross-image pixel contrast for semantic segmentation (2021)
Xia, Z., Pan, X., Song, S., Li, L.E., Huang, G.: Vision transformer with deformable attention (2022)
Yu, W., et al.: MetaFormer is actually what you need for vision (2022)
Zhang, F., Torr, P., Ranftl, R., Richter, S.R.: Looking beyond single images for contrastive semantic segmentation learning. In: Advances in Neural Information Processing Systems, vol. 34, pp. 3285–3297 (2021)
Zhao, X., et al.: Contrastive learning for label efficient semantic segmentation (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jyhne, S.R., Andersen, PA., Goodwin, M., Oveland, I. (2023). A Contrastive Learning Scheme with Transformer Innate Patches. In: Bramer, M., Stahl, F. (eds) Artificial Intelligence XL. SGAI 2023. Lecture Notes in Computer Science(), vol 14381. Springer, Cham. https://doi.org/10.1007/978-3-031-47994-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-47994-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47993-9
Online ISBN: 978-3-031-47994-6
eBook Packages: Computer ScienceComputer Science (R0)