Skip to main content
SpringerLink
Log in

Open Set Recognition in Real World

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Open set recognition (OSR) constitutes a critical endeavor within the domain of computer vision, frequently deployed in applications, such as autonomous driving and medical imaging recognition. Existing OSR methodologies predominantly center on the acquisition of a profound association between image data and corresponding labels, facilitating the extraction of discriminative features instrumental for distinguishing novel categories. Nevertheless, real-world scenarios often introduce not only novel classes (referred to semantic shift) but also intricate environmental modifications that engender alterations in the distribution of established classes (termed as covariate shift). The latter phenomenon has the potential to undermine the robust correlation between images and labels established by conventional statistical correlation modeling approaches, consequently resulting in significant degradation of OSR performance. Causal correlation stands as the fundamental linkage between entities, routinely harnessed by humans to enhance their cognitive capacities for a more profound comprehension of the intricate world. With inspiration drawn from this perspective, our work herein introduces the causal inference-inspired open set recognition (CISOR) approach tailored for real-world OSR (RWOSR). CISOR represents the pioneering initiative to leverage the stability inherent in causal correlation to construct two pivotal modules: the covariate causal independence (CCI) module and the semantic causal uniqueness (SCU) module, both instrumental in addressing the RWOSR problem. The CCI module adeptly confronts the challenge of covariate shift by imposing constraints on the correlations between inter-class causal features. This strategy effectively mitigates the impact of spurious correlations between distinct categories on the generalization capacity of discriminative features. Furthermore, in order to counteract the issue of semantic shift, the SCU module harnesses correlations between causal features within the same class as constraints, thereby facilitating the extraction of resilient causal features endowed with superior discriminative capabilities. Empirical findings substantiate the superior efficacy of the proposed CIOSR method when compared to state-of-the-art approaches across diverse RWOSR benchmark datasets. The source code of this article will be available at https://github.com/yangzhen1252/RWOSR1.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Algorithm 1
Algorithm 2
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author, [Leyuan Fang], upon reasonable request.

References

  • Bendale, A., & Boult, T. (2015). Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1893–1902).

  • Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563–1572).

  • Busto, P. P., & Gall, J. (2017). Open set domain adaptation. In Proceedings of IEEE/CVF international conference on computer vision (pp. 754–763).

  • Caesar, H., et al. (2020). nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11621–11631).

  • Cen, J., Luan, D., Zhang, S., et al. (2023). The devil is in the wrongly-classified samples: Towards unified open-set recognition. In Proceedings of ICLR.

  • Chen, G., Qiao, L., Shi, Y., et al. (2020). Learning open set network with discriminative reciprocal points. In Proceedings of European conference on computer vision (pp. 507–522).

  • Chen, G., Peng, P., Wang, X., & Tian, Y. (2022). Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8065–8081.

    PubMed  Google Scholar 

  • Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297.

    Article  Google Scholar 

  • Fang, L., Yan, Y., Yue, J., & Deng, Y. (2023). Toward the vectorization of hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 61, 1–14.

    Google Scholar 

  • Fukumizu, K., Bach, F., & Jordan, M. (2004). Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5, 73–99.

    MathSciNet  Google Scholar 

  • Ge, Z. Y., Demyanov, S., Chen, Z., & Garnavi, R. (2017). Generative OpenMax for multi-class open set classification [Online]. Available: arXiv:1707.07418.

  • Geng, C., Huang, S. J., & Chen, S. (2021). Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10), 3614–3631.

    Article  PubMed  Google Scholar 

  • Geng, X., Dong, G., Xia, Z., & Liu, H. (2022). SAR target recognition via random sampling combination in open-world environments. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 331–343.

    Article  ADS  Google Scholar 

  • Giusti, E., Ghio, S., Oveis, A. H., & Martorella, M. (2022). Proportional similarity-based openmax classiffer for open set recognition in SAR images. Remote Sensing, 14(18), 4665.

    Article  ADS  Google Scholar 

  • Gretton, A., Fukumizu, K., Teo, C., et al. (2008). A kernel statistical test of independence. In Proceedings of NeuralPS (pp. 585–592).

  • Hassen, M., & Chan, P. K. (2018). Learning a neural-network-based representation for open set recognition. [Online]. arXiv:1802.04365.

  • Huang, M., et al. (2022). SwinTextSpotter: Scene text spotting via better synergy between text detection and text recognition. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 4583–4593).

  • Jain, L. P., Scheirer, W. J., & Boult, T. E. (2014). Multi-class open set recognition using probability of inclusion. In Proceedings of the European Conference on Computer Vision (pp. 393–409).

  • Jo, I., Kim, J., Kang, H., Kim, Y. D., & Choi, S. (2018). Open set recognition by regularising classifier with fake data generated by generative adversarial networks. In Proceedings of international conference on acoustics, speech, and signal processing (pp. 2686–2690).

  • Kerner, H. R., Wellington, D. F., Wagstaff, K. L., Bell, J. F., Kwan, C., & Amor, H. B. (2019). Novelty detection for multispectral images with application to planetary exploration. In Proceedings of AAAI.

  • Kotz, S., & Nadarajah, S. (2000). Extreme value distributions: Theory and applications. World Scientific.

    Book  Google Scholar 

  • Kuang, K., Cui, P., Athey, S., et al. (2018). Stable prediction across unknown environments. In Proceedings of KDD.

  • Kuang, K., Li, L., Geng, Z., et al. (2020). Causal inference. Engineering, 6(3), 253–263.

    Article  Google Scholar 

  • Li, D., Yang, Y., Song, Y., & Hospedales, T. (2017). Deeper, broader and artier domain generalization. In Proceedings IEEE/CVF international conference on computer vision (pp. 5542–5550).

  • Li, H., Pan, S. J., Wang, S., & Kot, A. C. (2018). Domain generalization with adversarial feature learning. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 5400–5409).

  • Liu, B., Kang, H., Li, H., Hua, G., & Vasconcelos, N. (2020). Few-shot open-set recognition using meta-learning. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 8795–8804).

  • Liu, J., Tian, J., Han, W., et al. (2023). Learning multiple gaussian prototypes for open-set recognition. Information Sciences, 626, 738–753.

    Article  Google Scholar 

  • Liu, L., Ouyang, W., Wang, X., et al. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128, 261–318.

    Article  Google Scholar 

  • Liu, Z., Fu, Y., Pan, Q., & Zhang, Z. (2023). Orientational distribution learning with hierarchical spatial attention for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 8757–8772.

    PubMed  Google Scholar 

  • Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B. & Yu, S. X. (2019). Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2532–2541).

  • Lopez-Paz, D., Nishihara, R., Chintala, S., Scholkopf, B., & Bottou, L. (2017). Discovering causal signals in images. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 58–66).

  • Mensink, T., Verbeek, J., Perronnin, F., & Csurka, G. (2013). Distancebased image classification: Generalizing to new classes at nearzero cost. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(11), 2624–2637.

    Article  PubMed  Google Scholar 

  • Moon, W., Park, J., Seong, H. S., et al. (2022). Difficulty-aware simulator for open set recognition. In Proceedings of the European conference on computer vision.

  • Mousavian, A., Anguelov, D., Flynn, J., & Kosecka, J. (2017). 3D bounding box estimation using deep learning and geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7074–7082).

  • Neal, L., Olson, M., Fern, X., Wong, W., & Li, F. (2018). Open set learning with counterfactual images. In Proceedings of European conference on computer vision (pp. 613–628).

  • Nie, D., & Shen, D. (2020). Adversarial confidence learning for medical image segmentation and synthesis. International Journal of Computer Vision, 128, 2494–2513.

    Article  MathSciNet  PubMed  PubMed Central  Google Scholar 

  • Nirmal, S., Sowmya, V., & Soman, K. P. (2020). Open set domain adaptation for hyperspectral image classiffcation using generative adversarial network. In Proceedings of ICICCT (pp. 819–827).

  • Oliveira, H., Silva, C., Machado, G. L., Nogueira, K., & Dos Santos, J. A. (2021). Fully convolutional open set segmentation. Machine Learning 1–52.

  • Oza, P., & Patel, V. M. (2019). C2AE: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2307–2316).

  • Pal, D., Bose, S., Banerjee, B., & Jeppu, Y. (2023). MORGAN: Meta-learning based few-shot open-set recognition via generative adversarial network. In Proceedings of WACV (pp. 6295–6304).

  • Pang, S., et al. (2021). SpineParseNet: Spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation. IEEE Transactions on Medical Imaging, 40(1), 262–273.

    Article  PubMed  Google Scholar 

  • Peng, J., Li, L., & Tang, Y. Y. (2019). Maximum likelihood estimation based joint sparse representation for the classification of hyperspectral remote sensing images. IEEE Transactions on Neural Networks and Learning Systems, 30(6), 1790–1802.

    Article  MathSciNet  PubMed  Google Scholar 

  • Prakhya, S., Venkataram, V., & Kalita, J. (2017). Open set text classification using convolutional neural networks. In Proceedings of the international conference recent advances in natural language processing (pp. 466–475).

  • Rahimi, A., & Recht, B. (2008). Random features for large scale kernel machines. In Proceedings of NeuralPS (pp. 1177–1184).

  • Rahman, S., Khan, S. H., & Porikli, F. (2020). Zero-shot object detection: Joint recognition and localization of novel concepts. International Journal of Computer Vision, 128, 2979–2999.

    Article  Google Scholar 

  • Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149.

    Article  PubMed  Google Scholar 

  • Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., & Boult, T. E. (2013). Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757–1772.

    Article  PubMed  Google Scholar 

  • Scheirer, W. J., Jain, L. P., & Boult, T. E. (2014). Probability models for OSR. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2317–2324.

    Article  PubMed  Google Scholar 

  • Shen, Z., Cui, P., et al. (2020). Stable learning via sample reweighting. In Proceedings of AAAI conference on artificial intelligence (pp. 5692–5699).

  • Shu, L., Xu, H., & Liu, B. (2018). Unseen class discovery in open-world classification. [Online]. arXiv:1801.05609.

  • Song, Q., Sun, B., & Li, S. (2023). Multimodal sparse transformer network for audio-visual speech recognition. IEEE Transactions on Neural Networks and Learning Systems, 34(12), 10028–10038.

    Article  PubMed  Google Scholar 

  • Strobl, E., Zhang, K., & Visweswaran, S. (2019). Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference7(1).

  • Torralba, A., & Efros, A. (2011). Unbiased look at dataset bias. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1521–1528).

  • Wang, L., et al. (2023). Multi-Modal 3D object detection in autonomous driving: A survey and taxonomy. IEEE Transactions on Intelligent Vehicles, 8(7), 3781–3798.

    Article  Google Scholar 

  • Wang, R., Guo, J., Zhao, R., Su, L., Ye, Y., Zhang, X., Zhang, Y., & Feng, R. (2023). Class-aware variational auto-encoder for open set recognition. In Proceedings of IEEE ICME.

  • Wang, Z., Xu, Q., & Yang, Z. (2022). OpenAUC: Towards AUC-oriented open-set recognition. In Proceedings of NeurIPS.

  • Wu, L., Fang, L., He, X., He, M., Ma, J., & Zhong, Z. (2023a). Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 8827–8844.

  • Wu, J., Zhu, D., Fang, L., Deng, Y., & Zhong, Z. (2023b). Efficient layer compression without pruning. IEEE Transactions on Image Processing, 32, 4689–4700.

  • Xu, Y., Qin, L., Liu, X., Xie, J., & Zhu, S.-C. (2018). A causal and-or graph model for visibility fluent reasoning in tracking interacting objects. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2178–2187).

  • Yang, X., Feng, F., Ji, W., Wang, M., & Chua, T.-S. (2021). Deconfounded video moment retrieval with causal intervention. In Proceedings of SIGIR (pp. 1–10).

  • Yang, Y., Hou, C., Lang, Y., Guan, D., Huang, D., & Xu, J. (2019). Open-set human activity recognition based on micro-doppler signatures. Pattern Recognition, 85, 60–69.

    Article  ADS  Google Scholar 

  • Yao, L., Chu, Z., & Li, S. (2021). A survey on causal inference. ACM Transactions on Knowledge Discovery from Data, 15(5), 1556–4681.

    Article  Google Scholar 

  • Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., & Naemura, T. (2019). Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4016–4025).

  • Yu, H., Cui, P., et al. (2023). Stable learning via sparse variable independence. In Proceedings of AAAI conference on artificial intelligence

  • Yu, Y., Qu, W. Y., Li, N., & Guo, Z. (2017). Open-category classification by adversarial sample generation. In Proceedings of the international joint conferences on artificial intelligence

  • Yue, J., Fang, L., & He, M. (2022). Spectral-spatial latent reconstruction for open-set hyperspectral image classification. IEEE Transactions on Image Processing, 31, 5227–5241.

    Article  ADS  PubMed  Google Scholar 

  • Zhang, H., & Patel, V. (2017). Sparse representation-based open set brecognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8), 1690–1696.

    Article  PubMed  Google Scholar 

  • Zhang, J., Liu, J., Shi, L., Pan, B., & Xu, X. (2020). An open set domain adaptation network based on adversarial learning for remote sensing image scene classiffcation. In Proceedings of IEEE international geoscience and remote sensing symposium (pp. 1365–1368).

  • Zhang, S., Feng, X., Fan, W., et al. (2023). Video-audio domain generalization via confounder disentanglement. In Proceedings of AAAI conference on artificial intelligence.

  • Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y., & Shen, Z. (2021). Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE conference on computer vision and pattern Recognition (pp. 5368–5378).

  • Zhou, X., Tang, T., Cui, Y., & Kuang, G. (2022). SAR open set recognition based on counterfactual framework. In Proceedings of PIERS (pp. 249–253).

  • Zhou, Y., Shang, S., Song, X., Zhang, S., You, T., & Zhang, L. (2022). Intelligent radar jamming recognition in open set environment based on deep learning networks. Remote Sensing, 14(24), 6220.

    Article  ADS  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant U22B2014 and Grant 62101072, in part by the Science and Technology Plan Project Fund of Hunan Province under Grant 2022RSC3064, in part by the Hunan Provincial Natural Science Foundation of China under Grant 2021JJ40570.

Funding

National Natural Science Foundation of China under Grant U22B2014 and Grant 62101072; Science and Technology Plan Project Fund of Hunan Province under Grant 2022RSC3064; Hunan Provincial Natural Science Foundation of China under Grant 2021JJ40570.

Author information

Author notes

  1. Zhen Yang and Jun Yue have contributed equally to this work.

Authors and Affiliations

  1. College of Electrical and Information Engineering, Hunan University, Changsha, 410082, Hunan, China

    Zhen Yang & Leyuan Fang

  2. School of Automation, Central South University, Changsha, 410083, Hunan, China

    Jun Yue

  3. Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Helmholtz Institute Freiberg for Resource Technology, 09599, Freiberg, Germany

    Pedram Ghamisi

  4. Department of Comptuer Science, Peking University, Beijing, 100871, China

    Shiliang Zhang

  5. School of Electronic Information, Wuhan University, Wuhan, 430072, Hubei, China

    Jiayi Ma

Authors
  1. Zhen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pedram Ghamisi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shiliang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiayi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leyuan Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leyuan Fang.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Communicated by Zhun Zhong.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Z., Yue, J., Ghamisi, P. et al. Open Set Recognition in Real World. Int J Comput Vis (2024). https://doi.org/10.1007/s11263-024-02015-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11263-024-02015-9

Keywords

Search

Navigation