Skip to main content
SpringerLink
Log in

A general image orientation detection method by feature fusion

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The automatic detection of image orientation is an important part of computer vision research. It is widely used in a variety of intelligent devices and application software. In the existing research on orientation detection, low-level features used in classification model cannot accurately express the high-level semantics of the image, and fine-tuning the existing deep learning network does not consider whether the extracted features can express the human visual perception of the orientation. As a result, the generalization ability of the model is not high. Based on the above shortcomings, we propose an automatic image orientation detection method based on the fusion of attention features (AF) and rotation features (RF). Firstly, the AF is obtained by fusing the attention mechanism features, which are extracted from the feature maps of different scales of ResNet50. It can quickly screen out high-value information from a large amount of information by using limited attention resources. Secondly, the “rotating LBP” features of different scales that can better reflect the direction attribute are extracted. The RF is obtained by residual dilated convolution combing with ResNet50. It can more accurately express the directional characteristics of the image and improve the generalization ability of the model. Finally, AF and RF are fused to realize the detection of four orientations of the image. The proposed method is verified on five different types of data sets. The results show that this method can more comprehensively express the directional semantics of images and improve the classification accuracy and wide application of the model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Code availability The codes generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Notes

  1. http://www.wikiart.org.

  2. https://www.abstractartistgallery.org/; https://www.artsy.net/search?term=abstract.

  3. http://disi.unitn.it/yan-ulevskaya/mart.html.

  4. http://www.deviantart.com.

References

  1. Lyu, S.: Automatic image orientation determination with natural image statistics, pp. 491– 494 (2011)

  2. Cingovska, I., Ivanovski, Z.A., Martin, F.: Automatic image orientation detection with prior hierarchical content-based classification. In: 18th IEEE International Conference on Image Processing, ICIP 2011, Brussels, Belgium, September, pp. 11– 14 ( 2011)

  3. Borawski, M., Frejlichowski, D.: An algorithm for the automatic estimation of image orientation. Int. Conf. Mach. Learn. Data Min. Pattern Recogn. 7376, 336–344 (2012)

    Article  Google Scholar 

  4. Ciocca, G., Cusano, C., Schettini, R.: Image orientation detection using low-level features and faces, vol. 7537, pp. 75370–753708 (2010)

  5. Hollitt, C., Deeb, A.S.: Determining image orientation using the hough and fourier transforms. In: Conference on Image and Vision Computing New Zealand, pp. 346– 351 (2012)

  6. Cao, Z., Liu X, G.N.: A fast orientation estimation approach of natural images. IEEE Trans. Syst. Man Cybern. Syst. 46(11), 1589–1597 (2016)

    Article  Google Scholar 

  7. Ciocca, G., Cusano, C., Schettini, R.: Image orientation detection using lbp-based features and logistic regression. Multimed. Tools Appl. 74(9), 3013–3034 (2015)

    Article  Google Scholar 

  8. Liu, J., Dong, W., Zhang, X.: Orientation judgment for abstract paintings. Multimed. Tools Appl. 76, 1017–1036 (2017)

    Article  Google Scholar 

  9. Swami, K., Deshpande, P.P., Khandelwal, G., Vijayvargiya, A.: Why my photos look sideways or upside down? detecting canonical orientation of images using convolutional neural networks. In: International Conference on Multimedia and Expo, pp. 495–500 (2017)

  10. Joshi, U., Guerzhoy, M.: Automatic photo orientation detection with convolutional neural networks. In: 2017 14th Conference on Computer and Robot Vision (CRV), pp. 103–108 (2017)

  11. Morra, L., Famouri, S., Karakus, H.C., Lamberti, F.: Automatic detection of canonical image orientation by convolutional neural networks. In: 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT), pp. 113–128 (2019)

  12. Prince, M., Alsuhibany, S.A., Siddiqi, N.A.: A step towards the optimal estimation of image orientation. IEEE Access 7, 185750–185759 (2019)

    Article  Google Scholar 

  13. Lumini, A., Nanni, L., Scattolaro, L., Maguolo, G.: Image orientation detection by ensembles of stochastic CNNs. Mach. Learn. Appl. 6, 100090 (2021)

    Google Scholar 

  14. Soroush, R., Baleghi, Y.: Nir/rgb image fusion for scene classification using deep neural networks. Vis. Comput. (2022)

  15. Mohamed Hazgui, H.G., Barhoumi, W.: Genetic programming-based fusion of hog and lbp features for fully automated texture classification. Vis. Comput. 38, 457–476 (2022)

    Article  Google Scholar 

  16. Li, X., Pi, J., Lou, M., Qu, Y., et al.: Multi-level feature fusion network for nuclei segmentation in digital histopathological images. Vis. Comput. (2022)

  17. Wang, G., Gan, X., Cao, Q., Zhai, Q.: Mfanet: multi-scale feature fusion network with attention mechanism. Vis. Comput. (2022)

  18. Bai, R.Y., Guo, X.Y., Jai, C.H.: Orientation detection of abstract painting based on loacl binary pattern. Comput. Appl. Softw. 38(4), 239–244 (2021)

    Google Scholar 

  19. Bai, R.Y., Guo, X.Y., Jai, C.H.: What is the correct hanging orientation for abstract painting? Orientation judgment and detection. In: The 3rd International Conference on Computer Science and Application Engineering (2020)

  20. Bai, R.Y., Guo, X.Y.: Automatic orientation detection of abstract painting. Knowl. Based Syst. 227(3), 107240 (2021)

    Article  Google Scholar 

  21. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770– 78 (2016)

  22. Deng, J., Dong, W., Socher, R., Li, L.J., Li, F.F.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, Florida, USA, pp. 20–25 (2009)

  23. Woo, S., Park, J.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  24. Liu, L., Xie, Y.X., Wei, Y.M., Lao, S.Y.: Survey of local binary pattern method. J. Image Gr. 19(12), 1696–1720 (2014)

    Google Scholar 

  25. Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., Cottrell, G.: Understanding convolution for semantic segmentation. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1451–1460 (2018)

  26. Xiao, J., Ehinger, K.A., Hays, J., Torralba, A., Oliva, A.: Sun database: exploring a large collection of scene categories. Int. J. Comput. Vis. 119(1), 3–22 (2016)

    Article  MathSciNet  Google Scholar 

  27. Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric consistency for large scale image search (2008)

  28. Torralba, A., Sinha, P.: Recognizing indoor scenes (2009)

  29. Sartori, A., Yanulevskaya, V., Salah, A.A., Uijlings, J., Bruni, E., Sebe, N.: Affective analysis of professional and amateur abstract paintings using statistical analysis and art theory. ACM Trans. Interact. Intell. Syst. 5(2), 1–27 (2015)

    Article  Google Scholar 

  30. Alameda-Pineda, X., Ricci, E., Yan, Y., Sebe, N.: Recognizing emotions from abstract paintings using non-linear matrix completion. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  31. Peng, X., Zhu, H., Feng, J., Shen, C., Zhang, H., Zhou, J.: Deep clustering with sample-assignment invariance prior. IEEE Trans. Neural Netw. Learn. Syst. 31(11), 4857–4868 (2020)

    Article  MathSciNet  Google Scholar 

  32. Peng, X., Feng, J., Xiao, S., Yau, W.Y., Zhou, J.T., Yang, S.: Structured autoencoders for subspace clustering. IEEE Trans. Image Process. 27(10), 5076–5086 (2018)

    Article  MathSciNet  Google Scholar 

  33. Hu, P., Zhu, H., Lin, J., Peng, D., Zhao, Y.-P., Peng, X.: Unsupervised contrastive cross-modal hashing. IEEE Trans. Pattern Anal. Mach. Intell. (2022). https://doi.org/10.1109/TPAMI.2022.3177356

    Article  Google Scholar 

  34. Hu, P., Peng, X., Zhu, H., Zhen, L., Lin, J., Yan, H., Peng, D.: Deep semisupervised multiview learning with increasing views. IEEE Trans. Cybern. (2021)

Download references

Acknowledgements

This work is partially supported by the Youth Program of the National Natural Science Foundation of China (61603228), Fundamental Research Program of Shanxi Province (202103021223030), Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi (2020L0036).

Author information

Authors and Affiliations

  1. College of Automation and Software, Shanxi University, Taiyuan, 030013, Shanxi, People’s Republic of China

    Bai Ruyi

Authors
  1. Bai Ruyi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by CJ and XG. The first draft of the manuscript was written by RB, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Bai Ruyi.

Ethics declarations

Conflict of interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Ethical approval

We declare that this manuscript is original, has not been published before and is not currently considered for publication elsewhere.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ruyi, B. A general image orientation detection method by feature fusion. Vis Comput 40, 287–302 (2024). https://doi.org/10.1007/s00371-023-02782-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02782-5

Keywords

Search

Navigation