Skip to main content
SpringerLink
Log in

Super-Resolution by Predicting Offsets: An Ultra-Efficient Super-Resolution Network for Rasterized Images

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13679))

Included in the following conference series:

Abstract

Rendering high-resolution (HR) graphics brings substantial computational costs. Efficient graphics super-resolution (SR) methods may achieve HR rendering with small computing resources and have attracted extensive research interests in industry and research communities. We present a new method for real-time SR for computer graphics, namely Super-Resolution by Predicting Offsets (SRPO). Our algorithm divides the image into two parts for processing, i.e., sharp edges and flatter areas. For edges, different from the previous SR methods that take the anti-aliased images as inputs, our proposed SRPO takes advantage of the characteristics of rasterized images to conduct SR on the rasterized images. To complement the residual between HR and low-resolution (LR) rasterized images, we train an ultra-efficient network to predict the offset maps to move the appropriate surrounding pixels to the new positions. For flat areas, we found simple interpolation methods can already generate reasonable output. We finally use a guided fusion operation to integrate the sharp edges generated by the network and flat areas by the interpolation method to get the final SR image. The proposed network only contains 8,434 parameters and can be accelerated by network quantization. Extensive experiments show that the proposed SRPO can achieve superior visual effects at a smaller computational cost than the existing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For FSRCNN, we use the hyper-parameter of \(D=56\), \(S=12\) and \(M=4\).

  2. 2.

    For ESPCN, we use the hyper-parameter of \(D=22\) and \(S=32\).

  3. 3.

    For eSR family, we select four versions of eSR: eSR-MAX with \(K=3\) and \(C=1\); eSR-TR with \(K=5\) and \(C=2\); eSR-TM with \(K=7\) and \(C=4\); eSR-CNN with \(C=4\), \(D=3\) and \(S=6\).

References

  1. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: ECCV, pp. 252–268 (2018)

    Google Scholar 

  2. Akenine-Moller, T., Haines, E., Hoffman, N.: Real-Time Rendering. AK Peters/CRC Press (2019)

    Google Scholar 

  3. Chen, H., Gu, J., Zhang, Z.: Attention in attention network for image super-resolution. arXiv preprint arXiv:2104.09497 (2021)

  4. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: CVPR, pp. 11065–11074 (2019)

    Google Scholar 

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI 38(2), 295–307 (2015)

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  7. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323. JMLR Workshop and Conference Proceedings (2011)

    Google Scholar 

  8. Jinjin, G., Haoming, C., Haoyu, C., Xiaoxing, Y., Ren, J.S., Chao, D.: PIPAL: a large-scale image quality assessment dataset for perceptual image restoration. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 633–651. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_37

    Chapter  Google Scholar 

  9. Gu, J., et al.: NTIRE 2022 challenge on perceptual image quality assessment. In: CVPR Workshops, pp. 951–967 (2022)

    Google Scholar 

  10. Gu, J., Shen, Y., Zhou, B.: Image processing using multi-code GAN prior. In: CVPR, pp. 3012–3021 (2020)

    Google Scholar 

  11. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: ACM MM, pp. 2024–2032 (2019)

    Google Scholar 

  12. Jimenez, J., Echevarria, J.I., Sousa, T., Gutierrez, D.: SMAA: enhanced subpixel morphological antialiasing. In: Computer Graphics Forum, vol. 31, pp. 355–364. Wiley Online Library (2012)

    Google Scholar 

  13. Kaplanyan, A.S., Sochenov, A., Leimkühler, T., Okunev, M., Goodall, T., Rufo, G.: DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. TOG 38(6), 1–13 (2019)

    Article  Google Scholar 

  14. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654 (2016)

    Google Scholar 

  15. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: CVPR, pp. 1637–1645 (2016)

    Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  17. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 4681–4690 (2017)

    Google Scholar 

  18. Li, Z., et al.: Blueprint separable residual network for efficient image super-resolution. In: CVPR, pp. 833–843 (2022)

    Google Scholar 

  19. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops, pp. 136–144 (2017)

    Google Scholar 

  20. Michelini, P.N., Lu, Y., Jiang, X.: edge-SR: super-resolution for the masses. In: WACV, pp. 1078–1087 (2022)

    Google Scholar 

  21. Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)

  22. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: NeurIPS, vol. 32, pp. 8026–8037 (2019)

    Google Scholar 

  23. Reshetov, A.: Morphological antialiasing. In: Proceedings of the Conference on High Performance Graphics, pp. 109–116 (2009)

    Google Scholar 

  24. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)

    Google Scholar 

  25. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR, pp. 3147–3155 (2017)

    Google Scholar 

  26. Thomas, M.M., Vaidyanathan, K., Liktor, G., Forbes, A.G.: A reduced-precision network for image reconstruction. TOG 39(6), 1–12 (2020)

    Article  Google Scholar 

  27. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  28. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP 13(4), 600–612 (2004)

    Google Scholar 

  29. Xiao, L., Nouri, S., Chapman, M., Fix, A., Lanman, D., Kaplanyan, A.: Neural supersampling for real-time rendering. TOG 39(4), 142-1 (2020)

    Google Scholar 

  30. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)

    Google Scholar 

  31. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV, pp. 286–301 (2018)

    Google Scholar 

  32. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)

    Article  Google Scholar 

  33. Zhao, H., Kong, X., He, J., Qiao, Yu., Dong, C.: Efficient image super-resolution using pixel attention. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 56–72. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_3

    Chapter  Google Scholar 

Download references

Acknowledgement

This work was partly supported by SZSTC Grant No. JCYJ20190809172201639 and WDZC20200820200655001, Shenzhen Key Laboratory ZDSYS20210623092001004. We sincerely thank Yongfei Pu, Yuanlong Li, Jieming Li and Yuanlin Chen for contributing to this study.

Author information

Authors and Affiliations

  1. The University of Sydney, Camperdown, Australia

    Jinjin Gu

  2. University of Maryland, College Park, USA

    Haoming Cai

  3. Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

    Chenyu Dong, Ruofan Zhang, Wenming Yang & Chun Yuan

  4. ETH Zürich, Zürich, Switzerland

    Yulun Zhang

  5. Peng Cheng National Laboratory, Shenzhen, China

    Chun Yuan

Authors
  1. Jinjin Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoming Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenyu Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruofan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yulun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wenming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chun Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jinjin Gu or Chun Yuan .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gu, J. et al. (2022). Super-Resolution by Predicting Offsets: An Ultra-Efficient Super-Resolution Network for Rasterized Images. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19800-7_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19799-4

  • Online ISBN: 978-3-031-19800-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation