Advertisement

Video Enhancement via Super-Resolution Using Deep Quality Transfer Network

  • Pai-Heng HsiaoEmail author
  • Ping-Lin Chang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10113)

Abstract

Streaming low bitrate while preserving high-quality video content is a crucial topic in multimedia and video surveillance. In this work, we explore the problem of spatially and temporally reconstructing high-resolution (HR) frames from a high frame-rate low-resolution (LR) sequence and a few temporally subsampled HR frames. The targeted problem is essentially different from the problems handled by typical super-resolution (SR) methods such as single-image SR and video SR, which attempt to reconstruct HR images using only LR images. To tackle the targeted problem, we propose a deep quality transfer network, based on the convolutional neural network (CNN), which consists of modules including generation and selection of HR pixel candidates, fusion with LR input, residual learning and bidirectional architecture. The proposed CNN model has real-time performance at inference stage. The empirical studies have verified the generality of the proposed CNN model showing significant quality gains for video enhancement.

Keywords

Optical Flow Convolutional Neural Network Bicubic Interpolation Deep Convolutional Neural Network Convolution Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Wiegand, T., Sullivan, G.J., Bjøntegaard, G., Luthra, A.: Overview of the h. 264/avc video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13, 560–576 (2003)CrossRefGoogle Scholar
  2. 2.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intel. (PAMI) 32, 295–307 (2016)CrossRefGoogle Scholar
  3. 3.
    Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  4. 4.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015)Google Scholar
  5. 5.
    Huang, Y., Wang, W., Wang, L.: Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Neural Information Processing Systems (NIPS), pp. 235–243 (2015)Google Scholar
  6. 6.
    Liao, R., Tao, X., Li, R., Ma, Z., Jia, J.: Video super-resolution via deep draft-ensemble learning. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 531–539 (2015)Google Scholar
  7. 7.
    Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5224–5232 (2015)Google Scholar
  8. 8.
    Shahar, O., Faktor, A., Irani, M.: Space-time super-resolution from a single video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3353–3360 (2011)Google Scholar
  9. 9.
    Gupta, A., Bhat, P., Dontcheva, M., Deussen, O., Curless, B., Cohen, M.: Enhancing and experiencing spacetime resolution with videos and stills. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–9 (2009)Google Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  11. 11.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)Google Scholar
  12. 12.
    Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 2758–2766 (2015)Google Scholar
  13. 13.
    Protter, M., Elad, M., Takeda, H., Milanfar, P.: Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process. (TIP) 18, 36–51 (2009)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. (TIP) 18, 1958–1975 (2009)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Yang, C.Y., Ma, C., Yang, M.H.: Single-image super-resolution: a benchmark. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 372–386. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10593-2_25 Google Scholar
  16. 16.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  17. 17.
    Baker, S., Kanade, T.: Super-resolution optical flow. Technical report (1999)Google Scholar
  18. 18.
    Tarvainen, J., Mikko, N., Pirkko, O.: Spatial and temporal information as camera parameters for super-resolution video. In: Proceedings of the International Symposium on Multimedia (ISM), pp. 302–305 (2012)Google Scholar
  19. 19.
    Ancuti, C., Ancuti, C.O., Bekaert, P.: A patch-based approach to restore videos using additional stills. In: Proceedings of the International Symposium ELMAR, pp. 143–146 (2010)Google Scholar
  20. 20.
    Horn, B.K., Schunck, B.G.: Determining optical flow. In: Technical Symposium East. International Society for Optics and Photonics, pp. 319–331 (1981)Google Scholar
  21. 21.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Deepflow: large displacement optical flow with deep matching. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 1385–1392 (2013)Google Scholar
  22. 22.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1164–1172 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Umbo CV Inc.Taipei CityTaiwan

Personalised recommendations