Skip to main content

Video enhancement using reference photographs

Abstract

Digital video cameras are becoming commonplace in many households, but they still leave something to be desired in terms of image quality. Their poor light sensitivity make images noisy and blurry, and internal storage bandwidth limits the frame resolution. We present a technique for enhancing a low quality video sequence, using a set of high quality reference photographs, taken of the same scene. Our technique generates a high quality frame by copying information from the photographs in a patch-wise fashion. The copying is guided by a sparse set of reliable correspondences between the video frames and photographs. Our technique is purely image-based, and does not require depth estimation. A robust descriptor is employed for establishing valid matches between the video frames and the photographs. Then, the geometric transformation is estimated between every corresponding patch. With only a few reference photographs, we are able to reduce noise and motion blur, and more important, increase resolution by a factor of 6 (see Fig. 1).

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Ancuti, C., Haber, T., Mertens, T., Bekaert, P.: Video enhancement using reference photographs. In: Conference Abstracts and Applications of ACM SIGGRAPH 2007, (Sketch session), San Diego. ACM, New York (2007)

    Google Scholar 

  2. 2.

    Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of Symposium on Interactive 3D graphics, pp. 217–226. ACM, New York (2001)

    Chapter  Google Scholar 

  3. 3.

    Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 24(9), 1167–1182 (2002)

    Article  Google Scholar 

  4. 4.

    Bhat, P., Zitnick, C.L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., Kang, S.B.: Using photographs to enhance videos of a static scene. In: Proceedings of Eurographics Symposium on Rendering (EGSR), pp. 327–338. ACM, New York (2007)

    Google Scholar 

  5. 5.

    Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: Proceedings of International Conference on Pattern Recognition, pp. 600–605. IEEE Press, Washington, DC (2000)

    Google Scholar 

  6. 6.

    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of ACM SIGGRAPH, pp. 341–346. ACM, New York (2001)

    Google Scholar 

  7. 7.

    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 1033–1038. IEEE Press, Washington, DC (1999)

    Chapter  Google Scholar 

  8. 8.

    Fattal, R.: Upsampling via imposed edges statistics. ACM Trans. Graph. (SIGGRAPH) 26(3), (2007)

  9. 9.

    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  10. 10.

    Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of 4th Alvey Vision Conference, vol. 18, pp. 147–151. ACM, New York (1988)

    Google Scholar 

  11. 11.

    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  12. 12.

    Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of ACM SIGGRAPH 2001, pp. 327–340. ACM, New York (2001)

    Google Scholar 

  13. 13.

    Irani, M., Peleg, S.: Improving resolution by image registration. J. Comput. Vis. Graph. Image Process. 55(3), 231–239 (1991)

    Google Scholar 

  14. 14.

    Jiang, Z., Wong, T.-T., Bao, H.: Practical super-resolution from dynamic video sequences. In: Proceedings of IEEE Computer Video and Pattern Recognition. IEEE Press, Washington, DC (2003)

    Google Scholar 

  15. 15.

    Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoustics Speech Signal Process 26(9), 1153–1160 (1981)

    Article  MathSciNet  Google Scholar 

  16. 16.

    Kopf, J., Cohen, M., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. Proc. ACM Trans. Graph. (SIGGRAPH) 26(3), 96–100 (2007)

    Article  Google Scholar 

  17. 17.

    Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. Proc. ACM SIGGRAPH 2003 22(3), 277–286 (2003)

    Article  Google Scholar 

  18. 18.

    Liang, L., Liu, C., Xu, Y.-Q., Guo, B., Shum, H.-Y.: Real-time texture synthesis by patch-based sampling. In: Proceedings of ACM SIGGRAPH 2001. ACM, New York (2001)

    Google Scholar 

  19. 19.

    Lindeberg, T.: Feature detection with automatic scale selection. Int. J. Comput. Vis. 30(2), 77–116 (1999)

    Google Scholar 

  20. 20.

    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  21. 21.

    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Comput. Vis. Pattern Recogn. (CVPR) 30(2), 257–263 (2004)

    Google Scholar 

  22. 22.

    Moreels, P., Perona, P.: Evaluation of features detectors and descriptors based on 3D objects. Int. J. Comput. Vis. 73(3), 263–284 (2007)

    Article  Google Scholar 

  23. 23.

    Schultz, R., Stevenson, R.: Extraction of high-resolution frames from video sequences. J. IEEE Trans. Image Process. 5(6), 996–1011 (1996)

    Article  Google Scholar 

  24. 24.

    Shechtman, E., Caspi, Y., Irani, M.: Space-time super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 27(4), 531–545 (2005)

    Article  Google Scholar 

  25. 25.

    Shi, J., Tomasi, C.: Good features to track. IEEE Comput. Vis. Pattern Recogn. (CVPR), 593–600 (1994)

  26. 26.

    Sun, J., Zheng, N.-N., Tao, H., Shum, H.-Y.: Image hallucination with primal sketch priors. IEEE Comput. Vis. Pattern Recogn. (CVPR), 729 (2003)

  27. 27.

    Wei, L.-Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of ACM SIGGRAPH 2000, pp. 479–488. ACM, New York (2000)

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Cosmin Ancuti.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ancuti, C., Haber, T., Mertens, T. et al. Video enhancement using reference photographs. Visual Comput 24, 709–717 (2008). https://doi.org/10.1007/s00371-008-0251-y

Download citation

Keywords

  • Picture/image generation
  • Graphics utilities
  • Applications