Is Super-Resolution with Optical Flow Feasible?
Reconstruction-based super-resolution from motion video has been an active area of study in computer vision and video analysis. Image alignment is a key component of super-resolution algorithms. Almost all previous super-resolution algorithms have assumed that standard methods of image alignment can provide accurate enough alignment for creating super-resolution images. However, a systematic study of the demands on accuracy of multi-image alignment and its effects on super-resolution has been lacking. Furthermore, implicitly or explicitly most algorithms have assumed that the multiple video frames or specific regions of interest are related through global parametric transformations. From previous works, it is not at all clear how super-resolution performs under alignment with piecewise parametric or local optical flow based methods. This paper is an attempt at understanding the influence of image alignment and warping errors on super-resolution. Requirements on the consistency of optical flow across multiple images are studied and it is shown that errors resulting from traditional flow algorithms may render super-resolution infeasible.
KeywordsOptical Flow Motion Estimation Motion Error Image Alignment Reprojection Error
Unable to display preview. Download preview PDF.
- Elad, M. and Feuer, A. 1997. Restoration of a single superresolution image form several blurred, noisy and undersampled measured images. IEEE Trans. on Image Processing, pp. 1646–1658.Google Scholar
- Patti, A., Sezan, M., and Tekalp, M. 1997. Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Trans. on Image Processing, pp. 1064–1078.Google Scholar
- Bascle, B., Blake, A., and Zisserman, A. 1996. Motion deblurring and super-resolution from an image sequence. In Proc. European Conf. Comp. Vision, pp 573–581.Google Scholar
- Schultz, R.R. and Stevenson, R.L. 1996. Extraction of high resolution frames from video sequences. IEEE Trans. on Image Processing, pp. 996–1011.Google Scholar
- Hardie, R., Barrard, K., and Armstrong E. 1997. Joint MAP Registration and High-resolution Image Estimation Using a Sequence of Undersampled Images. IEEE Trans. on Image Processing, pp. 1621–1633. Multi-resolutionGoogle Scholar
- Marziliano, P. and Vetterlli, M. 1999. Reconstruction of Irregularly Sampled Discrete-Time Bandlimited Signals. IEEE Trans. on Signal Processing, pp. 3462–3471.Google Scholar
- Shekarforoush, H. and Chellappa, R. 1999. Data-driven multi-channel super-resolution with application to video sequences. Journal of the Optical Society of America A, pp. 481–492.Google Scholar
- Tsai, R.Y. and Huang, T.S. 1984. Multi-frame Image Restoration and Registration. Advances in Computer Vision and Image Processing, JAI Press Inc.Google Scholar
- Baker, S. and Kanade, T. 1999. Super-Resolution Optical Flow. CMU-RI-TR-9936.Google Scholar
- Baker, S. and Kanade, T. 2000. Limits on Super-Resolution and How to Break Them. In Proc. Conf. Comp. Vision and Patt. Recog. Google Scholar
- Freeman, W. and Pasztor, E. 1999. Learning low-level vision. In Proc. Int. Conf. Comp. Vision.Google Scholar
- Bonet, J.S.D. 1997. sampling procedure for analysis and synthesis of texture images. In Proceedings of SIGGRAPH, pp. 361–368.Google Scholar
- B. D. Lucas and T. Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proc. 7th Int. Joint Conf. on Art. Intell..Google Scholar
- Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. 1992. Hierarchical Model-Based Motion Estimation. In Proc. European Conf. Comp. Vision, pp. 237–252.Google Scholar
- Simoncelli, E.P. and Adelson, E.H. 1990. Computing Optical Flow Distribution Using Spatio-Temporal Filters. MIT Media Lab Technical Report 165.Google Scholar
- Sha, N. R. and Zakhor, A. 1999. Resolution Enhancement of Color Video Sequences. In IEEE Trans. on IP, Vol. 8, No. 6, June, 1999, pp. 879–885.Google Scholar
- Birchfield, S. Derivation of Kanade-Lucas-Tomasi Tracking Equation. Unpublished, May 1996. http://vision.stanford.edu/birch/klt/.