Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion

  • Jiale He
  • Gaobo Yang
  • Jingyu Song
  • Xiangling Ding
  • Ran Li
Original Research Paper


Motion-compensated frame-rate up-conversion (MC-FRUC) often exploits either bilateral motion estimation (ME) or unidirectional ME with a fixed block size, which constrains the perceptual quality of up-converted video. In this paper, an advanced MC-FRUC approach is proposed by exploiting hierarchical prediction-based motion vector refinement. To reduce block mismatching in texture regions and color areas, an adaptive multi-layered block matching criterion is designed to extract color and edge information, which is integrated with motion information as constraint term. A hierarchical prediction-based motion vector refinement approach is proposed to obtain more accurate and dense motion vector fields (MVFs). To eliminate the outliers of MVFs, a robust dual-weighted motion vector smoothing scheme is adopted by using both spatial correlation and reliability of neighboring blocks. Experimental results show that the proposed approach has low computational complexity and outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames.


Frame-rate up-conversion Adaptive multi-layered block matching criterion Hierarchical prediction-based motion vector refinement Robust dual-weighted motion vector smoothing 



This work is supported in part by the National Natural Science Foundation of China (61572183, 61379143, 61232016, U1405254).


  1. 1.
    Dai, W., Shen, Y., Tang, X., et al.: Sparse representation with spatio-temporal online dictionary learning for promising video coding. IEEE Trans. Image Process. 25(10), 4580–4595 (2016)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Kaviani, H.R., Shirani, S.: Frame rate upconversion using optical flow and patch-based reconstruction. IEEE Trans. Circuits Syst. Video Technol. 26(9), 1581–1594 (2016)CrossRefGoogle Scholar
  3. 3.
    Li, R., Liu, H., Liu, Z., et al.: Motion-compensated frame interpolation using patch-based sparseland model. Signal Process. Image Commun. 54, 36–48 (2017)CrossRefGoogle Scholar
  4. 4.
    Dikbas, S., Arici, T., Altunbasak, Y.: Fast motion estimation with interpolation-free sub-sample accuracy. IEEE Trans. Circuits Syst. Video Technol. 20(7), 1047–1051 (2010)CrossRefGoogle Scholar
  5. 5.
    Wang, D., Vincent, A., Blanchfield, P., et al.: Motion-compensated frame rate up-conversion—part II: new algorithms for frame interpolation. IEEE Trans. Broadcast. 56(2), 142–149 (2010)CrossRefGoogle Scholar
  6. 6.
    Wang, C., Zhang, L., He, Y., et al.: Frame rate up-conversion using trilateral filtering. IEEE Trans. Circuits Syst. Video Technol. 20(6), 886–893 (2010)CrossRefGoogle Scholar
  7. 7.
    Kang, S.J., Cho, K.R., Kim, Y.H.: Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Trans. Consum. Electron. 53(4), 1759–1767 (2007)CrossRefGoogle Scholar
  8. 8.
    Li, R., Liu, H., Chen, J., et al.: Wavelet pyramid based multi-resolution bilateral motion estimation for frame rate up-conversion. IEICE Trans. Inf. Syst. 99(1), 208–218 (2016)CrossRefGoogle Scholar
  9. 9.
    Choi, B.D., Han, J.W., Kim, C.S., et al.: Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Trans. Circuits Syst. Video Technol. 17(4), 407–416 (2007)CrossRefGoogle Scholar
  10. 10.
    Kang, S.J., Yoo, S., Kim, Y.H.: Dual motion estimation for frame rate up-conversion. IEEE Trans. Circuits Syst. Video Technol. 20(12), 1909–1914 (2010)CrossRefGoogle Scholar
  11. 11.
    Tsai, T.H., Lin, H.Y.: High visual quality particle based frame rate up conversion with acceleration assisted motion trajectory calibration. J. Disp. Technol. 8(6), 341–351 (2012)CrossRefGoogle Scholar
  12. 12.
    Zhai, J., Yu, K., Li, J., et al.: A low complexity motion compensated frame interpolation method. In: IEEE International Symposium on Circuits and Systems (ISCAS), pp. 4927–4930. IEEE (2005)Google Scholar
  13. 13.
    Yoo, D.G., Kang, S.J., Kim, Y.H.: Direction-select motion estimation for motion-compensated frame rate up-conversion. J. Disp. Technol. 9(10), 840–850 (2013)CrossRefGoogle Scholar
  14. 14.
    Kim, U.S., Sunwoo, M.H.: New frame rate up-conversion algorithms with low computational complexity. IEEE Trans. Circuits Syst. Video Technol. 24(3), 384–393 (2014)CrossRefGoogle Scholar
  15. 15.
    Jeong, S.G., Lee, C., Kim, C.S.: Exemplar-based frame rate up-conversion with congruent segmentation. In: IEEE International Conference on Image Processing (ICIP), pp. 845–848 (2012)Google Scholar
  16. 16.
    Benois-Pineau, J., Nicolas, H.: A new method for region-based depth ordering in a video sequence: application to frame interpolation. J. Vis. Commun. Image Represent. 13(3), 363–385 (2002)CrossRefGoogle Scholar
  17. 17.
    Orchard, M.T., Sullivan, G.J.: Overlapped block motion compensation: an estimation-theoretic approach. IEEE Trans. Image Process. 3(5), 693–699 (1994)CrossRefGoogle Scholar
  18. 18.
    Zhang, Y., Zhao, D., Ma, S., et al.: A motion-aligned auto-regressive model for frame rate up conversion. IEEE Trans. Image Process. 19(5), 1248–1258 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Li, R., Gan, Z., Cui, Z., et al.: Multi-channel mixed-pattern based frame rate up-conversion using spatio-temporal motion vector refinement and dual-weighted overlapped block motion compensation. J. Disp. Technol. 10(12), 1010–1023 (2014)CrossRefGoogle Scholar
  20. 20.
    Bayer, B.E.: Color imaging array: U.S. Patent 3,971,065[P]. 1976-7-20Google Scholar
  21. 21.
    Castagno, R., Haavisto, P., Ramponi, G.: A method for motion adaptive frame rate up-conversion. IEEE Trans. Circuits Syst. Video Technol. 6(5), 436–446 (1996)CrossRefGoogle Scholar
  22. 22.
    Liu, H., Xiong, R., Zhao, D., et al.: Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Trans. Circuits Syst. Video Technol. 22(8), 1188–1198 (2012)CrossRefGoogle Scholar
  23. 23.
    Choi, B.T., Lee, S.H., Park, Y.J., et al.: Frame rate up-conversion using the wavelet transform. In: IEEE International Conference on Consumer Electronics (ICCE), pp. 172–173 (2000)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Jiale He
    • 1
  • Gaobo Yang
    • 1
  • Jingyu Song
    • 2
  • Xiangling Ding
    • 1
  • Ran Li
    • 3
  1. 1.School of Information Science and EngineeringHunan UniversityChangshaChina
  2. 2.System Engineering Research Institute (SERI)BeijingChina
  3. 3.School of Computer and Information TechnologyXinyang Normal UniversityXinyangChina

Personalised recommendations