Skip to main content
Log in

A review of temporal video error concealment techniques and their suitability for HEVC and VVC

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Despite of the recent progresses in reliable and high bandwidth communication, packet loss is still probable and needs special attention in real-time video streaming applications. Congestion and bit error rate, which sometimes are more than the protection capability of the channel codes, are the sources of packet loss in video communication. One common approach to deal with video packet loss is to use error concealment techniques, which estimate the non-received data as close as possible to the actual data. This article reviews the temporal video error concealment methods that have been developed over the past 30 years. The techniques are categorized into 8 groups, and the methods are covered with enough details. The strengths and weaknesses of the 8 groups are also tabulated, and some suggestions for future work and open areas for research are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31

Similar content being viewed by others

Notes

  1. VVC is set to be finalized by mid-2020 [64]

References

  1. Agrafiotis D, Bull R, Canagarajah CN (2006) Enhanced error concealment with mode selection. IEEE Transactions On Circuits and Systems for Video Technology 16(8):960–973

    Article  Google Scholar 

  2. Akbari MT, Granado B (2017) Sparse recovery-based error concealment. IEEE Transactions on Multimedia 19(6):1339–1350

    Article  Google Scholar 

  3. Arthofer F et al. (2016) The future of television. The Boston Consulting Group

  4. Atzori L, De Natale FGB, Perra C (2001) A spatio-temporal concealment technique using boundary matching algorithm and mesh-based warping (BMA-MBW). IEEE Trans Multimedia 3(3):326–338

    Article  Google Scholar 

  5. Belfiore S, Grangetto M, Magli E, Olmo G (2005) Concealment of whole-frame losses for wireless low bit-rate video based on multiframe optical flow estimation. IEEE Transactions on Multimedia 7(2):316–329

    Article  Google Scholar 

  6. Bopardikar S, Hillestad OI And a. Perkis (2005) a temporal error concealment algorithm based on structural alignment for packet video. International Conference on Signal Processing & Communications (SPCOM)

  7. Bormann C, Castellani A, Shelby Z (2012) CoAP: an application protocol for billions of tiny internet nodes. IEEE Internet Comput 16(2):62–67

    Article  Google Scholar 

  8. Bross B, Chen J, Liu S, Wang YK (2020) Versatile video coding (Draft 10). Joint video experts team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11

  9. Brown R (1994) Dishing up full-power DBS. Broadcasting & Cable 124(13):48–50

    Google Scholar 

  10. Carreira J, Ekmekcioglu E, Kondoz A, Assuncao P, Faria S, De Silva V (2014) Selective motion vector redundancies for improved error resilience in HEVC. IEEE International Conference on Image Processing (ICIP):2457–2461

  11. Chang YL, Reznick YA, Chen Z, Cosman PC (2013) Motion compensated error concealment for HEVC based on block-merging and residual energy. International Packet Video Workshop (PV), San Jose

    Book  Google Scholar 

  12. Chen M-J, Chen L-G, Weng R-M (1997) Error concealment of lost motion vectors with overlapped motion compensation. IEEE Trans Circuits Syst Video Technol 7(3):560–563

    Article  Google Scholar 

  13. Chen Y, Yu K, Li J, Li S (2004) An error concealment algorithm for entire frame loss in video transmission. Picture Coding Symposium, San Francisco, USA

    Google Scholar 

  14. Chen M, Chen C, Chi M (2005) Temporal error concealment algorithm by recursive block-matching principle. IEEE Trans Circuits Syst Video Technol 15(11):1385–1393

    Article  Google Scholar 

  15. Chen C, Liu Y, Yang Z, Bu J, Deng X (2008) Multi-frame error concealment for H.264/AVC frames with complexity adaptation. IEEE Trans Consum Electron 54(3):1422–1429

    Article  Google Scholar 

  16. Chen Y, Hu Y, Au OC, Li H, Chen CW (2008) Video error concealment using Spatio-temporal boundary matching and partial differential equation. IEEE Transactions on Multimedia 10(1):2–15

    Article  Google Scholar 

  17. Chen X, Chung YY, Bae C, He X, Yeh WC (2010) An efficient error concealment algorithm for H.264/AVC using regression modeling-based prediction. IEEE Trans Consum Electron 56(4):2694–2701

    Article  Google Scholar 

  18. Jianle Chen, Yan Ye, Seung Hwan Kim (2020) Algorithm description for Versatile Video Coding and Test Model 8 (VTM 8). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11

  19. Chien JT, Li GL, Chen MJ (2010) Effective error concealment algorithm of whole frame loss for H.264 video coding standard by recursive motion vector refinement. IEEE Trans Consum Electron 56(3):1689–1695

    Article  Google Scholar 

  20. Choe G, Nam C, Chu C (2018) An effective temporal error concealment in H.264 video sequences based on scene change detection-PCA model. Multimed Tools Appl 77(24):31953–31967

    Article  Google Scholar 

  21. Chung B, Yim C (2019) Bi-sequential video error concealment method using adaptive Homography-based registration. IEEE Transactions on circuits and systems for video technology 30(6):1535–1549

    Article  Google Scholar 

  22. Feng J, Lo K, Mehropour H, Karbowiak AE (1997) Error concealment for MPEG video transmissions. IEEE Trans Consumer Electron 43(2):183–187

    Article  Google Scholar 

  23. Fleury M, Moiron S, Ghanbari M (2011) Innovations in video error resilience and concealment. Recent Patents Signal Process 1(2):1–11

    Google Scholar 

  24. Fleury M, Altaf M, Moiron S, Qadri N, Ghanbari M (2012) Source coding methods for robust wireless video streaming. Multimedia Networking and Coding:175–207

  25. Ghahremani S, Ghanbari M (2017) Error resilient video transmission in ad hoc networks using layered and multiple description coding. Multimed Tools Appl 76(6):9033–9049

    Article  Google Scholar 

  26. Ghanbari M (2011) Standard codecs: image compression to advanced video coding. IET publication, 3rd edition. Ch 9:265–271

    Google Scholar 

  27. Ghanbari S, Bobber MZ (2002) A cluster-based method for the recovery of lost motion vectors in video coding. 4th IEEE Conference on Mobile and Wireless Communications Networks:9–11

  28. Ghotbou A, Khansari M (2020) Comparing application layer protocols for video transmission in IoT low power lossy networks: an analytic comparison. Wireless Netw doi. https://doi.org/10.1007/s11276-020-02453-6

  29. Global Internet Phenomena Report, Sandvine, September 2019, page 5, “The massive impact of OTT fragmentation”, Retrieved April 1, 2020 from: https://www.sandvine.com/press-releases/sandvine-releases-2019-global-internet-phenomena-report

  30. Hadizadeh H, Bajic I, Cheung G (2013) Video error concealment using a computation efficient low saliency prior. IEEE Transactions on Multimedia 15(8):2099–2113

    Article  Google Scholar 

  31. Haskell P, Messerschmitt D (1992) Resynchronization of motion compensated video affected by ATM cell loss. IEEE Int. Conf. Acoustics, Speech, and Signal Processing:545–548

  32. Herrero R (2019) A comparison of mechanisms for RTC in the context of IoT. Internet Things 8:100110

    Article  Google Scholar 

  33. Herrero R (2020) Analysis of IoT mechanisms for media streaming. Internet Things 9:100168

    Article  Google Scholar 

  34. Ho C-L, Chang L-W (2014) Temporal and spatial error concealment using cooperative game. Proc. ISEEE, Sapporo, Japan, pp 380–384

    Google Scholar 

  35. Hojati S, Kazemi M, Moallem P (2020) Error concealment with parallelogram partitioning of the lost area. Multimed Tools Appl 79:7449–7469

    Article  Google Scholar 

  36. Hsia SC, Hsiao CH (2016) Fast-efficient shape error concealment technique based on block classification. IET Image Process 10(10):693–700

    Article  Google Scholar 

  37. Hsia SC, Cheng SC, Chou SW (2005) Efficient adaptive error concealment technique for video decoding system. IEEE Transactions on Multimedia 7(5):860–868

    Article  Google Scholar 

  38. Huang Z, Cai Q (2018) A video data recovery algorithm in wireless communication networks. IEEE 18th International Conference on Communication Technology:727–731

  39. Huang S-C, Kuo S-Y (2008) Optimization of hybridized error concealment for H.264. IEEE Trans Broadcast 54(3):499–516

    Article  Google Scholar 

  40. Huang YL, Lien HY (2006) Temporal error concealment for MPEG coded video using a self-organizing map. IEEE Trans Consum Electron 52(2):676–681

    Article  Google Scholar 

  41. Hwang MC, Kim JH, Yang HY, Ko SJ, Morales AW (2008) Frame error concealment technique using adaptive inter-mode estimation for H.264/AVC. IEEE Trans Consum Electron 54(1):163–170

    Article  Google Scholar 

  42. Hwang MC, Kim JH, Duong DT, Ko SJ (2008) Hybrid temporal error concealment methods for block-based compressed video transmission. IEEE Trans Broadcast 54(2):198–207

    Article  Google Scholar 

  43. International Telecommunication Union. Information technology - Generic coding of moving pictures and associated audio information: Systems. ITU-T Recommendation H.222.0, approved and In force 2018-08-29

  44. Kazemi M, Shirmohammadi S, Sadeghi KH (2013) A review of multiple description coding techniques for error-resilient video delivery. Multimedia Systems, Springer 20(3):283–309

    Article  Google Scholar 

  45. Kazemi M, Ghanbari M, Shirmohammadi S (2020) The performance of quality metrics in assessing error-concealed video quality. IEEE Trans Image Process 29:5937–5952

    Article  Google Scholar 

  46. Khan E, Lehmann S, Gunji H, Ghanbari M (2004) Iterative error detection and correction of H.263 coded video for wireless networks. IEEE Transactions on Circuits and Systems for Video Technology 14(12):1294–1307

    Article  Google Scholar 

  47. Kim DH, Kwon YJ, Choi KH (2018) Motion-vector refinement for video error concealment using downhill simplex approach. ETRI J 40(2):266–274

    Article  Google Scholar 

  48. Kung WY, Kim CS, Jay Kuo C-C (2006) Spatial and temporal error concealment techniques for video transmission over Noisy channels. IEEE Trans Circuits Syst Video Technol 16(7):789–802

    Article  Google Scholar 

  49. Lam WM, Reibman AR, Liu B (1993) Recovery of lost or erroneously received motion vectors. in Proc IEEE Int Conf Acoust Speech Signal Process:417–420

  50. Lee PJ, Lin ML (2006) Fuzzy logic based temporal error concealment for H.264 video. ETRI J 28(5):574–582

    Article  Google Scholar 

  51. Lee YC, Altunbasak Y, Mersereau RM (2002) Multi-frame error concealment for MPEG-coded video delivery over error-prone networks. IEEE Trans Image Process 11(11):1314–1331

    Article  Google Scholar 

  52. Li Y, Chen R (2017) Motion vector recovery for video error concealment based on the plane fitting. Springer Multimedia Tools and Applications 76(13):14993–15006

    Article  Google Scholar 

  53. Li H, Zhong Y (2013) Motion characteristic differentiated error concealment. Multimed Tools Appl 65(2):297–320

    Article  Google Scholar 

  54. Lie WN, Gao ZW (2006) Video error concealment by integrating greedy suboptimization and Kalman filtering techniques. IEEE Transactions on Circuits and Systems for Video Technology 16(8):982–992

    Article  Google Scholar 

  55. Lie W, Lee C, Yeh C, Gao Z (2014) Motion vector recovery for video error concealment by using iterative dynamic programming optimization. IEEE Transactions on Multimedia 16(1):216–227

    Article  Google Scholar 

  56. Lim CP, Tan EAW, Ghanbari M, Ghanbari S (1999) Cell loss concealment and packetization in packet video. International Journal of Imaging Systems and Technology, Special Issue on Image and Video Compression:54–58

  57. Lin T, Yang N, Syu R (2013) Error concealment algorithm for HEVC coded video using block partition decisions. IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC)

    Book  Google Scholar 

  58. Lin TL, Chen WC, Lai CK (2013) Recovery of lost motion vectors using encoded residual signals. IEEE Trans Broadcast 59(4):705–716

    Article  Google Scholar 

  59. Lin TL, Ding TL, Yang NC, Wu PY, Tung KH, Lai CK, Chang TE (2016) Video motion vector recovery method using decoding partition information. J Disp Technol 12(11):1451–1463

    Article  Google Scholar 

  60. Lin T-L, Ding T-L, Fan C-Y, Chen W-C (2017) Error concealment algorithm based on sparse optimization. Springer Multimedia Tools and Applications 76(1):397–413

    Article  Google Scholar 

  61. Liu X, Zhai D, Zhou J, Wang S, Zhao D, Guo H (2016) Sparsity-based image error concealment via adaptive dual dictionary learning and regularization. IEEE Trans Image Process 26(2):782–796

    Article  MathSciNet  MATH  Google Scholar 

  62. Ma M, Au O, Chan SG, Sun M (2010) Edge-directed error concealment. IEEE Transactions on Circuits and Systems for Video Technology 20(3):382–394

    Article  Google Scholar 

  63. Nam C, Chu C, Kim T, Han S (2020) A novel motion recovery using temporal and spatial correlation for a fast temporal error concealment over H.264 video sequences. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-019-08176-x

  64. Nauman A, Qadri YA, Amjad M, Zikria YB, Afzal MK, Kim SW (2020) Multimedia internet of things: a comprehensive survey. IEEE ACCESS 8:8202–8250

    Article  Google Scholar 

  65. Oztas B, Pourazad MT, Nasiopoulos P, Leung VCM (2012) A study on the HEVC performance over lossy networks. 19th IEEE International Conference on Electronics, Circuits and Systems (ICECS):785–788

  66. Peng Y-T, Cosman PC (2014) Weighted boundary matching error concealment for HEVC using block partition decisions. 48th Asilomar Conference on Signals, Systems and Computers:921–925

  67. Peng Q, Yang T, Zhu C (2002) Block-based temporal error concealment for video packet using motion vector extrapolation. IEEE Int Conf Commun Circuits Syst West Sino Expo 1:10–14

    Google Scholar 

  68. Pereira R, Pereira EG (2016) Video streaming: overview and challenges in the internet of things. Pervasive Computing, Next Generation Platforms for Intelligent Data Collection, In, pp 417–444

    Google Scholar 

  69. Persson D, Eriksson T (2009) Mixture model- and least squares-based packet video error concealment. IEEE Trans Image Process 18(5):1048–1054

    Article  MathSciNet  MATH  Google Scholar 

  70. Persson D, Eriksson T, Hedelin P (2008) Packet video error concealment with Gaussian mixture models. IEEE Trans Image Process 17(2):145–154

    Article  MathSciNet  Google Scholar 

  71. Pongpadpinit W, Pearmain A (2006) Recovery of motion vectors for error concealment based on an edge-detection approach. IEE Proc-Vis Image Signal Process 153(1):63–69

    Article  Google Scholar 

  72. Punjabi SA, Katsaggelos AK (2018) Video error concealment using deep neural networks. In Proceedings of the IEEE International Conference on Image Processing, Athens, Greece, pp 380–384

    Google Scholar 

  73. Pyun J-Y, Lee J-S, Jeong J-W, Jeong J-H, Ko S-J (2003) Robust error concealment for visual communications in burst-packet-loss networks. IEEE Trans Consumer Electron 49(4):1013–1019

    Article  Google Scholar 

  74. Qian X, Liu G, Wang H (2009) Recovering connected error region based on adaptive error concealment order determination. IEEE Trans Multimedia 11(4):683–695

    Article  Google Scholar 

  75. Radmehr AG (2016) Error concealment via particle filter by Gaussian mixture modeling of motion vectors for H.264/AVC. SIViP 10(2):311–318

    Article  Google Scholar 

  76. Rahman WU, Choi Y-S, Chung K (2019) Performance evaluation of video streaming application over CoAP in IoT. IEEE Access 7:39852–39861

    Article  Google Scholar 

  77. Salama P, Shroff NB, Delp EJ (2000) Error concealment in MPEG video streams over ATM networks. IEEE Journal on Selected Areas in Communications 18(6):1129–1144

    Article  Google Scholar 

  78. Schuster GM, Katsaggelos AK (2006) Motion compensated shape error concealment. IEEE Trans Image Process 15(2):501–510

    Article  Google Scholar 

  79. Seth K, Kamakoti V, Srinivasan S (2010) Efficient motion vector recovery algorithm for H.264 using B-spline approximation. IEEE Trans Broadcast 56(4):467–480

    Article  Google Scholar 

  80. Shanableh T, Ghanbari M (2003) Loss concealment using b-pictures motion information. IEEE Trans Multimedia 5(2):257–266

    Article  Google Scholar 

  81. Shirani S, Kossentini F, Ward R (2000) A concealment method for video communications in an error-prone environment. IEEE Journal On Selected Areas In Communications 18(6):1122–1128

    Article  Google Scholar 

  82. Shirani S, Erol B, Kossentin F (2000) A concealment method for shape information in MPEG-4 coded video sequences. IEEE Transactions on Multimedia 2(3):185–190

    Article  Google Scholar 

  83. Song K, Chung T, Kim Y, Oh Y, Kim CS (2007) Error concealment of H.264/AVC video frames for Mobile video broadcasting. IEEE Trans Consum Electron 53(2):704–711

    Article  Google Scholar 

  84. Suh JW, Ho YS (2002) Error Concealment Techniques for Digital TV. IEEE Trans Broadcast 48(4):299–306

    Article  Google Scholar 

  85. Sullivan GJ, Ohm J-R, Han W-J, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22(12):1648–1667

    Article  Google Scholar 

  86. Topiwala P, Krishnan M, Dai W (2019) Performance comparison of VVC, AV1 and EVC. Proc. SPIE Vol. 11137, applications of digital image processing XLII, document 1113715, San Diego, California. USA

  87. Tsekeridou S, Pitas I (2000) MPEG-2 error concealment based on block-matching principles. IEEE Trans Circuits Syst Video Technol 10(4):646–658

    Article  Google Scholar 

  88. Turaga DS, Chen T (2002) Model-based error concealment for wireless video. IEEE Transactions on Circuits and Systems for Video Technology 12(6):483–495

    Article  Google Scholar 

  89. Usman M, He X, Xu M, Lam K (2015) Survey of error concealment techniques: research directions and open issues. IEEE Picture Coding Symposium:233–238

  90. Usman M, He X, Lam KM, Xu M, Bokhari SMM, Chen J (2016) Frame interpolation for cloud-based Mobile video streaming. IEEE Transactions on Multimedia 18(5):831–839

    Article  Google Scholar 

  91. Wada M (1989) Selective recovery of video packet loss using error concealment. IEEE Journal on Selected Areas in Communications 7(5):807–814

    Article  Google Scholar 

  92. Wang Y, Zhu QF (1998) Error control and concealment for video communication: a review. Proc IEEE 86(5):974–997

    Article  Google Scholar 

  93. Wang Y, Wenger S, Wen J, Katsaggelos AK (2000) Review of error resilient coding techniques for real-time video communications. IEEE signal process. Mag. 1(4):61–82

    Google Scholar 

  94. Wang Y, Hannuksela MM, Varsa V, Hourunranta A, Gabbouj M (2002) The error concealment feature in the H.26L test model. In Proc. Int. Conf. Image Processing 2:729–732

    Google Scholar 

  95. Wang Y, Reibman A, Lin S (2005) Multiple description coding for video delivery. Proc IEEE 93(1):57–70

    Article  Google Scholar 

  96. Wu Z, Boyce J (2006) An error concealment scheme for entire frame losses based on H.264/AVC. in Proc. ISCAS’06, Island of Kos, Greece: 4463–4466

  97. Wu J, Liu X, Yoo KY (2008) A temporal error concealment method for H.264/AVC using motion vector recovery. IEEE Trans Consum Electron 54(4):1880–1885

    Article  Google Scholar 

  98. Xiang J, Xu C, Yan Q, Peng XW (2019) Generative adversarial networks based error concealment for low resolution video. IEEE International Conference On Acoustics, Speech And Signal Processing (ICASSP)

    Book  Google Scholar 

  99. Xu Y, Zhou Y (2004) H.264 video communication based refined error concealment schemes. IEEE Trans Consum Electron 50(4):1135–1141

    Article  Google Scholar 

  100. Xu Y, Zhou Y (2008) Adaptive temporal error concealment scheme for H.264/AVC video decoder. IEEE Trans Consum Electron 54(4):1846–1851

    Article  Google Scholar 

  101. Yan B, Gharavi H (2009) A hybrid frame concealment algorithm for H.264/AVC. IEEE Trans Image Process 19(1):98–107

    Article  MathSciNet  MATH  Google Scholar 

  102. Yu W, Sun H G. He and Z. Zhang (2017) a multi-step temporal error concealment method. International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)

  103. Zhang Y, Ma KK (2003) Error concealment for video transmission with dual multiscale Markov random field modeling. IEEE Trans Image Process 12(2):236–242

    Article  Google Scholar 

  104. Zhang J, Arnold JF, Frater MR (2000) A cell-loss concealment technique for MPEG-2 coded video. IEEE Transactions on Circuits and Systems for Video Technology 10(4):659–665

    Article  Google Scholar 

  105. Zhang Y, Xiang X, Zhao D, Ma S, Gao W (2012) Packet video error concealment with auto regressive model. IEEE Trans Circuits Syst Video Technol 22(1):12–27

    Article  Google Scholar 

  106. Zhao C, Ma S, Zhang J, Gao W (2013) A highly effective error concealment method for whole frame loss. IEEE international symposium on circuits and systems (ISCAS), Beijing, China

  107. Zheng J, Chau LP (2004) A temporal error concealment algorithm for H.264/AVC using Lagrange interpolation. In Int. Symp. Circuits Syst (ISCAS):133–136

  108. Zheng J, Chau LP (2004) Error-concealment algorithm for H.26L using first-order plane estimation. IEEE Transactions on Multimedia 6(6):801–805

    Article  Google Scholar 

  109. Zheng J, Chau L-P (2005) Efficient motion vector recovery algorithm for H.264 based on a polynomial model. IEEE Trans Multimedia 7(3):507–513

    Article  Google Scholar 

  110. Zhiliang X, Zhiheng Z, Weijun L (2009) Fuzzy optical flow using in error concealment. International Conference on Fuzzy Systems and Knowledge Discovery

  111. Zhou J, Yan B, Gharavi H (2011) Efficient motion vector interpolation for error concealment of H.264/AVC. IEEE Trans Broadcast 57(1):75–80

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mohammad Kazemi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kazemi, M., Ghanbari, M. & Shirmohammadi, S. A review of temporal video error concealment techniques and their suitability for HEVC and VVC. Multimed Tools Appl 80, 12685–12730 (2021). https://doi.org/10.1007/s11042-020-10333-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10333-6

Keywords

Navigation