Abstract
Despite of the recent progresses in reliable and high bandwidth communication, packet loss is still probable and needs special attention in real-time video streaming applications. Congestion and bit error rate, which sometimes are more than the protection capability of the channel codes, are the sources of packet loss in video communication. One common approach to deal with video packet loss is to use error concealment techniques, which estimate the non-received data as close as possible to the actual data. This article reviews the temporal video error concealment methods that have been developed over the past 30 years. The techniques are categorized into 8 groups, and the methods are covered with enough details. The strengths and weaknesses of the 8 groups are also tabulated, and some suggestions for future work and open areas for research are provided.
Similar content being viewed by others
Notes
VVC is set to be finalized by mid-2020 [64]
References
Agrafiotis D, Bull R, Canagarajah CN (2006) Enhanced error concealment with mode selection. IEEE Transactions On Circuits and Systems for Video Technology 16(8):960–973
Akbari MT, Granado B (2017) Sparse recovery-based error concealment. IEEE Transactions on Multimedia 19(6):1339–1350
Arthofer F et al. (2016) The future of television. The Boston Consulting Group
Atzori L, De Natale FGB, Perra C (2001) A spatio-temporal concealment technique using boundary matching algorithm and mesh-based warping (BMA-MBW). IEEE Trans Multimedia 3(3):326–338
Belfiore S, Grangetto M, Magli E, Olmo G (2005) Concealment of whole-frame losses for wireless low bit-rate video based on multiframe optical flow estimation. IEEE Transactions on Multimedia 7(2):316–329
Bopardikar S, Hillestad OI And a. Perkis (2005) a temporal error concealment algorithm based on structural alignment for packet video. International Conference on Signal Processing & Communications (SPCOM)
Bormann C, Castellani A, Shelby Z (2012) CoAP: an application protocol for billions of tiny internet nodes. IEEE Internet Comput 16(2):62–67
Bross B, Chen J, Liu S, Wang YK (2020) Versatile video coding (Draft 10). Joint video experts team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11
Brown R (1994) Dishing up full-power DBS. Broadcasting & Cable 124(13):48–50
Carreira J, Ekmekcioglu E, Kondoz A, Assuncao P, Faria S, De Silva V (2014) Selective motion vector redundancies for improved error resilience in HEVC. IEEE International Conference on Image Processing (ICIP):2457–2461
Chang YL, Reznick YA, Chen Z, Cosman PC (2013) Motion compensated error concealment for HEVC based on block-merging and residual energy. International Packet Video Workshop (PV), San Jose
Chen M-J, Chen L-G, Weng R-M (1997) Error concealment of lost motion vectors with overlapped motion compensation. IEEE Trans Circuits Syst Video Technol 7(3):560–563
Chen Y, Yu K, Li J, Li S (2004) An error concealment algorithm for entire frame loss in video transmission. Picture Coding Symposium, San Francisco, USA
Chen M, Chen C, Chi M (2005) Temporal error concealment algorithm by recursive block-matching principle. IEEE Trans Circuits Syst Video Technol 15(11):1385–1393
Chen C, Liu Y, Yang Z, Bu J, Deng X (2008) Multi-frame error concealment for H.264/AVC frames with complexity adaptation. IEEE Trans Consum Electron 54(3):1422–1429
Chen Y, Hu Y, Au OC, Li H, Chen CW (2008) Video error concealment using Spatio-temporal boundary matching and partial differential equation. IEEE Transactions on Multimedia 10(1):2–15
Chen X, Chung YY, Bae C, He X, Yeh WC (2010) An efficient error concealment algorithm for H.264/AVC using regression modeling-based prediction. IEEE Trans Consum Electron 56(4):2694–2701
Jianle Chen, Yan Ye, Seung Hwan Kim (2020) Algorithm description for Versatile Video Coding and Test Model 8 (VTM 8). Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11
Chien JT, Li GL, Chen MJ (2010) Effective error concealment algorithm of whole frame loss for H.264 video coding standard by recursive motion vector refinement. IEEE Trans Consum Electron 56(3):1689–1695
Choe G, Nam C, Chu C (2018) An effective temporal error concealment in H.264 video sequences based on scene change detection-PCA model. Multimed Tools Appl 77(24):31953–31967
Chung B, Yim C (2019) Bi-sequential video error concealment method using adaptive Homography-based registration. IEEE Transactions on circuits and systems for video technology 30(6):1535–1549
Feng J, Lo K, Mehropour H, Karbowiak AE (1997) Error concealment for MPEG video transmissions. IEEE Trans Consumer Electron 43(2):183–187
Fleury M, Moiron S, Ghanbari M (2011) Innovations in video error resilience and concealment. Recent Patents Signal Process 1(2):1–11
Fleury M, Altaf M, Moiron S, Qadri N, Ghanbari M (2012) Source coding methods for robust wireless video streaming. Multimedia Networking and Coding:175–207
Ghahremani S, Ghanbari M (2017) Error resilient video transmission in ad hoc networks using layered and multiple description coding. Multimed Tools Appl 76(6):9033–9049
Ghanbari M (2011) Standard codecs: image compression to advanced video coding. IET publication, 3rd edition. Ch 9:265–271
Ghanbari S, Bobber MZ (2002) A cluster-based method for the recovery of lost motion vectors in video coding. 4th IEEE Conference on Mobile and Wireless Communications Networks:9–11
Ghotbou A, Khansari M (2020) Comparing application layer protocols for video transmission in IoT low power lossy networks: an analytic comparison. Wireless Netw doi. https://doi.org/10.1007/s11276-020-02453-6
Global Internet Phenomena Report, Sandvine, September 2019, page 5, “The massive impact of OTT fragmentation”, Retrieved April 1, 2020 from: https://www.sandvine.com/press-releases/sandvine-releases-2019-global-internet-phenomena-report
Hadizadeh H, Bajic I, Cheung G (2013) Video error concealment using a computation efficient low saliency prior. IEEE Transactions on Multimedia 15(8):2099–2113
Haskell P, Messerschmitt D (1992) Resynchronization of motion compensated video affected by ATM cell loss. IEEE Int. Conf. Acoustics, Speech, and Signal Processing:545–548
Herrero R (2019) A comparison of mechanisms for RTC in the context of IoT. Internet Things 8:100110
Herrero R (2020) Analysis of IoT mechanisms for media streaming. Internet Things 9:100168
Ho C-L, Chang L-W (2014) Temporal and spatial error concealment using cooperative game. Proc. ISEEE, Sapporo, Japan, pp 380–384
Hojati S, Kazemi M, Moallem P (2020) Error concealment with parallelogram partitioning of the lost area. Multimed Tools Appl 79:7449–7469
Hsia SC, Hsiao CH (2016) Fast-efficient shape error concealment technique based on block classification. IET Image Process 10(10):693–700
Hsia SC, Cheng SC, Chou SW (2005) Efficient adaptive error concealment technique for video decoding system. IEEE Transactions on Multimedia 7(5):860–868
Huang Z, Cai Q (2018) A video data recovery algorithm in wireless communication networks. IEEE 18th International Conference on Communication Technology:727–731
Huang S-C, Kuo S-Y (2008) Optimization of hybridized error concealment for H.264. IEEE Trans Broadcast 54(3):499–516
Huang YL, Lien HY (2006) Temporal error concealment for MPEG coded video using a self-organizing map. IEEE Trans Consum Electron 52(2):676–681
Hwang MC, Kim JH, Yang HY, Ko SJ, Morales AW (2008) Frame error concealment technique using adaptive inter-mode estimation for H.264/AVC. IEEE Trans Consum Electron 54(1):163–170
Hwang MC, Kim JH, Duong DT, Ko SJ (2008) Hybrid temporal error concealment methods for block-based compressed video transmission. IEEE Trans Broadcast 54(2):198–207
International Telecommunication Union. Information technology - Generic coding of moving pictures and associated audio information: Systems. ITU-T Recommendation H.222.0, approved and In force 2018-08-29
Kazemi M, Shirmohammadi S, Sadeghi KH (2013) A review of multiple description coding techniques for error-resilient video delivery. Multimedia Systems, Springer 20(3):283–309
Kazemi M, Ghanbari M, Shirmohammadi S (2020) The performance of quality metrics in assessing error-concealed video quality. IEEE Trans Image Process 29:5937–5952
Khan E, Lehmann S, Gunji H, Ghanbari M (2004) Iterative error detection and correction of H.263 coded video for wireless networks. IEEE Transactions on Circuits and Systems for Video Technology 14(12):1294–1307
Kim DH, Kwon YJ, Choi KH (2018) Motion-vector refinement for video error concealment using downhill simplex approach. ETRI J 40(2):266–274
Kung WY, Kim CS, Jay Kuo C-C (2006) Spatial and temporal error concealment techniques for video transmission over Noisy channels. IEEE Trans Circuits Syst Video Technol 16(7):789–802
Lam WM, Reibman AR, Liu B (1993) Recovery of lost or erroneously received motion vectors. in Proc IEEE Int Conf Acoust Speech Signal Process:417–420
Lee PJ, Lin ML (2006) Fuzzy logic based temporal error concealment for H.264 video. ETRI J 28(5):574–582
Lee YC, Altunbasak Y, Mersereau RM (2002) Multi-frame error concealment for MPEG-coded video delivery over error-prone networks. IEEE Trans Image Process 11(11):1314–1331
Li Y, Chen R (2017) Motion vector recovery for video error concealment based on the plane fitting. Springer Multimedia Tools and Applications 76(13):14993–15006
Li H, Zhong Y (2013) Motion characteristic differentiated error concealment. Multimed Tools Appl 65(2):297–320
Lie WN, Gao ZW (2006) Video error concealment by integrating greedy suboptimization and Kalman filtering techniques. IEEE Transactions on Circuits and Systems for Video Technology 16(8):982–992
Lie W, Lee C, Yeh C, Gao Z (2014) Motion vector recovery for video error concealment by using iterative dynamic programming optimization. IEEE Transactions on Multimedia 16(1):216–227
Lim CP, Tan EAW, Ghanbari M, Ghanbari S (1999) Cell loss concealment and packetization in packet video. International Journal of Imaging Systems and Technology, Special Issue on Image and Video Compression:54–58
Lin T, Yang N, Syu R (2013) Error concealment algorithm for HEVC coded video using block partition decisions. IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC)
Lin TL, Chen WC, Lai CK (2013) Recovery of lost motion vectors using encoded residual signals. IEEE Trans Broadcast 59(4):705–716
Lin TL, Ding TL, Yang NC, Wu PY, Tung KH, Lai CK, Chang TE (2016) Video motion vector recovery method using decoding partition information. J Disp Technol 12(11):1451–1463
Lin T-L, Ding T-L, Fan C-Y, Chen W-C (2017) Error concealment algorithm based on sparse optimization. Springer Multimedia Tools and Applications 76(1):397–413
Liu X, Zhai D, Zhou J, Wang S, Zhao D, Guo H (2016) Sparsity-based image error concealment via adaptive dual dictionary learning and regularization. IEEE Trans Image Process 26(2):782–796
Ma M, Au O, Chan SG, Sun M (2010) Edge-directed error concealment. IEEE Transactions on Circuits and Systems for Video Technology 20(3):382–394
Nam C, Chu C, Kim T, Han S (2020) A novel motion recovery using temporal and spatial correlation for a fast temporal error concealment over H.264 video sequences. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-019-08176-x
Nauman A, Qadri YA, Amjad M, Zikria YB, Afzal MK, Kim SW (2020) Multimedia internet of things: a comprehensive survey. IEEE ACCESS 8:8202–8250
Oztas B, Pourazad MT, Nasiopoulos P, Leung VCM (2012) A study on the HEVC performance over lossy networks. 19th IEEE International Conference on Electronics, Circuits and Systems (ICECS):785–788
Peng Y-T, Cosman PC (2014) Weighted boundary matching error concealment for HEVC using block partition decisions. 48th Asilomar Conference on Signals, Systems and Computers:921–925
Peng Q, Yang T, Zhu C (2002) Block-based temporal error concealment for video packet using motion vector extrapolation. IEEE Int Conf Commun Circuits Syst West Sino Expo 1:10–14
Pereira R, Pereira EG (2016) Video streaming: overview and challenges in the internet of things. Pervasive Computing, Next Generation Platforms for Intelligent Data Collection, In, pp 417–444
Persson D, Eriksson T (2009) Mixture model- and least squares-based packet video error concealment. IEEE Trans Image Process 18(5):1048–1054
Persson D, Eriksson T, Hedelin P (2008) Packet video error concealment with Gaussian mixture models. IEEE Trans Image Process 17(2):145–154
Pongpadpinit W, Pearmain A (2006) Recovery of motion vectors for error concealment based on an edge-detection approach. IEE Proc-Vis Image Signal Process 153(1):63–69
Punjabi SA, Katsaggelos AK (2018) Video error concealment using deep neural networks. In Proceedings of the IEEE International Conference on Image Processing, Athens, Greece, pp 380–384
Pyun J-Y, Lee J-S, Jeong J-W, Jeong J-H, Ko S-J (2003) Robust error concealment for visual communications in burst-packet-loss networks. IEEE Trans Consumer Electron 49(4):1013–1019
Qian X, Liu G, Wang H (2009) Recovering connected error region based on adaptive error concealment order determination. IEEE Trans Multimedia 11(4):683–695
Radmehr AG (2016) Error concealment via particle filter by Gaussian mixture modeling of motion vectors for H.264/AVC. SIViP 10(2):311–318
Rahman WU, Choi Y-S, Chung K (2019) Performance evaluation of video streaming application over CoAP in IoT. IEEE Access 7:39852–39861
Salama P, Shroff NB, Delp EJ (2000) Error concealment in MPEG video streams over ATM networks. IEEE Journal on Selected Areas in Communications 18(6):1129–1144
Schuster GM, Katsaggelos AK (2006) Motion compensated shape error concealment. IEEE Trans Image Process 15(2):501–510
Seth K, Kamakoti V, Srinivasan S (2010) Efficient motion vector recovery algorithm for H.264 using B-spline approximation. IEEE Trans Broadcast 56(4):467–480
Shanableh T, Ghanbari M (2003) Loss concealment using b-pictures motion information. IEEE Trans Multimedia 5(2):257–266
Shirani S, Kossentini F, Ward R (2000) A concealment method for video communications in an error-prone environment. IEEE Journal On Selected Areas In Communications 18(6):1122–1128
Shirani S, Erol B, Kossentin F (2000) A concealment method for shape information in MPEG-4 coded video sequences. IEEE Transactions on Multimedia 2(3):185–190
Song K, Chung T, Kim Y, Oh Y, Kim CS (2007) Error concealment of H.264/AVC video frames for Mobile video broadcasting. IEEE Trans Consum Electron 53(2):704–711
Suh JW, Ho YS (2002) Error Concealment Techniques for Digital TV. IEEE Trans Broadcast 48(4):299–306
Sullivan GJ, Ohm J-R, Han W-J, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22(12):1648–1667
Topiwala P, Krishnan M, Dai W (2019) Performance comparison of VVC, AV1 and EVC. Proc. SPIE Vol. 11137, applications of digital image processing XLII, document 1113715, San Diego, California. USA
Tsekeridou S, Pitas I (2000) MPEG-2 error concealment based on block-matching principles. IEEE Trans Circuits Syst Video Technol 10(4):646–658
Turaga DS, Chen T (2002) Model-based error concealment for wireless video. IEEE Transactions on Circuits and Systems for Video Technology 12(6):483–495
Usman M, He X, Xu M, Lam K (2015) Survey of error concealment techniques: research directions and open issues. IEEE Picture Coding Symposium:233–238
Usman M, He X, Lam KM, Xu M, Bokhari SMM, Chen J (2016) Frame interpolation for cloud-based Mobile video streaming. IEEE Transactions on Multimedia 18(5):831–839
Wada M (1989) Selective recovery of video packet loss using error concealment. IEEE Journal on Selected Areas in Communications 7(5):807–814
Wang Y, Zhu QF (1998) Error control and concealment for video communication: a review. Proc IEEE 86(5):974–997
Wang Y, Wenger S, Wen J, Katsaggelos AK (2000) Review of error resilient coding techniques for real-time video communications. IEEE signal process. Mag. 1(4):61–82
Wang Y, Hannuksela MM, Varsa V, Hourunranta A, Gabbouj M (2002) The error concealment feature in the H.26L test model. In Proc. Int. Conf. Image Processing 2:729–732
Wang Y, Reibman A, Lin S (2005) Multiple description coding for video delivery. Proc IEEE 93(1):57–70
Wu Z, Boyce J (2006) An error concealment scheme for entire frame losses based on H.264/AVC. in Proc. ISCAS’06, Island of Kos, Greece: 4463–4466
Wu J, Liu X, Yoo KY (2008) A temporal error concealment method for H.264/AVC using motion vector recovery. IEEE Trans Consum Electron 54(4):1880–1885
Xiang J, Xu C, Yan Q, Peng XW (2019) Generative adversarial networks based error concealment for low resolution video. IEEE International Conference On Acoustics, Speech And Signal Processing (ICASSP)
Xu Y, Zhou Y (2004) H.264 video communication based refined error concealment schemes. IEEE Trans Consum Electron 50(4):1135–1141
Xu Y, Zhou Y (2008) Adaptive temporal error concealment scheme for H.264/AVC video decoder. IEEE Trans Consum Electron 54(4):1846–1851
Yan B, Gharavi H (2009) A hybrid frame concealment algorithm for H.264/AVC. IEEE Trans Image Process 19(1):98–107
Yu W, Sun H G. He and Z. Zhang (2017) a multi-step temporal error concealment method. International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)
Zhang Y, Ma KK (2003) Error concealment for video transmission with dual multiscale Markov random field modeling. IEEE Trans Image Process 12(2):236–242
Zhang J, Arnold JF, Frater MR (2000) A cell-loss concealment technique for MPEG-2 coded video. IEEE Transactions on Circuits and Systems for Video Technology 10(4):659–665
Zhang Y, Xiang X, Zhao D, Ma S, Gao W (2012) Packet video error concealment with auto regressive model. IEEE Trans Circuits Syst Video Technol 22(1):12–27
Zhao C, Ma S, Zhang J, Gao W (2013) A highly effective error concealment method for whole frame loss. IEEE international symposium on circuits and systems (ISCAS), Beijing, China
Zheng J, Chau LP (2004) A temporal error concealment algorithm for H.264/AVC using Lagrange interpolation. In Int. Symp. Circuits Syst (ISCAS):133–136
Zheng J, Chau LP (2004) Error-concealment algorithm for H.26L using first-order plane estimation. IEEE Transactions on Multimedia 6(6):801–805
Zheng J, Chau L-P (2005) Efficient motion vector recovery algorithm for H.264 based on a polynomial model. IEEE Trans Multimedia 7(3):507–513
Zhiliang X, Zhiheng Z, Weijun L (2009) Fuzzy optical flow using in error concealment. International Conference on Fuzzy Systems and Knowledge Discovery
Zhou J, Yan B, Gharavi H (2011) Efficient motion vector interpolation for error concealment of H.264/AVC. IEEE Trans Broadcast 57(1):75–80
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kazemi, M., Ghanbari, M. & Shirmohammadi, S. A review of temporal video error concealment techniques and their suitability for HEVC and VVC. Multimed Tools Appl 80, 12685–12730 (2021). https://doi.org/10.1007/s11042-020-10333-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10333-6