Skip to main content
SpringerLink
Log in

Color calibration of multi-view video plus depth for advanced 3D video

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Multi-view video plus depth (MVD) format is considered as the next-generation standard for advanced 3D video systems. MVD consists of multiple color videos with a depth value associated with each texture pixel. Relying on this representation and by using depth-image-based rendering techniques, new viewpoints for multi-view video applications can be generated. However, since MVD is captured from different viewing angles with different cameras, significant illumination and color differences can be observed between views. These color mismatches degrade the performance of view rendering algorithms by introducing visible artifacts leading to a reduced view synthesis quality. To cope with this issue, we propose an effective method for correcting color inconsistencies in MVD. Firstly, to avoid occlusion problems and allow performing correction in the most accurate way, we consider only the overlapping region when calculating the color mapping function. These common regions are determined using a reliable feature matching technique. Also, to maintain the temporal coherence, correction is applied on a temporal sliding window. Experimental results show that the proposed method reduces the color difference between views and improves view rendering process providing high-quality results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Vetro, A., Tourapis, A.M., Müller, K., Chen, T.: 3D-TV content storage and transmission. IEEE Trans. Broadcast. 57(2), 384–394 (2011)

    Article  Google Scholar 

  2. Tanimoto, M.: FTV: free-viewpoint television. Sig. Process. Image Commun. 27(7), 555–570 (2012)

    Article  Google Scholar 

  3. Smolic, A., Müller, K., Merkle, P., Kauff, P., Wiegand, T.: An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution. In: Proceedings of the Picture Coding Symposium (PCS), Chicago, IL, USA, pp. 1–4 (2009)

  4. Vetro, A., Yea, S., Smolic, A.: Towards a 3D video format for auto-stereoscopic displays. In: Proceedings of the SPIE Conference on Applications of Digital Image Processing XXXI, San Diego, CA, USA (2008)

  5. Müller, K., Merkle, P., Wiegand, T.: 3-D video representation using depth maps. Proc. IEEE 99(4), 643–656 (2011)

    Article  Google Scholar 

  6. Kauff, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Smolic, A., Tanger, R.: Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability. Signal Process. Image Commun. 22(2), 217–234 (2007)

    Article  Google Scholar 

  7. Smolic, A., Müller, K., Merkle, P., Atzpadin, N., Fehn, C., Mller, M., Schreer, O., Tanger, R., Kauff, P., Wiegand, T., Megyesi, Z.: Multi-view video plus depth (MVD) format for advanced 3D video systems. In: Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, JVT-W100, San Jose, CA, USA (2007)

  8. Fehn, C.: Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. In: Proceedings of the SPIE Conference on Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, USA, pp. 93–104 (2004)

  9. Smolic, A.: 3D video and free viewpoint video—from capture to display. Pattern Recogn. 44(9), 1958–1968 (2011)

    Article  MathSciNet  Google Scholar 

  10. Reiter, U., Brunnström, K., De Moor, K., Larabi, M.-C., Pereira, M., Pinheiro, A., You, J., Zgank, A.: Factors influencing quality of experience. In: Möller, S., Raake, A. (eds.) Quality of Experience: Advanced Concepts, Applications, and Methods, pp. 55–72. Springer International Publishing, Berlin (2014)

    Chapter  Google Scholar 

  11. Pölönen, M., Hakala, J., Bilcu, R., Järvenpää, T., Häkkinen, J., Salmimaa, M.: Color asymmetry in 3D imaging: influence on the viewing experience. 3D Res. 3(3), 1–10 (2012)

    Article  Google Scholar 

  12. Chen, J., Zhou, J., Sun, J., Bovik, A. C.: Binocular mismatch induced by luminance discrepancies on stereoscopic images. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2014), pp. 1–6 (2014)

  13. Salmimaa, M., Hakala, J., Pölönen, M., Järvenpää, T., Bilcu, R., Häkkinen, J.: Luminance asymmetry in stereoscopic content: binocular rivalry or Luster. In: Proceedings of SID Symposium Digest of Technical Papers, pp. 801–804 (2014)

  14. Winkler, S., Min, D.: Stereo/multiview picture quality: overview and recent advances. Signal Process. Image Commun. 28(10), 1358–1373 (2013)

    Article  Google Scholar 

  15. Zhong, J., Kleijn, B., Hu, X.: Camera control in multi-camera systems for video quality enhancement. IEEE Sens. J. 14(9), 2955–2966 (2014)

    Article  Google Scholar 

  16. Ilie, A., Welch, G.: Ensuring color consistency across multiple cameras. In: Proceedings of International Conference on Computer Vision (ICCV 2005), Washington, DC, USA, pp. 1268–1275 (2005)

  17. Jung, J., Ho, Y.: Color correction for multi-view images using relative Luminance and chrominance mapping curves. J Signal Process. Syst. 72(2), 107–117 (2013)

    Article  Google Scholar 

  18. Pitié, F., Kokaram, A. C., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), Beijing, China, pp. 1434–1439 (2005)

  19. Pitié, F., Kokaram, A.C., Dahyot, R.: Automated colour grading using colour distribution transfer. Comput. Vis. Image Underst. 107(1), 123–137 (2007)

    Article  Google Scholar 

  20. Doutre, C., Nasiopoulos, P.: Color correction preprocessing for multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 19(9), 1400–1406 (2009)

    Article  Google Scholar 

  21. Fecker, U., Barkowsky, M., Kaup, A.: Histogram-based pre-filtering for luminance and chrominance compensation of multi-view video. IEEE Trans. Circuits Syst. Video Technol. 18(9), 1258–1267 (2008)

    Article  Google Scholar 

  22. Chen, Y., Cai, C., Liu, J.: YUV correction for multi-view video compression. In: Proceedings of the International Conference Pattern Recognition (ICPR), Hong Kong, pp. 734–737 (2006)

  23. Hur, J.H., Cho, S., Lee, Y.L.: Adaptive local illumination change compensation method for H.264-based multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1496–1505 (2007)

    Article  Google Scholar 

  24. Li, X., Jiang, L., Ma, S., Zhao, D., Gao, W.: Template based illumination compensation algorithm for multiview video coding. In: Proceedings of the SPIE Conference on Visual Communications and Image Processing (VCIP), Huangshan, China (2010)

  25. Shi, B., Li, Y., Liu, L., Xu, C.: Color correction and compression for multi-view video using h.264 features. In: Proceedings of the 9th Asian Conference on Computer Vision (ACCV), Xi’an, China, pp. 43–52 (2009)

  26. Yamamoto, K., Kitahara, M., Kimata, H., Yendo, T., Fujii, T., Tanimoto, M., Shimizu, S., Kamikura, K., Yashima, Y.: Multiview video coding using view interpolation and color correction. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1436–1449 (2007)

    Article  Google Scholar 

  27. Faridul, H.S., Pouli, T., Chamaret, C., Stauder, J., Tremeau, A., Reinhard, E.: A Survey of Color Mapping and Its Applications. Eurographics State of the Art Report, Strasbourg (2014)

    Google Scholar 

  28. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Surf: speeded up robust features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  29. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  30. Hirschmller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2009)

    Article  Google Scholar 

  31. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)

    Article  MATH  Google Scholar 

  32. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

  33. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: Proceedings of the British Machine Vision Conference (BMVC), Cardiff, UK, pp. 384–396 (2002)

  34. Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: Proceedings of the 7th European Conference on Computer Vision (ECCV), Copenhagen, Denmark, pp. 128–142 (2002)

  35. Ke, Y., Sukthankar, R.: PCA-SIFT: A more distinctive representation for local image descriptors. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 506–513 (2004)

  36. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W. T.: SIFT flow: dense correspondence across different scenes. In: Proceedings of the 10th European Conference on Computer Vision (ECCV), Marseille, France, pp. 28–42 (2008)

  37. Juan, L., Gwun, O.: A comparison of SIFT, PCA-SIFT and SURF. Int. J. Image Process. 3(4), 143–152 (2009)

    Google Scholar 

  38. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005)

    Article  Google Scholar 

  39. Nauge, M., Larabi, M.-C., Fernandez-Maloigne, C.: A statistical study of the correlation between interest points and gaze points. In: Proceedings of the SPIE Conference Human Vision and Electronic Imaging XVII, Burlingame, California, USA (2012)

  40. Harding, P., Robertson, N. M.: A Comparison of Feature Detectors with Passive and Task-Based Visual Saliency. In: Proceedings of the 16th Scandinavian Conference on Image Analysis (SCIA), Oslo, Norway, pp. 716–725 (2009)

  41. Harding, P., Robertson, N.M.: Visual saliency from image features with application to compression. Cogn. Comput. 5(1), 76–98 (2013)

    Article  Google Scholar 

  42. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice Hall, New Jersey (2007)

    Google Scholar 

  43. ISO/IEC JTC1/SC29/WG11: Call for Proposals on 3D Video Coding Technology. Doc. N12036, Geneva, Switzerland (2011)

  44. ISO/IEC JTC1/SC29/WG11: Report on Experimental Framework for 3D Video Coding. Doc. N11631, Guangzhou, China (2010)

  45. Corrigan, D., Pitié, F., Marcin, G., Kearney, G., Morris, V., Rankin, A; Linnane, M., O’Deax, M., Leez, C., Kokaram, A.: A video database for the development of stereo-3D post-production algorithms. J. Virtual Real. Broadcast. 10 (2013). https://www.jvrb.org/past-issues/10.2013/3780/

  46. Bosc, E., Hanhart, P., Le Callet, P., Ebrahimi, T.: A quality assessment protocol for Free-viewpoint video sequences synthesized from decompressed depth data. In: Proceedings of the Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wrthersee, Austria, pp. 100–105 (2013)

  47. ITU-R Rec. BT.500.: Methodology for the subjective assessment of the quality of television pictures, 46 pp. Geneva, Switzerland (2012)

  48. Sharma, G., Wu, W., Dalal, E.N.: The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations. Color. Res. Appl. 30(1), 21–30 (2005)

    Article  Google Scholar 

  49. Westland, S., Ripamonti, C., Cheung, V.: Computational Colour Science Using MATLAB, 2nd edn. Wiley-ISandT series in Imaging Science and Technology, New York (2012)

    Book  Google Scholar 

  50. Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph 30(4), 1–40 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Oran 2, Oran, Algeria

    Sid Ahmed Fezza

  2. XLIM Laboratory, SIC Department, University of Poitiers, Poitiers, France

    Mohamed-Chaker Larabi

Authors
  1. Sid Ahmed Fezza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed-Chaker Larabi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sid Ahmed Fezza.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fezza, S.A., Larabi, MC. Color calibration of multi-view video plus depth for advanced 3D video. SIViP 9 (Suppl 1), 177–191 (2015). https://doi.org/10.1007/s11760-015-0761-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-015-0761-9

Keywords

Search

Navigation