Signal, Image and Video Processing

, Volume 9, Supplement 1, pp 177–191 | Cite as

Color calibration of multi-view video plus depth for advanced 3D video

Original Paper

Abstract

Multi-view video plus depth (MVD) format is considered as the next-generation standard for advanced 3D video systems. MVD consists of multiple color videos with a depth value associated with each texture pixel. Relying on this representation and by using depth-image-based rendering techniques, new viewpoints for multi-view video applications can be generated. However, since MVD is captured from different viewing angles with different cameras, significant illumination and color differences can be observed between views. These color mismatches degrade the performance of view rendering algorithms by introducing visible artifacts leading to a reduced view synthesis quality. To cope with this issue, we propose an effective method for correcting color inconsistencies in MVD. Firstly, to avoid occlusion problems and allow performing correction in the most accurate way, we consider only the overlapping region when calculating the color mapping function. These common regions are determined using a reliable feature matching technique. Also, to maintain the temporal coherence, correction is applied on a temporal sliding window. Experimental results show that the proposed method reduces the color difference between views and improves view rendering process providing high-quality results.

Keywords

Color correction Multi-view video plus depth View rendering SURF RANSAC QoE 

References

  1. 1.
    Vetro, A., Tourapis, A.M., Müller, K., Chen, T.: 3D-TV content storage and transmission. IEEE Trans. Broadcast. 57(2), 384–394 (2011)CrossRefGoogle Scholar
  2. 2.
    Tanimoto, M.: FTV: free-viewpoint television. Sig. Process. Image Commun. 27(7), 555–570 (2012)CrossRefGoogle Scholar
  3. 3.
    Smolic, A., Müller, K., Merkle, P., Kauff, P., Wiegand, T.: An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution. In: Proceedings of the Picture Coding Symposium (PCS), Chicago, IL, USA, pp. 1–4 (2009)Google Scholar
  4. 4.
    Vetro, A., Yea, S., Smolic, A.: Towards a 3D video format for auto-stereoscopic displays. In: Proceedings of the SPIE Conference on Applications of Digital Image Processing XXXI, San Diego, CA, USA (2008)Google Scholar
  5. 5.
    Müller, K., Merkle, P., Wiegand, T.: 3-D video representation using depth maps. Proc. IEEE 99(4), 643–656 (2011)CrossRefGoogle Scholar
  6. 6.
    Kauff, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Smolic, A., Tanger, R.: Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability. Signal Process. Image Commun. 22(2), 217–234 (2007)CrossRefGoogle Scholar
  7. 7.
    Smolic, A., Müller, K., Merkle, P., Atzpadin, N., Fehn, C., Mller, M., Schreer, O., Tanger, R., Kauff, P., Wiegand, T., Megyesi, Z.: Multi-view video plus depth (MVD) format for advanced 3D video systems. In: Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, JVT-W100, San Jose, CA, USA (2007)Google Scholar
  8. 8.
    Fehn, C.: Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. In: Proceedings of the SPIE Conference on Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, USA, pp. 93–104 (2004)Google Scholar
  9. 9.
    Smolic, A.: 3D video and free viewpoint video—from capture to display. Pattern Recogn. 44(9), 1958–1968 (2011)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Reiter, U., Brunnström, K., De Moor, K., Larabi, M.-C., Pereira, M., Pinheiro, A., You, J., Zgank, A.: Factors influencing quality of experience. In: Möller, S., Raake, A. (eds.) Quality of Experience: Advanced Concepts, Applications, and Methods, pp. 55–72. Springer International Publishing, Berlin (2014)CrossRefGoogle Scholar
  11. 11.
    Pölönen, M., Hakala, J., Bilcu, R., Järvenpää, T., Häkkinen, J., Salmimaa, M.: Color asymmetry in 3D imaging: influence on the viewing experience. 3D Res. 3(3), 1–10 (2012)CrossRefGoogle Scholar
  12. 12.
    Chen, J., Zhou, J., Sun, J., Bovik, A. C.: Binocular mismatch induced by luminance discrepancies on stereoscopic images. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2014), pp. 1–6 (2014)Google Scholar
  13. 13.
    Salmimaa, M., Hakala, J., Pölönen, M., Järvenpää, T., Bilcu, R., Häkkinen, J.: Luminance asymmetry in stereoscopic content: binocular rivalry or Luster. In: Proceedings of SID Symposium Digest of Technical Papers, pp. 801–804 (2014)Google Scholar
  14. 14.
    Winkler, S., Min, D.: Stereo/multiview picture quality: overview and recent advances. Signal Process. Image Commun. 28(10), 1358–1373 (2013)CrossRefGoogle Scholar
  15. 15.
    Zhong, J., Kleijn, B., Hu, X.: Camera control in multi-camera systems for video quality enhancement. IEEE Sens. J. 14(9), 2955–2966 (2014)CrossRefGoogle Scholar
  16. 16.
    Ilie, A., Welch, G.: Ensuring color consistency across multiple cameras. In: Proceedings of International Conference on Computer Vision (ICCV 2005), Washington, DC, USA, pp. 1268–1275 (2005)Google Scholar
  17. 17.
    Jung, J., Ho, Y.: Color correction for multi-view images using relative Luminance and chrominance mapping curves. J Signal Process. Syst. 72(2), 107–117 (2013)CrossRefGoogle Scholar
  18. 18.
    Pitié, F., Kokaram, A. C., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), Beijing, China, pp. 1434–1439 (2005)Google Scholar
  19. 19.
    Pitié, F., Kokaram, A.C., Dahyot, R.: Automated colour grading using colour distribution transfer. Comput. Vis. Image Underst. 107(1), 123–137 (2007)CrossRefGoogle Scholar
  20. 20.
    Doutre, C., Nasiopoulos, P.: Color correction preprocessing for multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 19(9), 1400–1406 (2009)CrossRefGoogle Scholar
  21. 21.
    Fecker, U., Barkowsky, M., Kaup, A.: Histogram-based pre-filtering for luminance and chrominance compensation of multi-view video. IEEE Trans. Circuits Syst. Video Technol. 18(9), 1258–1267 (2008)CrossRefGoogle Scholar
  22. 22.
    Chen, Y., Cai, C., Liu, J.: YUV correction for multi-view video compression. In: Proceedings of the International Conference Pattern Recognition (ICPR), Hong Kong, pp. 734–737 (2006)Google Scholar
  23. 23.
    Hur, J.H., Cho, S., Lee, Y.L.: Adaptive local illumination change compensation method for H.264-based multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1496–1505 (2007)CrossRefGoogle Scholar
  24. 24.
    Li, X., Jiang, L., Ma, S., Zhao, D., Gao, W.: Template based illumination compensation algorithm for multiview video coding. In: Proceedings of the SPIE Conference on Visual Communications and Image Processing (VCIP), Huangshan, China (2010)Google Scholar
  25. 25.
    Shi, B., Li, Y., Liu, L., Xu, C.: Color correction and compression for multi-view video using h.264 features. In: Proceedings of the 9th Asian Conference on Computer Vision (ACCV), Xi’an, China, pp. 43–52 (2009)Google Scholar
  26. 26.
    Yamamoto, K., Kitahara, M., Kimata, H., Yendo, T., Fujii, T., Tanimoto, M., Shimizu, S., Kamikura, K., Yashima, Y.: Multiview video coding using view interpolation and color correction. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1436–1449 (2007)CrossRefGoogle Scholar
  27. 27.
    Faridul, H.S., Pouli, T., Chamaret, C., Stauder, J., Tremeau, A., Reinhard, E.: A Survey of Color Mapping and Its Applications. Eurographics State of the Art Report, Strasbourg (2014)Google Scholar
  28. 28.
    Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Surf: speeded up robust features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)CrossRefGoogle Scholar
  29. 29.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Hirschmller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2009)CrossRefGoogle Scholar
  31. 31.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)MATHCrossRefGoogle Scholar
  32. 32.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)Google Scholar
  33. 33.
    Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: Proceedings of the British Machine Vision Conference (BMVC), Cardiff, UK, pp. 384–396 (2002)Google Scholar
  34. 34.
    Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: Proceedings of the 7th European Conference on Computer Vision (ECCV), Copenhagen, Denmark, pp. 128–142 (2002)Google Scholar
  35. 35.
    Ke, Y., Sukthankar, R.: PCA-SIFT: A more distinctive representation for local image descriptors. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 506–513 (2004)Google Scholar
  36. 36.
    Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W. T.: SIFT flow: dense correspondence across different scenes. In: Proceedings of the 10th European Conference on Computer Vision (ECCV), Marseille, France, pp. 28–42 (2008)Google Scholar
  37. 37.
    Juan, L., Gwun, O.: A comparison of SIFT, PCA-SIFT and SURF. Int. J. Image Process. 3(4), 143–152 (2009)Google Scholar
  38. 38.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005)CrossRefGoogle Scholar
  39. 39.
    Nauge, M., Larabi, M.-C., Fernandez-Maloigne, C.: A statistical study of the correlation between interest points and gaze points. In: Proceedings of the SPIE Conference Human Vision and Electronic Imaging XVII, Burlingame, California, USA (2012)Google Scholar
  40. 40.
    Harding, P., Robertson, N. M.: A Comparison of Feature Detectors with Passive and Task-Based Visual Saliency. In: Proceedings of the 16th Scandinavian Conference on Image Analysis (SCIA), Oslo, Norway, pp. 716–725 (2009)Google Scholar
  41. 41.
    Harding, P., Robertson, N.M.: Visual saliency from image features with application to compression. Cogn. Comput. 5(1), 76–98 (2013)CrossRefGoogle Scholar
  42. 42.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice Hall, New Jersey (2007)Google Scholar
  43. 43.
    ISO/IEC JTC1/SC29/WG11: Call for Proposals on 3D Video Coding Technology. Doc. N12036, Geneva, Switzerland (2011)Google Scholar
  44. 44.
    ISO/IEC JTC1/SC29/WG11: Report on Experimental Framework for 3D Video Coding. Doc. N11631, Guangzhou, China (2010)Google Scholar
  45. 45.
    Corrigan, D., Pitié, F., Marcin, G., Kearney, G., Morris, V., Rankin, A; Linnane, M., O’Deax, M., Leez, C., Kokaram, A.: A video database for the development of stereo-3D post-production algorithms. J. Virtual Real. Broadcast. 10 (2013). https://www.jvrb.org/past-issues/10.2013/3780/
  46. 46.
    Bosc, E., Hanhart, P., Le Callet, P., Ebrahimi, T.: A quality assessment protocol for Free-viewpoint video sequences synthesized from decompressed depth data. In: Proceedings of the Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wrthersee, Austria, pp. 100–105 (2013)Google Scholar
  47. 47.
    ITU-R Rec. BT.500.: Methodology for the subjective assessment of the quality of television pictures, 46 pp. Geneva, Switzerland (2012)Google Scholar
  48. 48.
    Sharma, G., Wu, W., Dalal, E.N.: The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations. Color. Res. Appl. 30(1), 21–30 (2005)CrossRefGoogle Scholar
  49. 49.
    Westland, S., Ripamonti, C., Cheung, V.: Computational Colour Science Using MATLAB, 2nd edn. Wiley-ISandT series in Imaging Science and Technology, New York (2012)CrossRefGoogle Scholar
  50. 50.
    Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph 30(4), 1–40 (2011)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  1. 1.University of Oran 2OranAlgeria
  2. 2.XLIM Laboratory, SIC DepartmentUniversity of PoitiersPoitiersFrance

Personalised recommendations