Color Correction for Stereo and Multi-view Coding

Chapter

Abstract

Nowadays, various multimedia applications use multi-view video (MVV), which is obtained by capturing the same scene with multiple cameras from varying viewpoints. Therefore, illumination and color variations can be observed among the different views. These color inconsistencies can significantly reduce compression efficiency and rendering quality. Various methods have been proposed in the literature to compensate for these color mismatches. In this chapter, we review the commonly used color correction techniques for MVV, with a focus on coding applications. Experimental evaluations of the most prominent methods are provided to give the reader the opportunity to compare their performances in terms of visual effect, coding performance, and rendering quality.

References

  1. 1.
    Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Surf: speeded up robust features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)Google Scholar
  2. 2.
    Boyce, J. M.: Weighted prediction in the H.264/MPEG AVC video coding standard. In: Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), Vancouver, Canada, 789–792 (2004)Google Scholar
  3. 3.
    Chen, Y., Cai, C., Liu, J.: YUV correction for multi-view video compression. In: Proceedings of the International Conference Pattern Recognition (ICPR), Hong Kong, 734–737 (2006)Google Scholar
  4. 4.
    Chen, Y., Wang, Y.-K., Ugur, K., Hannuksela, M. M., Lainema, J., Gabbouj, M.: 3D video services with the emerging MVC standard. EURASIP J. Adv. Signal Process. Article ID 786015 (2009)Google Scholar
  5. 5.
    Chen, Y., Ma, K. K., Cai, C.: Histogram-offset-based color correction for multi-view video coding. In: Proceedings of the 17th IEEE International Conference on Image Processing (ICIP), 977–980 (2010)Google Scholar
  6. 6.
    Chen, J., Zhou, J., Sun, J., Bovik, A. C.: Binocular mismatch induced by luminance discrepancies on stereoscopic images. In: IEEE International Conference on Multimedia and Expo (ICME 2014), 1–6 (2014)Google Scholar
  7. 7.
    Doutre, C., Nasiopoulos, P.: Color correction preprocessing for multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 19(9), 1400–1406 (2009)Google Scholar
  8. 8.
    Faridul, H. S., Pouli, T., Chamaret, C., Stauder, J., Tremeau, A., Reinhard, E.: A survey of color mapping and its applications. Eurographics State of the Art Report, 43–67 (2014)Google Scholar
  9. 9.
    Fecker, U., Barkowsky, M., Kaup, A.: Histogram-based pre-filtering for luminance and chrominance compensation of multi-view video. IEEE Trans. Circuits Syst. Video Technol. 18(9), 1258–1267 (2008)Google Scholar
  10. 10.
    Fehn, C.: Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. In: Proceedings of the SPIE Conference on Stereoscopic Displays and Virtual Reality Systems XI, San Jose, CA, USA, 93–104 (2004)Google Scholar
  11. 11.
    Fezza, S. A., Larabi, M.-C., Faraoun, K. M.: Feature-based color correction of multiview video for coding and rendering enhancement. IEEE Trans. Circuits Syst. Video Technol. 24(9), 1486–1498 (2014)Google Scholar
  12. 12.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM. 24(6), 381–395 (1981)Google Scholar
  13. 13.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing. 3rd ed. Prentice-Hall, Englewood Cliffs (2007)Google Scholar
  14. 14.
    Hong, W., Zhao, Y., Yu, L., Zhu, C.: Detection model of luster effect in binocular rivalry. In Proceedings 19th International Conference on Digital Signal Processing (DSP 2014), 866–869 (2014)Google Scholar
  15. 15.
    Hur, J. H., Cho, S., Lee, Y. L.: Adaptive local illumination change compensation method for H.264-based multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1496–1505 (2007)Google Scholar
  16. 16.
    Iatsun, I., Larabi, M.-C., Fernandez, C.: Study of visual fatigue/discomfort generated by S3D video using eye-tracking data. In: Proceedings Stereoscopic Displays Appl. XXIV, Burlingame, CA, USA (2013)Google Scholar
  17. 17.
    Ilie, A., Welch, G.: Ensuring color consistency across multiple cameras. In: Proceedings of International Conference on Computer Vision (ICCV), Washington, DC, USA, 1268–1275 (2005)Google Scholar
  18. 18.
    ISO/IEC JTC1/SC29/WG11: Report on Experimental Framework for 3D Video Coding. Doc. N11631, Guangzhou, China (2010)Google Scholar
  19. 19.
    ISO/IEC JTC1/SC29/WG11: Call for Proposals on 3D Video Coding Technology. Doc. N12036, Geneva, Switzerland (2011)Google Scholar
  20. 20.
    ITU-T Recommendation H.264: Advanced Video Coding for Generic Audiovisual Services, ITU-T Rec. H.264 (2013)Google Scholar
  21. 21.
    Joshi, N. S.: Color calibration for arrays of inexpensive image sensors. Masters thesis, Stanford University (2004) Google Scholar
  22. 22.
    Joshi, N. S., Wilburn, B., Vaish, V., Levoy, M. Horowitz, M.: Automatic Color Calibration for Large Camera Arrays. Technical Report CS2005-0821, UCSD CSE (2005)Google Scholar
  23. 23.
    Jung, J. I., Ho, Y. S.: Color correction algorithm based on camera characteristics for multi-view video coding. Signal Image Video Process. 8(5),1–12 (2014)Google Scholar
  24. 24.
    Jung, J.I., Ho, Y.S.: Geometric and colorimetric error compensation for multi-view images. J. Vis. Commun. Image Represent. 25(4), 698–708 (2014)Google Scholar
  25. 25.
    Kamikura, K., Watanabe, H., Jozawa, H., Kotera, H., Ichinose, S.: Global brightness-variation compensation for video coding. IEEE Trans. Circuits Syst. Video Technol. 8(8), 988–1000 (1998)Google Scholar
  26. 26.
    Kang, Y. S., Ho, Y. S.: An efficient image rectification method for parallel multi-camera arrangement. IEEE Trans. Consumer Electron. 57(3), 1041–1048 (2011)Google Scholar
  27. 27.
    Kauff, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Smolic, A., Tanger, R.: Depth map creation and image based rendering for advanced 3DTV services providing interoperability and scalability. Signal Process. Image Commun. 22(2), 217–234 (2007)Google Scholar
  28. 28.
    Kaup, A., Fecker, U.: Analysis of multireference block matching for multiview video coding. In: Proceedings 7th Workshop Digital Broadcasting, Erlangen, Germany, 33–39 (2006)Google Scholar
  29. 29.
    Kim, J.-H., Lai, P.-L., Ortega, A., Su, Y., Yin, P., Gomila, C.: Results of CE2 on multi-view video coding. In Joint Video Team (JVT), Doc. JVT-T117. Klagenfurt, Austria (2006)Google Scholar
  30. 30.
    Kim, J.-H., Lai, P.-L., Lopez, J., Ortega, A., Su, Y., Yin, P., Gomila, C.: New coding tools for illumination and focus mismatch compensation in multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1519–1535 (2007)Google Scholar
  31. 31.
    Lambooij, M.T., IJsselsteijn, W.A., Heynderickx, I.: Visual discomfort in stereoscopic displays: a review. In: Proceedings of Stereoscopic Displays and Virtual Reality Systems XIV, San Jose, CA, USA (2007)Google Scholar
  32. 32.
    Lee, Y.L., Hur, J.H., Lee, Y.K., Cho, S.H., Kwon, H.J., Hur, N.H., Kim, J.W.: Results of CE2 on multi-view video coding. Joint Video Team (JVT) Doc. JVT-T110. Klagenfurt, Austria (2006)Google Scholar
  33. 33.
    Lee, D. S., Ko, M. S., Seo, Y. H., Kim, D. W., Yoo, J.: Illumination compensation for multi-view video based on layered histogram matching with depth information. Opt. Commun. 286, 74–84 (2013)Google Scholar
  34. 34.
    Li, X., Jiang, L., Ma, S., Zhao, D., Gao, W.: Template based illumination compensation algorithm for multiview video coding. In: Proceedings Conference on Visual Communications and Image Processing (VCIP), Huangshan, China (2010)Google Scholar
  35. 35.
    Li, K., Dai, Q., Xu, W.: Collaborative color calibration for multi-camera systems. Signal Process. Image Commun. 26(1), 48–60 (2011)Google Scholar
  36. 36.
    Lou, J.G., Cai, H., Li, J.: A real-time interactive multi-view video system. In Proceedings of the 13th Annual ACM International Conference on Multimedia, Singapore, 161–170 (2005)Google Scholar
  37. 37.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)Google Scholar
  38. 38.
    Marpe, D., Schwarz, H., Wiegand, T.: Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 620–636 (2003)Google Scholar
  39. 39.
    Merkle, P., Smolic, A., Müller, K., Wiegand, T.: Efficient prediction structures for multiview video coding. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1461–1473 (2007)Google Scholar
  40. 40.
    Müller, K., Merkle, P., Wiegand, T.: 3-D video representation using depth maps. Proc. IEEE. 99(4), 643–656 (2011)Google Scholar
  41. 41.
    Nanda, H., Cutler, R.: Practical calibrations for a real-time digital omnidirectional camera. In Technical Sketches, Computer Vision and Pattern Recognition (2001)Google Scholar
  42. 42.
    Park, S., Sim, D.: Core experiment results on CE2 for multi-view video coding. Joint Video Team (JVT) Doc. JVT-T113, Klagenfurt, Austria (2006)Google Scholar
  43. 43.
    Pedersini, F., Sarti, A., Tubaro, S.: Accurate and simple geometric calibration of multi-camera systems. Signal Process. 77(3), 309–334 (1999)Google Scholar
  44. 44.
    Pitié, F., Kokaram, A. C., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), Beijing, China, 1434–1439 (2005)Google Scholar
  45. 45.
    Pitié, F., Kokaram, A. C., Dahyot, R.: Automated colour grading using colour distribution transfer. Comput. Vis. Image Underst. 107(1), 123–137 (2007)Google Scholar
  46. 46.
    Pölönen, M., Hakala, J., Bilcu, R., Järvenpää, T., Häkkinen, J., Salmimaa, M.: Color asymmetry in 3D imaging: influence on the viewing experience. 3D Res. 3(3), 1–10 (2012)Google Scholar
  47. 47.
    Reiter, U., Brunnström, K., De Moor, K., Larabi, M.-C., Pereira, M., Pinheiro, A., You, J., Zgank,A.: Factors influencing quality of experience. In: Möller, S., Raake, A. (eds.) Quality of Experience: Advanced Concepts, Applications, and Methods, pp.55–72. Springer (2014)Google Scholar
  48. 48.
    Salmimaa, M., Hakala, J., Pölönen, M., Järvenpää, T., Bilcu, R., Hkkinen, J.: Luminance asymmetry in stereoscopic Content: binocular rivalry or luster. In: Proceedings SID Symposium Digest of Technical Papers, 801–804 (2014)Google Scholar
  49. 49.
    Shao, F., Jiang, G.Y., Yu, M., Ho, Y.S.: Fast color correction for multi-view video by modeling spatio-temporal variation. J. Vis. Commun. Image Represent. 21(5), 392–403 (2010)Google Scholar
  50. 50.
    Sharma, G., Wu, W., Dalal, E.N.: The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations. Color Res. Appl. 30(1), 21–30 (2005)Google Scholar
  51. 51.
    Slavkovikj, V.: Color calibration of a multi-camera array. Masters thesis, Gjøvik university college, Norway (2011)Google Scholar
  52. 52.
    Slavkovikj, V., Hardeberg, J. Y., Eichhorn, A.: Characterizing the response of charge-couple device digital color cameras. In: Proceedings SPIE Sensors, Cameras, and Systems for Industrial and Scientific Applications XIII, Burlingame, California, USA (2012)Google Scholar
  53. 53.
    Smolic, A.: 3D video and free viewpoint video—from capture to display. Pattern Recognit. 44(9), 1958–1968 (2011)Google Scholar
  54. 54.
    Smolic, A., Müller, K., Merkle, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Tanger, R., Kauff, P., Wiegand, T., Megyesi, Z.: Multi-view video plus depth (MVD) format for advanced 3D video systems. Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, JVT-W100, San Jose, CA, USA (2007)Google Scholar
  55. 55.
    Sohn, K., Kim, Y., Seo, J., Yoon, J., Kim, G.: Results on CE2 using IBDE for multi-view video coding. Joint Video Team (JVT) Doc. JVT-T114, Klagenfurt, Austria (2006)Google Scholar
  56. 56.
    Su, Y., Vetro, A., Smolic, A.: Common test conditions for multiview video coding. Joint Video Team (JVT) Doc. JVT-U211, Hangzhou, China (2006)Google Scholar
  57. 57.
    Tanimoto, M.: FTV: free-viewpoint television. Signal Process. Image Commun. 27(7), 555–570 (2012)Google Scholar
  58. 58.
    Tehrani, M.P., Ishikawa, A., Sakazawa, S., Koike, A.: Iterative colour correction of multicamera systems using corresponding feature points. J. Vis. Commun. Image Represent. 21(5), 377–391 (2010)Google Scholar
  59. 59.
    Tian, D., Pandit, P., Yin, P., Gomila, C.: Study of MVC coding tools, Joint Video Team (JVT) Doc, JVT-Y044, Shenzhen, China (2007)Google Scholar
  60. 60.
    Vetro, A., Yea, S., Smolic, A.: Towards a 3D video format for auto-stereoscopic displays. In Proceedings of the SPIE Conference on Applications of Digital Image Processing XXXI, San Diego, CA, USA (2008)Google Scholar
  61. 61.
    Vetro, A., Pandit, P., Kimata, H., Smolic, A., Wang, Y. K.: Joint multiview video model (JMVM) 8.0. Joint Video Team (JVT), Doc. JVT-AA207, Geneva, Switzerland (2008)Google Scholar
  62. 62.
    Vetro, A., Tourapis, A. M., Müller, K., Chen, T.: 3D-TV content storage and transmission. IEEE Trans. Broadcast. 57(2), 384–394 (2011)Google Scholar
  63. 63.
    Vetro, A., Wiegand, T., Sullivan, G.J.: Overview of the stereo and multiview video coding extensions of the H.264/AVC standard. Proc. IEEE. 99(4), 626–642 (2011)Google Scholar
  64. 64.
    Westland, S., Ripamonti, C., Cheung, V.: Computational Colour Science Using MATLAB. 2nd ed. Wiley Chichester, West Sussex, England (2012)Google Scholar
  65. 65.
    Winkler, S., Min, D.: Stereo/multiview picture quality: overview and recent advances. Signal Process. Image Commun. 28(10), 1358–1373 (2013)Google Scholar
  66. 66.
    Woods, A., Docherty, T., Koch, R.: Image distortions in stereoscopic video systems. In: Proceedings Stereoscopic Displays and Applications IV, San Jose, CA, USA (1993)Google Scholar
  67. 67.
    Yamamoto, K., Kitahara, M., Kimata, H., Yendo, T., Fujii, T., Tanimoto, M., Shimizu, S., Kamikura, K., Yashima, Y.: Multiview video coding using view interpolation and color correction. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1436–1449 (2007)Google Scholar
  68. 68.
    Zhong, J., Kleijn, B., Hu, X.: Camera control in multi-camera systems for video quality enhancement. IEEE Sens. J. 14(9), 2955–2966 (2014)Google Scholar
  69. 69.
    Zilly, F., Müller, M., Eisert, P., Kauff, P.: The stereoscopic analyzer—an image-based assistance tool for stereo shooting and 3D production. In: Proceeding 17th IEEE International Conference on Image Processing (ICIP), Hong Kong, 4029–4032 (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.University of Oran 2OranAlgeria
  2. 2.XLIM Institute, SIC DepartmentUniversity of PoitiersPoitiersFrance

Personalised recommendations