Parallax correction via disparity estimation in a multi-aperture camera

Abstract

In this paper, an image fusion algorithm is proposed for a multi-aperture camera. Such camera is a feasible alternative to traditional Bayer filter camera in terms of image quality, camera size and camera features. The camera consists of several camera units, each having dedicated optics and color filter. The main challenge of a multi-aperture camera arises from the fact that each camera unit has a slightly different viewpoint. Our image fusion algorithm corrects the parallax error between the sub-images using a disparity map, which is estimated from the single-spectral images. We improve the disparity estimation by combining matching costs over multiple views using trifocal tensors. Images are matched using two alternative matching costs, mutual information and Census transform. We also compare two different disparity estimation methods, graph cuts and semi-global matching. The results show that the overall quality of the fused images is near the reference images.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. 1.

    Kolehmainen, T., Rytivaara, M., Tokkonen, T., Mäkelä, J., Ojala, K.: Imaging device. US Patent No. 7453510 (2008)

  2. 2.

    Hirakawa, K.: Cross-talk explained. In: 15th IEEE International Conference on Image Processing, pp. 677–680 (2008)

  3. 3.

    van Walree, P.: Chromatic aberrations. http://toothwalker.org/optics/chromatic.html (2016). Accessed 5 Apr (2016)

  4. 4.

    Hernández, C.: Lens blur in the new google camera app (2014), googleresearch.blogspot.com/2014/04/lens-blur-in-new-google-camera-app.html

  5. 5.

    LinX Imaging. Technology presentation. http://linximaging.com/imaging/ (2014)

  6. 6.

    Venkataraman, K., Lelescu, D., Duparre, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., Nayar, S.: PiCam: an ultra-thin high performance monolithic camera array. ACM Trans. Graph. 32(6), 13 (2013)

    Article  Google Scholar 

  7. 7.

    Suda, Y.: Image sensing apparatus and its control method, control program, and storage medium for correcting position deviation of images. US Patent No. 7847843 (2010)

  8. 8.

    Yu, Y., Zhang, Z.: Digital cameras using multiple sensors with multiple lenses. US Patent No. 6611289 (2003)

  9. 9.

    Gere, D.S.: Image capture using luminance and chrominance sensors. US Patent No. 8497897 (2013)

  10. 10.

    Sung, G.-Y., Park, D.-S., Lee, H.-Y., Kim, S.-S., Kim, C.-Y.: Camera module. European Patent No. 1871091 (2007)

  11. 11.

    Light. https://light.co/camera 2015. Accessed 25 Nov (2015)

  12. 12.

    Kim, J., Kolmogorov, V., Zabih, R.: Visual correspondence using energy minimization and mutual information. In: The Proceedings of the 9th IEEE International Conference on Computer Vision, 2, pp. 1033–1040 (2003)

  13. 13.

    Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)

    Article  Google Scholar 

  14. 14.

    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, United States of America (2003)

    Google Scholar 

  15. 15.

    Hirschmüller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2008)

    Article  Google Scholar 

  16. 16.

    Egnal, G.: Mutual information as a stereo correspondence measure. University of Pennsylvania, Department of Computer and Information Science, Technical Report No. MS-CIS-00-20 (2000)

  17. 17.

    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Lecture Notes in Computer Science vol. 801, pp. 151–158 (1994)

  18. 18.

    Middlebury Stereo Evaluation. http://vision.middlebury.edu/stereo/eval/. Accessed 5 Apr (2016)

  19. 19.

    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

    Article  Google Scholar 

  20. 20.

    Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)

    Article  MATH  Google Scholar 

  21. 21.

    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)

    Article  MATH  Google Scholar 

  22. 22.

    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)

    Article  MATH  Google Scholar 

  23. 23.

    Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 195–202 (2003)

    Google Scholar 

  24. 24.

    Kolmogorov, V., Zabih, R., Gortler, S.: Generalized Multi-camera Scene Reconstruction Using Graph Cuts. In: Energy Minimization Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 2683, pp. 501–516 (2003)

  25. 25.

    Levin, A., Lischinski, D., Weiss, Y.: Colorization Using Optimization. ACM SIGGRAPH, 689–694 (2004)

  26. 26.

    Banz, C., Blume, H., Pirsch, P.: Real-time semi-global matching disparity estimation on the GPU. In IEEE International Conference on Computer Vision Workshops, pp. 514-521 (2011)

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Janne Mustaniemi.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mustaniemi, J., Kannala, J. & Heikkilä, J. Parallax correction via disparity estimation in a multi-aperture camera. Machine Vision and Applications 27, 1313–1323 (2016). https://doi.org/10.1007/s00138-016-0773-7

Download citation

Keywords

  • Mutual information
  • Census transform
  • Trifocal tensor
  • Graph cuts
  • Semi-global matching