In this paper, an image fusion algorithm is proposed for a multi-aperture camera. Such camera is a feasible alternative to traditional Bayer filter camera in terms of image quality, camera size and camera features. The camera consists of several camera units, each having dedicated optics and color filter. The main challenge of a multi-aperture camera arises from the fact that each camera unit has a slightly different viewpoint. Our image fusion algorithm corrects the parallax error between the sub-images using a disparity map, which is estimated from the single-spectral images. We improve the disparity estimation by combining matching costs over multiple views using trifocal tensors. Images are matched using two alternative matching costs, mutual information and Census transform. We also compare two different disparity estimation methods, graph cuts and semi-global matching. The results show that the overall quality of the fused images is near the reference images.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Kolehmainen, T., Rytivaara, M., Tokkonen, T., Mäkelä, J., Ojala, K.: Imaging device. US Patent No. 7453510 (2008)
Hirakawa, K.: Cross-talk explained. In: 15th IEEE International Conference on Image Processing, pp. 677–680 (2008)
van Walree, P.: Chromatic aberrations. http://toothwalker.org/optics/chromatic.html (2016). Accessed 5 Apr (2016)
Hernández, C.: Lens blur in the new google camera app (2014), googleresearch.blogspot.com/2014/04/lens-blur-in-new-google-camera-app.html
LinX Imaging. Technology presentation. http://linximaging.com/imaging/ (2014)
Venkataraman, K., Lelescu, D., Duparre, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., Nayar, S.: PiCam: an ultra-thin high performance monolithic camera array. ACM Trans. Graph. 32(6), 13 (2013)
Suda, Y.: Image sensing apparatus and its control method, control program, and storage medium for correcting position deviation of images. US Patent No. 7847843 (2010)
Yu, Y., Zhang, Z.: Digital cameras using multiple sensors with multiple lenses. US Patent No. 6611289 (2003)
Gere, D.S.: Image capture using luminance and chrominance sensors. US Patent No. 8497897 (2013)
Sung, G.-Y., Park, D.-S., Lee, H.-Y., Kim, S.-S., Kim, C.-Y.: Camera module. European Patent No. 1871091 (2007)
Light. https://light.co/camera 2015. Accessed 25 Nov (2015)
Kim, J., Kolmogorov, V., Zabih, R.: Visual correspondence using energy minimization and mutual information. In: The Proceedings of the 9th IEEE International Conference on Computer Vision, 2, pp. 1033–1040 (2003)
Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)
Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, United States of America (2003)
Hirschmüller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2008)
Egnal, G.: Mutual information as a stereo correspondence measure. University of Pennsylvania, Department of Computer and Information Science, Technical Report No. MS-CIS-00-20 (2000)
Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Lecture Notes in Computer Science vol. 801, pp. 151–158 (1994)
Middlebury Stereo Evaluation. http://vision.middlebury.edu/stereo/eval/. Accessed 5 Apr (2016)
Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)
Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)
Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)
Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)
Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 195–202 (2003)
Kolmogorov, V., Zabih, R., Gortler, S.: Generalized Multi-camera Scene Reconstruction Using Graph Cuts. In: Energy Minimization Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 2683, pp. 501–516 (2003)
Levin, A., Lischinski, D., Weiss, Y.: Colorization Using Optimization. ACM SIGGRAPH, 689–694 (2004)
Banz, C., Blume, H., Pirsch, P.: Real-time semi-global matching disparity estimation on the GPU. In IEEE International Conference on Computer Vision Workshops, pp. 514-521 (2011)
About this article
Cite this article
Mustaniemi, J., Kannala, J. & Heikkilä, J. Parallax correction via disparity estimation in a multi-aperture camera. Machine Vision and Applications 27, 1313–1323 (2016). https://doi.org/10.1007/s00138-016-0773-7
- Mutual information
- Census transform
- Trifocal tensor
- Graph cuts
- Semi-global matching