Skip to main content
SpringerLink
Log in

A Shape-Adaptive Low-Complexity Technique for 3D Free-Viewpoint Visual Applications

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

View synthesis (VS) and depth map generation (DMG) are two of the most crucial operations for facilitating the development of free-viewpoint 3D video systems. This demand for high computational complexity has motivated the development of the proposed design. Conventional designs have directly adopted dedicated hardware to increase the speed of these operations independently, which has resulted in algorithmic-level redundancies. Thus, the conventional per-pixel computation results in high redundancy in both VS and DMG operations. Accordingly, this paper proposes a shape-adaptive low-complexity (SALC) technique for removing such algorithmic-level redundancies for both VS and DMG operations. Moreover, adopting shape-adaptive searching blocks can improve the accuracy of stereo matching (SM), which subsequently enhances the resulting VS. To overcome the problem of border misalignment in VS, a novel edge alignment technique is presented for detecting and aligning misaligned borders. In addition, an unreliable region repairing (URR) technique was adopted to enhance the accuracy of DMG operations, thereby optimizing the image quality. The proposed SALC technique was verified using \(C\)/Matlab simulations. Through the presented complexity reduction without any hardware acceleration, this design can achieve a high performance of 48 fps for synthesizing the virtual views of the CIF format (352 \(\times \) 288) based on a portable platform equipped with a 1 GHz ARM Cortex-A9 RISC processor only. The experimental results revealed that the proposed SALC technique saves up to 96 % computational complexity for VS, and the computation time of DMG operations is reduced to approximately 2 % of that required for conventional SM. The average improved image quality was 32.46 dB (i.e., 7.33 % improvement from the original 30.46 dB) when using the presented edge alignment technique. When considering the overall efficiency evaluation, the proposed SALC-based SM method is 172 times more efficient than the conventional fixed block-sized SM based on the combined indices of the BPCT product. Compared with the designs of previous studies, the proposed SALC-based design obtained superior efficiency in both VS and DMG as a result of the computation reuse property.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. D. Aliprandi, E. Piccinelli, Image-based three-dimensional free viewpoint video synthesis, in Proceedings of the IEEE 3DTV Conference (2009), pp. 1–4

  2. J. Canny, A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. pami–8(6), 679–698 (1986)

    Article  Google Scholar 

  3. T.S. Chang, J.I. Guo, C.W. Jen, Hardware-efficient DFT designs with cyclic convolution and subexpression sharing. IEEE Trans. Circuits Syst. II 47(9), 886–892 (2000)

    Article  Google Scholar 

  4. T.S. Chang, C.S. Kung, C.W. Jen, A simple processor core design for DCT/IDCT. IEEE Trans. Circuits Syst. Video Technol. 10(3), 439–447 (2000)

    Article  Google Scholar 

  5. N. Chang, T.H. Tsai, B.H. Hsu, Y.C. Chen, T.S. Chang, Algorithm and architecture of disparity estimation with mini-census adaptive support weight. IEEE Trans. Circuits Syst. Video Technol. 20(6), 792–805 (2010)

    Article  Google Scholar 

  6. S. Chen, L. Williams, View interpolation for image synthesis, in Proceedings of the 20th Annual Conference on Computer Graphics Interactive Techniques (1993), pp. 279–288

  7. S.S. Chen, H.M. Hang, New view generation from a video sequence, in Proceedings of the IEEE International Symposium on Circuits and Systems (1998), pp. 81–84

  8. K.H. Chen, J.I. Guo, J.S. Wang, C.W. Yeh, J.W. Chen, An energy-aware IP core design for the variable-length DCT/IDCT targeting at MPEG4 shape-adaptive transforms. IEEE Trans. Circuits Syst. Video Technol. 15(5), 704–715 (2005)

    Article  Google Scholar 

  9. C.M. Cheng, S.J. Lin, S.H. Lai, K. Lee, Efficient multiple virtual view generation based on reduced depth stereo image for advanced autostereoscopic displays, in Proceedings of the IEEE International Conference on Multimedia Expo (2009), pp. 241–244

  10. K.H. Chen, C.H. Chen, C.H. Chang, Y.C. Yang, Choose your own viewpoint: a high-quality/low-complexity free-viewpoint 3D visual system, in Proceedings of the IEEE International Emerging Signal Processing Applications Conference (Las Vegas, 2012), pp. 1–4

  11. A.K.R. Chowdhury, A. Kale, R. Chellappa, Video synthesis of arbitrary views for approximately planar scenes, in Proceedings of the IEEE International Conference on Acoustics, Speech, Signal Processing (2003), pp. 497–500

  12. D. Donoho, Wedgelet: nearly minimax estimation of edges. Ann. Stat. 27(3), 859–897 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  13. Y. Feng, D. Li, K. Luo, M. Zhang, Depth-image based view synthesis for three-dimensional television, in Proceedings of the IEEE Conference on Industrial Electronics Applications (2009), pp. 2428–2431

  14. M. Gong, R. Yang, L. Wang, M. Gong, A performance study on different cost aggregation approaches used in real-time stereo matching. Int. J. Comput. Vis. 75(2), 283–296 (2007)

    Article  Google Scholar 

  15. M. Hariyama, H. Sasaki, M. Kameyama, Architecture of a stereo matching VLSI processor based on hierarchically parallel memory access, in Proceedings of the IEEE International Midwest Symposium on Circuits Systems (2004), pp. II-245-247

  16. M. Hariyama, N. Yokoyama, M. Kameyama, Y. Kobayashi, FPGA implementation of a stereo matching processor based on window-parallel-and-pixel-parallel architecture, in Proceedings of the IEEE Midwest Symposium on Circuits Systems (2005), pp. 1219–1222

  17. A. Hore, D. Ziou, Image quality metrics: PSNR vs. SSIM, in Proceedings of the International Conference on Pattern Recognition (2010), pp. 2366–2369

  18. Y.R. Horng, Y.C. Tseng, T.S. Chang, VLSI architecture for real-time HD1080p view synthesis engine. IEEE Trans. Circuits Syst. Video Technol. 21(9), 1329–1340 (2011)

    Article  Google Scholar 

  19. T. Hosokawa, S. Jarusirisawad, H. Saito, Online video synthesis for removing occluding objects using multiple uncalibrated cameras via plane sweep algorithm, in Proceedings of the 3rd ACM/IEEE International Conference on Distributed Smart Cameras (2009), pp. 1–8

  20. N. Inamoto, H. Saito, Free viewpoint video synthesis and presentation from multiple sporting videos, in Proceedings of the IEEE International Conference on Multimedia Expo (2005), pp. 1–4

  21. N. Inamoto, H. Saito, Virtual viewpoint replay for a soccer match by view interpolation from multiple cameras. IEEE Trans. Multimedia 9(6), 1155–1166 (2007)

    Article  Google Scholar 

  22. S. Jarusirisawad, H. Saito, 3DTV view generation using uncalibrated cameras, in Proceedings of the 3DTV Conference (2008), pp. 57–60

  23. H. Jiang, S. Fels, J.J. Little, Optimizing multiple object tracking and best view video synthesis. IEEE Trans. Multimedia 10(6), 997–1012 (2008)

    Article  Google Scholar 

  24. C. Jin, H. Jeong, Intermediate view synthesis for multi-view 3D displays using belief propagation-based stereo matching, in Proceedings of the 3rd International Conference on Convergence Hybrid Information Technology (2008), pp. 919–924

  25. H.C. Karcithanasis, J.A. Vlontws, VLSI architectures for accurate aotion and disparity estimation using full-search block-matching and edge preserving non-linear smoothing, in Proceedings of the IEEE Workshop VLSI, Signal Processing (1993), pp. 75–83

  26. A. Kubota, K. Aizawa, Virtual view generation by linear processing of two differently focused images, in Proceedings of the International Conference Pattern Recognition (2002), pp. 504–507

  27. W.N. Lie, B.E. Wei, Intermediate view synthesis from binocular images for stereoscopic applications, in Proceedings of the IEEE International Symposium on Circuits Systems (2001), pp. 287–290

  28. Z. Liu, P. An, S. Liu, Z. Zhang, Arbitrary view generation based on DIBR, in Proceedings of the International Symposium on Intelligent Signal Processing Communication Systems (2007), pp. 168–171

  29. J. Lu, S. Rogmans, G. Lafruit, F. Catthoor, Stream-centric stereo matching and view synthesis: a high-speed approach on GPUs. IEEE Trans. Circuits Syst. Video Technol. 19(11), 1598–1611 (2009)

    Article  Google Scholar 

  30. P. Merkle, Y. Morvan, A. Smolic, D. Farin, K. Muller, P.H.N. de With, T. Wiegand, The effects of multiview depth video compression on multiview rendering. Signal Process. Image Commun. 24, 73–88 (2009)

    Article  Google Scholar 

  31. Middlebury Stereo Vision Page [online]. http://vision.middlebury.edu/stereo/

  32. Y. Mori, N. Fukushima, T. Fujii, M. Tanimoto, View generation with 3D warping using depth information for FTV, in Proceedings of the 3DTV Conference (2008), pp. 229–232

  33. Y. Mori, N. Fukushima, T. Yendo, T. Fujii, M. Tanimoto, View generation with 3D warping using depth information for FTV. Signal Process. Image Commun. 24, 65–72 (2009)

    Article  Google Scholar 

  34. K. Muller, A. Smolic, K. Dix, P. Kauff, T. Wiegand, Reliability-based generation and view synthesis in layered depth video, in Proceedings of the IEEE Workshop on Multimedia, Signal Processing (2008), pp. 34–39

  35. L. Nalpantidis, A. Amanatiadis, G. Sirakoulis, N. Kyriakoulis, A. Gasteratos: dense disparity estimation using a hierarchical matching technique from uncalibrated stereo vision, in Proceedings of the International Workshop on Imaging Systems and Techniques (2009)

  36. P. Ndjiki-Nya, M. Koppel, D. Doshkov, H. Lakshman, P. Merkle, K. Muller, T. Wiegand, Depth image based rendering with advanced texture synthesis, in Proceedings of the International Conference on Multimedia Expo (2010), pp. 424–429

  37. J.H. Park, H.W. Park, Fast view interpolation of stereo images using image gradient and disparity triangulation, in Proceedings of the International Conference on Image Processing (2003), pp. 381–384

  38. J.I. Park, S. Inoue, Arbitrary view generation from multiple cameras, in Proceedings of the International Conference on Image Processing (1997), pp. 149–152

  39. S. Pelletier, S.P. Spackman, J.R. Cooperstock, High-resolution video synthesis from mixed-resolution video based on the estimate-and-correct method, in Proceedings of the 7th IEEE Workshops Application Computer Vision (2005), pp. 172–177

  40. N. Plath, S. Knorr, L. Goldmann, T. Sikora, Adaptive image warping for hole prevention in 3D view synthesis. IEEE Trans. Image Process. 22(9), 3420–3432 (2013)

    Article  MathSciNet  Google Scholar 

  41. H. Saito, S. Baba, T. Kanade, Appearance-based virtual view generation from multicamera videos captured in the 3-D room. IEEE Trans. Multimedia 5(3), 303–316 (2003)

    Article  Google Scholar 

  42. A. Smolic, K. Mueller, P. Merkle, P. Kauff, T. Wiegand, An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution, in Proceedings of the Picture Coding Symposium (2009), pp. 1–4

  43. A. Smolic, K. Muller, K. Dix, P. Merkle, P. Kauff, T. Wiegand, Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems, in Proceedings of the 15th IEEE International Conference on Image Processing (2008), pp. 2448–2451

  44. M. Solh, G. AlRegib, Hierarchical hole filling for depth-based view synthesis in FTV and 3D video. IEEE J. Sel. Top. Signal Process. 6(5), 495–504 (2012)

    Article  Google Scholar 

  45. J. Sun, N.N. Zheng, H.Y. Shum, Stereo matching using belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7), 787–800 (2003)

    Article  Google Scholar 

  46. F. Tombari, S. Mattoccia, L. Di Stefano, Segmentation-based adaptive support for accurate stereo correspondence, in Proceedings of the Pacific-Rim Symposium Image Video Technology (2007), pp. 427–438

  47. F. Tombari, S. Mattoccia, L.D. Stefano, E. Addimanda, Classification and evaluation of cost aggregation methods for stereo correspondence, in Proceedings of the IEEE Conference on Computer Vision, Pattern Recognition (2008), pp. 1–8

  48. T.H. Tsai, N. Chang, T.S. Chang, Data reuse analysis of local stereo matching, in Proceedings of the International Symposium on Circuits Systems (2008), pp. 812–815

  49. Y.C. Tseng, N.Y.C. Chang, T.S. Chang, Low-memory cost belief propagation architecture for disparity estimation, in Proceedings of the IEEE International Symposium on Circuits Systems (2009), pp. 153–157

  50. Y.C. Tseng, P.H. Hsu, T.S. Chang, Fast stereo matching with predictive search range, in Proceedings of the Picture Coding Symposium (2010), pp. 506–509

  51. P.K. Tsung, W.Y. Chen, L.F. Ding, S.Y. Chien, L.G. Chen, Cache-based integer motion/disparity estimation for quad-HD H.264/AVC and HD multiview video coding, in Proceedings of the IEEE International Conference on Acoustic Speech Signal Processing (2009), pp. 2013–2017

  52. P.K. Tsung, P.C. Lin, L.F. Ding, S.Y. Chien, L.G. Chen, Single iteration view interpolation for multiview video applications, in Proceedings of the 3DTV Conference (2009), pp. 1–4

  53. T. Uemori, T. Yendo, T. Fujii, M. Tanimoto, View generation by ray-space method in circular camera setup for FTV, in Proceedings of the 3DTV Conference (2008), pp. 233–236

  54. Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 1–14 (2004)

    Article  Google Scholar 

  55. S.A. White, Applications of distributed arithmetic to digital signal processing: a tutorial review, IEEE ASSP MAGAZINE JULY. (1989)

  56. R.M. Willett, R.D. Nowak, Platelet: a multiscale approach for recovering edges and surfaces in photon-limited medical imaging. IEEE Trans. Med. Imag. 22(3), 332–350 (2003)

    Article  Google Scholar 

  57. X. Xiu, J. Liang, Projective rectification-based view interpolation for multiview video coding and free viewpoint generation, in Proceedings of the Picture Coding Symposium (2009), pp. 1–4

  58. K. Yamamoto, M. Kitahara, H. Kimata, T. Yendo, T. Fujii, M. Tanimoto, S. Shimizu, K. Kamikura, Yashima: multiview video coding using view interpolation and color correction. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1436–1449 (2007)

    Article  Google Scholar 

  59. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto, Error suppression in view synthesis using reliability reasoning for FTV in Proceedings of the 3DTV-Conference (2010), pp. 1–4

  60. S. Yea, A. Vetro, RD-optimized view synthesis prediction for multiview video coding, in Proceedings of the IEEE International Conference on Image Processing (2007), pp. 209–212

  61. S. Yea, A. Vetro, View synthesis prediction for rate-overhead reduction in FTV, in Proceedings of the 3DTV Conference (2008), pp. 145–148

  62. G. Zhang, W. Hua, X. Qin, T.T. Wong, H. Bao, Stereoscopic video synthesis from a monocular video. IEEE Trans. Vis. Comput. Graph. 13(4), 686–696 (2007)

    Article  Google Scholar 

  63. K. Zhang, J. Lu, F. Catthoor, Cross-based local stereo matching using orthogonal integral images. IEEE Trans. Circuits Syst. Video Technol. 19(7), 1073–1079 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Feng-Chia University, 100 Wen-hwa Rd., Sea-twen, Taichung, 40724, Taiwan, ROC

    Kuan-Hung Chen, Cheng-Hao Chen, Chih-Hao Chang & Jun-Yu Liu

  2. Department of Electronic Engineering, National Yunlin University of Science and Technology, 123 University Road, Section 3, Douliou, Yunlin, 64002, Taiwan, ROC

    Ching-Lung Su

Authors
  1. Kuan-Hung Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cheng-Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chih-Hao Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun-Yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ching-Lung Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kuan-Hung Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, KH., Chen, CH., Chang, CH. et al. A Shape-Adaptive Low-Complexity Technique for 3D Free-Viewpoint Visual Applications. Circuits Syst Signal Process 34, 579–604 (2015). https://doi.org/10.1007/s00034-014-9870-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-014-9870-x

Keywords

Search

Navigation