Data-Driven Tight Frame for Multi-channel Images and Its Application to Joint Color-Depth Image Reconstruction



In image restoration, we usually assume that the underlying image has a good sparse approximation under a certain system. Wavelet tight frame system has been proven to be such an efficient system to sparsely approximate piecewise smooth images. Thus, it has been widely used in many practical image restoration problems. However, images from different scenarios are so diverse that no static wavelet tight frame system can sparsely approximate all of them well. To overcome this, recently, Cai et. al. (Appl Comput Harmon Anal 37:89–105, 2014) proposed a method that derives a data-driven tight frame adapted to the specific input image, leading to a better sparse approximation. The data-driven tight frame has been applied successfully to image denoising and CT image reconstruction. In this paper, we extend this data-driven tight frame construction method to multi-channel images. We construct a discrete tight frame system for each channel and assume their sparse coefficients have a joint sparsity. The multi-channel data-driven tight frame construction scheme is applied to joint color and depth image reconstruction. Experimental results show that the proposed approach has a better performance than state-of-the-art joint color and depth image reconstruction approaches.


Data-driven tight frame Group sparsity Image reconstruction 

Mathematics Subject Classification

65T60 94A08 90C90 


  1. 1.
    Bao, C., Cai, J.-F., and Ji, H.: Fast sparsity-based orthogonal dictionary learning for image restoration, In: IEEE International Conference on Computer Vision, pp. 3384–3391, Sydney, (2013)Google Scholar
  2. 2.
    Bao, C., Ji, H., Shen, Z.: Convergence analysis for iterative data-driven tight frame construction scheme, Appl. Comput. Harmon. Anal. (2014). doi:10.1016/j.acha.2014.06.007
  3. 3.
    Cai, J.-F., Chan, R.H., Shen, L., Shen, Z.: Restoration of chopped and nodded images by framelets. SIAM J. Sci. Comput. 30, 1205–1227 (2008)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Cai, J.-F., Chan, R.H., Shen, L., Shen, Z.: Convergence analysis of tight framelet approach for missing data recovery. Adv. Comput. Math. 31, 87–113 (2009)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Cai, J.-F., Chan, R.H., Shen, Z.: A framelet-based image inpainting algorithm. Appl. Comput. Harmon. Anal. 24, 131–149 (2008)CrossRefMATHMathSciNetGoogle Scholar
  6. 6.
    Cai, J.-F., Dong, B., Osher, S., Shen, Z.: Image restoration: total variation, wavelet frames, and beyond. J. Amer. Math. Soc. 25, 1033–1089 (2012)CrossRefMATHMathSciNetGoogle Scholar
  7. 7.
    Cai, J.-F., Ji, H., Shen, Z., Ye, G.-B.: Data-driven tight frame construction and image denoising. Appl. Comput. Harmon. Anal. 37, 89–105 (2014)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Cai, J.-F., Osher, S., Shen, Z.: Linearized Bregman iterations for compressed sensing. Math. Comp. 78, 1515–1536 (2009)CrossRefMATHMathSciNetGoogle Scholar
  9. 9.
    Cai, J.-F., Osher, S., Shen, Z.: Linearized Bregman iterations for frame-based image deblurring. SIAM J. Imaging Sci. 2, 226–252 (2009)CrossRefMATHMathSciNetGoogle Scholar
  10. 10.
    Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise \(C^2\) singularities. Comm. Pure Appl. Math. 57, 219–266 (2004)CrossRefMATHMathSciNetGoogle Scholar
  11. 11.
    Chan, R.H., Chan, T.F., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24, 1408–1432 (2003)Google Scholar
  12. 12.
    Chan, T.F., Shen, J.: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods. Society for Industrial and Applied Mathematics, Philadelphia (2005)Google Scholar
  13. 13.
    Daubechies, I.: Ten Lectures on Wavelets, vol. 61 of CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1992)Google Scholar
  14. 14.
    Daubechies, I., Han, B., Ron, A., Shen, Z.: Framelets: MRA-based constructions of wavelet frames. Appl. Comput. Harmon. Anal. 14, 1–46 (2003)CrossRefMATHMathSciNetGoogle Scholar
  15. 15.
    Fornasier, M., Ramlau, R., Teschke, G.: The application of joint sparsity and total variation minimization algorithms to a real-life art restoration problem. Adv. Comput. Math. 31, 157–184 (2009)CrossRefMATHMathSciNetGoogle Scholar
  16. 16.
    Fornasier, M., Rauhut, H.: Recovery algorithms for vector-valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46, 577–613 (2008)CrossRefMATHMathSciNetGoogle Scholar
  17. 17.
    Han, B., Kutyniok, G., Shen, Z.: Adaptive multiresolution analysis structures and shearlet systems. SIAM J. Numer. Anal. 49, 1921–1946 (2011)CrossRefMATHMathSciNetGoogle Scholar
  18. 18.
    Hui, T.-W., and Ngi Ngan, K.: Motion-Depth: Rgb-d Depth Map Enhancement with Motion and Depth in Complement, In: IEEE conference on Computer Vision and Pattern Recognition, pp. 3962–3969, Columbus (2014)Google Scholar
  19. 19.
    Jain, R., Kasturi, R., Schunck, B.G.: Machine Vision, vol. 5. McGraw-Hill, New York (1995)Google Scholar
  20. 20.
    Ji, H., Shen, Z., Xu, Y.: Wavelet frame based scene reconstruction from range data. J. Comput. Phys. 229, 2093–2108 (2010)CrossRefMATHMathSciNetGoogle Scholar
  21. 21.
    Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M. H., Eisert, P., Hornegger, J., and Polthier, K.: Freecam: A Hybrid Camera System for Interactive Free-viewpoint Video. In: Proceedings of International Workshop on Vision, Modeling and Visualization, pp. 17–24 (2011)Google Scholar
  22. 22.
    Labate, D., Lim, W.-Q., Kutyniok, G., Weiss, G.: Sparse multidimensional representation using shearlets, In: Optics & Photonics 2005, International Society for Optics and Photonics, pp. 59140U–59140U, San Diego, (2005)Google Scholar
  23. 23.
    Le Pennec, E., Mallat, S.: Sparse geometric image representations with bandelets. IEEE Trans. Image Process. 14, 423–438 (2005)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Liang, J., Ma, J., Zhang, X.: Seismic data restoration via data-driven tight frame. Geophysics 79, V65–V74 (2014)CrossRefGoogle Scholar
  25. 25.
    Lu, S., Ren, X., and Liu, F.: Depth enhancement via low-rank matrix completion, In: IEEE Conference on Computer Vision and Pattern Recognition, Columbus, (2014)Google Scholar
  26. 26.
    Maimone, A., Fuchs, H.: Encumbrance-free telepresence system with real-time 3d capture and display using commodity depth cameras, In: 10th IEEE international symposium on Mixed and Augmented Reality, pp. 137–146, Basel, (2011)Google Scholar
  27. 27.
    Mallat, S.: A wavelet tour of signal processing, Elsevier/Academic Press, Amsterdam, 3rd edn. The sparse way, With contributions from Gabriel Peyré (2009)Google Scholar
  28. 28.
    Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C.: Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos, In: Proceedings of Eurographics, 31 (2012)Google Scholar
  29. 29.
    Ron, A., Shen, Z.: Affine systems in \(L_2({{\bf R^d}})\): the analysis of the analysis operator. J. Funct. Anal. 148, 408–447 (1997)CrossRefMATHMathSciNetGoogle Scholar
  30. 30.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)CrossRefMATHGoogle Scholar
  31. 31.
    Seok Lee H., and Mu Lee, K.: Simultaneous super-resolution of depth and images using a single camera, In: The IEEE Conference on Computer Vision and Pattern Recognition, Sydney, (2013)Google Scholar
  32. 32.
    Shekhar, S., Patel, V.M., Nasrabadi, N.M., Chellappa, R., Joint sparsity-based robust multimodal biometrics recognition, In: Computer Vision-ECCV 2012 Workshops and Demonstrations. Springer, pp. 365–374 (2012)Google Scholar
  33. 33.
    Shen, Z.: Wavelet frames and image restorations, In: Proceedings of the International Congress of Mathematicians, Volume IV, pp. 2834–2863. Hindustan Book Agency, New Delhi (2010)Google Scholar
  34. 34.
    Yang, X., Kim, S., and Xing, E. P.: Heterogeneous multitask learning with joint sparsity constraints, In: Advances in neural information processing systems, pp. 2151–2159, Vancouver, B.C., (2009)Google Scholar
  35. 35.
    Zhou, W., Cai, J.-F., Gao, H.: Adaptive tight frame based medical image reconstruction: a proof-of-concept study for computed tomography. Inverse Prob. (2013). doi:10.1088/0266-5611/29/12/125006

Copyright information

© Operations Research Society of China, Periodicals Agency of Shanghai University, Science Press and Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Beijing Key Laboratory of Multimedia and Intelligent Software Technology, College of Metropolitan TransportationBeijing University of TechnologyBeijingChina
  2. 2.Department of MathematicsUniversity of IowaIowa CityUSA

Personalised recommendations