Super resolution of single depth image based on multi-dictionary learning with edge feature regularization

  • Sihan Li
  • Anhong WangEmail author
  • Hong Shangguan
  • Yingchun Wu
  • Donghong Li
  • Youcheng Wu
  • Jie Liang


Currently, the acquisition and application of depth images are attracting a lot of attention thanks to the rapid development of 3D video. However, the current depth cameras cannot obtain high quality depth images due to the limitations in the imaging system. In this paper, to address this issue, we propose a scheme for single depth image super resolution based on multi-dictionary learning with edge regularization model. In the training stage, we focus on the edge information that represents the structure of the image, and extract the edge part, the low frequency and high frequency parts from the high resolution depth image set respectively. After that, three dictionaries are learned for the three parts with the constraint of the same sparse representation. In the image synthesis stage, we employ an edge-preserving regularization model as a reconstruction constraint to preserve the sharp structure, and reconstruct the depth image via the dictionaries learned from the training stage. Experimental results show that our proposed method can achieve good results in edge preservation, and both PSNR and SSIM values of the reconstructed depth images are superior to the state-of-art methods.


Super resolution reconstruction Sparse representation Depth image Dictionary learning Edge regularization 



This work has been supported in part by National Natural Science Foundation of China (No.61672373 and No.61501315), Scientific and Technological Innovation Team of Shanxi Province (No. 201705D131025), Key Innovation Team of Shanxi 1331 Project(2017015), Collaborative Innovation Center of Internet+3D Printing in Shanxi Province(201708), The Program of “One hundred Talented People” of Shanxi Province, Shanxi Province Science Foundation for Youths (201701D221106); Taiyuan University of Science and Technology doctoral promoter (20162044). The authors thank the anonymous reviewers for their valuable comments and suggestions to improve the quality of this paper.


  1. 1.
    Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel ML (2012) Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding. In: British machine vision conferenceGoogle Scholar
  2. 2.
    Buades A, Coll B, Morel JM (2005) A non-local algorithm for image denoising. Computer Vision and Pattern Recognition. CVPR 2005. IEEE Computer Society Conference. 2005:60–65 vol. 2Google Scholar
  3. 3.
    Coates A, Ng AY (2012) Learning feature representations with K-Means. Neural networks: Tricks of the trade. Springer, Berlin Heidelberg, p 561–580Google Scholar
  4. 4.
    Dong C, Chen CL, He K et al (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307CrossRefGoogle Scholar
  5. 5.
    Dong Y et al (2017) Depth map upsampling using joint edge-guided convolutional neural network for virtual view synthesizing. J Electronic Imaging 26(4):043004CrossRefGoogle Scholar
  6. 6.
    Elad M, Aharon M (2006) Image Denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15(12):3736–3745MathSciNetCrossRefGoogle Scholar
  7. 7.
    Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65CrossRefGoogle Scholar
  8. 8.
    Gui Z, Liu Y, He J (2012) PML algorithm for positron emission tomography combined with nonlocal fuzzy anisotropic diffusion filtering. IEEE Trans Nucl Sci 59(5):1984–1989CrossRefGoogle Scholar
  9. 9.
    Ha S, Mueller K (2015) Low dose CT image restoration using a database of image patches. J Phys Med Biol 60(2):869–882CrossRefGoogle Scholar
  10. 10.
    Han JW, Kim JH, Cheon SH et al (2010) A novel image interpolation method using the bilateral filter. IEEE Trans Consum Electron 56(1):175–181CrossRefGoogle Scholar
  11. 11.
    Keys RG (2003) Cubic convolution interpolation for digital image processing. IEEE Trans Acoust Speech Signal Process 29(6):1153–1160MathSciNetCrossRefGoogle Scholar
  12. 12.
    Köhler T, Maier A, Christlein V (2015) Binarization driven blind Deconvolution for document image restoration. German conference on. Pattern Recogn:91–102Google Scholar
  13. 13.
    Lee C, Eden M, Unser M (1998) High quality image resizing using oblique projection operators. IEEE Trans Image Process 7(5):679–692CrossRefGoogle Scholar
  14. 14.
    Li S, Jiang H, Pang W (2017) Joint multiple fully connected convolutional neural network with extreme learning machine for hepatocellular carcinoma nuclei grading. Comput Biol Med 84:145–167CrossRefGoogle Scholar
  15. 15.
    Li Y, Xue T, Sun L et al (2012) Joint Example-Based Depth Map Super-Resolution. IEEE International Conference on Multimedia and Expo, p 152–157Google Scholar
  16. 16.
    Li C, Yung NHC, Sun X et al (2017) Human arm pose modeling with learned features using joint convolutional neural network. Mach Vis Appl 28(1–2):1–14CrossRefGoogle Scholar
  17. 17.
    Pati YC, Rezaiifar R, Krishnaprasad PS (2002) Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Signals, Systems and Computers, 1993. 1993 Conference Record of The Twenty-Seventh Asilomar Conference on. IEEE, 40–44 vol.1Google Scholar
  18. 18.
    Peleg T, Elad M (2014) A statistical prediction model based on sparse representations for single image super-resolution. IEEE PressGoogle Scholar
  19. 19.
    Protter M, Elad M, Takeda H et al (2008) Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans Image Process 18(1):36–51MathSciNetCrossRefGoogle Scholar
  20. 20.
    RaduTimofte, Vincent De Smet (2014) Luc Van Gool:A+: adjusted anchored neighborhood regression for fast super-resolution, ACCVGoogle Scholar
  21. 21.
    Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326CrossRefGoogle Scholar
  22. 22.
    Santosh KC, Vajda S, Antani S (2016) Edge map analysis in chest X-rays for automatic pulmonary abnormality screening. Int J Comput Assist Radiol Surg 11(9):1637–1646CrossRefGoogle Scholar
  23. 23.
    Santosh K C , Aafaque A , Antani S (2017) Line segment-based stitched multipanel figure separation for effective biomedical CBIR. IJPRAI 31(6):1–18CrossRefGoogle Scholar
  24. 24.
    Aafaque A, Santosh KCA (2016) Automatic compound figure separation in scientific articles: a study of edge map and its role for stitched panel boundary detection. RTIP2R: 319–332Google Scholar
  25. 25.
    Santosh KC, Vajda S, Antani S (2015) Automatic pulmonary abnormality screening using thoracic edge map CBMS, 360–361Google Scholar
  26. 26.
    R. Timofte, V. De, and L. V. Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 1920–1927Google Scholar
  27. 27.
    Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color image. computer vision, 1998. Sixth International Conference on IEEE:839–846Google Scholar
  28. 28.
    Tronicke J, Böniger U (2013) Steering kernel regression: An adaptive denoising tool to process GPR data. IEEE International Workshop on Advanced Ground Penetrating Radar, p 1–4Google Scholar
  29. 29.
    Wang R (2016) Edge Deterction using convolutional neural network. international symposium on neural networks. Springer International PublishingGoogle Scholar
  30. 30.
    Xie J, Feris RS, Sun MT (2015) Edge guided single depth image super resolution. IEEE International Conference on Image Processing, p 3773–37777Google Scholar
  31. 31.
    Xiong Z, Xu D, Sun X et al (2013) Example-based super-resolution with soft information and decision. IEEE Trans Multimedia 15(6):1458–1465CrossRefGoogle Scholar
  32. 32.
    Yang Y, Wang Z (2011) A new image super-resolution method in the wavelet domain. International conference on image & graphics. IEEE:163–167Google Scholar
  33. 33.
    Yang Y, Wang Z (2012) Range image super-resolution via guided image filter. International conference on internet multimedia computing and service. ACM:200–203Google Scholar
  34. 34.
    Yang S, Wang Z, Zhang L, Wang M (2014) Dual-geometric neighbor embedding for image super resolution with sparse tensor. IEEE Trans Image Process 23(7):2793–2803MathSciNetCrossRefGoogle Scholar
  35. 35.
    Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873MathSciNetCrossRefGoogle Scholar
  36. 36.
    Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. International Conference on Curves and Surfaces. Springer-Verlag, 711–730Google Scholar
  37. 37.
    Zhang J, Chen Z, Xiong R, et al. Image super-resolution via dual-dictionary learning and sparse representation. IEEE International Symposium on Circuits and Systems. 2012:1688–1691.Google Scholar
  38. 38.
    Zhang A, Jiang H, Ma L et al (2016) A Shearlet-based algorithm for quantum noise removal in low-doseCT images, SPIE Medical Imaging. International Society for Optics and Photonics, p 97843O-97843O-7Google Scholar
  39. 39.
    Zhang Y, Zhang Y, Zhang J et al (2015) Single image super-resolution via iterative collaborative representation. Proceedings, Part II, of the 16th Pacific-Rim Conference on Advances in Multimedia Information Processing -- PCM 2015 - Volume 9315. Springer-Verlag New York, Inc., 63–73Google Scholar
  40. 40.
    Zhang Y, Zhang Y, Zhang J et al (2016) CCR: clustering and collaborative representation for fast single image super-resolution. IEEE Trans Multimedia 18(3):405–417CrossRefGoogle Scholar
  41. 41.
    Zhao L, Bai H, Liang J et al (2017) Single depth image super-resolution with multiple residual dictionary learning and refinement. IEEE International Conference on Multimedia and Expo, p 739–744Google Scholar
  42. 42.
    Zheng H, Bouzerdoum A, Phung SL (2013) Depth image super-resolution using multi-dictionary sparse representation. IEEE International Conference on Image Processing, p 957–961Google Scholar
  43. 43.
    Zohora FT (2017) Foreign circular element detection in Chest X-Rays for Effective Automated Pulmonary Abnormality Screening. IJCVIP 7(2):36–49CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  • Sihan Li
    • 1
  • Anhong Wang
    • 1
    Email author
  • Hong Shangguan
    • 1
  • Yingchun Wu
    • 1
  • Donghong Li
    • 1
  • Youcheng Wu
    • 1
  • Jie Liang
    • 2
  1. 1.Institute of Digital Multimedia and CommunicationTaiyuan University of Science and TechnologyTaiyuanChina
  2. 2.Engineering ScienceSimon Fraser UniversityBurnabyCanada

Personalised recommendations