Skip to main content
SpringerLink
Log in

Image fusion using online convolutional sparse coding

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

As signal enhancement technique, image fusion alleviates limitation single sensor in terms to information presentation and enhance visual quality. Extracting affluent features to accurately represent image is crucial for fusion. However, filters via convolutional sparse coding (CSC) have disadvantages of heavy computation cost and low representation. Superior signal representation and low spatial complexity of online convolutional sparse coding are exploited to image fusion to compensate for shortcomings of CSC. The detail and low-frequency components of source images are firstly decomposed using two-layer decomposition. Then each layers use rules to obtain fused components. Finally, fused image can be reconstructed by both high-frequency and low-frequency layers. To verify performance of proposed method, 9 infrared-visible fusion methods and 5 medical fusion methods are used as comparison experiments. The quantitative (\(Q^{ABF}\), \(Q^{E}\), \(Q^{M}\) and \(Q^{P}\)) assessments confirm superiority of method. In addition, qualitative results exhibit powerful information preservation and better visualization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Availability of data and materials

The datasets used or analysed during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029.

  2. https://drive.google.com/drive/folders/0BzXT0LnoyRqlazdSdUFFSUw4anc.

  3. http://www.med.harvard.edu/AANLIB/home.html.

References

  • Boyd S et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, New York

    Google Scholar 

  • Chang X et al (2015) Compound rank-$ k $ projections for bilinear analysis. IEEE Trans Neural Netw Learn Syst 27(7):1502–1513

    Article  MathSciNet  Google Scholar 

  • Heide F et al (2015) Fast and flexible convolutional sparse coding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 5135–5143

  • Hu Q et al (2021) Multi-modality image fusion combining sparse representation with guidance filtering. Soft Comput 25(6):4393–4407

    Article  Google Scholar 

  • Kaur M, Singh D (2021) Multi-modality medical image fusion technique using multi-objective differential evolution based deep neural networks. J Ambient Intell Humaniz Comput 12(2):2483–2493

    Article  Google Scholar 

  • Khare A et al (2021) Shearlet transform based technique for image fusion using median fusion rule. Multimed Tools Appl 80(8):11491–11522

    Article  Google Scholar 

  • Li S, Yin H (2011) Multimodal image fusion with joint sparsity model. Opt Eng 50(6):067007

    Article  Google Scholar 

  • Li H et al (2018a) Infrared and visible image fusion using a deep learning framework. In: 2018 24th international conference on pattern recognition (ICPR). IEEE, pp 2705–2710

  • Li H et al (2018b) Infrared and visible image fusion using latent low-rank representation. arXiv preprint arXiv:1804.08992

  • Li Z et al (2018c) Rank-constrained spectral clustering with flexible embedding. IEEE Trans Neural Netw Learn Syst 29(12):6073–6082

    Article  MathSciNet  Google Scholar 

  • Li Z et al (2018d) Dynamic affinity graph construction for spectral clustering using multiple features. IEEE Trans Neural Netw Learn Syst 29(12):6323–6332

    Article  MathSciNet  Google Scholar 

  • Li Z et al (2019) Zero-shot event detection via event-adaptive concept relevance mining. Pattern Recognit 88:595–603

    Article  Google Scholar 

  • Li G et al (2021a) An infrared and visible image fusion method based on multi-scale transformation and norm optimization. Inf Fusion 71:109–129

    Article  Google Scholar 

  • Li X et al (2021b) Joint image fusion and denoising via three-layer decomposition and sparse representation. Knowl Based Syst 224:107087

    Article  Google Scholar 

  • Liu Y, Wang Z (2015) Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Process 9(5):347–357

    Article  Google Scholar 

  • Liu Z et al (2011) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans Pattern Anal Mach Intell 34(1):94–109

    Article  Google Scholar 

  • Liu Y et al (2015) A general framework for image fusion based on multi-scale transform and sparse representation. Inf Fusion 24:147–164

    Article  Google Scholar 

  • Liu Y et al (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886

    Article  Google Scholar 

  • Liu CH et al (2017) Infrared and visible image fusion method based on saliency detection in sparse domain. Infrared Phys Technol 83:94–102

    Article  Google Scholar 

  • Luo M et al (2021) Reliable shot identification for complex event detection via visual-semantic embedding. Comput Vis Image Underst 213:103300

    Article  Google Scholar 

  • Ma J et al (2017) Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys Technol 82:8–17

    Article  Google Scholar 

  • Mairal J et al (2010) Online learning for matrix factorization and sparse coding. J Mach Learn Res 11(1):1–45

    MathSciNet  MATH  Google Scholar 

  • Naidu VPS (2011) Image fusion technique using multi-resolution singular value decomposition. Def Sci J 61(5):479

    Article  MathSciNet  Google Scholar 

  • Piella G, Heijmans H (2003) A new quality metric for image fusion. In: Proceedings 2003 international conference on image processing (Cat. No. 03CH37429), vol 3. IEEE, pp III–173

  • Qu X et al (2008) Image fusion algorithm based on orientation information motivated pulse coupled neural networks. In: 2008 7th World Congress on intelligent control and automation. IEEE, pp 2437–2441

  • Ren P et al (2020) A comprehensive survey of neural architecture search: challenges and solutions. arXiv preprint arXiv:2006.02903

  • Ren P et al (2021) A survey of deep active learning. ACM Comput Surv (CSUR) 54(9):1–40

    Article  Google Scholar 

  • Tan W et al (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Comput Appl 3:1–21

    Google Scholar 

  • Wang P, Liu B (2008) A novel image fusion metric based on multi-scale analysis. In: 2008 9th international conference on signal processing. IEEE, pp 965–968

  • Wang X et al (2017) A new infrared and visible image fusion algorithm in nsct domain. In: International conference on intelligent computing. Springer, pp 420–431

  • Wang Y et al (2018) Scalable online convolutional sparse coding. IEEE Trans Image Process 27(10):4850–4859

    Article  MathSciNet  Google Scholar 

  • Wang L et al (2021) Multimodal medical image fusion based on nonsubsampled shearlet transform and convolutional sparse representation. Multimed Tools Appl 80(30):36401–36421

    Article  Google Scholar 

  • Wohlberg B (2015) Efficient algorithms for convolutional sparse representations. IEEE Trans Image Process 25(1):301–315

    Article  MathSciNet  MATH  Google Scholar 

  • Xu F et al (2013) An enhanced infrared and visible image fusion method based on wavelet transform. In: 2013 5th international conference on intelligent human–machine systems and cybernetics, vol 2. IEEE, pp 453–456

  • Xydeas CS et al (2000) Objective image fusion performance measure. Electron Lett 36(4):308–309

    Article  Google Scholar 

  • Yan C et al (2020) Self-weighted robust LDA for multiclass classification with edge classes. ACM Trans Intell Syst Technol (TIST) 12(1):1–19

    Google Scholar 

  • Yang B, Li S (2014) Visual attention guided image fusion with sparse representation. Optik 125(17):4881–4888

    Article  Google Scholar 

  • Yin M et al (2018) Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas 68(1):49–64

    Article  MathSciNet  Google Scholar 

  • Yu E et al (2018) Adaptive semi-supervised feature selection for cross-modal retrieval. IEEE Trans Multimed 21(5):1276–1288

    Article  MathSciNet  Google Scholar 

  • Zhang C, Yang X (2019) Image fusion based on masked online convolutional dictionary learning with surrogate function approach. In: International conference on intelligent computing, communication and devices. Springer, pp 70–74

  • Zhao J et al (2007) Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement. Int J Innov Comput Inf Control 3(6):1433–1447

    Google Scholar 

  • Zhou Z et al (2016a) Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters. Inf Fusion 30:15–26

    Article  Google Scholar 

  • Zhou Z et al (2016b) Fusion of infrared and visible images for night-vision context enhancement. Appl Opt 55(23):6480–6490

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Sichuan Science and Technology Program (2020YFS0351), the Sichuan University and Luzhou Municipal People’s Government Strategic cooperation projects (2020CDLZ-10) and Intelligent Policing Key Laboratory of Sichuan Province (ZNJW2022ZZMS001).

Funding

This work was supported by the Sichuan Science and Technology Program (2020YFS0351), the Sichuan University and Luzhou Municipal People’s Government Strategic cooperation projects (2020CDLZ-10) and Intelligent Policing Key Laboratory of Sichuan Province (ZNJW2022ZZMS001).

Author information

Authors and Affiliations

  1. National Key Laboratory of Fundamental Science on Synthetic Vision, College of Computer Science, Sichuan University, South 1st Ring Road, Chengdu, 610065, Sichuan, China

    Chengfang Zhang, Ziyou Zhang & Ziliang Feng

  2. Intelligent Policing Key Laboratory of Sichuan Province, Sichuan Police College, 186 Longtouguan Road, Luzhou, 646000, Sichuan, China

    Chengfang Zhang

  3. Experiment and Equipment Center, Sichuan Police College, 186 Longtouguan Road, Luzhou, 646000, Sichuan, China

    Chengfang Zhang

Authors
  1. Chengfang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziyou Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziliang Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZC completed experiment and wrote of paper. ZZ finished language polishing. FZ completed overall framework.

Corresponding author

Correspondence to Chengfang Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any study performed on humans or animals by the authors.

Consent to participate

This article does not contain any study performed on humans or animals by the authors.

Consent for publication

Not applicable

Code availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Zhang, Z. & Feng, Z. Image fusion using online convolutional sparse coding. J Ambient Intell Human Comput 14, 13559–13570 (2023). https://doi.org/10.1007/s12652-022-03822-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-03822-z

Keywords

Search

Navigation