Skip to main content
SpringerLink
Log in

Compressed Video Sensing Based on Deep Generative Adversarial Network

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

This paper considers the deep-learning-aided compressed video sensing problem. To this end, a deep generative adversarial network has been proposed to provide an approximation of the non-reference frame using its corresponding reference frame. The tests confirm the superiority of this scheme over the conventional methods used earlier. Furthermore, two scenarios have been suggested for deep compressed video sensing and recovery. In the first scenario, the difference between the non-reference frame and its approximation obtained from the pre-trained network is compressively sampled and transmitted to the receiver where the proposed residual reconstruction network is adopted to reconstruct the signal. The second scenario utilizes a pre-trained network followed by an augmented layer to approximate the non-reference frames. In the transmitter, the parameters of the augmented layer are trained for the current non-reference block. Instead of transmitting the samples of the block, the parameters of its trained augmented layer are sent to the receiver where the reconstruction is done using the same pre-trained network. The performances of the proposed scenarios demonstrate their objective and subjective superiority over the state-of-the-art algorithms in both the reconstruction quality and run time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The data can be available upon reasonable request.

Notes

  1. http://toflow.csail.mit.edu/.

References

  1. A. Agrawal, R. Verschueren, S. Diamond, S. Boyd, A rewriting system for convex optimization problems. J. Control Decis. 5(1), 42–60 (2018)

    Article  MathSciNet  Google Scholar 

  2. M. Azghani, A. Esmaeili, K. Behdin, F. Marvasti, Missing low-rank and sparse decomposition based on smoothed nuclear norm. IEEE Trans. Circuits Syst. Video Technol. 30(6), 1550–1558 (2020). https://doi.org/10.1109/TCSVT.2019.2907467

    Article  Google Scholar 

  3. M. Azghani, M. Karimi, F. Marvasti, Multihypothesis compressed video sensing technique. IEEE Trans. Circuits Syst. Video Technol. 26(4), 627–635 (2015)

    Article  Google Scholar 

  4. M. Azghani, F. Marvasti, L 2-regularized iterative weighted algorithm for inverse scattering. IEEE Trans. Antennas Propag. 64(6), 2293–2300 (2016)

    Article  MathSciNet  Google Scholar 

  5. E.J. Candès, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  Google Scholar 

  6. E.J. Candes, J.K. Romberg, T. Tao, Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. A J. Issue Courant Inst. Math. Sci. 59(8), 1207–1223 (2006)

    Article  MathSciNet  Google Scholar 

  7. C. Chen, E.W. Tramel, J.E. Fowler, Compressed-sensing recovery of images and video using multihypothesis predictions. In 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp. 1193–1198. IEEE (2011)

  8. K. Dabov, A. Foi, V. Katkovnik, K, Egiazarian, Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In 2007 IEEE International Conference on Image Processing, vol. 1 ( IEEE, 2007), pp. I–313

  9. Z. Gao, Y. Guo, J. Zhang, T. Zeng, G. Yang, Hierarchical perception adversarial learning framework for compressed sensing MRI. IEEE Trans. Med. Imaging (2023)

  10. I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks. arXiv:1406.2661 (2014)

  11. D.P. Kingma, J. Ba, Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  12. C. Li, M. Wand, Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision, (Springer, 2016), pp. 702–716

  13. Y. Li, F. Xiao, W. Liang, L. Gui, Multiply complementary priors for image compressive sensing reconstruction in impulsive noise (ACM Transactions on Multimedia Computing, Communications and Applications, 2024)

  14. M. Mirza, S. Osindero, Conditional generative adversarial nets. arXiv:1411.1784 (2014)

  15. S. Mun, J.E. Fowler, Block compressed sensing of images using directional transforms. In 2009 16th IEEE International Conference on Image Processing (ICIP) (IEEE, 2009), pp. 3021–3024

  16. V. Nair, G.E. Hinton, Rectified linear units improve restricted Boltzmann machines. In Icml (2010)

  17. H. Rajoriya, R. Sadiwala, Deep compressive sensing and reconstruction algorithm in wireless internet of things. J. Integr. Sci. Technol. 11(2), 487 (2023)

    Google Scholar 

  18. O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention (Springer, 2015), pp. 234–241

  19. N. Sadeghi, M. Azghani, Multi-user massive mimo channel estimation using joint sparsity and non-ideal feedback modeling. Digit. Signal Process. 100, 102640 (2020)

    Article  Google Scholar 

  20. G.B. Satrya, I.N.A. Ramatryana, S.Y. Shin, Compressive sensing of medical images based on HSV color space. Sensors 23(5), 2616 (2023)

    Article  Google Scholar 

  21. R. Sedghi, M. Azghani, Sparsity-based mimo interference suppression technique in the presence of imperfect channel state information. IET Commun. 13(19), 3201–3206 (2019)

    Article  Google Scholar 

  22. X. Sun, C. Tian, W. Tian, Y. Zhang, Privacy-enhanced and verifiable compressed sensing reconstruction for medical image processing on the cloud. IEEE Access 10, 18134–18145 (2022)

    Article  Google Scholar 

  23. S. Tang, C.F. Cheang, X. Yu, Y. Liang, Q. Feng, Z. Chen, Transcs-net: a hybrid transformer-based privacy-protecting network using compressed sensing for medical image segmentation. Biomed. Signal Process. Control 86, 105131 (2023)

    Article  Google Scholar 

  24. E.W. Tramel, J.E. Fowler, Video compressed sensing with multihypothesis. In 2011 Data Compression Conference (IEEE, 2011), pp. 193–202

  25. V. Upadhyaya, G. Sharma, T.A. Tran, M. Salim, Compressive sensing-based medical imaging techniques to detect the type of pneumonia in lungs. In: Computational Intelligence in Medical Decision Making and Diagnosis, (CRC Press, 2023), pp. 151–168

  26. M. Wakin, J.N. Laska, M.F. Duarte, D. Baron, S. Sarvotham, D. Takhar, K.F. Kelly, R.G. Baraniuk, Compressive imaging for video representation and coding. In Picture Coding Symposium, vol 13 (2006)

  27. Z. Wang, Z. Wang, C. Zeng, Y. Yu, X. Wan, High-quality image compressed sensing and reconstruction with multi-scale dilated convolutional neural network. Circuits Syst. Signal Process. 42(3), 1593–1616 (2023)

    Article  Google Scholar 

  28. K. Xu, F. Ren, Csvideonet: a real-time end-to-end learning framework for high-frame-rate video compressive sensing. In: 2018 IEEE winter conference on applications of computer vision (WACV). (IEEE, 2018), pp. 1680–1688

  29. H. Yao, F. Dai, S. Zhang, Y. Zhang, Q. Tian, C. Xu, Dr2-net: Deep residual reconstruction network for image compressive sensing. Neurocomputing 359, 483–493 (2019)

    Article  Google Scholar 

  30. D. Ye, Z. Ni, H. Wang, J. Zhang, S. Wang, S. Kwong, Csformer: bridging convolution and transformer for compressive sensing. IEEE Trans. Image Process. (2023)

  31. Z. Zha, B. Wen, X. Yuan, S. Ravishankar, J. Zhou, C. Zhu, Learning nonlocal sparse and low-rank models for image compressive sensing: Nonlocal sparse and low-rank modeling. IEEE Signal Process. Mag. 40(1), 32–44 (2023)

    Article  Google Scholar 

  32. C. Zhao, S. Ma, J. Zhang, R. Xiong, W. Gao, Video compressive sensing reconstruction via reweighted residual sparsity. IEEE Trans. Circuits Syst. Video Technol. 27(6), 1182–1195 (2016)

    Article  Google Scholar 

  33. Y. Zhao, Q. Zeng, E.Y. Lam, Adaptive compressed sensing for real-time video compression, transmission, and reconstruction. In: 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA), pp. 1–10 (2023). https://doi.org/10.1109/DSAA60987.2023.10302598

  34. C. Zhou, C. Chen, Y. Zhang, F. Ding, D. Zhang, Mh-net: A learnable multi-hypothesis network for compressed video sensing. IEEE Access 7, 166606–166613 (2019)

    Article  Google Scholar 

Download references

Funding

The authors have received no fundings for this research.

Author information

Authors and Affiliations

  1. Laboratory of Wireless Communication and Signal Processing (WCSP), Faculty of Electrical Engineering, Sahand University of Technology, Sahand, Iran

    Masoumeh Azghani

  2. Advanced Communications Research Institute (ACRI), Electrical Engineering Department of Sharif University of Technology, Tehran, Iran

    Valiyeh Ansarian Nezhad & Farokh Marvasti

Authors
  1. Valiyeh Ansarian Nezhad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masoumeh Azghani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farokh Marvasti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masoumeh Azghani.

Ethics declarations

Conflict of interest

The authors declare that they have no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nezhad, V.A., Azghani, M. & Marvasti, F. Compressed Video Sensing Based on Deep Generative Adversarial Network. Circuits Syst Signal Process (2024). https://doi.org/10.1007/s00034-024-02672-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00034-024-02672-8

Keywords

Search

Navigation