Skip to main content
SpringerLink
Log in

Flexible architecture for real-time synchronized processing of multimedia signals

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Simultaneous processing of multiple multimedia appears in many applications. However, there is a lack of a generalized hardware platform that fits all application needs from the number to the format of the input and output multimedia. The processing is also associated with synchronization problems such as startup delays and deviating frame rates of the multimedia. This paper presents a flexible platform with co-design of hardware and software for the applications specific needs. On the hardware side, it presents modular and scalable architecture that considers: the required number of input and output multimedia signals, the mixed analog and digital multimedia signals and their processing hardware components crosstalk to minimize the signal-to-noise ratio on the platform, and finally the low power consumption. On the processing side, a synchronization module is proposed and efficiently implemented to handle the startup delays and the deviating frame rates of the input multimedia signals. The system hardware and software were implemented for two case studies. A case study for fusion of multimedia signals of different modalities (visible and near infra-red (RGBN)), that is needed for modern smart phone cameras, is presented. Another case study for producing a 4K format required for larger displays is included, that stitches 9 high-definition videos simultaneously. The multimedia pipeline: decoding, processing, encoding were all realized and implemented successfully. The system performed in real-time of 30 frames per second. The platform end-to-end signal-to-noise ratio where above 56 and reaching 102 decibels, and the power consumption was below 2 Watts, making it suitable for real-time embedded multimedia systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Ahmad I (2007) Multi-view video: get ready for next-generation television. IEEE Distrib Syst Online 8(3):6–6

    Article  Google Scholar 

  2. Ancuti C, Ancuti CO, Haber T, Bekaert P (2012) Enhancing underwater images and videos by fusion. In: Proc. IEEE Conf computer vision pattern recognition, pp 81–88

  3. Awad M, Elliethy A, Aly HA (2018) A real-time FPGA implementation of visible/near infrared fusion based image enhancement. In: Proc. IEEE Int conf image processing, pp 3968–3972

  4. Bennett EP, Mason JL, McMillan L (2007) Multispectral bilateral video fusion. IEEE Trans Image Process 16(5):1185–1194

    Article  MathSciNet  Google Scholar 

  5. Brooks D (2000) Splitting planes for speed and power. Printed Circuit Design Mag. CMP Media publication

  6. Cheney DP, Curley LD, Lee WR, Richardson LD, Svec RS (2003) Integrated video processing system having multiple video sources and implementing picture-in-picture with on-screen display graphics. U.S. Patent 6519,283

  7. Desmouliers C, Aslan S, Oruklu E, Saniie J, Vallina FM (2010) HW/SW co-design platform for image and video processing applications on Virtex-5 FPGA using PICO. In: Proc. IEEE Int conf electro inf technol, pp 1–6

  8. Eastman NL (1996) Considerations for mixed analog/digital PCB design. In: The 1996 IEEE Wescon Conf., pp 297–301

  9. El-Hashash MM, Aly HA (2019) High-speed video haze removal algorithm for embedded systems. J Real-Time Image Process 16(4):1117–1128

    Article  Google Scholar 

  10. Elliethy AS, Aly HA, Sharma G (2014) View synthesis from wide-baseline views using occlusion aware estimation of large disparities. Proc SPIE Stereoscopic Displays Applic 9011:90111U

    Article  Google Scholar 

  11. Fouad MM, Aly HA (2016) A modified multiview video streaming system using 3-tier architecture. Adv Electr Electron Eng 14(2):196–204

    Google Scholar 

  12. Guo X, Wei X, Liu Y (2017) An FPGA implementation of multi-channel video processing and 4K real-time display system. In: Proc. IEEE Int congress on image and signal process., biomed eng and informat, pp 1–6

  13. Jack K (2004) Video demystified: a handbook for the digital engineer, 4th edn. Elsevier

  14. Jain DK, Jacob S, Alzubi J, Menon V (2019) An efficient and adaptable multimedia system for converting PAL to VGA in real-time video processing. J Real-Time Image Process, pp 1–13

  15. Khalifa AA, Aly HA (2015) Near-infrared video projection system based on digital micromirror devices and digital signal processors. Opt Eng 54(12):123102

    Article  Google Scholar 

  16. Khodary AG, Aly HA (2014) A new image-sequence haze removal system based on DM6446 Davinci processor. In: Proc. IEEE Global conf signal and inf process, pp 703–706

  17. Muller J, Muller J, Tetzlaff R (2014) A new high-speed real-time video processing platform. In: Proc. IEEE Int workshop cellular nanosc netw appl, pp 1–2

  18. Nilsson M (2015) Ultra high definition video formats and standardisation. BT Media and Broadcast Research Paper

  19. Ogrinc MA, Card RA, Burns CR, Clarke CP, Collier RL, Collins KM, Crane SE, Hersh C, Knittel BC, Mayer ST et al (1995) Real time video image processing system. U.S. Patent 5384,912

  20. Ott HW (2001) Partitioning and layout of a mixed-signal PCB. Printed Circuit Design Mag 18(6):8–11

    Google Scholar 

  21. Pandey JG, Karmakar A, Shekhar C (2012) An embedded architecture for implementation of a video acquisition module of a smart camera system. In: Proc. IEEE Int conf devices, circuits syst, pp 191–194

  22. Park M-W, Park G-H (2012) Realistic multi-view scalable video coding scheme. IEEE Trans Consum Electron 58(2):535–543

    Article  Google Scholar 

  23. Radiocommunication Sector of ITU (2007) BT.656: Interface for digital component video signals in and television systems operating at the 4:2:2 level of recommendation ITU-R BT.601, BT Series, Broadcasting service (television)

  24. Roska T, Chua LO (1993) The CNN universal machine: an analogic array computer. IEEE Trans Circ Syst II: Analog Digit Signal Process 40(3):163–173

    Article  Google Scholar 

  25. Said Y, Saidani T, Smach F, Atri M, Snoussi H (2012) Embedded real-time video processing system on FPGA. In: Proc. Int conf on image and signal process, pp 85–92

  26. Taylor RJ (1979) Video processing system. U.S. Patent 4148,070

  27. Tian D, Zou F, Vetro A (2014) Method and system for processing multiview videos for view synthesis using motion vector predictor list. U.S. Patent 8823,821

  28. Toledo FJ, Martinez JJ, Ferrandez JM (2007) FPGA-based platform for image and video processing embedded systems. In: Proc. 3rd Southern Conf. on programmable logic, pp 171–176

  29. Touil L, Kechiche L, Ouni B (2014) Generic SOPC platform for video interactive system with MPMC controller. Int J Embedded Syst Appl 4:1

    Google Scholar 

  30. Van Krevelen R, Poelman R (2010) A survey of augmented reality technologies, applications and limitations. Int J Virt Real 9(2):1–20

    Article  Google Scholar 

  31. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 13(4):600–612

    Article  Google Scholar 

  32. Yamazaki S (1991) Method and device for processing multiple video signals. U.S. Patent 5040,067

  33. Yu F, Liu L, He B, Huang Y, Shi C, Cai S, Song Y, Du S, Wan Q (2019) Analysis and FPGA realization of a novel 5D hyperchaotic four-wing memristive system, active control synchronization, and secure communication application. Complexity, 2019

Download references

Author information

Authors and Affiliations

  1. Military Technical College, Cairo, Egypt

    Mohamed Awad, Islam T. Abougindia, Ahmed Elliethy & Hussein A. Aly

Authors
  1. Mohamed Awad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Islam T. Abougindia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Elliethy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hussein A. Aly
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hussein A. Aly.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Awad, M., Abougindia, I.T., Elliethy, A. et al. Flexible architecture for real-time synchronized processing of multimedia signals. Multimed Tools Appl 80, 18531–18551 (2021). https://doi.org/10.1007/s11042-021-10575-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10575-y

Keywords

Search

Navigation