Skip to main content
SpringerLink
Log in

Architectures for Multimedia Processing: A Cross-Layer Perspective

  • Living reference work entry
  • First Online:
Handbook of Computer Architecture

  • 147 Accesses

Abstract

Video streaming, as a domain, is one of the largest consumers of network bandwidth and accounts for more than 60% of downstream Internet traffic. As of January 2021, video streaming applications, such as Netflix, YouTube, Amazon Prime Video, HBO Max, etc., accounted for 66.2% of the global mobile data usage every month, widely overcoming other applications like social networking, web browsing, etc. This requires the research and development of energy-efficient hardware and software architectures that can be deployed on handheld battery-operated devices like smartphones, to ensure that they can satisfy user requirements and application quality constraints. This chapter discusses a heterogeneous hardware-software approach that can be used to develop architectures for video coding systems, including the investigation of quality-tolerance techniques that can maximize energy efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Bitmovin (2019) Video developer report. https://bitmovin.com/bitmovin-2019-video-developer-report-av1-codec-ai-machine-learning-low-latency/

  • Bjontegaard G (2001) Calculation of average PSNR differences between RD-curves. VCEG-M33

    Google Scholar 

  • Bossen F, Bross B, Suhring K, Flynn D (2012) HEVC complexity and implementation analysis. IEEE Trans Circuits Syst Video Technol 22(12):1685–1696

    Article  Google Scholar 

  • Bross B, Chen J, Ohm JR, Sullivan GJ, Wang YK (2021) Developments in international video coding standardization after AVC, with an overview of versatile video coding (VVC). In: Proceedings of the IEEE

    Book  Google Scholar 

  • Chi P, Li S, Xu C, Zhang T, Zhao J, Liu Y, Wang Y, Xie Y (2016) Prime: a novel processing-in-memory architecture for neural network computation in reram-based main memory. ACM SIGARCH Comput Archit News 44(3):27–39

    Article  Google Scholar 

  • Diniz CM, Shafique M, Bampi S, Henkel J (2013) High-throughput interpolation hardware architecture with coarse-grained reconfigurable datapaths for HEVC. In: 2013 IEEE international conference on image processing. IEEE, pp 2091–2095

    Google Scholar 

  • El-Harouni W, Rehman S, Prabakaran BS, Kumar A, Hafiz R, Shafique M (2017) Embracing approximate computing for energy-efficient motion estimation in high efficiency video coding. In: Design, automation & test in Europe conference & exhibition (DATE), 2017. IEEE, pp 1384–1389

    Google Scholar 

  • Financial Express (2020) Rise of paid subscribers. https://www.financialexpress.com/brandwagon/2020-rise-of-paid-subscribers/2172942/

  • Grellert M, Shafique M, Khan MUK, Agostini L, Mattos JC, Henkel J (2013) An adaptive workload management scheme for HEVC encoding. In: 2013 IEEE international conference on image processing. IEEE, pp 1850–1854

    Google Scholar 

  • Hanzo L, Cherriman P, Streit J (2007) Video compression and communications: from basics to H. 261, H. 263, H. 264, MPEG4 for DVB and HSDPA-style adaptive turbo-transceivers. Wiley, Hoboken

    Google Scholar 

  • Javaid H, Shafique M, Parameswaran S, Henkel J (2011) Low-power adaptive pipelined mpsocs for multimedia: an H. 264 video encoder case study. In: 2011 48th ACM/EDAC/IEEE design automation conference (DAC). IEEE, pp 1032–1037

    Google Scholar 

  • Khan MUK, Shafique M, Grellert M, Henkel J (2013a) Hardware-software collaborative complexity reduction scheme for the emerging HEVC intra encoder. In: 2013 design, automation & test in Europe conference & exhibition (DATE). IEEE, pp 125–128

    Google Scholar 

  • Khan MUK, Shafique M, Henkel J (2013b) An adaptive complexity reduction scheme with fast prediction unit decision for HEVC intra encoding. In: 2013 IEEE international conference on image processing. IEEE, pp 1578–1582

    Google Scholar 

  • Khan MUK, Shafique M, Henkel J (2014) Software architecture of high efficiency video coding for many-core systems with power-efficient workload balancing. In: 2014 design, automation & test in Europe conference & exhibition (DATE). IEEE, pp 1–6

    Google Scholar 

  • Khan MUK, Shafique M, Bauer L, Henkel J (2015) Multicast fullhd H. 264 intra video encoder architecture. IEEE Trans Comput-Aided Des Integr Circuits Syst 34(12):2049–2053

    Article  Google Scholar 

  • Khan MUK, Shafique M, Henkel J (2017) Energy efficient embedded video processing systems: a hardware-software collaborative approach. Springer, Berlin

    Google Scholar 

  • LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  • Lesniak F, Kreß F, Becker J (2021) Transparent near-memory computing with a reconfigurable processor. In: International symposium on applied reconfigurable computing. Springer, pp 221–231

    Google Scholar 

  • Matthew Ball (2020) The impact of COVID-19 on pay-tv and ott video. https://www.matthewball.vc/all/covidvideo

  • Palomino D, Shafique M, Amrouch H, Susin A, Henkel J (2014) HEVCDTM: application-driven dynamic thermal management for high efficiency video coding. In: 2014 design, automation & test in Europe conference & exhibition (DATE). IEEE, pp 1–4

    Google Scholar 

  • Sampaio F, Zatt B, Shafique M, Agostini L, Henkel J, Bampi S (2013) Content-adaptive reference frame compression based on intra-frame prediction for multiview video coding. In: 2013 IEEE international conference on image processing. IEEE, pp 1831–1835

    Google Scholar 

  • Sampaio F, Shafique M, Zatt B, Bampi S, Henkel J (2014) DSVM: energy-efficient distributed scratchpad video memory architecture for the next-generation high efficiency video coding. In: 2014 design, automation & test in Europe conference & exhibition (DATE). IEEE, pp 1–6

    Google Scholar 

  • Sampaio F, Shafique M, Zatt B, Bampi S, Henkel J (2015) Approximation-aware multi-level cells STT-RAM cache architecture. In: 2015 international conference on compilers, architecture and synthesis for embedded systems (CASES). IEEE, pp 79–88

    Google Scholar 

  • Sandvine (2019) Global internet phenomena report. https://www.sandvine.com/press-releases/sandvine-releases-2019-global-internet-phenomena-report

  • Shafique M, Henkel J (2011) Hardware/software architectures for low-power embedded multimedia systems. Springer Science & Business Media, Berlin

    Book  Google Scholar 

  • Shafique M, Henkel J (2014) Low power design of the next-generation high efficiency video coding. In: 2014 19th Asia and South Pacific design automation conference (ASP-DAC). IEEE, pp 274–281

    Google Scholar 

  • Shafique M, Zatt B (2012) A complexity reduction scheme with adaptive search direction and mode elimination for multiview video coding. In: 2012 picture coding symposium. IEEE, pp 105–108

    Google Scholar 

  • Shafique M, Bauer L, Henkel J (2007) An optimized application architecture of the H. 264 video encoder for application specific platforms. In: 2007 IEEE/ACM/IFIP workshop on embedded systems for real-time multimedia. IEEE, pp 119–124

    Google Scholar 

  • Shafique M, Bauer L, Henkel J (2008) 3-tier dynamically adaptive power-aware motion estimator for H. 264/avc video encoding. In: Proceeding of the 13th international symposium on low power electronics and design (ISLPED’08). IEEE, pp 147–152

    Google Scholar 

  • Shafique M, Bauer L, Henkel J (2009a) A parallel approach for high performance hardware design of intra prediction in H. 264/avc video CODEC. In: 2009 design, automation & test in Europe conference & exhibition. IEEE, pp 1434–1439

    Google Scholar 

  • Shafique M, Molkenthin B, Henkel J (2009b) Non-linear rate control for H. 264/avc video encoder with multiple picture types using image-statistics and motion-based macroblock prioritization. In: 2009 16th IEEE international conference on image processing (ICIP). IEEE, pp 3429–3432

    Google Scholar 

  • Shafique M, Bauer L, Henkel J (2010a) enbudget: a run-time adaptive predictive energy-budgeting scheme for energy-aware motion estimation in H. 264/mpeg-4 avc video encoder. In: 2010 design, automation & test in Europe conference & exhibition (DATE 2010). IEEE, pp 1725–1730

    Google Scholar 

  • Shafique M, Bauer L, Henkel J (2010b) Optimizing the H. 264/avc video encoder application structure for reconfigurable and application-specific platforms. J Sig Process Syst 60(2):183–210

    Article  Google Scholar 

  • Shafique M, Molkenthin B, Henkel J (2010c) An HVS-based adaptive computational complexity reduction scheme for H. 264/avc video encoder using prognostic early mode exclusion. In: 2010 design, automation & test in Europe conference & exhibition (DATE 2010). IEEE, pp 1713–1718

    Google Scholar 

  • Shafique M, Zatt B, Walter FL, Bampi S, Henkel J (2012) Adaptive power management of on-chip video memory for multiview video coding. In: DAC design automation conference 2012. IEEE, pp 866–875

    Google Scholar 

  • Singh G, Chelini L, Corda S, Awan AJ, Stuijk S, Jordans R, Corporaal H, Boonstra AJ (2019) Near-memory computing: past, present, and future. Microprocess Microsyst 71:102868

    Article  Google Scholar 

  • Statista Research Department (2021) Global mobile data traffic share. https://www.statista.com/statistics/383715/global-mobile-data-traffic-share/

  • Sullivan GJ, Ohm JR, Han WJ, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22(12):1649–1668

    Article  Google Scholar 

  • Teimoori MT, Hanif MA, Ejlali A, Shafique M (2018) Adam: adaptive approximation management for the non-volatile memory hierarchies. In: 2018 design, automation & test in Europe conference & exhibition (DATE). IEEE, pp 785–790

    Google Scholar 

  • Vanne J, Viitanen M, Hamalainen TD, Hallapuro A (2012) Comparative rate-distortion-complexity analysis of HEVC and avc video codecs. IEEE Trans Circuits Syst Video Technol 22(12):1885–1898

    Article  Google Scholar 

  • Vizzotto BB, Zatt B, Shafique M, Bampi S, Henkel J (2012) A model predictive controller for frame-level rate control in multiview video coding. In: 2012 IEEE international conference on multimedia and expo. IEEE, pp 485–490

    Google Scholar 

  • Wang TC, Mallya A, Liu MY (2021) One-shot free-view neural talking-head synthesis for video conferencing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10039–10049

    Google Scholar 

  • Wien M, Bross B (2020) Versatile video coding–algorithms and specification. In: 2020 IEEE international conference on visual communications and image processing (VCIP). IEEE, pp 1–3

    Google Scholar 

  • World Health Organization (2021) COVID-19 weekly epidemiological update, edition 46, 29 june 2021. https://www.financialexpress.com/brandwagon/2020-rise-of-paid-subscribers/2172942/

  • Zadtootaghaj S, Schmidt S, Barman N, Möller S, Martini MG (2018) A classification of video games based on game characteristics linked to video coding complexity. In: 2018 16th annual workshop on network and systems support for games (NetGames). IEEE, pp 1–6

    Google Scholar 

  • Zatt B, Shafique M, Bampi S, Henkel J (2011a) A low-power memory architecture with application-aware power management for motion & disparity estimation in multiview video coding. In: 2011 IEEE/ACM international conference on computer-aided design (ICCAD). IEEE, pp 40–47

    Google Scholar 

  • Zatt B, Shafique M, Sampaio F, Agostini L, Bampi S, Henkel J (2011b) Run-time adaptive energy-aware motion and disparity estimation in multiview video coding. In: 2011 48th ACM/EDAC/IEEE design automation conference (DAC). IEEE, pp 1026–1031

    Google Scholar 

  • Zhou Y, Tian L, Zhu C, Jin X, Sun Y (2019) Video coding optimization for virtual reality 360-degree source. IEEE J Sel Top Sig Process 14(1):118–129

    Article  Google Scholar 

Download references

Acknowledgements

We would like to explicitly thank Felipe Sampaio, Bruno Zatt, Sergio Bampi, Daniel Palomino, Muhammad Usman Karim Khan, and Jörg Henkel for their contributions to parts of the works cited in this chapter. We would also like to thank other researchers in industry and academic alike, especially the ones cited in this work, who contributed to this field to enable advancements that helped us realize the potential of video coding across multiple domains.

Author information

Authors and Affiliations

  1. Engineering Division, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates

    Muhammad Shafique

  2. Institute of Computer Engineering, Technische Universität Wien (TU Wien), Vienna, Austria

    Bharath Srinivas Prabakaran

Authors
  1. Muhammad Shafique
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bharath Srinivas Prabakaran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Shafique .

Editor information

Editors and Affiliations

  1. Sch of Computer Science & Engineering, Nanyang Technological University, Singapore, Singapore

    Anupam Chattopadhyay

Section Editor information

  1. Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore, Singapore

    Anupam Chattopadhyay

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Singapore Pte Ltd.

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Shafique, M., Prabakaran, B.S. (2022). Architectures for Multimedia Processing: A Cross-Layer Perspective. In: Chattopadhyay, A. (eds) Handbook of Computer Architecture. Springer, Singapore. https://doi.org/10.1007/978-981-15-6401-7_7-1

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-6401-7_7-1

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-6401-7

  • Online ISBN: 978-981-15-6401-7

  • eBook Packages: Springer Reference EngineeringReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation