The MPEG-4 standard was defined in the early days of broadband Internet, after successful deployments of digital television networks, with the goal of unifying both broadcast and broadband media architectures and protocols in a single standard, tackling natural media (audio, video, images) as well as synthetic 2D or 3D graphics and audio. As such MPEG-4 can be seen as one of the first attempt at building the so-called convergence of Web and TV. Some parts of the standard have changed the media world forever (AAC audio and AVC|H264 video compression, MP4 file format), and while other parts have not always met their markets in successful way, they paved the way for more recent works, including HTML5 media. In this chapter, we explain how the MPEG-4 standard manages playback and synchronization of audio-visual streams and graphics animations and how multiple timelines can be used to provide rich interactive presentation over broadband and broadcast.
KeywordsInteractivity Synchronization Graphics Media Broadcast
time label assigned to a block of data in a media delivery system; timestamps are usually integer numbers, converted back to seconds using a timescale or timestamp resolution value expressing the number of integers in a second; for example, a timestamp of 180000 in a timescale of 90000 indicates a time of 2 s.
time at which a media frame has to be available for the decoder.
time at which a media frame must be presented to the user; composition and decoding times usually only differ for video stream using bi-directional motion estimation schemes, hence requiring a coding order different from the display order.
process and signaling required to reproduce a time-accurate, jitter-free playback of a given media stream; for example, signaling and maintaining the exact time lapse between successive video frames in a fixed or variable frame rate stream.
process and signaling required to reproduce a time-accurate playback of multiple media streams; for example, signaling and maintaining stream timestamps for achieve lip synchronization between an audio and a video stream.
value against which decoding (resp. composition) timestamps are compared to trigger the decoding (resp. display) process; depending on the delivery systems, the clock reference may be delivered together with the content, as is the case in broadcast environments, derived from the wall-clock time another time source such as internet time, or derived from any other local system clock/counter (CPU cycle count, audio output sample rate, etc.).
- 1.Pereira, F.C., Ebrahimi, T.: The MPEG-4 Book. Prentice Hall PTR, Upper Saddle River, NJ, USA (2002)Google Scholar
- 2.Van der Meer, J.: Fundamentals and Evolution of Mpeg-2 Systems: Paving the MPEG Road, 1st edn. Wiley (2014)Google Scholar
- 3.Real time streaming protocol. https://www.ietf.org/rfc/rfc2326.txt
- 4.Synchronized multimedia integration language. https://www.w3.org/TR/SMIL3/
- 5.Dufourd, J.C., Avaro, O., Concolato, C.: LASeR: the MPEG standard for rich media services. IEEE Multimedia (2005)Google Scholar
- 6.Scalable vector graphics. https://www.w3.org/TR/SVG12/
- 7.Hyper-text transfer protocol. https://tools.ietf.org/html/rfc7230
- 8.RTP payload format for transport of MPEG-4 elementary streams. https://tools.ietf.org/html/rfc3640
- 9.Hyper text transfer protocol version 2 (HTTP/2). https://tools.ietf.org/html/rfc7540
- 10.File delivery over unidirectional transport. https://tools.ietf.org/html/rfc6726
- 11.FCAST: object delivery for the asynchronous layered coding (ALC) and NACK-oriented reliable multicast (NORM) protocols. https://tools.ietf.org/html/rfc6968
- 14.ISO/IEC 14496-5, reference software, ISO/IEC publically available standards. http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=36086
- 15.Le Feuvre, J., Concolato, C., Moissinac, J. C.: GPAC: open source multimedia framework. In: Proceedings of the 15th ACM International Conference on Multimedia, pp. 1009–1012. ACM, Sept 2007. https://gpac.io