Multimedia Synchronization I. Fatimah Alzahrani

Definitions Multimedia System : A system or application that supports the integrated processing of several media types. Multimedia Synchronization : Synchronization in multimedia system refers to the temporal relations between media objects in a multimedia system.

Data Streams Data Stream is any sequence of individual packets transmitted in a time-dependent fashion – Packets can carry information of either continuous or discrete media. Transmission modes :  Asynchronous.  Synchronous.

Asynchronous In asynchronous transmission, the transmitter transmits data bytes at any instant of time. Only one byte is sent at a time. There is ideal time between two data bytes. Transmitter and Receiver operate at different clock frequencies. To help receiver ‘start’ and ‘stop’ bits are used along with data in middle. Ideal time between byte is not constant. They are also known as gaps. In asynchronous transmission timing of signal is not important.

Synchronous In synchronous transmission, data transmission is carried out under the control of a common master clock. Bytes are transmitted as a block in a continuous stream of bits. Transmitter and Receiver operate at synchronised clock frequencies. No ‘start’ and ‘stop’ bits are used. No need of ideal time between data bytes. In synchronous transmission timing of signal is important.

Two types of media data : 1.Continuous : Time-Dependent Ex :- Real-Service (Live Video). 2.Discrete : Time-Independent Ex :- Text or Graphic. Three types of relationships : 1. Content. 2.Spatial. 3.Temporal.

Content Relationships Content Relation: defines dependency of media objects for some data.  Example: dependency between a filled spreadsheet and graphics that represent data listed in spreadsheet.

Spatial Relationship Spatial relation is represented through layout relation and defines space used for presentation of a media object on an output device at a certain point of time in a multimedia document. Example: desktop publishing  Layout frames are placed on an output device and a content is assigned to this frame Positioning of layout frames:  Fixed to a position of a document  Fixed to a position on a page  Relative to the position of other frame Concept of frames is used for positioning of time-dependent objects  Example: in window-based systems, layout frames correspond to windows and video can be positioned in a window

Temporal Relationship Temporal relation defines temporal dependencies between media objects. Example: lip synchronization Time-dependent objects – represent a media stream because there exists temporal relations between consecutive units of the stream

Cont. Temporal Relations Temporal synchronization is supported by many system components: OS (CPU scheduling). Communication systems (traffic shaping, network scheduling).

1.Intra-Media Synchronization: Time Relation Between Various Presentation Units of One Time-Dependent Media Object. Ex : Time relation between the single frames of a video sequence. For a video witha rate of 25 frames per second each of the frames has to be displayed for 40 ms. Frames of a video sequence that shows a jumping ball :

2.Inter-Media Synchronization : The Synchronization Between Media Objects. Ex : Time Relations of A Multimedia Synchronization That Starts With Audio/Video Sequence, Followed By Several Pictures and An Animation That is Commented By An Audio Sequence.

Logical Data Unit (LDU) Hierarchy : LDU is the unit of a media stream. Different streams have different LDU size. Closed LDU ▫Predictable duration ▫Ex: Stored media objects Open LDU ▫Duration not known before the presentation starts ▫Ex: Input from live source

Logic Data Unit Physical units as LDU or blocks of samples as LDU

Logic Data Unit User-defined vs capturing defined.

Live Synchronization Goal is to exactly reproduce at a presentation the temporal relations as they existed during the capturing process Need to capture temporal relation information during the capturing Live Sync is needed in conversational services Video Conferencing, Video Phone.

Lip Synchronization Lip synchronization ▫Coupling between audio and video.

Reference Models for Multimedia Synchronization Focus on the following four-layer model  Specification Layer - contains applications and tools for creating synchronization specification. It can be classified into :  (time) Interval-based specification  Axes-based specification  Control flow-based specification (given synchronization points)  Event-based specification.  Object Layer - presentation object synch  Stream Layer - inter-stream synch of media streams  Media Layer - intra-stream synch of continious media

Synchronization Specification Methods Interval-based Specification  presentation duration of an object is considered as interval Examples of operations: A before(0) B, A overlap B, A starts B, A equals B, A during B, A while(0,0) B Advantage: easy to handle open LDUs ad therefore user interactions Disadvantage: complex specifications

Audio1 while (0,0) Video Audio1 before RecordedInteraction RecordedInteraction before (0) B1 P1 before (0) P2 P2 before (0) P3 P3 before (0) Interaction P3 before (0) Animation Animation while (2,5) Audio2 Interaction before (0) P4.

Synchronization Specification Methods Control Flow-based Specification – Hierarchical Approach.  Basic hierarchical specification: 1. serial synchronization, 2. parallel synchronization of actions. Advantage: easy to understand, natural support of hierarchy and integration of interactive objects is easy Disadvantage: Additional description of skews and QoS is necessary, we must add presentation durations

Synchronization in Distributed Environment(1) Information of synchronization must be transmitted with audio and video streams, so that the receiver side can synchronize the streams Delivery of complete sync information can be done before the start of presentation  This is used in synthetic synchronization Advantage: simple implementation Disadvantage: presentation delay Deliver of complete sync information can be using out-of-band communication via a separate sync channel  This is used in live synchronization Advantage: no additional presentation delays Disadvantage: additional channel is needed; additional errors can occur

Synchronization in Distributed Environment(2) Delivery of complete synchronization information can be done using in-band communication via multiplexed data streams, i.e., synchronization information is in headers of the multimedia PDU Advantage: related sync information is delivered together with media units Disadvantage: difficult to use for multiple sources

Synchronization in Distributed Environment(3) (Location of Synchronization Operations) It is possible to synchronize media objects by recording objects together and leave them together as one object, i.e., combine objects into a new media object during creation; Synchronization operation happens then at the recording site. Synchronization operation can be placed at the sink. In this case the demand on bandwidth is larger because additional sync information must be transported. Synchronization operation can be placed at the source. In this case the demand on bandwidth is smaller because the streams are multiplexed according to synchronization requirements.

Synchronization in Distributed Environment(4) (Clock Synchronization) Consider synchronization accuracy between clocks at source and destination. Global time-based synchronization needs clock Synchronization. In order to re-synchronize, we can allocate buffers at the sink and start transmission of audio and video in advance, or use NTP (Network Time Protocol) to bound the maximum clock offset.

Audio1 while (0,0) Video Audio1 before RecordedInteraction RecordedInteraction before (0) B1 P1 before (0) P2 P2 before (0) P3 P3 before (0) Interaction P3 before (0) Animation Animation while (2,5) Audio2 Interaction before (0) P4.

Multimedia Applications Multimedia plays major role in following areas Instruction Business Advertisements Training materials Presentations Customer support services Entertainment Interactive Games

Multimedia Applications Enabling Technology Accessibility to web based materials. Teaching-learning disabled children & adults. Fine Arts & Humanities Museum tours. Art exhibitions. Presentations of literature.