E0262 MIS - Multimedia Playback Systems Anandi Giridharan Electrical Communication Engineering, Indian Institute of Science, Bangalore – , India Synchronization of multimedia Data for a multimedia News-on-Demand Application

E0262 MIS - Multimedia Playback Systems Software control architecture for synchronizing multiple data streams generated from distributed media-storing database servers without the use of a global clock. A stream synchronization protocol (SSP) allows for synchronization recovery, ensuring a high quality multimedia display at the receiver. Different media are transmitted through ATM Network for Presentation on client workstation. Since client Work station may not have large space dedicated to multimedia news retrieval, Multimedia news document is progressively transmitted over the network.

E0262 MIS - Multimedia Playback Systems The articles are stored in a remote multimedia database. The news articles are composed of media such as text, images, graphics, animation, video, and audio. Each multimedia article has an associated temporal presentation scenario that is also stored in the multimedia database.. The media servers are of different types, such as image server, voice server, text and graphic server, video server, etc. Multimedia data retrieval and document delivery are performed continuously in real time.

E0262 MIS - Multimedia Playback Systems A voice stream requires small delay variances but can tolerate packet loss. On the other hand a video stream in a nonscalable coding format requires high bandwidth and is very sensitive to packet loss. In order to support this real time application, a delivery schedule is computed before transmitting the media objects and a real-time recovery process at the receiving end, is required to preserve the synchronization among the data carried different connections.

E0262 MIS - Multimedia Playback Systems Several synchronization control methods have been posed for solving the problem of stream synchronization. By inserting synchronization points in each individual stream. the concept of logic data unit (LDU). delay-jitter control scheme By regulating the random delay that has been introduced End-to-end synchronization delay. Computing schedules before transmitting the data packets

E0262 MIS - Multimedia Playback Systems Rate mismatches among multiple streams generated from physical devices are adjusted by skipping and pausing the data. Flow synchronization control protocol is proposed implementing end to end synchronization delay. Random delay experienced by each packet is regulated at destination node. Correction of asynchrony is made at source by speeding up or slowing down traffic on various media streams.

E0262 MIS - Multimedia Playback Systems There are two basic approaches for delivering multimedia Data. a. Real time delivery scheme, b. Store-and-forward scheme. The multimedia document is played back after all its components arrive. In order to support real time applications, a real time synchronization delivery method is proposed in this work

E0262 MIS - Multimedia Playback Systems A stream delivery schedule specifies the time for the independent media-servers to deliver their objects in order to meet the temporal relationship requested in the scenario. The scheduling relies on the QoS guaranteed by the network's virtual connections such as network delay, delay variance, throughput, etc. Besides the traffic prediction and scheduling of the object delivery, synchronization recovery is performed at the receiver before the playback of the multiple data streams. The Stream synchronization SSP uses skew tolerance parameters as error thresholds to guarantee the control for the different types of application data streams. In addition, the differences between the characteristics of the coded and uncoded-streams are taken into account. A stream delivery schedule specifies the time for the independent media-servers to deliver their objects in order to meet the temporal relationship requested in the scenario. The scheduling relies on the QoS guaranteed by the network's virtual connections such as network delay, delay variance, throughput, etc. Besides the traffic prediction and scheduling of the object delivery, synchronization recovery is performed at the receiver before the playback of the multiple data streams. A stream synchronization protocol (SSP) prevents the synchronization errors from being rampant in the display, even during network.

E0262 MIS - Multimedia Playback Systems A Delivery scheduling scheme.

E0262 MIS - Multimedia Playback Systems A SYNCHRONIZATION SCHEME SCHEDULING In the multimedia news application, when the user requests to view a news document, the request is transmitted to the multimedia database. The database sends the presentational scenario to the scheduler on the client's site. This scenario specifies the temporal relationships among the media objects involved in a particular document. The scheduler determines which media types are involved in the scenario and creates the corresponding client media synchronization controllers (MSCs). A client MSC is responsible for receiving and transferring to the user interface the objects of a particular media type that arrive on a stream connection. The scheduler is responsible for computing a delivery schedule for each media object involved in the document.

E0262 MIS - Multimedia Playback Systems The scheduler needs to know the network delay and delay variance between the media server and the user's site in order to compute the delivery schedule. The client QoS manager triggers a three-party quality of service negotiation between the client, the servers and the transport system.. The client QoS manager specifies the transmission characteristics for each medium stream. The results of the negotiation is sent to the media servers that translate the QoS parameters of their own media objects to transport system QoS parameters.

E0262 MIS - Multimedia Playback Systems Based on the transport QoS parameter values, the server MSCs can each open a connection to the remote user's workstation. The transport service QoS parameters include the maximum transport service data unit (TSDU) size, the transmission rate, end- to-end average delay (,U) and the delay variance (n'). If the transport system can reserve the required resources, the negotiation succeeds. Otherwise the server MSCs are notified of the reject cause and renegotiation can take place.

E0262 MIS - Multimedia Playback Systems Text l is the first object to be delivered, the delivery time offset of Textl is tl = 0, the video/audio object has t 2, the voice object voice has t 3, and so on. The server MSCs are informed of these delivery time offsets. When the user requests to start the presentation, the scheduler informs the involved server MSCs and thus triggers the document delivery. Each server MSC then delivers the data objects according to the scheduled time offset specified in the delivery schedule. The client MSCs handle the start-out mismatching for the application. A synchronization delay is introduced before the presentation schedule is applied in order io compensate for the mismatch.

E0262 MIS - Multimedia Playback Systems A SYNCHRONIZATION RECOVERY SCHEME Random network delay destroys the continuity of the data stream by introducing gaps and jitters during the data transmission. Therefore certain compensations at the receiver are necessary when synchronization errors occur A stream synchronization protocol (SSP) describes the protocol recovery operations of the client MSCs. The concept of an intentional delay is used by the various streams in order to adjust their presentation time to recover from network fluctuations.

E0262 MIS - Multimedia Playback Systems 120 ms mismatching in lip-synchronization between an audio and video stream will be perceived as disturbing by the application users. On the other hand, if text annotation is played out 200 ms earlier than voice, the presentation is still well accepted. The synchronization errors that can be tolerated by human perception vary in different application scenarios

E0262 MIS - Multimedia Playback Systems (a) Parameters in the presentation requirement; (b) Mismatch and recovery (a) Parameters in the presentation requirement; (b) Mismatch and

E0262 MIS - Multimedia Playback Systems The MSCs involved in the first activity of the scenario, get the video duration time length T v starting time difference for the text and voice activity T s,, Text and audio duration time T a and T t The skew tolerance time between the audio and the text objects T d1 & between the video and the audio kext objects T d 2 are also retrieved. The skew tolerance T dl is smaller than T d2 An intentional delay T i, is placed on the objects is introduced. Ti helps to decrease the interruption caused by filling gaps. MSC associated with video fills in gaps by preparing last frame While other MSCs are simply pause their display to screen

E0262 MIS - Multimedia Playback Systems The Audio and text delayed by T daudio and T dtext in first activity In second activity Intentional delays of T daudio and T daudio -T dtext are placed on the picture and text objects to preserve temporal relationship

E0262 MIS - Multimedia Playback Systems. For instance, if the video object arrives late, the audio and text MSCs can adjust their presentation within this activity by adding a intentional delay equal to the video object's delay. When no unexpected delays occur, the signaling traffic between the MSCs is reduced to a minimum. Alternatively, when network delay fluctuations occur, the idea of rescheduling the presentation within or the end of an activity allows for the synchronization to be recovered. This method also referred to as a time expanding policy, is very well suited for applications such as multimedia news on-demand services.

E0262 MIS - Multimedia Playback Systems