Temporal relationships.

What is meant by temporal relationship ? Temporal (timing) relationships are important in a multimedia presentation. Ex: A speaker’s lip movement and the words she utters must match in a video clip Type of temporal relationship and accuracy depend upon: application and the type of media used. Deviations to the required temporal relationships can occur on the transmitting end, on the communication link, and on the receiving end.

Topic of study Fundamentals of temporal relationships Classification and specification of temporal relationships

. Temporal relationships based on object stream interaction Intramedia Interaction between the objects of a single stream. Ex: Animation without sound Let an animation clip is stored at the rate of 14 frames per sec During playback the rate will be 14 frames per sec Intermedia Interaction between the objects of two parallel streams. Ex: Video clip with sound A precise temporal relationship must be maintained between the image frames and the digitized voice to maintain lip sync

. User interaction based temporal relationship Interactivity being discussed here relates to the interaction between the user and the system The response time of a standalone or networked interactive multimedia system should be small and consistent

. Temporal relationships based on Media Levity Live presentations Information captured, transmitted, and presented in real-time. Ex: Video conference is an example of a live presentation Requirement: minimum end-to-end delay. Stored Presentations Captured and stored on secondary storage systems. Strict real-time constraint during retrieval. Video on demand is an example of a stored and retrieved non live presentation Mixed presentation Uses a combination of live and stored information. Example: Collaborative conference One window on the screen shows live pictures while other windows are used to show stored information

Temporal relationship and isochronous multimedia A video clip played without sound If it was stored at 24 fps,then it must be played back at 24 fps It is not acceptable to play the video at 20fps in one sec and then compensate by playing it at 28 fps in the next

Temporal relationship and synchronous multimedia Video clip when played along with audio is a synchronous media Sound and video streams should play isochronously at correct rate Must also be synchronized wrt each other

SYNCHRONIZATION ACCURACY SPECIFICATION The factors used to specify synchronization accuracy are: Delay, Jitter, Skew, Error rate

. Delay Interactive application: response time. Streamed application: time take to traverse the network. Jitter shaky picture. quivering voice. Skew slow or fast moving picture. lower or higher pitch than the normal.

. SYNCHRONIZATION MECHANISMS Introducing intentional delay - removing jitter by buffering: All delay jitter can be removed by buffering. Not desirable for most applications. Acceptable for One way communication applications such as: NVoD (near video on demand). Not acceptable for Interactive applications, such as: FVoD (full video on demand) and other Needs larger buffer.

Processing delays: Transmitting end Stored objects: Retrieval delays for stored objects: Tquery is the time taken to process and transform the query into the location of the objects on the storage medium Tseek is the time taken to locate the objects on the storage medium Taccess is the time taken to read and transfer the object from the medium These delays are not applicable to objects captured in real time

Capturing delays for real-time objects Apply to objects such as audio and video captured in real-time In natural form all real-life quantities are analog in nature For digital processing must be converted to digital values Tsample is the time taken for converting analog to digital value Tencode is the time for encoding like compression or encryption On the contrary stored objects are ready for transmission

Networking delays Networking delays apply to stored as well as captured objects Tpacketize is the time taken to break the object data into packets Ttransmit is the time taken to pass the object through the network interface Ex:Nbits/Rbps Tpropagate is the time delay from the instant the object is into the network on the transmitting end to the instant when it comes out of receiving end In ckt switching propagation delay=Distance/speed of transmission

Processing delays: receiving end Data must be buffered on the receiving end also Tbuffer is the delay added due to buffering at the receiving end Buffere delay may also happen in the transmitting end The recived data must be depacketized Tdepacketize delay Tdecode delay is time for decoding at the reciving end