1. 1 Synchronization TTM4142, 2007 Harald Øverby/Leif Arne Rønningen

2. 2 Outline What is synchronization Relationships Logical Data Units (LDU) Reference model for synchronization Specification

3. 3 Definition of synchronization Synchronization The act or result of synchronizing; the state of being synchronous Synchronize To happen at the same time; to represent or arrange (events) to indicate coincidence or coexistence; to make synchronous in operation; to make (motion or picture sound) exactly simultaneous with the action Source: Merriam-Webster dictionary

4. 4 Definition of synchronization Synchronization creates relationships between independent objects –Time dependent / time independent objects Three types of relationships –Content –Spatial –Temporal

5. 5 Relationships

6. 6 Content relationships Connection between media objects and data Example –Between data in a table and a picture Update in data triggers update in other views Consistency

7. 7 Spatial relationships Layout relationships Define space of a media object –Objects arranged in a two dimensional area –Output device Content assigned to frames –Positioning: fixed, relative Example –Powerpoint slide –Web page

8. 8 Temporal relationships Time relationships between objects –Important for time dependent media objects Examples –Timing relationships between audio object and video object Further on: focus on temporal relationships

9. 9 Intra object synchronization Time relation between various presentation units of one time-dependent media object Example: bouncing ball

10. 10 Inter object synchronization Time relation between two or more media objects Examples: Audio, Pictures (P), Video, Animation

11. 11 Logical data unit (LDU) LDU: Information unit of a time-dependent media object Granularity of LDU –Video: Frame, object, pixel

12. 12 Logical data unit (LDU) Closed LDU –Predictable duration –Ex: Stored media objects Open LDU –Duration not known before the presentation starts –Ex: Input from live source

13. 13 Logical data unit (LDU) Physical units as LDU or blocks of samples as LDU

14. 14 Logical data unit (LDU) User-defined vs capturing defined

15. 15 Logical data unit (LDU)

16. 16 Synchronization in multimedia systems Television systems –Sound and video are multiplexed –Delay affects both sound and video –Synchronization always maintained Focus on time-independent media –LDU to be put in relationship are independent media objects Quality of Service –Hard guarantees –Soft guarantees

17. 17 Synchronization examples Lip synchronization –Coupling between audio and video

18. 18 Lip synchronization experiment Skew between video and audio +/- 40 ms intervals Positive skew: audio ahead of video Short clips

19. 19 Example questionnaire

20. 20 Experiments - results

21. 21 Experiments - results

22. 22 Results summary Asymmetry –Positive skew more disturbing Difficult to detect errors when object far away Statistical errors Long movie –+/- 80 ms : no distraction –-240 ms, +160 ms : disturbing

23. 23 QoS: Skew values

24. 24 Reference Models for Multimedia Synchronization Support to understand requirements Many proposals Focus on the following four-layer model –Specification Layer –Object Layer - presentation object synch –Stream Layer - inter-stream synch of media streams –Media Layer - intra-stream synch of continious media

25. 25 Reference model for synchronization

26. 26 Media Layer Application handles only one continuous media stream as an LDU sequence Device-independent interface, file and device access –Read (device-handle, LDU) –Write (device-handle, LDU) –etc Application runs one process for each stream Guarantees single LDU processing

27. 27 Stream Layer Handles continuous media streams and media stream groups Guarantees intra-stream synch Group presents streams in parallel Gurarantees inter-stream synch of streams in a group Handles resource reservation and scheduling of LDU processing Typical operations –start (stream), stop (stream) –create_group (list_of_streams) –start (group), stop (group) –setcuepoint (stream/group, at, event)

28. 28 Object Layer 1 Deals with all kinds of media Hides the difference between discrete and continuous media Accepts a synchronization specification as input Media-oriented QoS (such as allowed skew, allowed jitter) Plans and coordinates presentation execution Initiates presentation of time-dependent and time- independent media objects 1 In DMP the term could have been Scene Layer

29. 29 Specification Layer Does not provide a specific interface, is open Tools for generating synch specification –Edit –Format Maps user interface QoS specifications to the Object Layer

30. 30 Synchronization Specification, main categories Interval-based –Temporal relationships between the presentation of media objects Axis-based –Presentation objects placed in common axes Control-flow-based –Presentation flow synchronized at certain points Event-based –Events trigger presentation actions

31. 31 Synchronization in a distributed environment More complex than a local environment –Distributed storage of media –Distributed storage of synchronization information –Communications delay and delay jitter –Multi-party communication patterns

32. 32 Transport of synchronization specification Additional synchronization channel Multiplexed channel

33. 33 Location of synchronization operations

34. 34 Multiple communications

35. 35 Multi-step Synchronization Synchronization during object acquirement of retrieval during delivery of the LDUs to the network during the transport at the sink within the output device

36. 36 Synchronization Specification Complex specification comprise: multiple object synchronization user interactions Requirements for a specification method: Support of object consistency and for maintenance of specifications Media objects should be kept as logical units in the specification All types of synchronization relations should be easily describable Support for the integration of time-dependent and time-independent media objects Quality of service requirements must be describable Support of hierarchical levels of a synchronization

37. 37 Interval-based Specification

38. 38 Synchronization Based on a Global Timer Media objects

39. 39 Synchronization Based on Virtual Axes

40. 40 Synchronization Based on Reference Points

41. 41 Petri Net Specification

42. 42

43. 43 Event-based Synchronization