1Intelligent Tools for Media Workshop in Media Authoring Automatic Generation of Animation and Video Contact Doron Friedman Mobile:

2Intelligent Tools for Media Agenda Introduction Motivation and Background Motivation and Background Course Material Course Material Logistics and Administration Description of Projects

3Intelligent Tools for Media Motivation – Media Authoring 3D: 3D: Professional animators – several years of study Professional software Low leve of abstaction Very little reuse Video Video Professional editors Professional software Pixel based (rather than object based) Pixel based (rather than object based)

4Intelligent Tools for Media Smart Graphics What this is NOT about How to achieve efficient 3D rendering How to achieve efficient 3D rendering What this IS about How to quickly generate animation from high level input How to quickly generate animation from high level input

5Intelligent Tools for Media Making movies (Simplified) Story/idea ScriptShooting ScriptAnimation File Story- understanding Storytelling Directing NLUReusable animation Animation composition Text-to-speech Theory of space and time Camera-control principles Model of actions

6Intelligent Tools for Media Script -> Animation Formal Script Shooting Script Animation File Theory of space and time Camera-control principles Model of actions Reusable animation Animation composition Text-to-speech

7Intelligent Tools for Media Interactive Virtual Environments Linear Animation Authoring Authoring Interactive Virtual Environments Authoring Authoring Agent Agent Real-time Director Real-time Director

8Intelligent Tools for Media Applications Production Industry Rapid prototyping Rapid prototyping Storyboarding Storyboarding Gaming Industry Engine for interactive games Engine for interactive games A movie-maker game A movie-maker game Toys (Lego/AIBO) Toys (Lego/AIBO)EducationTrainingSimulations Industrial Military Traffic Scientific Medical

9Intelligent Tools for Media State of the trade today Leading tools (Maya, RenderMan,..) Leading tools (Maya, RenderMan,..) Leading standards/trends (VRML, Java3D, …) Leading standards/trends (VRML, Java3D, …) How authors work today How authors work today

10Intelligent Tools for Media SMIL-1 - Time Par Par Seq Seq begin begin Layout –Relative/absolute positioning –Z-index –Fit (meet, slice, scroll)

11Intelligent Tools for Media SMIL-2 45 Modules / Language Profiles Integration with other languages Vector graphics Excl (exclusive par) Relative synchronization Time manipulation (speed, reverse) AnimationAudioLayoutTransitions

12Intelligent Tools for Media MPEG-4 as a Rich Media Representation Scheme Several goals Profile Approach

13Intelligent Tools for Media Mpeg-4 Relevant Features TermCap node: –Frame rate –Memory/CPU load BIFS switch node

14Intelligent Tools for Media MPEG-7 – the Bits about the Bits D – Descriptors – representation of features (syntax + semantics) DS – Description Schemes – structure and semantics of relationships between components DDL – Description Defintion Language – to create new DS, and perhaps new D

15Intelligent Tools for Media Example – Multiple Description Types Structural – right, left, moving towards Physical – color, shape, motion Conceptual – goalkeeper, player, ball

16Intelligent Tools for Media MPEG-21 Define a multimedia framework to enable transparent and augmented use of multimedia resources across a wide range of networks and devices 7 Items Digital Item Declaration Digital Item Declaration Digital Item Identification and Description Digital Item Identification and Description Content Handling and Usage Content Handling and Usage Intellectual Property Management and Protection Intellectual Property Management and Protection Terminals and Networks Terminals and Networks Content Representation Content Representation Event Reporting Event Reporting

17Intelligent Tools for Media Course Material Background: OO Design and Programming (C++/Java, tbd) 3D technologies: VRML VRML HANIM HANIM Java3D ? Java3D ? MPEG-4/BIFS/XMT MPEG-4/BIFS/XMT Video/Mixed Technologies MPEG-4 MPEG-4 JMF JMF SMIL-1/SMIL-2 SMIL-1/SMIL-2

18Intelligent Tools for Media Course Logistics Communication channels Groups Development methodology Dates

19Intelligent Tools for Media Communication Channels Mailing list: tba Web-site: 2.html 2.html 2.html Includes links to resources My

20Intelligent Tools for Media Groups 1-4 people in group (recommended: 2) Group gets one grade Inter-group integration – big bonus!

21Intelligent Tools for Media Development Methodology Some thoughts on Software Engineering: Writing programs is easy, writing software is impossible Writing programs is easy, writing software is impossible % of code used % of code used Legacy software Legacy software

22Intelligent Tools for Media Development Process (waterfall) MRD (Marketing Requirements Document) HLD (High Level Design) FRS (Functional Requirement Specification) ATP (Acceptance Testing Procedures) CodeDebug Testing / Q.A.

23Intelligent Tools for Media Projects 1.Animation Library 2.Automatic Summaries 3.Video Template Authoring 4.Object-based Video 5.Film & TV School projects

24Intelligent Tools for Media Animation Library Objects with animation Humanoid Humanoid Car Car Door Door 4-legged animal 4-legged animal Elevator Elevator Bottle, glass and drink Bottle, glass and drink Cigarette and smoke Cigarette and smoke Weapons Weapons …

25Intelligent Tools for Media Animation Library - Humanoid Stand Walk (from, to), (path) RunJump Sit (on) take (object) Put (object, location) Touch (object/organ) WaveTalkEat/drink Climb (wall, stairs,..)

26Intelligent Tools for Media Animation Library - Car Drive (from, to), (path) Open (door) Close (door) Traffic simulation

27Intelligent Tools for Media Animation Library - Speech Text-to-speech Lip animation (sync?)

28Intelligent Tools for Media Animation Library – Facial Expressions Basic emotions (smile, angry, afraid) Low level animation Eyes (look-at, close, open, blink) Eye-brows Mouth Face features (hair color, hair style, eye color, glasses)

29Intelligent Tools for Media Animation Library - Other Door (open, close) 4-legged (share animation with humanoid) Elevator (open, close, go-to-floor) Bottle, glass, drink (pour, person drinking, spill) Gun (shoot, bullet/missile) …

30Intelligent Tools for Media Animation Composition Sequence (states) Parallel (walk, run, Jump) x (hold, wave) (walk, run, Jump) x (hold, wave) Compound Animations (hierarchy) E.g: eat = take (food), touch(mouth) E.g: eat = take (food), touch(mouth) Inverse Kinematics

31Intelligent Tools for Media Smart Objects Different Complexity Levels of Behavior Default behaviors Default behaviors Rules and constraints (trigger -> action) Rules and constraints (trigger -> action) Behavior model (perception, motivation, planning) Behavior model (perception, motivation, planning) High level interface to expose capabilities Naïve physics Examples Avatar Avatar Tamaguchi/pet Tamaguchi/pet Car Car Elevator Elevator

32Intelligent Tools for Media Problem specification -example Smart lift in crowd simulation

33Intelligent Tools for Media Defining object behaviors - example smart table in party simulation graspable fruits get_closest_ fruit (and eat it) get_closest_ fruit (and eat it) replace_fruits replace_fruits

34Intelligent Tools for Media Script Parsing Define syntax Define internal representation Parse input, create representation, and write to file Challenges Action Action Space (near, behind, path planning, obstacles) Space (near, behind, path planning, obstacles) Time (before, after, concurrent, fast) Time (before, after, concurrent, fast)

35Intelligent Tools for Media Along Example

36Intelligent Tools for Media Animation Generation Define internal representation Read representation from file Generate VRML Challenges Re-use VRML code Re-use VRML code Performance and optimization Performance and optimization scalability scalability

37Intelligent Tools for Media Automatic Editing Formalize cinematic principles Analyze screenplay Hints (Annotations) in screenplay Editing Separate into shots Separate into shots Decide shot type Decide shot type Camera motion Camera motion Light Light Sound Sound Dissolves Dissolves Effects Effects

38Intelligent Tools for Media Example - shots Camera placement The Line External Internal Apex

39Intelligent Tools for Media Examples anim_files/frame.htm

40Intelligent Tools for Media Template-based Video Editing

41Intelligent Tools for Media Automated template-based generation Wizard approach: user fills in the slots Performance: static elements of presentation are pre-cached (unless very large) Allows for dynamic, on-the-fly generation of presentations DOM MPEG-4 Abstract Template XSL material 1…. XML XSL SMIL-1 caching material n DB

42Intelligent Tools for Media Rich Media Transcoding