SceneMaker: Multimodal Visualisation of Natural Language Film Scripts Dr. Minhua Eunice Ma School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee, Northern Ireland {p.mckevitt, tf.lunney, Eva Hanser Prof. Paul Mc Kevitt Dr. Tom Lunney Dr. Joan Condell School of Computing and Mathematics Faculty of Business, Computing and Law University of Derby, England

PRESENTATION OUTLINE Aims & Objectives Related Projects SceneMaker Design and Implementation Relation to Other Work Conclusion and Future Work

AIMS Automatically generate well-designed and affective virtual scenes from screenplays Realistic visualisation of emotional aspects Multimodal representation with 3D animation, speech, audio and cinematography Enhance believability of virtual actors and scene presentation : AIMS & OBJECTIVES Input: Screen- play SceneMaker System Output: Animation

OBJECTIVES Processing/inferencing emotions and semantic information within story context Common sense, affective and cinematic knowledge ontologies reflecting human cognitive reasoning rules Automatic genre recognition from text Design, implementation and evaluation of SceneMaker : AIMS & OBJECTIVES

Standardized format and language of screenplays Automatic annotation of formal screenplay elements (Jhala 2008) Semantic information on location, timing, props, actors, events, manners, dialogue and camera direction SEMANTIC TEXT PROCESSING : RELATED PROJECTS INT. M.I.T. HALLWAY -- NIGHT Lambeau and Tom come around a corner. His P.O.V. reveals a figure in silhouette blazing through the proof on the chalkboard. There is a mop and a bucket beside him. As Lambeau draws closer, reveal that the figure is Will, in his janitor's uniform. There is a look of intense concentration in his eyes. LAMBEAU Excuse me! WILL Oh, I'm sorry. LAMBEAU What're you doing? WILL (walking away) I'm sorry. Screenplay Extract from ‘Good Will Hunting (1997)’

Emotion recognition from text: keyword spotting, lexical affinity, statistical models, fuzzy logic rules, machine learning, commonsense knowledge, cognitive models XML-based annotations defining visual appearance of animated characters and scenes: BEAT – Behaviour Expression Animation Toolkit (Cassell et al. 2001) MSML – Movie Script Markup Language (Van Rijsselbergen et al. 2009) VISUAL AND EMOTIONAL SCRIPTING : RELATED PROJECTS

Automatic physical transformation and synchronisation of 3D models reflecting emotion Manner influences intensity, scale, force, fluency and timing of an action Multimodal annotated affective video or motion captured data (Gunes and Piccardi 2006) MODELLING AFFECTIVE BEHAVIOUR Personality & Emotion Engine (Su et al. 2007) Greta (Pelachaud 2005) : RELATED PROJECTS

WordsEye – Scene composition (Coyne and Sproat 2001) ScriptViz – Screenplay visualisation (Liu and Leung 2006) CONFUCIUS – Action, speech & scene animation (Ma 2006) CAMEO – Cinematic and genre visualisation (Shim and Kang 2008) VISUALISING 3D SCENES WordsEye CONFUCIUSScriptVizCAMEO : RELATED PROJECTS

Emotional speech synthesis (Schröder 2001) - Prosody rules Music recommendation systems - Categorisation of rhythm, chords, tempo, melody, loudness and tonality - Sad or happy music and genre membership (Cano et al. 2005) - Associations between emotions and music (Kuo et al. 2005) AUDIO GENERATION : RELATED PROJECTS

Context consideration through natural language processing, common sense knowledge and reasoning methods Extract genre and moods from screenplays Influence on all elements of visualisation Enhance naturalism and believability Text-to-animation software prototype, SceneMaker KEY OBJECTIVES : DESIGN AND IMPLEMENTATION

Animatio n Player Script Editor Screen- play Text & Language Processing Text & Language Processing Context Interpretation Context Interpretation Multimedia Generation Multimedia Generation } Genre Emotio n Action } ARCHITECTURE OF SCENEMAKER : DESIGN AND IMPLEMENTATION

SOFTWARE AND TOOLS : DESIGN AND IMPLEMENTATION LVSR (2) Lexical Visual Semantic Representation Script Format Ontology Unity (6) 3D Engine (JavaScript,XML) MSML (5) /SMIL Concept Net (3) Common Sense Knowledge Gate (1) ANNIE Onto-Gazetteer Genre Ontology RDFS/OWL Movie Ontology RDFS/OWL WordNet- Affect (4) Festival (7) Speech Synthesiser Natural Language Processing & Script Segmentation Context + Emotion Reasoning Event Synchronisation 3D Rendering + Multimedia 3D Models (3D Studio Max) Movie Script Automatic Sound & Music Selection (1) (2) Ma 2006 (3) Liu and Singh 2004 (4) Strapparava and Valitutti 2004 (5) Van Rijsselbergen et al (6) (7)

Evaluating 4 aspects of SceneMaker: EVALUATION OF SCENEMAKER AspectEvaluation Correctness of screenplay analysis & visual interpretation Hand-animating scenes Effectiveness of output scenes Existing feature film scenes Suitability for genre typeScenes of unknown scripts categorised by readers Functionality of interfaceTesting with drama students and directors : DESIGN AND IMPLEMENTATION

RELATION TO OTHER WORK

CONCLUSION AND FUTURE WORK Automatic expressive multi-media animation of screenplays Focus on: – automatic reasoning about story context and emotional interpretation – based on world knowledge and context memory – emotions influencing scene compositions and event execution – scene direction refined by genre-specifics Analysis of script format to access semantic information Automatic genre specification from script Heightened expressiveness, naturalness and artistic quality Assist directors, actors, drama students, script writers Future work: Implementation & Testing of SceneMaker

Thank you. QUESTIONS OR COMMENTS ?