1 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 University of East Anglia, UK Dicta-Sign Partner 3 Prof John Glauert Virtual Humans Group School of Computing Sciences

2 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Major Roles in Dicta-Sign WP3Synthesis and Animation Lead WP6Sign Language Corpora BSL Lexicon and Corpus WP7 Bidirectional Integration Lead

3 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Partnerships

4 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Virtual Signing at UEA Dicta- Sign John Glauert Richard Kennaway, Post Doc Vince Jennings, Engineer Face Tracking, Audio-Visual Speech Barry Theobald Crowd Modelling, Urban Modelling Prof Andy Day

5 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Independent Television Commission Televirtual University of East Anglia The Post Office Royal Institute for Deaf People Instituut voor Doven Hamburg University Institut für Rundfunktechnik Institut National des Télécommunications The ViSiCAST Consortium

6 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 The eSIGN Consortium Televirtual University of East Anglia Royal National Institute for Deaf People Viataal Hamburg University Norfolk County Council

7 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Sign Language Animation from Sign Gesture Notation Prof John Glauert University of East Anglia with: Norfolk County Council, RNID, Televirtual Ltd.

8 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Some Signing Avatars

9 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Virtual Characters

10 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Virtual Characters that can Sign

11 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Virtual Human Signing Approaches

12 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Motion Capture Very lifelike animation Time-consuming to set up Blending of signs Combining signs from different signers

13 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Hand-Crafted Animation Define Key Frames Interpolate between Key Frames Can give very lifelike animation Time consuming (maybe hours per sign) Blending of signs still required

14 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Synthetic Signing Synthesis from Abstract Representation Quick to create lexicon a few minutes to transcribe a sign Instantly retargettable to any avatar with humanoid topology Automatic Blending Low Bandwidth

15 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Uses of Signing Avatars: Supporting Communication

16 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Simon the Signer, Simon-the-Signer[Broadcast TV] Generate signed accompaniment to broadcast Teletext stream as source SSE – Sign Supported English Funded by the ITC

17 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Simon the Signer, Developing transmission technology virtual signer in set-top boxes transmission of signing from text subtitles TV Capture Card Teletext Stream Audio/Video Stream AvatarSoftware Mixer Computer

18 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 SiSi: Say it Sign it, 2007 IBM with UEA and RNID Uses Speech recogniser Might help some meetings Not pure BSL

19 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 SiSi: Say it Sign it, 2007 Ive seen an exciting movie

20 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 TESSA, TESSA[Retail, PO] Using speech recognizer Convert counter-clerks voice input to text Generate sign stream from text BSL – limited repertoire Funded by the Post Office

21 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Post Office Counter Services Post Offices transact business with almost all Deaf people Counter Clerk asks questions using speech - No back channel yet Customer Listensor Readsor Watches Virtual Human Signer Real trials at 5 sites

22 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 VANESSA, 2002 – 2005 Council Information Centres (CICs) Signing avatar conveys signed information Text version also displayed Signs created from special notation Developed and enhanced version of TESSA (Post Office)

23 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 VANESSA – Customers View

24 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 VANESSA – Clerks View

25 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Uses of Signing Avatars Information Services?

26 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Web Applications Weather Forecast: NGT,BSL,DGS

27 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Weather Forecasts Today Tonight Sunday Mostly cloudy and dry in the morning. Rain breaking out during the afternoon. Maximum temperature 13 °C. Met Office for London and South East England Rain becoming heavy and persistent in the evening and continuing through the night.

28 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Web Applications Weather Forecasts Grammar for normal Weather Phrases Sign Language version for each Phrase Forecast is sequence of Phrases Generate Common XML Weather Model XSLT processing for each Sign Language XSLT processing for Spoken Languages Content creation by non- signers

29 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Uses of Signing Avatars Education?

30 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Signing Avatars in Education PC Passport SQA computer skills qualification Need to be fair to all deaf candidates Not all interpreters have IT knowledge Different signs used by different schools Students will have a dictionary of IT signs

31 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 PC Passport Assessment

32 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 PC Passport Dictionary

33 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Performing Hands,

34 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Performing Hands, GameLab London, RNID, UEA Teach literacy for Deaf Children BSL and English literacy Mixes Video, 2D Animation, Avatars Games and Stories Shockwave Application BAFTA Nominated

35 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 LinguaSign

36 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 LinguaSign Primary Modern Foreign Language Learning EU Socrates Lingua 2 Vocabulary supported by iconic gestures (signs) Speech in French, Dutch, English Mix of conventional and synthetic animation

37 Dicta-Sign Kick Off Meeting, Athens, 06 February 2009 Thanks for your attention