1. Building an English to American Sign Language Translation System Matt Huenerfauth Presentation for CSE-391, March, 2004 Computer and Information Science University of Pennsylvania Advisors: Mitch Marcus & Martha Palmer

2. Outline Motivations Applications Why is English-to-ASL translation difficult? How can we approach the problem using some non-traditional technologies?

3. Motivations

4. Only half of Deaf high school graduates can read English at a fourth-grade level. –But most are fluent in American Sign Language. Are the languages that different? Yes, very different. –Hearing children’s English experience mostly oral/aural. –English is difficult to learn if you can only see/write it.

5. Motivations How different are English and ASL? Word order, grammar rules, but more fundamentally… English Add endings to a word. Add words to a sentence. ASL Do more at once, make little changes, use space. –Hands, face expression, eye gaze, head tilt, shoulder tilt, body posture… –Make little changes to hand shapes or to movements… –Imagine objects you’re talking about floating in space around you – point to them, show them moving around.

6. Motivations Many Deaf Accessibility Aids: –Expect Deaf person is fluent in English. –ASL is first language, English is second.. Human ASL Interpreters: –Can be too expensive/obtrusive. –Unexpected/spontaneous situations? Audiology Online

7. Input / Output What’s our input? English Text. What’s our output? Less obvious… Imagine a 3D virtual reality human being… Now imagine they can perform sign language… But this character needs a set of instructions telling it how to move! Our job: English  These Instructions. VCom3d

8. Input / Output What’s our input? English Text. What’s our output? Less obvious… Imagine a 3D virtual reality human being… Now imagine they can perform sign language… But this character needs a set of instructions telling it how to move! Our job: English  These Instructions. VCom3d

9. TV Closed Captioning - Before Audiology Online

10. TV Closed Captioning - After Audiology OnlineVCom3d

11. Handheld Translator Device This technology could be used to help build a portable English to Sign Language translation device. VCom3d

12. What else is this good for? Translate English to Sign Language in chat programs, over e-mail, or over TTYs. Educational programs for –Deaf children learning English. –Deaf children learning other subjects. –Hearing people learning sign language.

13. Virtual Signing Humans Photos: Seamless Solutions, Inc. Simon the Signer (Bangham et al. 2000.) Vcom3D Corporation

14. Wait a minute! Why do we need all these cartoon characters? Wouldn’t it just be better to videotape people performing all the signs in a sign language dictionary? When you want to translate something in English, you just paste the video clips together in English word order.

15. Seamless Solution, Inc.

16. What’s Wrong with This? This isn’t producing real sign language! –Breaks all the grammar rules. –Looks pasted together, choppy, irritating. –Not understandable/helpful to Deaf people. Imagine translating English into French this way… You would just get nonsense.

17. Cartoons are Better We can take the grammar and rules of real sign language into account, and build an animation piece by piece. Since we want to blend signs together and independently control the eyes, head, hands, and body, we don’t just use video clips: That’s why we use a cartoon character!

18. How to Translate Write a computer program that: Analyzes the English sentence that you type in. Re-arranges the elements of the sentence to match the grammar rules of sign language. Creates a script for the computer cartoon character to follow. Displays a cartoon character that performs the sign language according to the script you wrote.

19. Piece of cake? Maybe not. We’ve seen a lot of NLP approaches and tools for written languages, but they don’t work well for ASL –There’s no way to write down sign language. Unless you videotape it, it’s lost forever. Can’t collect corpus and use Wordfreak. –The interesting ways ASL uses the space around the signer also makes the job really difficult.

20. It’s all about space… The NLP approaches we’ve seen don’t explain how to handle the complex ways ASL uses space. Classifier predicates: drawing with your hands. –Very common, very communicatively useful. –Push boundaries of language: non-discrete information, spatial analogy, scene visualization, iconic/mime.

21. “Classifier Predicate” The car drove down the bumpy road past a cat. CAT ClassPred-V-{location of cat} CAR ClassPred-3-{drive on bumpy road} Where’s the cat, the road, and the car? How close? Where does the path start/stop? How show path is bumpy, winding, or hilly? The lexicons and grammar rules we’ve seen don’t help much…

22. We’re at Penn. And there’s a lot of interesting research… Some it involves virtual reality… Some of it has even been related to computational linguistics … Research in building a VR system that can accept English commands to make the objects in the scene move how you want.

23. Control a VR with English? Have a virtual reality model of characters and objects in a three-dimensional scene. Accepts English text input (directions for the characters or objects to follow). Produces an animation in which the characters obey the English commands. Updates the 3D scene to show changes. Badler, Bindiganavale, Allbeck, Schuler, Zhao, Lee, Shin, and Palmer. 2000.

24. An NL-Controlled 3D Scene http://hms.upenn.edu/software/PAR/images.html

25. Using this technology… An NL-Controlled 3D Scene http://hms.upenn.edu/software/PAR/images.html

26. Using this technology… An NL-Controlled 3D Scene

27. Using this technology… An NL-Controlled 3D Scene Original image from: Simon the Signer (Bangham et al. 2000.) Signing Character

28. Using this technology… An NL-Controlled 3D Scene Original image from: Simon the Signer (Bangham et al. 2000.) Signing Character

29. “Invisible World” Approach Mini VR scene in front of the signer containing entities from English text. (They’re invisible.) Interpret the English sentences as VR commands. The English-enabled VR software positions, moves, and orients the objects in this world. Our virtual signer can just put his hand on top of the moving invisible object in the scene! We just built a CLASSIFIER PREDICATE.

30. To Build a Whole System This works great for classifier predicates, but other ASL signs are less complicated. –Traditional NLP approaches work for them. –System needs multiple processing pathways that an input sentence can follow. –Sentences with a lot of spatial/movement vocabulary would get sent to the VR software.

31. Wrap Up A lot of good reasons to build an English to ASL translation system. But ASL is different than most other languages. This makes processing it using traditional computational linguistic software very difficult. Need a new approach for spatially complex ASL classifier predicates: invisible worlds! Make this one path inside a multi-path translation architecture to handle both the complicated sentences and the more traditional ones.

32. Photo Credits Some images taken from: “Audiology Online” article 5/13/2002 Seamless Solutions, Inc. Website Vcom3d Company Website J.A. Bangham, S J Cox, M Lincoln, I Marshall. 2000. Signing for the deaf using virtual humans. IEE2000

33. Extra Slides

34. Traditional vs. Complex Traditional Sentences: (No classifier predicates.) Where does Billy attend college? wh #BILLY IX x GO-TO UNIVERSITY WHERE Spatially Complex: (Uses classifier predicates.) I parked my car next to his cat. POSS x CAT ClassPred-bent-V-{locate cat in space} POSS 1s CAR ClassPred-3-{park next to cat} The truck drove down the windy road. IX x TRUCK ClassPred-3-{drive on windy road}

35. MT Pyramid of Designs MT Pyramid Dorr 1998.

36. Direct Pathway Just like the iCommunicator System. Word to sign look-up.

37. Transfer Pathway Handle Traditional Sentences Syntactically analyze English text before crossing over to ASL. –Capture more divergences and handle more complex phenomena. –Can successfully translate many English sentences into ASL.

38. Multi-Path Architecture Whenever possible, Use simpler easier-to-build MT approach. Only when needed, Use more sophisticated resource-intensive. We take advantage of the ‘breadth’ of one and the ‘depth’ of the other. If we add direct translation (to English-like signing) to the picture, we actually have three pathways.