1. What your avatar can reveal about your handwriting
Avatar gesture from pen gestures
Really the topic is controlling avatar gesture
Avatars are a graphical representation of a person,
function as a person’s body
Desktop virtual reality
Francesca A. Barrientos
Computer Science Division
23 May 2002  Ph.D. Dissertation Seminar

2. Interacting in avatar worlds
170,000 people in AlphaWorld
12 worlds reviewd in book by Bruce Damer
In Virtual places – from 6,000 to 18,000 people logged in at a time (Webb 2001)
A wedding May 8, 1996 (Damer ‘98)
Laurel’s herb farm in AlphaWorld (Damer ‘98)
23 May 2002
Ph.D. Dissertation seminar

3. Avatar nonverbal communication
In physical world, language embedded in matrix of sounds and visuals
Avatar, as a virtual body, can send nonverbal communication
Nonverbal is anything with no words/language
Language is learned with many channels received simultaneously
Invention of writing and reading makes verbal seem more important
23 May 2002
Ph.D. Dissertation seminar

4. Ph.D. Dissertation seminar
Problem statement
We want an interaction technique for controlling avatar gesture
Controls expressive movement
Seamless with verbal communication
23 May 2002
Ph.D. Dissertation seminar

5. Avatar gesture from pen gesture
Body gesture has symbolic and qualitative aspects
Pen gesture carries symbolic and continuous data
Pen gesture simultaneously selects avatar gesture and modulates multiple expressive qualities
Thesis statement
23 May 2002
Ph.D. Dissertation seminar

6. Control using handwritten letter
Writing pen gesture triggers animation
Body gesture is “sweep” to side
Symbol is letter ‘s’ l
Quality being varied is size
Closer look
23 May 2002
Ph.D. Dissertation seminar

7. Ph.D. Dissertation seminar
Overview
Background on nonverbal communication
Why it’s important
Why previous control techniques are inadequate
Description of interaction technique
Description of implementation
Conclusions
23 May 2002
Ph.D. Dissertation seminar

8. Kinds of nonverbal cues
Vocalics
Appearance
Proxemics & Haptics
Affective display
Kinesics
Facial expression
Gaze
Posture
Gesture
Thes
23 May 2002
Ph.D. Dissertation seminar

9. Ph.D. Dissertation seminar
Gesture types
Deictic
Iconic
Metaphoric
Beat
Adaptors
Regulators
23 May 2002
Ph.D. Dissertation seminar

10. Ph.D. Dissertation seminar
Importance of gesture
Encodes ideas shared with speech
Clarifies meaning when speech is ambiguous
Useful cue when outside noise interferes with speech
Aids utterance generation
Smooths over social intercourse
Communicates mood/emotion
23 May 2002
Ph.D. Dissertation seminar

11. Interaction with speech
Coverbal gesture shares meaning units with spoken language
Language-like gestures fill in for a word or phrase in a sentence
Emblems have standard forms and well understood within a group
The parents were all right but the kids were [gesture]
23 May 2002
Ph.D. Dissertation seminar

12. Observed virtual nonverbal displays
Presence
Appearance
Proxemics
Conversation group formation
Personal space
Avatars are treated like people – processed in same way
WorldsAway from Fujitsu
Avabar by Zuidema (Damer)
23 May 2002
Ph.D. Dissertation seminar

13. Ph.D. Dissertation seminar
Designed behaviors
Static expressions
Facial
Posture
Gesture
Animated motions
Entertaining dances
Custom animations
WorldsAway from Fujitsu
23 May 2002
Ph.D. Dissertation seminar

14. Ph.D. Dissertation seminar
Related work
Commercial worlds
Blend of text virtual communities and computer graphics
Worlds Chat – first 3D world in 1995
Multi-user virtual environment research
Vlnet (Guye-Vuillème et al ’98)
ComicChat (Kurlander+Skelly+Salesin ‘96)
Autonomous avatars (Vilhjálmsson+Cassell’98, Cassell et al ‘94)
Acting in virtual reality (Slater et al ‘00)
Synthetic characters
Improv (Perlin+Goldberg ’96)
Alive (Blumberg+Galyean ’95, Maes et al ‘97)
Jack (Badler ’97)
Computer mediated conversation visualization
Collaboration-at-a-glance (Donath ’95)
Chat Circles (Viegas+Donath ’99)
23 May 2002
Ph.D. Dissertation seminar

15. Vlnet gesture/mimics panel
Limitations of each method
> Want best of both worlds
(Guye-Vuillème et al)
Button for each expression or gesture
Modulate speed with slider
23 May 2002
Ph.D. Dissertation seminar

16. Ph.D. Dissertation seminar
ComicChat
Text inference
Emoticons
Sentence Structure
Keywords
Emotion wheel
(Kurlander et al)
23 May 2002
Ph.D. Dissertation seminar

17. BodyChat avatar agents
Automatic conversation regulation behaviors
Salutations
Envelope feedback
Facial expression
User specifies high-level intentions
Avatar expressions driven by chat text
Avatar software manages gestures and gaze behavior
(Vilhjálmsson+ Cassell)
23 May 2002
Ph.D. Dissertation seminar

18. Limitations of current techniques
Mainly speech independent nonverbal displays
Emotional facial display and posture
Emblematic gesture
Not general
Cumbersome interface
Graphical interface requires hunt and select
Cannot scale
Lack control over multiple expressive features of movement
23 May 2002
Ph.D. Dissertation seminar

19. Ph.D. Dissertation seminar
Why pen gestures
Natural
People doodle while talking and listening
Expressive
Reflects emotional state
Very personal
Can be intentionally manipulated
Dual nature
Analog and digital
Symbolic and qualitative
Discrete and continuous
Information and emotion
23 May 2002
Ph.D. Dissertation seminar

20. Continuous interaction
Computers fragment our thinking by substituting discrete events for continuous actions.
-Malcolm McCullough
(Abstracting Craft, p. 53)
[Gestures] can...enhance the experience of agency through kinesthetic involvement and the feeling of directness.
-Brenda Laurel
(Computers as Theatre, p. 158)
23 May 2002
Ph.D. Dissertation seminar

21. Interaction technique
Library Generated offline
Input User writes letter in GUI
Generation Selection and synthesis
Animation Gesture performed w- s- r-
See
library
GUI
Gesture generator
Avatar animator
23 May 2002
Ph.D. Dissertation seminar

22. Interaction schematic
23 May 2002
Ph.D. Dissertation seminar

23. Ph.D. Dissertation seminar
Design issues
Pen gesture input set
Using letters of the alphabet
Mapping from pen gesture features to avatar movement parameters
Handwriting features to extract
Gesture movement parameterization
Avatar gesture animation synthesis method
23 May 2002
Ph.D. Dissertation seminar

24. Ph.D. Dissertation seminar
Handwriting features
Desired properties
Controllable
Computable
Most important feature types (according to handwriting analysis)
Size
Speed
Pressure
23 May 2002
Ph.D. Dissertation seminar

25. Motion parameterization
Formal systems
Shape-Effort (Labanotation)
Formal sign language systems
Physical
Size
Speed
Sustain
Emotional
Emphatic
Listless
Tentative
23 May 2002
Ph.D. Dissertation seminar

26. Ph.D. Dissertation seminar
Physical mapping
Handwriting feature
Movement parameter
Size
Speed
Duration
However, combinations of physical features are often more psychologically meaningful than the physical measures themselves. For instance, emphasis is pressure and size.
23 May 2002
Ph.D. Dissertation seminar

27. Ph.D. Dissertation seminar
Gesture synthesis
Input: gesture and movement parameter values
Multi-linear interpolation from set of sample gestures
Each avatar gesture comprises a set of sample motions
Each motion sample has different expression
Samples are annotated with its movement parameter values
23 May 2002
Ph.D. Dissertation seminar

28. Modulation through multilinear interpolation
Joint trajectory I
Rotation angles over time
Gesture type β
Semantic category
Set of 2n prototypes - G
n style parameters
Prototypes represent extremal trajectories
Gesture instance Iβ(u,v )
Vector of joint trajectories
Multilinear interpolation on type produces instance
Speed u
Size v
Iβ(0,1)
Iβ(0,0)
Iβ(1,1)
Iβ(1,0)
Iβ(l,k)
Iβ(0,k)
Iβ(1,k) k l
Actually this is simplified
23 May 2002
Ph.D. Dissertation seminar

29. Ph.D. Dissertation seminar
Interpolating speed
Interpolation along curves of different durations
Time dilation step
Determine duration for interpolated curve
Choose sample rate on interpolated curve
Compress slower curve - sample at proportionately slower rate
Sample faster trajectory at proportionately faster rate
Angle θ
Briefly describe implementation t
23 May 2002
Ph.D. Dissertation seminar

30. Framework for applying pen gesture to avatar gestures
Subproblems
Expression map design
Gesture synthesis technique
Can explore other mappings and synthesis techniques
Labanotation parameterization of movement
Emotion parameterization
Customize toward particular domain
23 May 2002
Ph.D. Dissertation seminar

31. Ph.D. Dissertation seminar
Implementation
Cursive
Application for interactively controlling VRML avatars over the internet
Use to test animations
Architecture permits independence from specific avatar world software
Viewer can see animations without installing new software
Facilitate testing of shared object behaviors in virtual worlds
23 May 2002
Ph.D. Dissertation seminar

32. Avatar gesture samples
Built motion capture system
Magnetic position/orientation sensors
(Flock of Birds from Ascension Corp.)
Recorded gestures
Vary size and speed
Can create very personalized gesture
23 May 2002
Ph.D. Dissertation seminar

33. Ph.D. Dissertation seminar
Cursive screenshot
Cursive window
VRML browser
VRML avatar
Next slide is frameworks
23 May 2002
Ph.D. Dissertation seminar

34. Ph.D. Dissertation seminar
Architecture
23 May 2002
Ph.D. Dissertation seminar

35. Ph.D. Dissertation seminar
Communication
Cursive communicates with any copies of user’s avatar
viewer host
driver host
Driver logs into Vworld server
Other viewers receive notification
Other viewers request and download avatar copy
Avatar opens socket connection to Cursive
Cursive sends gesture commands via socket
Web server
request avatar
Download avatar
open socket
sends commands
Summarize
login
notification
VWorld server
23 May 2002
Ph.D. Dissertation seminar

36. Ph.D. Dissertation seminar
Evaluation
Very simple to control one parameter at a time
More complicated to control size and speed simultaneously
Effective usage requires practice
Require further investigation into mapping handwriting features to movement parameters
Viable technique for controlling avatar gesture
23 May 2002
Ph.D. Dissertation seminar

37. Ph.D. Dissertation seminar
Expert use
…the art of finding and executing an effective gesture is learned through the more indirect means of observation, experimentation, performance, and evaluation, and it is a skill that continues to grow over time.
- Brenda Laurel
(Computers as Theatre, p. 155)
23 May 2002
Ph.D. Dissertation seminar

38. Ph.D. Dissertation seminar
Summary
Want to control richly expressive, spontaneous gesture in avatar worlds
Solution is an interaction technique employing pen gesture input
Cursive: an implementation of this interaction technique
23 May 2002
Ph.D. Dissertation seminar

39. Ph.D. Dissertation seminar
Contributions
Novel interaction technique that augments the potential repertoire of avatar nonverbal communication
Possible to control spontaneous gesture
Control over expressive characteristics of gestural movement
Framework for applying pen gesture to avatar gesture
Mapping handwriting features to movement parameters
Synthesis of expressive gesture
The algorithms and methods used to implement the technique
Cursive, a working application that applies the interaction technique
23 May 2002
Ph.D. Dissertation seminar

40. Ph.D. Dissertation seminar
Future work
Explore other mappings
emotional expression in handwriting and gesture
Shape-Effort parameterization of movement
Explore other gesture synthesis methods
Reduce cost of obtaining gesture motion samples
Develop a framework for determining avatar gesture vocabulary
How many gestures
What types of gestures
23 May 2002
Ph.D. Dissertation seminar

41. Ph.D. Dissertation seminar
Conclusion
Transmitting bodily nonverbal communication through the internet is an exciting idea.
Think of the computer as a medium for personal expression
Continuous/rich interaction
Playful behavior
Sense of engagement
Handwriting has a place in affective computing
23 May 2002
Ph.D. Dissertation seminar

42. Ph.D. Dissertation seminar
Acknowledgements
A few of the people who helped with my research and with this talk.
My dissertation committee: Prof. John Canny, Prof. James Landay, Prof. John McWhorter.
My gesture model: Erin Dare.
My dissertation writing group: Blanca Gordo, Jeffrey Ow, Dr. Ellen Sacco-Fernandez, Lynne Horiuchi.
Other talk critics: Miriam Walker, Scott Klemmer, Dan Glaser, Jeremy Risner, James Lin, Jason Hong, Dr. Eric Paulos.
23 May 2002
Ph.D. Dissertation seminar

43. Ph.D. Dissertation seminar
Partial bibliography
Damer, B., Avatars! Exploring and Building Virtual Worlds on the Internet. Berkeley, CA: Peachpit Press, 1998.
Guye-Vuillème, A., T. Capin, I. Pandzic, N. Magnenat-Thalmann, and D. Thalmann, "Non-Verbal Communication Interface for Collaborative Virtual Environments," in Proc. CVE 98, June 1998, Manchester, 1998.
Vilhjálmsson, H. H. and J. Cassell, "BodyChat: Autonomous Communicative Behaviors in Avatars," in Proceedings of the Second International Conference on Autonomous Agents, May 9-13, 1998, Minneapolis: ACM, 1998, pp
Cassell, J., C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Becket, B. Douville, S. Prevost, and M. Stone, "Animated Conversation: Rule-Based Generation of Facial Expression, Gesture and Spoken Intonation for Multiple Conversational Agents," in Proceedings of SIGGRAPH '94., 1994.
Slater, M., J. Howell, A. Steed, D. P. Pertaub, M. Gaurau, and S. Springel, "Acting in Virtual Reality," in Collaborative Virtual Environments 2000: ACM, 2000, pp
Perlin, K. and A. Goldberg, "Improv: A System for Scripting Interactive Actors in Virtual Worlds," in Proc. Siggraph 96, H. Rushmeier, Ed., New York: ACM Press, 1996, pp
Blumberg, B. M. and T. A. Galyean, "Multi-level direction of autonomous creatures for real-time virtual environments," presented at SIGGRAPH, Los Angeles, CA, 1995.
Maes, P., T. Darrell, B. Blumberg, and A. Pentland, "The ALIVE system: wireless, full-body interaction with autonomous agents," Multimedia Systems, 5, no. 2, 1997, pp
Badler, N., "Virtual Humans for Animation, Ergonomics, and Simulation," in IEEE Workshop on Non-Rigid and Articulated Motion, June 1997, Puerto Rico, 1997.
Donath, J. S., "The illustrated conversation," Multimedia Tools and Applications, 1, no. 1, 1995, pp
Viegas, F. B. and J. S. Donath, "Chat Circles," in CHI 99: ACM, 1999, pp
Laurel, B., Computers as Theatre. Reading, Massachusetts: Addison-Wesley, 1993.
McCullough, M., Abstracting craft : the practiced digital hand. Cambridge, Massachusetts: The MIT Press, 1996.
23 May 2002
Ph.D. Dissertation seminar

44. Network communication
Driver host
Viewer host
World
browser
Animation commands sent via socket connection
23 May 2002
Ph.D. Dissertation seminar

45. Communicative power of gesture
Rich, continuous variation in time and space
Same gesture can have different expression
Can be combined with speech
Can be generated and varied spontaneously
Bodily outlet for personal expression
Relative to language
23 May 2002
Ph.D. Dissertation seminar

46. Ph.D. Dissertation seminar
Advantages
Single action selects and modulates
Transparent (no hunting)
Expressive modulation of multiple parameters
Compact
Natural (writing is previously learned skill)
23 May 2002
Ph.D. Dissertation seminar

47. Inputs modulate gesture
Extraction
Type library w- s- r- w-
Gesture Type s-
Character
recognizer r-
Inputs modulate gesture
Feature extractor
Speed .5 Size .3
Explain modulate
GUI
Style
parameters
23 May 2002
Ph.D. Dissertation seminar