Electronic Visualization Laboratory University of Illinois at Chicago Interaction between Real and Virtual Humans: Playing Checkers R. Torre, S. Balcisoy P. Fua, M. Ponder, and D. Thalmann Computer Graphics Lab (LIG) Swiss Federal Institute of Technology Atul Nayak

Electronic Visualization Laboratory University of Illinois at Chicago Virtual Humans Guided –Represent the user Autonomous –Act on their own

Electronic Visualization Laboratory University of Illinois at Chicago Virtual Worlds Virtual humans live in synthetic worlds

Electronic Visualization Laboratory University of Illinois at Chicago So what’s new? Make the virtual human interact with the user 2-way communication Real-time Non-invasive Vision-based recognition system

Electronic Visualization Laboratory University of Illinois at Chicago Playing checkers….

Electronic Visualization Laboratory University of Illinois at Chicago Real time tracking of 3D objects To track a 3D convex object (a cube, a checkers board) we need –A camera –A target –The wire frame 3D model of the target –A powerful computer

Electronic Visualization Laboratory University of Illinois at Chicago Model Based Tracking Create a mathematical model of the target World coordinates are a function of the state variables that control the object’s shape An objective function is minimized to find the best fit with image data

Electronic Visualization Laboratory University of Illinois at Chicago 3-D tracking algorithm Get a video image Process the image X(t+1) = X(t) + V(t).dT –X = position ; V = speed ; dT = time between t and t+1 Minimization of the objective function

Electronic Visualization Laboratory University of Illinois at Chicago Detecting the checkers If the sum of the intensities of each pixel in the real image of a white square is less than a certain threshold value, it means that a piece is present on that square.

Electronic Visualization Laboratory University of Illinois at Chicago Virtual Human Director (VHD) A system used to produce real-time multimedia products and TV shows Focuses on 2 issues –Fully integrated virtual humans with facial and body animation, and speech –A straightforward user interface for designers and directors

Electronic Visualization Laboratory University of Illinois at Chicago The VHD interface Virtual human animation and interaction capabilities integrated into one environment Control multiple actors Provide tools to create any scenery involving realistic virtual humans

Electronic Visualization Laboratory University of Illinois at Chicago The Augmented Reality System Synchronize the rendering of synthetic scenes with the real camera output VHD animates the virtual player and updates the scene – input from tracker and checker game controller

Electronic Visualization Laboratory University of Illinois at Chicago Image composition Chroma-keying : black is the key color All real objects represented in black Table’s virtual representation masks the virtual human Blend the real filmed scene and the synthetic 3D scene to get the mixed image Real player sees the mixed image on a large monitor

Electronic Visualization Laboratory University of Illinois at Chicago Comments Very realistic virtual humans will be widely used in film, games and interactive television Integrate VR with the Web – create 3D scenes with multimedia objects Replace checkers board with any other calibration object No extra devices – you can see yourself in the virtual world

Electronic Visualization Laboratory University of Illinois at Chicago More cool demos….

Electronic Visualization Laboratory University of Illinois at Chicago More cool demos….

Electronic Visualization Laboratory University of Illinois at Chicago References VHD: A System for Directing Real-Time Virtual Actors, G. Sannier, S. Balcisoy, N. Magnenat-Thalmann, D. Thalmann, The Visual Computer, Springer, Vol.15, No 7/8, 1999, pp All images and movies in this presentation have been taken from the LIG’s website URL :

Electronic Visualization Laboratory University of Illinois at Chicago Comments?