1. Accelerometer-based User Interfaces for the Control of a Physically Simulated Character Takaaki Shiratori Jessica K. Hodgins Carnegie Mellon University

2. Physical Simulation in Games Controllable interface for physically simulated character Jurassic Park: Trespasser, 1998 2 Simulation for everything. - Very limited. Little Big Planet, 2008 1 Simulation ONLY for passive objects. - Effective! 1 http://www.littlebigplanet.com/ 2 http://www.mobygames.com/game/trespasser-jurassic-park

3. Half speed Physically Simulated Character Natural-looking motion. Natural responses to environment/disturbance.

4. Natural-looking motion. Natural responses to environment/disturbance. Hard to control due to delayed response (anticipation). Physically Simulated Character Walk  Jump – Our Hypothesis – Performing similar actions might make the delay seem intuitive. 0 1.0 2.0 3.0 0 1 2 3 4 5 Speed [m/s] Time [sec] Walk Jump Anticipation

5. Our Approach Performance interface: User Imitates character’s motion with Wiimotes. – Wrist interface – Arm interface – Leg interface Test our hypothesis about delay with user study. Leg interface

6. Accelerometer IR sensor $40 / unit 30 million copies Wiimote TM 1 http://www.gizmag.com/go/6773/ 2 http://www.cs.cmu.edu/~johnny/projects/wii/ 12

7. Related Work Interface for controlling a character – Physically simulated character in 2D. Virtual navigation – Control “view point”. Wii games – Either dynamic or static measurement. [Johnson et al. 1999] [van de Panne and Lee 2003][Slyper and Hodgins 2008] [Chai and Hodgins 2005] [Templeman et al. 1999] [Razzaque et al. 2002] [Nintendo 2006][Nintendo 2007][Ubisoft 2007]

8. Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

9. Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

10. Focus on periodicity of character’s motion. User Input Walking RunningJumping Basic command: swing Wiimotes In-phase Out-of-phase

11. Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving? Variance

12. Frequency Auto-correlation function a : acceleration data T : current time t p : window size

13. Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving?

14. Phase Difference Cross-correlation function a : acceleration data a : mean acceleration T : current time t p : window size

15. Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving?

16. Inclination Estimation If Wiimote is not moving, If Wiimote is moving,  Wiimote’s local coordinate. x y z g t acc. x t mean

17. Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

18. Hopper Model Hip: Ball joint (3 DoFs) Knee: Slider joint (1 DoF) Simulation Rendering

19. Basic Motions Stepping in place (Stopping) RunningWalkingJumping

20. Consists of 3 contact states with Proportional- Derivative (PD) controller. Support foot passes under hip. Rear leg leaves ground. Swing leg contacts ground. FallRise Double Support Walking Controller [Raibert and Hodgins, 1991] Stepping-in-place controller: target velocity = 0

21. Consists of 4 contact states with PD controller. Running Controller [Raibert and Hodgins, 1991] Jumping controller: both legs in phase.

22. Based on the robustness of motion. Gait Transition Jumping Stepping in place WalkingRunning WalkingRunning Slow Fast

23. Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

24. Mapping Wiimotes to Physical Simulation Wiimotes not moving. Wiimotes out of phase. Wiimotes in phase. Height: Wiimote amplitude Jumping Stepping in place WalkingRunning WalkingRunning Slow Fast Wii frequency

25. Wrist Interface Imitate character’s leg motions with user’s wrists.

26. Wrist Interface Imitate character’s leg motions with user’s wrists. Walking (Slow swing) Running (Fast swing) JumpingTurning

27. Arm Interface Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body)

28. Arm Interface Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body) WalkingRunningJumpingTurning

29. Leg Interface Imitate character’s leg motion with user’s legs.

30. Leg Interface Imitate character’s leg motion with user’s legs. Walking (Slow step) Running (Fast step) JumpingTurning

31. Typical usage. Joystick Interface RightLeft Forward Run Walk Turn in place Step in place LocomotionJumping

32. User Study 15 subjects. Tasks – Straight track completion. – Test track completion. Questionnaire – Fun, ease of use, stress, familiarity, immersion, how much they liked it? – Free-form questions. Courses:

33. Task 1.Motion transition at line. 2.Keep straight walking/running until line. Straight Track Completion Average failure count Straight walking: All of our interfaces < Joystick

34. Failure for Straight Walking Approximate motions RightLeft Forward Run Walk Turn in place Step in place Precise manipulation Wrist ArmLeg Joystick Straight walk

35. Test Track Completion Simulation failureJump failureCurve failure + Time to completion Time to failure

36. Result of Test Track Completion Average countTime [sec] Curve failure: Leg interface < joystick Our interfaces are easier to control than joystick.

37. Result of Questionnaire “Fun”“Ease of use”“Stress” “Familiarity”“Immersion” “Like” Score

38. Free-form questions: Most subjects did not complain about the delay. A few subjects complained about the delay of all interfaces (including joystick). Questionnaire Rating score: “Immersion”: Wrist, Leg > Joystick “Like”: Wrist, Leg > Joystick

39. Insights from User Study Delay factors: – Acceleration analysis: 100 – 500 ms (Not included in joystick interface) – Physical simulation (anticipation): 200 – 500 ms Task completion: Easy to control: our interfaces > joystick. Questionnaire: “Immersive” and “Like”: our interfaces > joystick. Immersive performance interface might help with the delay issue.

40. Competitive Game

41. Conclusion Summary Performance interfaces for controlling a physically simulated character. User study mainly focusing on delay issues. – Performance interface might be able to help with delay issue. Future Work Reduce delay. – Improve acceleration analysis. Further user study. – Delay issue: Compare with data-driven control. – Other scenarios (e.g. fighting game, FPS).