1. Expressive Intelligence Studio // Center for Games and Playable Media http://eis.ucsc.edu // http://games.soe.ucsc.edu 3D User Interfaces Using the Kinect and Beyond John Murray Expressive Intelligence Studio Expressive Intelligence Studio // Center for Games and Playable Media http://eis.ucsc.edu // http://games.soe.ucsc.edu John Murray Title Here (Arial)

2. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Introduction to Interfaces User interface design (HCI) focuses on use cases and user studies Have only begun investigating 3D interfaces with the same level of rigor But have been actively researching them for the past two decades Divided into input devices, manipulation techniques, navigation techniques, and Expressive Intelligence Studio http://eis.ucsc.edu

3. 3D User Interfaces Input Devices Joysticks 3D Mice 2D Mice Buttons Brain Input Gloves

4. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Special Purpose Game Controllers Playstation Move Wiimote & Nunchuk Steel Battalion controller Wii Fit 360 Controller

5. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Playstation Move Uses optical tracking to locate sphere Color distinguishes from environment Uses 6dof (accelerometer & gyroscope) to provide rotation around point Up to 4 controllers Very fast

6. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Wiimote Began it all Uses IR LEDs in a tracker to handle exact position, but must be pointed toward “sensor” bar. Has accelerometer, but not gyroscope Extension adds additional resolution/detail to rotation

7. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Steel Battalion

8. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces New wave of input devices Touche $? Touche Leap $70 Sparkfun $400+ Sparkfun Emotiv EPOC neuroheadset $300

9. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Output Augmented reality (Foresight, Google Glass) 3D TVs and Monitors getting cheaper Tactile (Vibration, haptic feedback) Heat/cold

10. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces 3D MANIPULATION

11. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces 3D Selection & Manipulation Nonstandard manipulation – Deform – Puppet – Meshes 3D Manipulation “Spatial Rigid Object Manipulation” – Selection – Positioning: – Rotation: Changing orientation of the object

12. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Precision Grasp Small muscle groups (thumbs, fingers Thumbsticks on 360 and wiimote Better performance in 3d, faster, more accurate Reduce “clutching” Attached to hand (Kinect) Provides kinesthetic sense Uses larger muscle groups Exercise & imitates actual actions

13. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Isomorphic vs Isometric Think thumbpad vs mouse or Kinect. Measuring force vs position Relative vs absolute mapping of rotation and transform Think its more natural, but isomorphic can be limited by our own physical limitations. We do not have extendable arms, for instance.

14. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Selection Technique Indication of object – Occlusion – Object touching (list, voice set, automatic, iconic) – Pointing (2d, 3d gaze, 3d hand) – Indirect selection (1-1, pos->pos, vel->pos, pos->vel) Confirmation of selection (Event, gesture, voice command, no explicit command) Feedback (Text/Symbolic,Aural, visual, force/tacticle)

15. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Manipulation Techniques Exocentric Metaphors – World in miniature – Automatic scaling Egocentric Metaphors – Virtual Hand Metaphors “Classical” Go-Go Indirect Go-Go – Virtual Pointer Metaphors Ray Casting Aperture Flashlight Image plane

16. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces TRAVEL AND WAYFINDING

17. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Maneuvering Similar to search task: destination known. Need to have precise local changes to reach destination How far do you need to go? How many turns in path? – Turning torse when less frequent, other method when they are frequent Visibility of target from start location

18. Expressive Intelligence Studio http://eis.ucsc.edu 3D User Interfaces Travel Techniques Active vs Passive – Active is when the user moves their viewport consciously – Passive is when the user directs travel Direction/Target Selection – Gaze, pointing, orientation, Stop moving Velocity/Acceleration Selection – Constant, gesture, explicit, adaptive Input Conditions – Constant, continuous input, start and stop inputs, automatic start/stop