1. Probabilistic User interfaces Roderick Murray-Smith Department of Computing Science, University of Glasgow & Hamilton Institute, NUI Maynooth rod@dcs.gla.ac.uk http://www.dcs.gla.ac.uk/~rod http://www.dcs.gla.ac.uk/~rod With John Williamson, Parisa Eslambolchilar, Andy Crossan, Steve Strachan, Vuokko Lantz & Stephen Brewster

2. Overview 1.Dynamics & Statistics in Interaction 2.Uncertain, Dynamic Feedback mechanisms 3.Demos 1.Hex Entry: Intelligent adaptation of handling qualities during interaction 2.Pointing without a pointer: Control models in interaction design 4.Conclusions

3. Dynamics & Statistics in HCI? Why introduce dynamics – is that not harder? –We can only control what we can perceive. –Dependent on feedback, so upper limits on the speed of change of display. –Dynamics allows us to slip in intelligence which couldnt be done with a static interaction technique Why uncertain interaction? –Uncertainty in users mind about what to do next, and system uncertain about users intentions. –With mobile devices, interaction with the user is now continuous instead of discrete, and input devices are noisier.

4. Feedback & Inference A model of user interaction in a closed-loop system involving uncertainty. Feedback of the results of the inference process is provided to the users which they can then compare with their goals. Inference may often be about users beliefs, desires or intentions…

5. Display and control Display augmentation –Improve input to human to simplify control task Control augmentation –Change the effective dynamics between control input and system output The display is to provide the user with information needed to exercise control. i.e. predict consequences of control alternatives, evaluate status and plan control actions.

6. Ambiguous displays Used in psychophysics experiments (e.g Körding & Wolpert 2004) Transfer idea to user interface design. If the system is uncertain about inputs or user intentions, present data in an appropriately ambiguous fashion. Does it regularise user behaviour & improve usability appropriately? Pattern recognition and displays are interdependent and should be developed together

7. Mobile & Acceleration Sensing HP IPAQ Xsens accelerometer –3 DOF linear accelerometer –Samples up to ~100 Hz – Weight ~10.35g Potential for one handed / screen free interaction Mobile devices used in many contexts, subject to varying levels of disturbance –Ideal testbed for probabilistic interaction –Small screen –Vibrotactile/audio feedback

8. Audio displays - Granular Synthesis A structured approach to probabilistic audio Quantum theory of sound Accumulate short grains from waveforms sources Select grains according to some probability distribution

9. Sonification of Probability Distributions Straightforward sonification of probabilistic models –Associate distributions with collections of source waveforms –Continuous distributions can be sonified via sampling from a parametric synthesis algorithm Produces a smooth texture representing the changing probabilities

10. Feedback for gesture recognition Mapping from an input trajectory to an audio display via a number of gesture recognition models. Each gesture is associated with a model and the output probabilities are fed to the synthesis algorithm. Can be combined with direct sonification of gesture movements. Users can explore functionality –Feedback from the goals, depends on accuracy of gesture & estimated skill of user –User behaviour can be shaped, starting with simple, blurred gestures and progressing to sharper, more complex expression.

11. Outline of the gesture recognition and sonification system Parametric model of gestures (Dynamical motor primitives, Locally Weighted Learning) Model parameters fitted to the current observations Gesture recognition engine Audio feedback generation Audio source 1 Audio source 2 Audio source 3 … Audio source N Audio feedback generation Acceleration measurements of the phones movements Sonification of the recognition result and its confidence (Granular Synthesis) Sonification of the performed gesture (Granular Synthesis) A posteriori probabilities for different gestures

12. Feedback for gesture recognition Benefits of feedback –Feedback on how the gesture recognition engine is performing, i.e. recognition result and confidence –Gives the user insight into the pattern recognition mechanics. –Feedback on how the user is performing, i.e. sonification of the users actions Coupling of gesture recognition and feedback generation –Simple, parametric representation for the observed gesture data –Gesture model parameters can act as pattern features of the recognition engine –Recognition engine produces a posteriori probabilities for each gesture class –Parametrized feedback generation on the basis of gesture feature vectors or classification results, e.g. Granular Synthesis of audio/vibro sources

13. Haptic Targeting Spatially & Time Varying Vector Field Directional Grains using Vector Summation Highlights Areas of High Uncertainty

14. Quickening/Predictive displays Augmentation of a display with predictive information –Experience indicates that, by using a properly designed predictor instrument, a novice can in 10 minutes or less learn to operate a complex and difficult control system as well as or better than even the most highly skilled operator using standard indicators, from Kelley, C.R. Manual and Automatic Control 1968 Standard technique in manual control systems – e.g quickening of helicopter displays, Showing derivatives of current state Quicken the probabilistic audio display –Add predictions of change of probability to the display, e.g. if derivative of probability is increasing, decrease if derivative is decreasing… –Allows users to determine when they are moving towards regions of high probability; aids in targeting of modes Models such as Gaussian processes allow derivative uncertainty to be included.

15. Demo: Nonlinear dynamics & Monte-Carlo simulations

16. Feedback conclusions Provided examples of granular synthesis for sonifying probabilistic interfaces –with quickening, & Monte Carlo predictions can help improve interfaces to a continuously controlled environment which involves uncertainty –can be extended to force- and vibrotactile feedback –Helps users learn gestures for mobile devices –Allows flexibility to give feedback about different orders of derivatives, applications in rehabilitation engineering.

17. Hex Entry: Intelligent adaptation of handling qualities during interaction Flexibility brought by dynamic models allows intelligent interaction, – handling qualities of the dynamics of the interface are adapted depending on current inferred user goals. – actions require less effort, equivalent to a lower bit rate in communication terms, the more likely the systems interpretations of user intentions. Serves as example of continuous interaction system, –with gestures, augmented control and potential for audio feedback Predictions of future trajectories –Could be linked to sound as MC samples Quickening via velocity & acceleration info

18. Hex: The Aims A continuous input system – all entry is one single smooth sequence Incorporating a probabilistic model to represent uncertainty and increase performance But with a structure that can be learned – gestures must be repeatable –Support transition from novice (tightly closed control loop) to expert (open-loop, learned behaviour)

19. Example Words HelloGISTHexago ns

20. Augmented control System provides augmented, nonlinear control –Dont perform actions for user - help user reproduce ideal behaviour themselves –Adapts to context, changing properties of the control system –Minimises effect of disturbances and errors Goal is that although initial use is very dependent on feedback, user learns open-loop gesture-like behaviour.

21. Nonlinear dynamics Vector field adapts to current context –In this case Q has been chosen Handling qualities improved appropriately –Makes it easy to get to U

22. Semantic Pointing (Blanch, Guiard, Beaudouin-Lafon 2004.) Motor space and Display space have different properties Control-Display ratio adapted depending on proximity of target

23. Predictive control & Word Autocomplete Also show top k most probable paths –fitting a cubic spline through hexagon centres

24. Hex Conclusions: From control to gestures Progress from feedback control to open-loop gesturing –With audio/tactile feedback, can be less reliant on screen –Progression to higher-order control –Provides new users with a way to gradually learn system. Gestures can be used for simple common tasks (Autocomplete, delete etc) Dynamic representation allows intelligent systems make life easier for the user. Current system would need a lot of development before being a natural text-entry system (only ca. 17 words per minute)

25. The Selection Problem How can we determine user intention? Evaluate probability distribution over potential goals Closed-loop interaction –Goals are negotiated with the system in continuous time –Continuous feedback on user state with respect to goals Selection for devices for which pointing is non-intuitive

26. Perceptual Control Theory How can we determine intention? Perceptual Control Theory – Powers, et al. –Fundamental hypothesis: Humans act to control their perceptions –Test for this control behaviour Hypothesis: Identify intentions via correlations between input and known disturbance patterns System Feedback Control loop Goal

27. An Agent Perspective Reformulate selection in terms of agents –Each goal or item is considered an independent agent Agents probe user –experiment, look for response –Evaluate probabilities p(selected i ) –Akin to MCMC Sampling from users mind!

28. Demos

29. Example: Movement If no control, If controlled, Hence, we have: Agent Disturbance User Control Result Test: Compare distribution of histories over some time window

30. Interpretations Can be seen as control, damping, imitation, gesture recognition or excitation –Control/damping –Imitation of the motion of the disturbance (pursuit task) –Gesture recognition with dynamically created gestures –User excites modes of the object, inducing a meaningful disturbance in the object

31. Feedback Real-time feedback on potential goals –Visual example in demo –Mapping entropy to audio dissonance Log P(i) Entropy

32. Sampling from users mind E.g. Bubbles (Schyns et al 2003) Adapt idea for use with fisheye interfaces and continuous interaction instead of discrete accept/reject.

33. PCT Conclusions Probabilistic selection method suitable for non- conventional sensing and feedback systems, supporting real-time feedback on progress towards potential goals, incorporating models from manual control theory to optimize performance Much scope for extending and applying these general ideas to practical interfaces –More sophisticated user models –Disturbance/experiment design –Feedback design…

34. Outlook Dynamics allow intelligence to be sandwiched into an interface –look-and-feel of an interface, noisy channel, or in control terms, the adaptive handling qualities? Augment the display or the control? Adapt ambiguity in display to context –E.g. walking, in bus, at desk Exciting overlap between: –human motor control, –Statistics –manual control/dynamic systems –human-computer interaction

35. Other fun things…