1. Interaction techniques for post-WIMP interfaces Lawrence Sambrooks Supervisor: Dr Brett Wilkinson

2. 2 “WIMP GUI’s based on the keyboard and mouse are the perfect interface only for creatures with a single eye, one or more single-jointed fingers, and no other sensory organs.” – Bill Buxton [1]

3. Summary Research question Background Methodology Proposed architecture Technology User evaluation Termination criteria Timeline Questions 3

4. What is post-WIMP? 4 Microsoft Surface Bimanual High bandwidth Intuitive Invisible Multi-finger Multi-sensory Multi-user Task focussed

5. Research question Investigate and compare various interaction techniques for post-WIMP interfaces 5

6. Background WIMP GUI’s were popularised by the Mac in 1984 and later copied to PC and Unix workstations [1] Input and output devices remain largely unchanged from what was demonstrated by Douglas Engelbart in 1968 [4] Physical input devices limit the interactions that can take place [3] 6

7. Methodology Develop a simple, multi-modal 3D data visualisation application with support for three interfaces: Multi-touch Gesture Mouse and keyboard Provide the same functionality for each interface and allow the interactions to differentiate between them Test early and test often [2] 7

8. 8 Methodology SpaceStuntz (created by Martin Henschke) Utilise data sets provided by existing project [7]

9. The focus is HCI 9 HCI Data visualisation

10. Architecture 10

11. 11 Data visualisation

12. Technology Hardware: Windows 7 tablet PC Kinect Mouse and keyboard Software: Windows 7 SDK (multi-touch) Kinect SDK Visual Studio 2010 + XNA Game Studio 4.0 TechCraft block engine [5] 12 Microsoft Kinect

13. User evaluation Target groups: Honours, Masters, and PhD students Academic staff Feedback channels: Surveys NASA TLX (Task Load Index) [6] Performance tests Internal program statistics 13

14. Termination criteria All three interfaces are functional A sufficient sample of data has been collected for each of the interfaces such that a reasonable conclusion can be reached 14

15. Timeline 15

16. Conclusion In summary: Investigate three interaction techniques for a post-WIMP interface Develop a simple 3D visualisation application Focus on the HCI aspects rather than data visualisation Collect feedback for each interface via user evaluation More information: http://wiki.csem.flinders.edu.au/bin/view/Projects/ProjectSamb0014 16

17. Questions?

18. Hauptmann, A. G. Speech and gestures for graphic image manipulation. SIGCHI Bull., 20, SI 1989), 241- 245. Bolt, R. A. and Herranz, E. Two-handed gesture in multi-modal natural dialog. In Proceedings of the Proceedings of the 5th annual ACM symposium on User interface software and technology (Monteray, California, United States, 1992). Thorisson, K. R., Koons, D. B. and Bolt, R. A. Multi-modal natural dialogue. In Proceedings of the Proceedings of the SIGCHI conference on Human factors in computing systems (Monterey, California, United States, 1992). Nielsen, J. Noncommand user interfaces. Commun. ACM, 36, 4 1993), 83-99. Jacob, R. J. K. Human-computer interaction: Input devices. ACM Comput. Surv., 28, 1 (Mar 1996), 177- 179. van Dam, A. Beyond WIMP. Computer Graphics and Applications, IEEE, 20, 1 2000), 50-51. van Dam, A. User interfaces: disappearing, dissolving, and evolving. Commun. ACM, 44, 3 2001), 50-52. [1] van Dam, A. Post-WIMP user interfaces: the human connection. Springer-Verlag London Ltd, Godalming, 2001. Shen, C., Ryall, K., Forlines, C., Esenther, A., Vernier, F. D., Everitt, K., Wu, M., Wigdor, D., Morris, M. R., Hancock, M. and Tse, E. Informing the Design of Direct-Touch Tabletops. Computer Graphics and Applications, IEEE, 26, 5 2006), 36-46. Aoyama, D. A., Hsiao, J. T. T., Cardenas, A. F. and Pon, R. K. TimeLine and visualization of multiple-data sets and the visualization querying challenge. J. Vis. Lang. Comput., 18, 1 (Feb 2007), 1-21. Cawthon, N. and Moere, A. V. The effect of aesthetic on the usability of data visualization. IEEE Computer Soc, City, 2007. Hornecker, E. "I don't understand it either, but it is cool:” visitor interactions with a multi-touch table in a museum. City, 2008. Chan, A. T. S., Leong, H. V., and Kong, S. H. Real-time Tracking of Hand Gestures for Interactive Game Design. IEEE, New York, 2009. Edelmann, J., Schilling, A. and Fleck, S. The DabR: A multitouch system for intuitive 3D scene navigation. City, 2009. Petersen, N. and Stricker, D. Continuous natural user interface: Reducing the gap between real and digital world. City, 2009. Reisman, J. L., Davidson, P. L. and Han, J. Y. A screen-space formulation for 2D and 3D direct manipulation. In Proceedings of the Proceedings of the 22nd annual ACM symposium on User interface software and technology (Victoria, BC, Canada, 2009). [2] Yee, W. Potential Limitations of Multi-touch Gesture Vocabulary: Differentiation, Adoption, Fatigue. Springer-Verlag Berlin, City, 2009. [3] Andrews, D. and Teoh, S. T. MTVis: Tree Exploration Using a Multi-Touch Interface. Spie-Int Soc Optical Engineering, City, 2010. [4] Lee, J. C. In search of a natural gesture. XRDS, 16, 4 2010), 9-12. Knoedel, S. and Hachet, M. Multi-touch RST in 2D and 3D spaces: Studying the impact of directness on user performance. City, 2011. [5] TechCraft block engine: http://techcraft.codeplex.com [6] NASA TLX: http://human-factors.arc.nasa.gov/groups/TLX [7] Hobbs, D. Rehabilitation gaming for children with Cerebral Palsy (ongoing PhD research). References