Using Tweek to Create Graphical User Interfaces in Virtual Reality Patrick Hartling IEEE VR 2003

VR Juggler — Overview Problem description Goals for Tweek software Design overview Implementation overview Future work

VR Juggler — Problem Description Interaction in VR is primarily spatial Not all interactions map well to this method Alphanumeric input Fine-grained interaction List selection Per-application custom interfaces Map abilities of specific devices to interaction Typically use 3D input devices Why abandon effective 2D interaction techniques in 3D space?

VR Juggler — Motivation Why have GUIs in virtual reality? GUIs are everywhere in the computing world Most computer users understand how to use a GUI Why not have GUIs in VR?

VR Juggler — Issues with GUIs in Virtual Reality (1/2) Positives Typically more flexible and more extensible than other input devices – More input methods means less overloading of button presses, gestures, etc. More portable between VR systems than hardware input devices Familiarity from desktop experience

VR Juggler — Issues with GUIs in Virtual Reality (2/2) Negatives No direct physical correspondence – In VR, a user could grab an object with his/her hand or use a GUI – No GUI in real world for moving the object Remote users may not see GUI [Park00] Use of the interface may be difficult – Collision detection – Placement Immersive interfaces have to re-invent GUI technology

VR Juggler — Previous Work (1/3) Many areas of previous work Widget sets for VR PDAs as input devices Floating menus and heads-up displays Virtual User Interface (VUI) Originally developed at Iowa State University [Heath98] 3D immersive user interface May be an open source release from Fuel Tech

VR Juggler — Previous Work (2/3) Java Interface to Virtual Environment (JAIVE) Used Java interface on handheld computer [Hill00] Communicated with C++ applications Interface could be extended from C++ using JAIVE protocol

VR Juggler — Previous Work (3/3) PDA in a CAVE™ system Kent Watsen, Naval Postgraduate School Experimented with use of Palm PDA in CAVE [Watsen99] 3DI Group Doug Bowman, Virginia Tech Investigating differences in system interaction techniques between VE displays Active on 3DUI mailing list

VR Juggler — Tweek Framework for the development of cross- platform, cross-language, scalable GUIs GUIs communicate with remote applications GUIs can be use on the desktop, on a PDA, or in an immersive VR environment Features GUI panels are dynamically loaded components – Panels use standard 2D GUI “widgets” Dynamic interface update and extension Multi-language capabilities

VR Juggler — Goals Allow GUI scalability Same GUI runs on the desktop, a PDA, and a projection-based VR system Must provide an easily extendable framework for cross-platform, cross-device GUIs Interact with VR applications written in any programming language Fully separate the GUI from the application Reuse existing GUI technology We do not want to re-invent scrollbars, menus, etc. Avoid cluttering immersive space

VR Juggler — Same GUI, Multiple Uses (1/2) PDA in Projection System Offers force feedback High-resolution display – Back-lit display is needed Widely available – A person with a PDA can walk into the VR system and get the GUI dynamically

VR Juggler — Same GUI, Multiple Uses (2/2) Desktop simulation Standard GUI interface used to control VR software in simulator mode Immersive 3D space VRJ VNC incorporates 2D GUI into 3D visualization – Modeled after 3Dwm ( use of VNC and other VR VNC viewers

VR Juggler — Design Overview (1/3) Distributed Model/View/Controller

VR Juggler — Design Overview (2/3) Observer design pattern Simplified MVC: subject, observer VR application is the subject – Maintains user-defined state information – Per-application customizations GUI is the observer – Provides a view of subject’s state – GUI interactions manipulate application state Benefits – Distinct separation of application and GUI – Allows multiple GUIs to observe the same application

VR Juggler — Design Overview (3/3) Two-way communication between application and GUI Application can send data to the GUI GUI flexibility A GUI can load any component it needs to make interaction more effective Not limited to direct manipulation of application state

VR Juggler — Implementation Overview Language tools C++ VR applications Java-based GUI composed of JavaBeans – Beans can be “pushed” to the GUI while the user is in the virtual space – Dynamic extension of interface CORBA communication between C++ and Java

VR Juggler — Example Tweek Interface Multi-purpose tool – Navigation – Map visualization – Data visualization – Information storage to spreadsheet- friendly format

VR Juggler — VRJ VNC: Embedding 2D GUIs in 3D Space Embed any 2D GUI in 3D world Uses VNC Can be added to existing VR Juggler applications

VR Juggler — VRJ VNC: Presentations in VR Bring PowerPoint presentation into the 3D environment

VR Juggler — VRJ VNC: Embedding Tweek Tweek Java GUI is just another window on a desktop Set VNC desktop size to give the desired embedded frame size Interaction with the GUI is translated to the desktop via VNC

VR Juggler — Future Work Use Java Applets to allow Tweek to be used from a web browser – Make use of a familiar interface – Easy access to application GUI for remote demonstrations

VR Juggler — Online Information 3DUI Tweek Contains publications, guides, and programmer references

VR Juggler — References [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, 1999.