1. Direct Manipulation and Vision and Touch based User Interfaces Martin Jägersand Johns Hopkins University CIPS and CISST groups (Computational Interaction with Physical Systems and Computer Integrated Surgical Systems and Technology)

2. Overview 1. Direct Manipulation From 1 st HCI course 2. Vision and Touch based User Interfaces From advanced UI course Interaction styles Command Form menu

3. Interaction styles If A then end Conventional: Low bandwidth interaction Partial or indirect system state displayed User works from internal mental model

4. Interaction styles Direct Manipulation Direct Manipulation: High bandwidth interactionHigh bandwidth interaction Interact directly and intuitively with objects (affordance)Interact directly and intuitively with objects (affordance) See system state (visibility)See system state (visibility) (Reversible actions)(Reversible actions)

5. Examples of Direct Manipulation Drawing programs e.g. Mac Paint Drawing programs e.g. Mac Paint Video games, flight simulator Video games, flight simulator Robot/machine teaching by showing Robot/machine teaching by showing Spreadsheet programs Spreadsheet programs Some window system desktops Some window system desktops But can you always see effects (visibility)?

6. xfig drawing program Icons afford use Icons afford use Results visible Results visible Direct spatial action- result mapping Direct spatial action- result mapping line([10, 20],[30, 85]); patch([35, 22],[15, 35], C); % C complex structure text(70,30,'Kalle'); % Potentially add font, size, etc matlab drawing:

7. Why direct manipulation? Recognition quicker than recall. Recognition quicker than recall. Human uses “the world” as memory/model Human uses “the world” as memory/model Human skilled at interacting spatially Human skilled at interacting spatially How quick is direct? Subsecond! Experiments show human performance decreased at 0.4s delay. Subsecond! Experiments show human performance decreased at 0.4s delay.

8. Vision and Touch based UI Typical UI today: Symbolic, 1D (slider), 2D Typical UI today: Symbolic, 1D (slider), 2D But human skilled at 3D, 6D, n-D spatial interaction with the world But human skilled at 3D, 6D, n-D spatial interaction with the world

9. User Interfaces for Physical Interaction Observe and Interpret Physical Interactions Make Machines work with Humans Soon most appliances will have embedded computers

10. On-line Visual-Motor function estimation Linear model around current state: Observed motion: Secant condition: Rank 1 Jacobian update

11. Visual space trajectory plan Reaching Reaching Fine manipulation

12. Visual programming for manip Like drawing program but with action primitives Like drawing program but with action primitives

13. Animation and Image Synthesis Train on a real video sequence. Learn appearance representation. Learn visual-motor modulation function Animate any other motion sequence. Real training video Synthesized and animated movie

14. Take home: Direct Manipulation Interact directly and intuitively with objects (affordance) Interact directly and intuitively with objects (affordance) See system state (visibility) See system state (visibility) Much of human communication is through spatial interaction using vision and touch Much of human communication is through spatial interaction using vision and touch