2. Vision-Based Interactive Systems Martin Jagersand c610

3. Applications for vision in User Interfaces Interaction with machines and robots – Service robotics – Surgical robots – Emergency response Interaction with software – A store or museum information kiosk

4. Service robots Mobile manipulators, semi-autonomous DIST TU BerlinKAIST

5. TORSO with 2 WAMs

6. Service tasks This is completely hardwired! Found no real task on WWW

7. But Maybe first applications in tasks humans can’t do?

8. Why is humanlike robotics so hard to achieve? See human task: – Tracking motion, seeing gestures Understand: – Motion understanding: Translate to correct reference frame – High level task understanding? Do: – Vision based control

9. Types of robotic systems Auton omy Generality Supervisory control Tele-assistance Programming by demonstration Preprogrammed systems

10. Interaction styles If A then end Conventional : Low bandwidth interaction Partial or indirect system state displayed User works from internal mental model

11. Interaction styles Direct Manipulation Direct Manipulation: High bandwidth interactionHigh bandwidth interaction Interact directly and intuitively with objects (affordance)Interact directly and intuitively with objects (affordance) See system state (visibility)See system state (visibility) (Reversible actions)(Reversible actions)

12. Examples of Direct Manipulation Drawing programs e.g. Mac Paint Video games, flight simulator Robot/machine teaching by showing Tele-assistance Spreadsheet programs Some window system desktops But can you always see effects (visibility)?

13. xfig drawing program Icons afford use Results visible Direct spatial action- result mapping line([10, 20],[30, 85]); patch([35, 22],[15, 35], C); % C complex structure text(70,30,'Kalle'); % Potentially add font, size, etc matlab drawing:

14. Why direct manipulation? Recognition quicker than recall. Human uses “the world” as memory/model Human skilled at interacting spatially How quick is direct? Subsecond! Experiments show human performance decreased at 0.4s delay. Subsecond! Experiments show human performance decreased at 0.4s delay.

15. Vision and Touch based UI Typical UI today: Symbolic, 1D (slider), 2D But human skilled at 3D, 6D, n-D spatial interaction with the world Supports Direct Manip!

16. Seeing a task Tracking movement – See directions, movements in tasks Recognizing gestures – Static hand and body postures Combination: Spatio-temporal gestures

17. Tracking movement Tracking the human is hard: – Appearance varies – Large search space, 60 parameters – Unobservable: Joint angles have to be inffered from limb positions, clothing etc. – Motion is non-linear. – Difficult to track 3D from 2D image plane info – Self occlusion of limbs

18. Trick 1: Physical model Reduce number of DOF’s by coupled model of articulated motion (Hedvig, Mike)

20. Trick 2: Use uniqueness of skin color Can be tracked at real time

21. Gestures: Identifying gestures is hard – Hard to segment hand parts – Self occlusion – Variability in viewpoints

22. Trick 3: Scale space Define hand gesture in course to fine terms

23. Trick 4: Variability filters

24. Programming by Demonstration From assembly relations From temporal assembly sequence – Segmenting manipulation sequence into parts (subtasks) is hard Using a gesture language

25. Tele-assistance: Gestures + context

26. Robust manipulations

27. Conclusions Most aspects of Robot see – robot do are hard Conventional methods are – Incapable of seeing task – Incapable of understanding what’s going on – Incapable of performing human manipulation tasks Uncalibrated methods are more promising