1. Hand-Eye Coordination and Vision-based Interaction Martin Jagersand Collaborators: Zach Dodds, Greg Hager, Andreas Pichler

2. Vision-based Tasks

3. Everyday Physical Interactions: Installing a Light Bulb

4. Sensory-Motor Coordination Humans interact effortlessly with the world through vision and physical contact Fundamental: Sensory-Motor transforms

5. Different issues What tasks can be performed? –Camera models, geometry, visual encodings How to do vision guided movement? –H-E transform estimation, feedback, feedforward motion control How to plan, decompose and perform whole tasks?

6. Tasks and Image Specifications task space error image space error image space satisfied task space satisfied

7. Task ambiguity Will the scissors cut the paper in the middle?

8. Task ambiguity Will the scissors cut the paper in the middle? NO!

9. Task Ambiguity Is the probe contacting the wire?

10. Task Ambiguity Is the probe contacting the wire? NO!

11. Task Ambiguity Is the probe contacting the wire?

12. Task Ambiguity Is the probe contacting the wire? NO!

13. Camera Models Internal calibration: Longuet-Higgins 1981 Weak calibration: Faugeras 1992, Hartley 1992 Affine calibration: Horaud et al. 1998, 1999 Stratification of stereo vision: Faugeras 1995 - characterizes the reconstructive certainty of weakly, affinely, and internally calibrated stereo rigs C aff C inj C proj C sim reconstruction up to a bijection of task space up to a projective transformation of task space up to an affine transformation of task space up to a similarity (scaled Euclidean transformation)

14. Visual Invariance  inj     proj  sim  aff   proj 

15. Task Invariance A task T ( f )=0 is invariant under a group G of transformations, iff x  f  V, g  G with g( f )  V T ( f )=0  T (g( f ))=0 n x n If T ( f ) = 0 here, T ( f ) must be 0 here, if T is G invariant proj inj sim aff

16. Jacobian Estimation

17. Estimating a Force Frame

18. Composite Task: Solving a Puzzle

19. Modulating Basis Vectors

20. Human Arm Animation Training sequence Simulated Animation