1. A M M W O R K S H O P John Hollerbach Oussama Khatib Vijay Kumar Al Rizzi Daniela Rus Control and Representation Vijay Kumar University of Pennsylvania NSF/NASA AMM Workshop March 10-11, 2005 Houston.

2. NSF/NASA AMM Workshop Outline State-of-art  Historical perspective (nostalgic memories) Accomplishments in robot control  Summary of last 21 years (WTEC study)  Recent, specific contributions (somewhat biased) Challenges  Panelists Discussion  What are the intellectual problem areas we should address? Infrastructure? Can we can rally around these?

3. NSF/NASA AMM Workshop Historical Perspective  40+ years of industrial robotics  >20 years of robotics as an academic discipline  ~13 years of mobile manipulation 40 years of industrial robotics General Motors 1961 Unimate Rus SarcosARC Hollerbach Mobility & Manipulation

4. NSF/NASA AMM Workshop The Real Agenda for AMM Mobility  Unstructured environments Manipulation  Physical interaction with the environment  Closely coupled perception/action  Not physically grounded  Dynamics is important Autonomy  Teleoperation (and therefore haptics)  Supervised Autonomy  Autonomy Haptics John Hollerbach Humanoids Oussama Khatib Perception/Action Al Rizzi Distributed/Modular Daniela Rus

5. NSF/NASA AMM Workshop Robotics in the news this week WSJ, 3/7 “…teleoperation with time delays is a vexing problem in robotics…” “…because of the lag, it’s inevitable that the human operator will make tiny errors - errors that will in turn cascade into much bigger ones…”

6. NSF/NASA AMM Workshop Literature Domain  ~8-10% manipulation  ~3-4% grasping  ~30-35% mobility Remaining are on medical, manufacturing, industrial, sensor or “methodology” Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984-86, 1998-2004 Control/representation  Model based (~15%)  Data driven approaches (~5%) Counted papers relevant to manipulation and mobility

7. NSF/NASA AMM Workshop Literature (Compared to 1984) Domain  ~10% manipulation  ~4% grasping  ~35% mobility Remaining are on medical, manufacturing, industrial, sensor or “methodology” Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984-86, 1998-2004 Control/representation  Model based (~15%)  Data driven approaches (~5%) Counted papers relevant to manipulation and mobility (40%) (4%) (40%) (3 %) Total number of papers = 74 ~9880 ICRA papers to date

8. NSF/NASA AMM Workshop Major Advances Academic/Government Labs  Inverse dynamics: application of feedback linearization to serial robots, now routinely used in industrial manipulators (e.g., ABB)  Time optimal control: along a path subject to dynamics, velocity and acceleration constraints, also used in industrial manipulators  Adaptive robot control: model based adaptive control with global stability guarantee  Nonholonomic control: control using time varying feedback or cyclic input, application of differential flat system theory, mostly applied to mobile robots and under-actuated robots. [Wen and Maciejewski, 04] !!! !? !!! Disclaimer: Not a survey of accomplishments/needs for AMM

9. NSF/NASA AMM Workshop Major Advances (Cont.)  Flexible joint robot modeling and control: Application of feedback linearization to flexible joint robots, applied to some industrial arms.  Teleoperation: wave variable based control for delay robustness. Guarantee stability, but user would feel delayed response.  Order N simulation: Application of order N computation to forward and inverse dynamics. Essential for large number degrees of freedom, e.g., robot with flexible link, micro-robots.  Hybrid force/position, impedance control: Simultaneous regulation of motion and force, applied to machining, assembly, haptic feedback, multi-finger control ?! ! !!!

10. NSF/NASA AMM Workshop AMM Survey (?) ICRA 2000: Grasping and Manipulation Review [Bicchi and Kumar, 2000] Saturation of the area?  All problems solved  Not interesting  Not relevant

11. NSF/NASA AMM Workshop Two other possibilities Problems are too hard Or Nobody is interested in funding this work!

12. NSF/NASA AMM Workshop Significant Accomplishments: Industry Fanuc 20% market share 1800 employees (1300 in research labs, 10 Ph.Ds) 10,000 robots Technology provides the competitive edge  Before z servo motors/amplifiers  Now z collision detection, compliance control, payload inertia/weight identification, force/vision sensing/integration  robots assemble/test robots  beyond human performance And mobile manipulation! Technology transfer does happen! Remember those ~9880 ICRA papers?

13. NSF/NASA AMM Workshop Results we can build on… (a parochial view) Modeling/controlling humanoids Dynamic manipulation and locomotion Cooperative mobile manipulation Distributed locomotion (and manipulation) systems Haptics and teleoperation

14. NSF/NASA AMM Workshop Humanoid dynamics and control Biomechanics for robotics  Realistic models  Minimum principles leading to realistic motions [Khatib] Integration (composition)  Integrated control of reach and posture  Task space versus posture space

15. NSF/NASA AMM Workshop Humanoid dynamics and control Whole-body multi-contact control  Multiple frictional contacts  Models z Posture z Legs z Locomotion [Khatib]

16. NSF/NASA AMM Workshop Locomotion and Dexterous Manipulation Dynamic manipulation and locomotion  Intermittent interaction  Passive dynamics  Reactive control [Rizzi]

17. NSF/NASA AMM Workshop Significant Accomplishments: Academia Multiple Mobile Manipulators  Multiple frictional contacts  Maintaining closure [Khatib] [Kumar][Rus]

18. NSF/NASA AMM Workshop M 3 Modular Mobile Manipulation Self-organizing, self-assembling, self-repair  Adapt structure  Multiple Functionalities  Can do work [Rus]

19. NSF/NASA AMM Workshop Teleoperation and Haptics High-DOF telemanipulators Locomotion Interfaces [Hollerbach]

20. NSF/NASA AMM Workshop And yet significant challenges remain! No successful field deployment of mobile manipulators  Example: Robotic servicing of Hubble (NAS Committee: Brooks, Rock, Kumar)  ETS-VII (JAXA/NASA) z Model-based tele-manipulation z Visual servoing for acquisition of non cooperative targets No robot (product) capable of physical interactions in unstructured environment  Example: Assistive Robotics

21. NSF/NASA AMM Workshop Assistive Robotics Impact  > 5 million wheelchair users* in the U.S.  > 730,000 strokes/year (2/3 disabled five years after stroke), > $50B/year  > 10,000 SCI/year (most < 20 yrs old) Realistic  Human-in-the-loop  No competing technology z Many other overarching challenges *Inter Agency Working Group on Assistive Technology Mobility Devices

22. NSF/NASA AMM Workshop Current technology  Artificial limbs: peg legs, hook hand  Crutches, canes, walkers  Wheelchairs  Environmental control systems  Remote control  Many, many customized products

23. NSF/NASA AMM Workshop Significant Challenges, Problems 1. New hardware, systems 2. Modeling/control 3. Composition, synthesis 4. Model-based versus data-based

24. NSF/NASA AMM Workshop pHRI: Safety and Performance >20 cm compliant covering Challenge: 10x reduction in effective inertia [Khatib]

25. NSF/NASA AMM Workshop Haptic Interfaces and Mobility Energetic/force interactions between robots and humans  Control simulations or real devices  Personal assist or amplification devices  Rehabilitation or exercise robots Need haptic interfaces that allow manipulation while walking  Psychological argument for VR  Need to control robots that can reach/grasp/manipulate/lean/kick/push [Hollerbach]

26. NSF/NASA AMM Workshop Portable Haptic Interfaces Body-worn systems  Powered exoskeleton  Ground-based system with locomotion interface

27. NSF/NASA AMM Workshop Representation and Control  Physics of environmental interaction  Distributed interaction z Whole arm/leg/body  Task representation for non-rigid interaction and manipulation  Control and task allocation of multi-function appendages (feet, legs, hands, arms, etc.)  Composition of closed-loop (perception/action) behaviors [Rizzi]

28. NSF/NASA AMM Workshop Composition of Behaviors: Example Four behaviors (closed-loop controllers)  Pre-shape (open/close)  Grasp/release  Reach/retract  Go to (move)

29. NSF/NASA AMM Workshop Composition Pre-shape (close) > Retract

30. NSF/NASA AMM Workshop Composition Retract > Move

31. NSF/NASA AMM Workshop Composition Move || Pre-shape (open)

32. NSF/NASA AMM Workshop Composition Move || Pre-shape (open)

33. NSF/NASA AMM Workshop Composition Pre-shape (open) > Grasp

34. NSF/NASA AMM Workshop Composition Grasp > Retract || Move

35. NSF/NASA AMM Workshop Composition Move

36. NSF/NASA AMM Workshop Composition Move > Reach > Release

37. NSF/NASA AMM Workshop Composition

38. NSF/NASA AMM Workshop Distributed Approaches and Modularity Distributed Control  Heterogeneous systems with active modules, passive modules, and tools for mobile manipulation  Mobile sub-assemblies and hierarchical control Thanks to Hod Lipson

39. NSF/NASA AMM Workshop Future Concept for Modular Robots in Mobile Manipulation Concept: self-assembly with active grippers and rods Concept: mobile sub-assemblies note: mobile manipulation with dynamic kinematic topology for c-space Concept: self-inspection and self-repair with tools

40. NSF/NASA AMM Workshop Distributed Approaches and Modularity Challenges Control for systems with dynamic kinematic topology  Under-constraint systems with continuum of solutions  Control for systems with changing c-space  Geometrically-driven posture control  Control for keeping balance and structural integrity  Optimal morphologies for tasks Uncertainty and Error in Modular Systems  Cooperative approach to error recovery in module and structure alignment, connections, assembly, and repair  Dynamical models with uncertainty

41. NSF/NASA AMM Workshop Model-based vs. Data Driven Control/representation  Model based (~15%)  Data driven approaches (~5%)  Dynamic models are getting more complicated and increasingly sensitive to parameters (uncertainty)  Emphasize completely data-driven approaches

42. NSF/NASA AMM Workshop Discussion Are there a set of basic research questions that  We can rally around?  Are unique to autonomous mobile manipulation?  Are critical? High-impact? If so, can we create a new research program?  How do we sell it?  How do we take this to the next step? Balance  basic research  high-caliber applied research How do we make robotics a “big science”?

43. NSF/NASA AMM Workshop Intellectual Basis for New Program in Autonomous Mobile Manipulation Closed-loop behaviors  Perception-action loops  Vision-based control Composition of behaviors  Sequential  Parallel, hierarchical Task description language  Formal semantics Uncertainty  Understanding and characterizing uncertainty  Data-driven approaches Teleoperation and haptics  Integration mobility with manipulation Can it be a Tether-esque program?