1. Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Mitul Saha and Pekka Isto Research supported by NSF Artificial Intelligence Lab Stanford University Research Institute for Technology University of Vaasa, Finland

2. The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning There has not been much development in manipulation planning for deformable objects because Manipulation Planning Research so far… So far, manipulation planning research has mainly focused on manipulating rigid objects We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

3. The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning There has not been much development in manipulation planning for deformable objects because Manipulation Planning Research so far… So far, manipulation planning research has mainly focused on manipulating rigid objects We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

4. The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning There has not been much development in manipulation planning for deformable objects because Manipulation Planning Research so far… So far, manipulation planning research has mainly focused on manipulating rigid objects We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

5. The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades –it is difficult to model and predict the deforming nature of deformable objects –struggle in basic motion planning There has not been much development in manipulation planning for deformable objects because Manipulation Planning Research so far… So far, manipulation planning research has mainly focused on manipulating rigid objects We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

6. Manipulation Planning for Deformable Linear Objects (DLOs) GOAL: to develop a motion planner that would enable robots to autonomously manipulate Deformable Linear Objects (ropes, cables, sutures) in various settings. bowline knot figure-8 knot sailing knot autonomous robotic DLO manipulation knot tying in daily/recreational life laying/loading cables in industrial settings suturing in medical surgery robot dress

7. Manipulation Planning for Deformable Linear Objects (DLOs) The DLO manipulation problem is extremely challenging for robotics because obeing highly deformable, they can exhibit a much greater diversity of behaviors, which are hard to model and predict oidentifying topological states of DLOs is coupled with some unsolved problems in knot-theory/ mathematics Interesting Challenging The DLO manipulation problem has a nice structure. It brings together robotics, knot theory, and computational mechanics.

8. Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. - knot simplified using Reidemeister moves (RM) from knot theory -one robot used to execute the RMs -assumes DLO resting on a plane

9. Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. Our contribution: -DLO need not be in a plane -We use more than one robot in coordination -We consider collision constraints (robot-DLO, robot-obstacle) -We consider the physical behavior of the DLO while planning -We consider interaction of the DLO with other objects - knot simplified using Reidemeister moves (RM) from knot theory -one robot used to execute the RMs -assumes DLO resting on a plane

10. The Manipulation Problem How do we define goal configurations? available robot arms

11. Goal configurations are defined in terms of topology instead of exact geometry Geometrically different but topologically same: Bowline knot Defining Goal Configurations while winding, number of wounds more important

12. In knot theory, crossing configuration of a curve is used to characterize its topology Defining Goal Configurations planar projection of the DLO central axis

13. In knot theory, crossing configuration of a curve is used to characterize its topology Crossing Configuration: (C 1, C 2, C 3, C 4 ): ((1,-6) -, (-2,5) -, (3,-8) -, (-4,7) - ) crossing: local self-intersections Defining Goal Configurations C 1 : (1,-6) - C 2 : (-2,5) - C 3 : (3,-8) - C 4 : (-4,7) - sign of a crossing planar projection of the DLO central axis how to account for interactions with other objects? make them part the DLO semi-deformable linear object (sDLO)

14. We take as input the physical model of the DLO in the form of a state transition function f: Physical modeling of the DLO Suture model: [Brown, et al., 04] Elastic thread model: [Wang, et al., 05]Nylon thread model: [Dhanik, 05] Recent successes in computational mechanics:

15. Manipulation using 2 cooperating robot arms Manipulation Tools

16. Manipulation using 2 cooperating robot arms Use of static sliding supports (“tri-needles”) to provide structural support Manipulation Tools

17. Defining “Forming Sequence” Forming Sequence: C 2, C 1, C 4, C 3 Basis of our Planning Approach walk along the DLO; crossing “formed” when encountered the second time

18. Defining “Forming Sequence” Forming Sequence: C 2, C 1, C 4, C 3 Basis of our Planning Approach walk along the DLO; crossing “formed” when encountered the second time C2C2 C1C1 C4C4 C3C3 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence”

19. Defining “Forming Sequence” Forming Sequence: C 2, C 1, C 4, C 3 Basis of our Planning Approach Defining “loop hierarchy” used to determine the placementof static sliding supports (“tri-needles”) walk along the DLO; crossing “formed” when encountered the second time C2C2 C1C1 C4C4 C3C3 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence”

20. Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use forming sequence to bias search -use physical model to sample new DLO shapes -use the loop hierarchy to place static sliding supports (tri-needles) search tree forbidden region

21. Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use forming sequence to bias search -use physical model to sample new DLO shapes -use the loop hierarchy to place static sliding supports (tri-needles) search tree forbidden region grasping robot fails Robot A DLO Robot A Robot B

22. Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use forming sequence to bias search -use physical model to sample new DLO shapes -use the loop hierarchy to place static sliding supports (tri-needles) search tree forbidden region

23. Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use forming sequence to bias search -use physical model to sample new DLO shapes -use the loop hierarchy to place static sliding supports (tri-needles) search tree forbidden region tri-needles loop hierarchy

24. Results bowline knot sailing knot bow neck-tie -Planner implemented in C++ -Took 15-20 minutes on a 1GB, 1GHz processor to generate manipulation plans for tying popular knots: bowline, neck-tie, bow (shoe-lace), and stunsail -Videos: http://ai.stanford.edu/~mitul/dlo

25. Results

26. neck-tie

27. In the real-life, we have tested the ability of the planner to generate robust plans by tying the popular Bowline knot with various household ropes on a hardware platform with two PUMA robots, using the manipulation plan generated by the planner. Results bowline knot robustness dues to tri-needles

28. Conclusion We have developed a motion planner for manipulating deformable linear objects (such as ropes, cables, sutures) in 3D using cooperating robots. - it can tie self-knots and knots around rigid objects - unlike in traditional motion planning, goals are topological and not geometric - we account for the physical behavior of the DLO - it is robust to imperfections in the physical model of the DLO - it is first of its kind (we not aware of any other planner for computing collision- free robot motions to manipulate a DLO in environments with obstacles) - the implemented planner has been tested both in graphic simulation and in real-life on a dual-PUMA-560 hardware platform suturing in medical surgery collaboration with General Motors Future Plans

29. Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Acknowledgement: Advisory: Jean-Claude Latombe PUMA experiments: Oussama Khatib, Irena, Jaehueng Park, Jin Sung Physical models of ropes: Etienne Burdet, Wang Fei (EPFL) Useful comments: anonymous reviewers

30. - Tight knots-Semi-tight knots We focus on two types of common knots: Crossing Configuration: ((1,-6) -, (-2,5) -, (3,-8) -, (-4,7) - ) over under over

31. Needle Placement