1. Efficient 3D Object Perception and Grasp Planning for Mobile Manipulation in Domestic Environments
Jorg Stuckler, Ricarda Steffens, Dirk Holz, and Sven Behnke
University of Bonn, Germany (2012)
Presented by Natalia Jacobowitz
Introduce self

2. Introduction
The paper explains how a group of engineers came up with efficient and fast methods for perceiving objects and planning a way to grasp the objects.
Utilizing:
Real-time object perception
Efficient grasp planning
Motion control
Allowing for continuous task execution something that had not been a focus in prior research
Tested in lab experiments and at the competitions

3. Related Work
Other people have attempted to develop systems for mobile manipulation in everyday environments but none are exactly like the research presented in this paper
Similar work has been done with a PR2
Other work was limited to recognizable objects, unlike the research being presented here which can handle previously unseen objects
Different research similarly performed top grasps along the object's principal axis.
Related work is overall slower than what was achieved here in the paper

4. Cosero
Robot

5. System Overview Cognitive Service Robot Cosero: 2 anthropomorphic arms
Yaw joint in the torso to enlarge workspace of the arms
Linear actuator in the trunk will move the upper body vertical .9m
Omnidirectional drive
*** Festo FinRay fingers on rotary joints (see next slide)
Microsoft Kinect RGB-D camera for sensing 3D environment
Infrared distance sensors attached to the palm in each gripper for measuring object’s distance from the gripper

6. Festo FinRay Fingers Lightweight plastics material
Fingers adapt its shape to the object surface
Anti-skidding material onto finger surface for grip

7. Festo FinRay Fingers

8. Motion Control & Mobile Manipulation
Omnidirectional drive
8 wheeled mobile base
The linear and angular velocity can be set independently and changed
Anthropomorphic arms
Differential inverse kinematics with redundancy resolution
Enables the robot to open doors and carry large objects
Robot will approach the horizontal surface with an object and will adjust its height and distance to align itself with the object
The robot can do a variety of tasks such as deliver an item to a specific person through recognition but this is not the focus of the paper
While the paper mentions these features- it is not the focus

9. Real-time 3D Perception
Utilizing the Microsoft Kinect for its depth camera
High frame rates allows for processing of images with 160x120 resolution at 20Hz
Compute Normals
Extract Horizontal Points
Detect Support Plane
Detect Object Candidates
Track Objects
5 steps will be elaborated in the following slides

10. Object Perception
Some of the calculations can be eliminated because we know that household objects are found in a finite set of locations- specifically on horizontal support planes.
Tabletops
Shelves
The experiments conducted are limited to the above two assumed locations

11. Compute Normals
Directly compute the normal vector over the neighboring pixels in x and y image space
For every point compute local surface normals from the cross product of two tangents to the surface
Create two maps of tangential vectors- one for rows and one for columns
For each map compute the difference vectors between corresponding 3D points
Compute an integral image for each channel and then compute the average tangential vector.
Runtime = linear

12. 2. Extract Horizontal Points
Extract all points with vertical normals
The resulting points are lying on the horizontal surface
Allows for efficiently segmenting the complete depth into planes
Looking at just horizontal planes saves computations

13. 3. Detect Support Plane
Find the most dominant horizontal plane and the points supporting it
The support plane will be horizontal
Only look at points and support planes lower than 1 meter and closer than 3 meters from the robot

14. 4. Detect Object Candidate
Extract the points above the horizontal support plane
Apply Euclidean clustering to get sets of points and object candidates

15. 5. Track Objects
Because of the fast segmentation the ability to detect and track objects in depth images at high frame rates - over several frames is possible
All object candidates will have their principal axis computed
Kalman filters, hungarian method

16. Efficient Grasp Planning
Overview: Finding feasible, collision-free grasps from the raw object point cloud
Collision free
Feasible

17. Efficient Grasp Planning
Paper’s approach to grasp planning is particularly good for:
Rigid objects
Object symmetrical in shape along the vertical axis
Center of gravity is at the center of the object
But the approach is still good for objects that don’t fit that mold

18. Efficient Grasp Planning
Grasping can be from the side or from above
Side grasp:
Approach the object along its vertical axis with the grippers aligned horizontally
Pre-grasp pose is reached, side-grasp motion primitive approaches the object, close gripper
Top grasp:
Pitch the end effector by 45 degrees downward to grasp object with the fingertips
Reach pre-grasp pose and then establish pitch orientation
Use IR distance sensors in the grippers to determine any premature contact with the object or support plane

19. Planning Collision-Free Grasps
Grasp planner selects a feasible collision-free grasp for the object of interest
Will sample grasp candidates, remove infeasible and colliding grasps, and ranks the remaining grasps to find the best one
First find out the shape and pose of the object
Side grasp: pre-grasp pose is sampled on an ellipse in the horizontal plane and grasp the object as low as possible (half the height of the gripper + .03m)
Top grasp: sample it on all points in height range of 2cm below the highest point of the object

20. Filtering for Feasible and Collision-Free Grasps
Post processing: filter out grasps based on the following:
Grasp width - reject grasps if the object will not fit into the gripper
Object height - reject side grasps if the object is too low
Reachability - reject grasps outside of the arm’s workspace
Collisions - reject grasps that collide during the reaching or grasping
Grasps
that
satisfy
Criteria

21. Searching For Collisions

22. Ranking of Grasps
Once there are collision-free grasps then the grasps get ranked:
Distance to object center - favor grasps with smaller distance to object center
Grasp width - favor grasps widths closest to .08m
Grasp orientation - favor grasps with smallest angle between line toward shoulder and grasping direction
Distance from robot - favor grasps with smaller distance to the shoulder

23. Using Grasp Ranking
Then using the rankings find the best side and top grasps and then choose the more appropriate of the two.
The object’s height is relevant for this.
Side grasps are favored slightly because they are faster

24. Experiments
All runtimes for each step were calculated

25. Experiments & Robustness
8 household objects and executed 12 grasps each
Didn’t just use rigid objects- tried clothes
Later tried this with a shelf, not just table
Sometimes only top grasp or only side grasp are feasible (not both)

26. Experiments & Robustness: Table

27. Experiments & Robustness: Shelf

28. Public Demonstration
No longer just tested in the controlled lab environment
competition:
GermanOpen 2011 winner *
GermanOpen 2012 winner
Istanbul 2011 winner
* Corsero and Dynamaid prepared breakfast (see video next slide)

29. Cosero Robot https://www.youtube.com/watch?v=zR_6IrJswU4
Grasp bottle of milk, open bottle, pour milk, dispose of bottle, garasp spoon (note top grasp) and place next to bowl

30. Conclusion Fast & Robust
The motivation behind all of this is to have more anthropomorphic robots- robots that can grasp every day items in your home
Paper presents efficient means to “perceive objects on planar surfaces and to plan feasible, collision-free graps on the object of interest”
Achieves this very fast: in real-time
Analyzes the many grasp options, rejects the bad, and prioritizes the better
Robust: can pick up a variety of household objects
If there is a failure the robot will try to pick up the object again

31. Opinions & Thoughts Repeatedly uses words “efficient” and “fast”
Expected a little more focus on the grasping
I was interested in learning more about grasping
Paper’s researchers haven’t done much more research relating to grasp planning but have researched more about 3D modeling and RGB-D cameras
Holz & Behnke have continued writing papers together
Stuckler & Benkhe have also continued writing papers together
Seems like they were successful especially since they won the two years prior to the publication of this paper

32. Sources: https://dl.acm.org/citation.cfm?id=2527951

33. Questions?