1. Action-Grounded Push Affordance Bootstrapping of Unknown Objects
Barry Ridge & Aleš Ude
Humanoid and Cognitive Robotics Lab
Department of Automation, Biocybernetics & Robotics
IROS Nov 4th, 2013 1

2. Introduction

3. Grounding 3-D Shape Features in Push Actions
RGB image + segmented 3-D point cloud
Shape primitives fitted to object parts
Learn object push affordances when objects are pushed from different positions & directions.
Dense 3D point clouds from modern depth sensors capture object surface shape with reasonably high accuracy.
Object models can be formed dynamically by fitting simple shape primitives to object parts in such point clouds.
Shape & position of object parts can be used to learn & predict affordances when objects are pushed from different positions & directions.

4. Point Cloud Library (PCL)

5. Calibrated Point Clouds & Push Trajectories

6. Action Coordinate Frame
Object point clouds before/after interaction + hand push trajectory
Point clouds + trajectory transformed to push action coordinate frame
Transform object point clouds before & after interaction to push action coordinate frame.
We define the push action coordinate frame w.r.t. the push contact point & direction:
- Origin centred at object contact point.
- Frame orientation found by fitting line to push trajectory using least squares linear regression.

7. Action-Grounded Shape Features
Fitting Planes to Object Parts Before Interaction
Fitting Planes to Object Parts After Interaction
Point Clouds + Trajectory transformed to Push Action Coordinate Frame
Divide object point cloud into part cells:
1 cell for overall point cloud.
2 cells by evenly splitting point cloud by its extent in X-dimension.
- 2 cells by evenly splitting point cloud by its extent in Y-dimension.
- 2 cells by evenly splitting point cloud by its extent in Z-dimension.
Find point centroids + Fit planes in each cell.
Derive 35-dimensional object shape feature vector.

8. Action-Grounded Shape Features

9. Learning Goal Cross-View Co-Occurrences Input Feature Space (Pre-Push)
Output Feature Space
(Post-Push)

10. Self-supervised Multi-view Learning Vector Quantization

11. Class Discovery
In order to discover affordance classes in the output layer, we perform unsupervised clustering of the prototypes using the X-means algorithm to find the optimal k* clustering...

12. Class Prediction
...then finally, class labels may be assigned to input layer prototypes using the following labeling function:
...and input samples may then be classified by matching them to their nearest-neighbour input layer prototype.

13. Experiments 5 Push Types 4 Affordance Types
5 push types: push forward from centre of object, top of object, bottom of object, left of object and right of object.
4 affordance types: forward translation, forward topple, left rotation and right rotation.

14. Experiments
134 object push trials performed on 5 household objects in various positions and orientations on table surface.
10-Fold Incremental Cross Validation.
Comparison with incremental supervised learning methods based on vector quantization, including:
GLVQ (Generalized Learning Vector Quantization)
GRLVQ (Generalized Relevance Learning Vector Quantization)
SRNG (Supervised Relevance Neural Gas)

15. Results: Short-Term Training

16. Results: Long-Term Training

17. Results: Reduced Feature Set, Short-Term Training

18. Results: Reduced Feature Set, Long-Term Training

19. Conclusions
Main contribution: an action grounded 3-D visual feature descriptor to be used for bootstrapping object affordances when objects are unknown.
When used in combination with a self-supervised learning algorithm, this feature descriptor is effective at facilitating both affordance class discovery and prediction in an online learning setting with a number of initially unknown objects and object affordances.
A feature relevance determination extension to the self-supervised algorithm was also shown to boost affordance class prediction results by emphasizing the discriminative contribution of particular features within the descriptor.