Bryan Willimon, Steven Hickson, Ian Walker, and Stan Birchfield Clemson University IROS Vilamoura, Portugal An Energy Minimization Approach to 3D Non- Rigid Deformable Surface Estimation Using RGBD Data

 We propose an algorithm that uses energy minimization to estimate the current configuration of a highly non-rigid object.  Our approach relies on a 3D nonlinear energy minimization framework to solve for the configuration using a semi-implicit scheme adapted from Fua and colleagues (Pilet et al. 2005). Overview

Previous Work on Pose Estimation for Robotics Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper. Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm. Both approaches require predefined fiducial markers.

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms: S moothness term Energy Minimization Approach C orrespondence term D epth term B oundary term

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms: Energy Minimization Approach

Mesh Initialization Energy Minimization Approach

S moothness term Energy Minimization Approach

S moothness term C orrespondence term D epth term B oundary term

S moothness term Energy Minimization Approach

S moothness term Energy Minimization Approach

C orrespondence term Energy Minimization Approach

C orrespondence term Energy Minimization Approach

D epth term Energy Minimization Approach Front ViewTop View

D epth term Energy Minimization Approach Front ViewTop View

B oundary term Without BoundaryWith Boundary Energy Minimization Approach

B oundary term Without BoundaryWith Boundary Energy Minimization Approach

Minimize energy equation

Experimental Results We captured RGBD video sequences of shirts and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios. Four experiments were conducted: 1)Illustrating the contribution of the depth term 2)Illustrating the contribution of the boundary term 3)Partial self-occlusion 4)Textureless shirt sequence

Experimental Results Illustrating the contribution of the depth term

Experimental Results Illustrating the contribution of the boundary term

Experimental Results Partial self-occlusion

Experimental Results Textureless shirt sequence

Experimental Results Video

Conclusion  We have presented an algorithm to estimate the 3D configuration of a highly non-rigid object through a video sequence using feature point correspondence, depth, and boundary information.  We plan to extend this research to handle a two-sided 3D triangular mesh that covers both the front and the back of the object.

Questions?