Natural Cost functions for contact selection Paul R. Schrater University of Minnesota Collaborators: Erik Schlicht, Erik Flister, Charles Sloane

Task-dependent Perception Importance of visual information varies with task Potential state space of every natural scene is huge.

Example: Information for Distance How far the ball on the right from you? Which ball is closer to you? dede dbdb dtdt Which ball is closer to the table? Questions like these can involve different state spaces and optimal inference strategies (Schrater & Kersten, 2000)

Information for Inter-object distances Which ball is closer to the table?

Example: Information for Contact

By Construction Only reliable features are Shadows

Example: Information for surface smoothness Difference Image Which Ball is smoother? Dominant Information Specularities

Task-dependent Perception Importance of visual information varies with task, even within the same scene. Potential state space of every natural scene is huge. Computational resources limited in the brain. How does the brain determine how to allocate perceptual resources? Need a theory of goal-oriented perception Proposal : The brain turns perceptual and motor goals into loss functions to filter information relevant to the current task

Goal of a task can be encoded in a loss function Perceptual tasks- –Actions = judgments –Typical goal- minimize error in perceptual judgments –Loss functions Error rate (discrete) Estimation error (continuous) Motor control tasks- –Actions=Movement of body parts –Typical goal- achieve desired body position/motion with little effort. –Loss functions Sequential control decisions Cost-to-go to achieve goal position

Loss functions as Information Filters Risk = Expected Loss Taylor series in L around the expected state Plugging into the Risk Assume posterior distribution is approximately Gaussian After simplifying

Loss functions as Information Filters Each state variable contributes differentially to the risk: Thus, the impact of loss on the i th component of the state vector is to re-weight its reliability by the curvature of the loss in that direction.

Loss function for contact selection If relevant information is filtered by task goals, it should work for some non-trivial case: Where should you place your fingers when grasping an object? ? ? Depends on movement goal lift, rotate, flip, push, slide Seems like a visual task (Can’t wait till contact to decide) but has motor goal. Grasp quality depends on non- visual factors. (Mass, COM, Intertia, Friction, Geometry w.r.t. COM, etc) Why Non-Trivial

Perception for contact selection State variables needed: –Vision does well: Object boundary Object configuration w.r.t. the body Surface material properties of the object Collision time/velocity –Vision does poorly or not at all Mass Center of mass location Body inertia Friction open question - Is vision simply performing recognition of previously felt objects, or is it also making inferences based on assumptions- –mass uniformity+material inference? –Shininess => slippery

Goal of contact- Manipulation The goal of object contact is typically to manipulate the object. Therefore the information relevant to determining contact should be derived from that goal. Thus, we will try to connect manipulation with contact. Proposal: Brain has internal models for object manipulation. –Forward kinematic model to predict object motion given finger motions –Need inverse dynamic model for computing controls required to produce object motion and maintain contact. –Need observer model that can supply feedback Contact state Object state Finger state

Internal models for object motion Key ideas: –Once an object is grasped, it is effectively part of the hand. –The motor system has learned internal models for manipulation (maintaining contact and applying appropriate forces) Internal model idea well supported in motor control –(e.g. Neilson et al, 1985; Ghahramani & Wolpert, 1997, etc.) Evidence for internal models for contact- –Eye fixations directed to intended grasp locations (Johansson et. al, 2001) –Plan for object shape (Santello & Soechting, 2000; Jenmalm & Johansson, 1997) –Plan for grip force for familiar object weight (Johansson & Westling, Gordon et. al, Witney et. al, 2001, etc ) –Plan for center of mass –Plan for familiar friction (Burstedt et al, 2000)

Contact as a Kinematic Linkage Contact without slippage acts Like a 5-dof joint with transformations Given link parameters, object velocity determined by finger & contact velocities

Kinematic Grasp Constraint -> Dynamics For details, see Murray, Li, Sastry (1994) –Equations for the dynamics of the object can be written down using the Pfaffian kinematic constraints Standard robotics equation But defined on object variables and transformed by adding the object via the virtual linkage

Planning for Object Motion The goal of the task is to move the object. Proposal: Cost for desired object motion induces contact selection

Optimal Stochastic Control Model for Perception and Action Generative World ModelGenerative Perception Model Actions Beliefs Goals L( x t,u k ) P( x t |Data) ykyk Sensory Data

Stochastic Optimal Control (Abstract) Noise Model System Model LossOptimal Cost-to-go Control law Posterior distrib. on state Control lossGoal loss Typical Loss Function

Loss function for Object Motion Given: Goal states for object: Min control cost on motion, both without and with object Write Cost-to-go to penalize desired object via-states and control cost where Rewrite in terms of three control models: pre-,mid-,post-contact

Loss function for Object Motion After rewriting, the cost-to-go explicitly incorporates contact locations (and potentially velocities). Contact locations are determined by finger positions, and finger positions by controls Thus, in principle, minimizing cost-to-go with contact model would allow automatic online selection of contact conditions. Alternatively, contact points could be pre-picked (estimated )by partially optimizing the cost function.

Preliminary Tests More Torque Less Torque More Torque Less Torque Predictions: Optimal finger positions vary with cylinder orientation Contact velocities will produce impulses in desired direction of motion.

Qualitative Difference Between Touch and Lift 0 Degree: Touch Y X Z Gravity Touch

Y X Z Gravity 0 Degree: Lift Qualitative Difference Between Touch and Lift Lift

Preliminary testing of the idea

How optimal are observed contacts? Compute the cost function and compare to human behavior. –Full model too hard- Cheated by assuming object motion under min control would take a minimum Jerk path on average. Computed forces required to lift object 5cm in 500msec along min jerk path over a dense set of finger contact locations Simulated a contact dynamics model (frictional rolling without slip). Used kinematic constraints and required object forces to compute the finger controls needed to accomplish the task. Cost could then be computed on the set of finger controls.

Contact Cost for touch task

Object motion risk

Object Jerk Cost 180 mm 40 mm

Human vs. Loss function prediction Purple: Touch contact positions Black: Lift contact positions

Conclusions- Speculations Goals come first- perception should be driven by satisfying task demands Contact selection is a difficult problem (or maybe I made it into one). Perceptual principle of least commitment- don’t commit to an estimate when a distribution will do. New role for vision- Contact model estimation? What are the goals of natural conscious perception?