University of Pennsylvania 1 GRASP Control of Multiple Autonomous Robot Systems Vijay Kumar Camillo Taylor Aveek Das Guilherme Pereira John Spletzer GRASP Laboratory

University of Pennsylvania 2 GRASP Multiple Autonomous Robots Hybrid Systems Approach to Robot Software l modes as behaviors l composition of modes Cooperative Control of Multiple Robots l cooperative manipulation l formation control l tracking l pursuit Human interaction l visualization

University of Pennsylvania 3 GRASP Vision for Multi-Robot Teams l Mobile platforms for deploying cameras into an environment l The case for cameras t Small t Cheap t Passive t Low Power l Uses for imagery t Visualization of remote environments t Obtaining information about targets w Position, level of activity etc. t Basis for convenient human robot interfaces

University of Pennsylvania 4 GRASP Visualization of Remote Environments l Registered omnidirectional images can be used to visualize remote scenes

University of Pennsylvania 5 GRASP Visualizing the scene l Scene can be interactively explored and/or revisited with a new camera trajectory specified by the user

University of Pennsylvania 6 GRASP GRASP Laboratory

University of Pennsylvania 7 GRASP View Synthesis with Quasi-Sparse Correspondences Dense correspondences can be difficult to obtain due to..  Occluded regions  Homogenous image regions l Strategy t Focus on accurately reproducing the motion of edges in the scene t Use interpolation to estimate the motion of the other points l Basis for visualization in MARS 2020

University of Pennsylvania 8 GRASP Novel view movies

University of Pennsylvania 9 GRASP Freespace Reasoning l We can reason about the structure of space by considering the union of the freespace volumes induced by a collection of triangulated disparity maps.

University of Pennsylvania 10 GRASP Results with 3D reasoning

University of Pennsylvania 11 GRASP Multi-Eyed Stereo Systems l Locations of targets and objects in the environment can be deduced from image measurements acquired by the robots l The robot team can effectively be viewed as a multi eyed stereo system

University of Pennsylvania 12 GRASP Sensor Planning and Control l Interesting property of these robot teams, estimates for various parameters of interest are obtained by combining measurements from multiple, distributed sensors l We could choose to view our team as a multi-eyed stereo system where the eyes can be moved l Question t Given that the sensor platforms are mobile, how should they be deployed in order to produce the best estimates?

University of Pennsylvania 13 GRASP C R denotes the robot configuration space, and  is an element of C R and denotes an element of this configuration space C W denotes the feature configuration space, and  is an element of C W and denotes an element of this configuration space denotes the measurements obtained by the robot team 22 [x t, y t ] T [x 2, y 2,  2 ] T 11 [x 1, y 1,  1 ] T x y   = [ x 1, y 1,  1, x 2, y 2,  2 ] T  = [ x t, y t ] T = [  1,  2 ] T Theoretical Framework

University of Pennsylvania 14 GRASP Given this terminology, one can define a quality function: which reflects the expected error in estimating the feature state from a given robot configuration  Objective: : is a function that provides an estimate of the feature state given the robots configuration and the sensor measurements : is a function that returns the expected error between the estimate returned by Est and the actual feature state for a particular robot configuration,  This will depend upon our model of sensor errors P(  ) : is a probability density function on C W Optimization Problem

University of Pennsylvania 15 GRASP The optimization problem of minimizing Q(  ) is typically difficult to solve analytically Particle Filtering approach: Approximate P(  ) by a set (  j,  j ), where  j is a single sample from C W, and  j a weight reflecting the probability of  j representing the state  The integral can then be approximated by a tractable summation. Computational Approach The resulting function is typically piecewise continuous in  and can be optimized using standard techniques

University of Pennsylvania 16 GRASP Implementation Example Target Position (  ) {  i,  i } Particle Set Estimated Particle Positions Particle Disparities

University of Pennsylvania 17 GRASP Integrating Sensing and Control Sense Predict Particle Filter Q Optimization Sensor Planner {  i,  i } Piggyback on particle filtering approach for sensing to obtain the particle set {  i,  i } Framework offers a complementary relationship between sensing and control

University of Pennsylvania 18 GRASP Tracking Targets

University of Pennsylvania 19 GRASP Tracking Targets cont’d

University of Pennsylvania 20 GRASP Handling Obstacles

University of Pennsylvania 21 GRASP Technology Transfer and Transition Robot hardware and software l University of Colorado l Oklahoma State University l Georgia Tech Evolution Robotics l Jim Ostrowski DoD Programs l MARS Teams (2020)