_ Daniela Doroftei Royal Military School Av. De la Renaissance 30, B1000 Brussels, Belgium Contribution to ViewFinder – WP2

 The goal of this research project is to prepare the RobuDem robot for an outdoor crisis management task. 2

 To achieve this goal, the robot must be able to:  Be tele-operated by a remote user  Ensure its own safety by avoiding obstacles detected by its sensors (sonar, stereo, …)  Navigate autonomously in an unknown environment by mapping the surroundings  Detect chemical contamination  Navigate to pre-defined goal positions  Execute complex tasks like searching for human victims  Here we will focus on the design of the control architecture for such a robot 3

 RobuDem-platform:  Outdoor all-terrain robot  4 driving / steering wheels allowing for different drive modes  Supports heavy loads (up to 300kg) 4

 Ultrasound Sensors for obstacle detection  Joystick for remote control  Chemical sensor with integrated temperature sensor 5

 Differential GPS (and wheel encoders) for positioning 6

 Quad-cam vision system consisting of a Bumble- bee stereo vision system and 2 digital cameras. 7

 Sensors: : Green rounded rectangle  Processors: : Violet rectangle : Gold rectangle : Blue rectangle  Actuators: : Black oval SteerRobudem Sensors BehaviorProcessors VisualProcessors Positioning & Mapping 8

SteerRobudem Joystick  Transmission of joystick commands to the robot: FilterCommands  Transmission of filtered commands to the robot: 9

SteerRobudem Joystick FilterCommands Obey Joystick Behavior Controller  In semi-autonomous control, there is no longer a direct link between joystick and robot actuators:  Instead, there is a Behavior-Based Controller managing the commands sent to the robot  The “Obey Joystick” behavior is only 1 of the multiple behaviors contributing to the global robot control strategy 10

These commands are fused by a behavior-based controller, taking into account both behaviors to come to a globally optimal and consistent command to be sent to the robot The joystick returns the user commands, telling the robot where to go. The “ObeyJoystick” behavior calculates the best action to perform in order to follow the users’ commands. ObeyJoystick The on-board sonars deliver information about obstacles in the robots’ path. The “AvoidObstaclesUsingSonar” behavior calculates the best action to perform in order not to bump into the detected obstacles. FilterCommands SteerRobudem FuseBehaviors AvoidObstaclesUsingSonar SonarJoystick These commands are filtered and sent to the robot 11

 General requirements:  The robot needs accurate positioning  The robot needs to build up a model (map) of its environment to reason with this data  Robot Safety:  The robot needs to avoid areas with excessive heat  The robot needs to detect chemicals and avoid areas which are too contaminated  The robot needs to detect obstacles using its sonar sensors and its stereo vision system and avoid these obstacles  The robot needs to avoid all previously detected obstacles stored in the environmental model (map) 12

 Robot Goals:  The robot must be tele-operable  The robot needs to detect chemicals and find the source of contamination  The robot should maximize the knowledge about the environment it is put in.  The robot needs to search for human victims on the disaster site  The robot needs to be able to execute a user-defined trajectory, given through a set of waypoints  In the event of a loss of network connection, the robot should be able to return to the base station 13

These 2 positioning estimates are fused a first time to come to a more accurate and robust position estimate A Visual Simultaneous Localization and Mapping module analyses all this input data. It builds up an environmental model (map) and places the robot accurately on this map. Data from the rotation of the wheels, measured by wheel encoders, is used to estimate the position VisualSLAM CameraFramegrabberGISMap GPSOdometry PositionEstimation A real-time differential GPS system provides absolute positioning data by acquiring signals from at least 4 space-based satellites If available, GIS (Geographic Information System) data is used to initialize the maps The environ- ment is observed by a camera, which detects and tracks features in the environment to estimate the robot motion and to update the map Environmental model 6-dimensional position estimate 14

 A Visual SLAM module delivers robot position and a map  A chemical sensor instantaneously measures contaminant concentrations  A temperature sensor measures the temperature  This data is used to build up 2 maps containing the chemical and heat distribution  Using this map, 2 behaviors “AvoidChemicals” and “AvoidHotZones” try to steer the robot away from danger zones VisualSLAM AvoidChemicalsAvoidHotZones LocalHeatMap ChemicalSensorTempe-rature LocalChemicalMap 15

 The robot is controlled using a behavior-based controller which sends steering commands.  To generate these commands, the controller aims to fuse a number of objectives / tasks / requirements:  General requirements:  The robot needs accurate positioning  The robot needs to build up a model (map) of its environment to reason with this data VisualSLAM PositionEstimation CameraFramegrabber GISMap GPS Odometry FilterCommands SteerRobudem FuseBehaviors 16

 Robot Safety:  The robot needs to avoid areas with excessive heat  The robot needs to detect chemicals and avoid areas which are too contaminated FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature 17

 Robot Safety:  The robot needs to avoid all previously detected obstacles stored in the environmental model (map) FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM 18

 Robot Safety:  The robot needs to detect obstacles using its sonar sensors and avoid these obstacles FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar 19

 Robot Safety:  The robot needs to detect obstacles using its stereo vision system and avoid these obstacles FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber 20

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick  Robot Goals:  The robot must be tele-operable 21

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals  Robot Goals:  The robot needs to detect chemicals and find the source of contamination 22

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals  Robot Goals:  The robot should maximize the knowledge about the environment it is put in MaximizeTerrainKnowledge 23

 Humans are searched for in each of the 4 camera images  A “Search Humans” behavior directs the robot in the persons’ direction FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals  Robot Goals:  The robot needs to search for human victims MaximizeTerrainKnowledge PersonDetector PersonDetector PersonDetector PersonDetector CameraFramegrabber SearchHumans 24

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals MaximizeTerrainKnowledge PersonDetector PersonDetector PersonDetector PersonDetector CameraFramegrabber SearchHumans GoToGoals GlobalPathPlanner GoalAssigner 25

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals MaximizeTerrainKnowledge PersonDetector PersonDetector PersonDetector PersonDetector CameraFramegrabber SearchHumans GoToGoals GlobalPathPlanner GoalAssigner ReturnToBase ConnectionChecker 26

FilterCommands SteerRobudem FuseBehaviors VisualSLAM PositionEstimation AvoidChemicals AvoidHotZones LocalHeatMap LocalChemicalMap ChemicalSensor CameraFramegrabber GISMap GPS Odometry Tempe-rature AvoidObstaclesUsingSLAM AvoidObstaclesUsingSonar Sonar AvoidObstaclesUsingStereo StereoFramegrabber ObeyJoystick Joystick GoToChemicals MaximizeTerrainKnowledge PersonDetector PersonDetector PersonDetector PersonDetector CameraFramegrabber SearchHumans GoToGoals GlobalPathPlanner GoalAssigner ReturnToBase ConnectionChecker TaskAssigner 27

FilterCommands SteerRobudem FuseBehaviors AvoidObstaclesUsingSLAM GoToChemicals VisualSLAM PositionEstimation AvoidObstaclesUsingSonar AvoidObstaclesUsingStereo PersonDetector SearchHumans AvoidChemicals AvoidHotZones GoToGoals ReturnToBase ObeyJoystick GlobalPathPlanner LocalHeatMap LocalChemicalMap PersonDetector PersonDetector PersonDetector ChemicalSensor CameraFramegrabber CameraFramegrabber StereoFramegrabber Sonar Joystick TaskAssigner ConnectionChecker GISMap GPS Odometry Tempe-rature GoalAssigner MaximizeTerrainKnowledge 28

 Using this modular behavior based control framework the robot can:  Be tele-operated by a remote user  Ensure its own safety by avoiding obstacles detected by its sensors (sonar, stereo, …)  Navigate autonomously in an unknown environment by mapping the surroundings  Detect chemical contamination  Navigate to pre-defined goal positions  Execute complex tasks like searching for human victims 29

30

FilterCommands SteerRobudem FuseBehaviors AvoidObstaclesUsingSLAM GoToChemicals VisualSLAM PositionEstimation AvoidObstaclesUsingSonar AvoidObstaclesUsingStereo PersonDetector SearchHumans AvoidChemicals AvoidHotZones GoToGoals ReturnToBase ObeyJoystick GlobalPathPlanner LocalHeatMap LocalChemicalMap PersonDetector PersonDetector PersonDetector ChemicalSensor CameraFramegrabber CameraFramegrabber StereoFramegrabber Sonar Joystick TaskAssigner ConnectionChecker GISMap GPS Odometry Tempe-rature GoalAssigner MaximizeTerrainKnowledge 31