Chapter 10. The Explorer System in Cognitive Systems, Christensen et al. Course: Robots Learning from Humans On, Kyoung-Woon Biointelligence Laboratory School of Computer Science and Engineering Seoul National University

Contents Introduction System Overview Navigation SA Object SA Place SA Conceptual Mapping SA Spatial Modeling and Reasoning Map Acquisition Acquiring the Conceptual Map Cross-Modal Spatial Knowledge Sharing Planning Scenario: Find Object Conclusion © 2011, SNU CSE Biointelligence Lab.,

Introduction Chapter 10. The Explore System © 2011, SNU CSE Biointelligence Lab.,

Introduction PROBLEM: Modeling space, Acting in this space and Reasoning about it. SETTING: Robot moving around in an initially Unknown, Large scale, environment Inhabited by humans. MOTIVATION: To study the problems that occur when an intelligent robot must interact with humans in a rich and complex environment. © 2011, SNU CSE Biointelligence Lab.,

Introduction © 2011, SNU CSE Biointelligence Lab., Construction of Spatial models from sensor data Simultaneous Localization And Mapping

Introduction Spatial model: Sensors input + Human input Two main modes : Robot explores space together with a user in a home tour fashion. Human Augmented Mapping paradigm. Fully autonomous exploration where the robot moves with the purpose of covering space. Ability to perform tasks autonomously : Performing a particular task to perfection << Acting within a flexible framework Investigate two problems within this context: What information must be exchanged by different parts of the system to make this possible How the current state of the world should be represented during such exchanges. © 2011, SNU CSE Biointelligence Lab.,

System Overview Chapter 10. The Explore System © 2011, SNU CSE Biointelligence Lab.,

System Overview © 2011, SNU CSE Biointelligence Lab.,

System Overview © 2011, SNU CSE Biointelligence Lab.,

System Overview Navigation SA The metric map (produced by the SLAM) The navigation graph (NavGraphProcess) Topological map (NavGraphProcess) Detecting and Tracking People Motion Control Object SA The view planning component creates a plan for which nodes to visit. The visual search can be performed using a pan-tilt- zoom camera where an attention mechanism gradually guides the robot to zoom in closer. © 2011, SNU CSE Biointelligence Lab.,

System Overview Place SA Assign nodes and areas to one of predefined semantic place categories (e.g. an office, a corridor etc.) The results for both sensors are integrated by a cue integration component, which provides beliefs about a place category for the current viewpoint. The area categorization provides important information when reasoning about space. Conceptual Mapping SA Maintains a symbolic representation of space suitable for situated action and interaction. It represents spatial areas, objects in the environment, and abstract properties of persons. © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning Chapter 10. The Explore System © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Map Acquisition Human Augmented Mapping Guided tour scenario. User walks up to the robot and initiates the mapping process with a command like “follow me!”. The robot continuously tracks the position of the user and follows them through the environment. As the robot moves, the spatial model is built from the sensor data. Interaction is not master/slave-like. User can query the spatial knowledge of the robot throughout the HAM sessions. © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Map Acquisition Autonomous Exploration Explorer uses a frontier-based strategy [18] for autonomous exploration: FREE, OCCUPIED, UNKNOWN (Frontier: boarder between FREE & UNKNOWN space) Exploration is considered complete when there are no more reachable frontiers. Because the way the navigation graph is built requires the robot to move, exploration is not only about having the sensor see all parts of the room, but the robot needs to move there as well © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Acquiring the Conceptual Map System comes with a rich conceptual ontology taxonomies of indoor area types, of commonly found objects, and of different spatio-topological relations between area instances and object instances. the system is started with a blank map, the A-Box of the reasoner is empty. it will contain area, person and object instances, and their relations, as contained in the loaded map. Infer new or more specific knowledge based on partial information By combining and reasoning over instances and their relations, the reasoned can infer more specific concepts for those instances. Make the information inside the Conceptual Mapping SA available to other sub-architectures. © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Cross-Modal Spatial Knowledge Sharing Current Spatial Context A robot proxy representing the physical robot itself An area proxy for the area the robot is currently in A position relation connecting the robot proxy with its area A person proxy for each person currently being tracked by the people tracking module area proxies for each person position relation proxies between the above An object proxy for each object belonging to an Area that is being represented position proxies connecting each object and its corresponding Area Close relation proxies between the robot and persons that are near to it © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Cross-Modal Spatial Knowledge Sharing The Robot Proxy exactly one robot proxy on the binder. The only feature of this proxy is its Concept: robot. Area Proxies The navigation subsystem divides space into areas, based on door nodes the sole binding feature AreaID Object Proxies feature Concept, describing the particular class of object that it belongs to – book or mug. The feature can be used both for binding and for executing a plan. © 2011, SNU CSE Biointelligence Lab.,

Spatial Modeling and Reasoning : Cross-Modal Spatial Knowledge Sharing Person Proxies Concept: always set to person Location: last observed metric position PersonID: unique identifier Position Relation Proxies Label: always set to position OtherSourceID: set to the ID of the navigation sub-architecture TemporalFrame: PERCEIVED for directly perceived entities; ASSERTED for others Closeness Relation Proxies Label: always set to close OtherSourceID: set to the ID of the navigation sub-architecture; negated TemporalFrame: always PERCEIVED © 2011, SNU CSE Biointelligence Lab.,

Planning Chapter 10. The Explore System © 2011, SNU CSE Biointelligence Lab.,