Scene Based Reasoning Cognitive Architecture Frank Bergmann, Brian Fenton, Reasoning Scenes Planner Plan Recognition Plan Execution Persistent Plans 3D Reconstruction Persistent Plans Attention Subsystem Persistent Plans Episodic Memory

(cc) Frank Bergmann, Brian Fenton, 2 Problems Addressed  We propose an integrated architecture for implementing a number of “self-models”.  We provide a model for talking about modalities (auxiliary verbs) without the use of higher-order or modal logics. Approach  Use Thomas Metzinger’s “Self-Model Theory of Subjectivity” as a kind of requirement statement.  Extend/generalize existing knowledge representation and reasoning formalisms to accommodate self-models. Implementation Status  Plays “Towers of Hanoi”  Working on a “Stone age prey/hunter” sandbox  Problems Addressed & Approach Reasoning Scenes Planner Plan Recognition Plan Execution Persistent Plans 3D Reconstruction Persistent Plans Attention Subsystem Persistent Plans Episodic Memory

(cc) Frank Bergmann, Brian Fenton, 3  Physical Volume Self-Model“This is the room I’m in, this is my leg.“ I dentify the 3D volume of the robot representing SBR and model its capabilities  Capabilities Self-Model„I am good at this.“ P erformance statistics of task decompositions  Planning Self-Model„I usually hit the goal in 30% of all cases.“ P erformance statistics of plans performed  Intention Self-Model”I currently try to do this.“ Introspection into the goals currently pursued  Goal Self-Model“I would like to be to do this.“ Introspection into active “Persistent Goals”  Social Self-Model“Other group members respect me.“ Role of self in group activities.  Behavioral Self-Model“I usually react like this.“ Episodic Memory recordings of past SBR actions  Emotional Self-Model“I am happy to hear the news.” Introspection into current emotions and historic model of emotions  Historical Self-Model“I used to do a lot of this.” Episodic Memory recorded SBR past actions.  Terminological Self-Model“I know that a penguin is not a bird.” Export the Description Logic TBox into a 2d scene diagram.  Flow of Mental Events Self-Model“I just had an idea how to solve plan X” Short-term memory of “mental events” Different Levels of Self-Models Physical Planning Social Internal

(cc) Frank Bergmann, Brian Fenton, 4 Scene Based Reasoning “Core” Architecture Reasoning Scenes Planner Plan Recognition Plan Execution Persistent Plans 3D Reconstruction Persistent Plans Attention Subsystem Persistent Plans Episodic Memory  20+ subsystems  Explained at

(cc) Frank Bergmann, Brian Fenton, 5 SBR „Core “ Reasoning „Scenes“ Planner „Scene“ „Plan“  We use “Scenes” as a unified representation of internal and world states and as “situations” for the planner. – 3D Scene Graph level – suitable for 3D reconstruction and low-level planning. Reasoning using physics simulation. Reasoning using “diagram reasoning”. – Semantic Network level – “1st Mind”, provides “concepts” and “roles” as an abstraction from object and their attributes. Reasoning using Description Logic reasoning – 2D Graph level – suitable to represent 2D maps, meta-representations of internal data- structure. – “Situation” level – Scenes act as world states and internal states to the planner, adding “dynamics” Eat Dinner Start Goal... Get pizza from Fridge Clean Up Eat Obtain Food Table1 Human1... Type: Table Color: green Position:... Size: 1 x 1... Type: Human Gender: male Position: in- front -of Task/ Action Scene Scene Based Reasoning “Core” Architecture

(cc) Frank Bergmann, Brian Fenton, 6 SBR „Core “ Reasoning „Scenes“ Planner „Scene“ „Plan“  We use “Scenes” as a unified representation of internal and world states and as “situations” for the planner.  Reasoning is performed by several specialized subsystems. – We use Description Logic as a “First Mind” convenient “knowledge assembler“ to classify objects and their attributes into concepts and roles, but not for “higher-level” reasoning. – 2D graph reasoning is a special type of planning. – “Two Minds”: Planning together with the attention, motivational and “persistent goal” subsystems forms a 2nd reasoning system in addition to DL Eat Dinner Start Goal... Get pizza from Fridge Clean Up Eat Obtain Food Table1 Human1... Type: Table Color: green Position:... Size: 1 x 1... Type: Human Gender: male Position: in- front -of Task/ Action Scene Scene Based Reasoning “Core” Architecture 2 2

(cc) Frank Bergmann, Brian Fenton, 7 SBR „Core “ Reasoning „Scenes“ Planner „Scene“ „Plan“  We use “Scenes” as a unified representation of internal and world states and as “situations” for the planner.  Reasoning is performed by several specialized subsystems.  The „Planner“ uses Scenes as situations and goals. – “Task decompositions” (HTNs) provide a middle ground between STRIPS style planning and procedural execution and are easy to learn. – Actions may have multiple deterministic outcomes – statistics about actions are collected in the episodic memory. – Physics simulations cover a range of planning capabilities that are difficult to handle using FOL and derivates – A “simulation subsystem” allows for what-if analysis of plans Eat Dinner Start Goal... Get pizza from Fridge Clean Up Eat Obtain Food Table1 Human1... Type: Table Color: green Position:... Size: 1 x 1... Type: Human Gender: male Position: in- front -of Task/ Action Scene Scene Based Reasoning “Core” Architecture

(cc) Frank Bergmann, Brian Fenton, 8  Attention Subsystem controls sensor focus  Persistent Plans for setting plan priorities  Episodic Memory stores plans indexed by content  Plan Recognition allows for 1-shot learning  20+ subsystems defined at Reasoning Scenes Planner Plan Recognition Plan Execution Persistent Plans 3D Reconstruction Persistent Plans Attention Subsystem Persistent Goal Hierarchy Episodic Memory Architecture Summary and “Two Minds”

(cc) Frank Bergmann, Brian Fenton, 9  Physical Volume Self-Model“This is my leg.“ I dentify the 3D volume of the robot representing SBR and model its capabilities  Capabilities Self-Model„I am good at this.“ P erformance statistics of task decompositions  Planning Self-Model„I usually hit the goal in 30% of all cases.“ P erformance statistics of plans performed  Intention Self-Model”I currently try to do this.“ Introspection into the goals currently pursued  Goal Self-Model“I would like to be to do this.“ Introspection into active “Persistent Goals”  Social Self-Model“Other group members respect me.“ Role of self in group activities.  Behavioral Self-Model“I usually react like this.“ Episodic Memory recordings of past SBR actions  Emotional Self-Model“I am happy to hear the news.” Introspection into current emotions and historic model of emotions  Historical Self-Model“I used to do a lot of this.” Episodic Memory recorded SBR past actions.  Terminological Self-Model“I know that a penguin is not a bird.” Export the Description Logic TBox into a 2d scene diagram.  Flow of “Thoughts” Self-Model“I was surprised by this event” Short-term memory (“time-line”) of “mental events” Self-Models and Self-Referentiality

(cc) Frank Bergmann, Brian Fenton, 10 Examples of Scenes representing Self-Models: Plan & DL-TBox  Modalities examples: – „My goal is...“ – „I believe that...“  Explicit representation of data-structures  No need to use modal or higher order logic  Apply the same reasoning engine as to object-level data Dinner Start Goal... Get pizza from Fridge Clean Up Eat Obtain Food... Bird Canary Penguin Animal Fish  has skin  eats  breathes  moves  !can fly  can swim  can sing  is yellow  has wings  can fly  has feathers  has fins  can swim  has gills

(cc) Frank Bergmann, Brian Fenton, 11  Physical Volume Self-Model“This is my leg.“ I dentify the 3D volume of the robot representing SBR and model its capabilities  Capabilities Self-Model„I am good at this.“ P erformance statistics of task decompositions  Planning Self-Model„I usually hit the goal in 30% of all cases.“ P erformance statistics of plans performed  Intention Self-Model”I currently try to do this.“ Introspection into the goals currently pursued  Goal Self-Model“I would like to be to do this.“ Introspection into active “Persistent Goals”  Social Self-Model“Other group members respect me.“ Role of self in group activities.  Behavioral Self-Model“I usually react like this.“ Episodic Memory recordings of past SBR actions  Emotional Self-Model“I am happy to hear the news.” Introspection into current emotions and historic model of emotions  Historical Self-Model“I used to do a lot of this.” Episodic Memory recorded SBR past actions.  Terminological Self-Model“I know that a penguin is not a bird.” Export the Description Logic TBox into a 2d scene diagram.  Flow of “Thoughts” Self-Model“I was surprised by this event” Short-term memory (“time-line”) of “mental events” Self-Models and Self-Referentiality Physical Planning Social Internal