S.C. Shapiro The SNePS Approach to Cognitive Robotics Stuart C. Shapiro Department of Computer Science and Engineering and Center for Cognitive Science University at Buffalo

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Goal A computational cognitive agent that can: –Understand and communicate in English; –Discuss specific, generic, and “rule-like” information; –Reason; –Discuss acts and plans; –Sense; –Act; –Remember and report what it has sensed and done.

S.C. Shapiro Embodied Cassie A computational cognitive agent –Embodied in hardware –or Software-Simulated –Based on SNePS and GLAIR.

S.C. Shapiro SNePS Knowledge Representation and Reasoning –Propositions as Terms SNIP: SNePS Inference Package –Specialized connectives and quantifiers SNeBR: SNePS Belief Revision SNeRE: SNePS Rational Engine Interface Languages –SNePSUL: Lisp-Like –SNePSLOG: Logic-Like –GATN for Fragments of English.

S.C. Shapiro GLAIR Architecture Knowledge Level Perceptuo-Motor Level Sensory-Actuator Level NL ActionPercepts Grounded Layered Architecture with Integrated Reasoning SNePS

S.C. Shapiro Interaction with Cassie English (Statement, Question, Command) (Current) Set of Beliefs [SNePS] (Updated) Set of Beliefs [SNePS] Actions [SNeRE] (New Belief) [SNePS] English sentence expressing new belief answering question reporting actions Answer [SNIP] GATN Parser GATN Generator Reasoning Clarification Dialogue Looking in World Reasoning

S.C. Shapiro Cassie, the BlocksWorld Robot

S.C. Shapiro Cassie, the FEVAHR

S.C. Shapiro FEVAHR/Cassie in the Lab

S.C. Shapiro FEVAHRWorld Simulation

S.C. Shapiro UXO Remediation Cassie Cassie Corner flag NonUXO object Corner flag UXO Battery meter Corner flag Drop-off zone Field Safe zone Recharging Station

S.C. Shapiro Crystal Space Environment

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Entities, Terms, Symbols, Objects Cassie’s mental entity: a person named Bill SNePS term: B5 Object in world:

S.C. Shapiro Intensional Representation Intensional entities are distinct even if coreferential. “The morning star is the evening star.” “George IV wondered if Scott was the author of Waverly.”

S.C. Shapiro McCarthy’s Telephone Number Problem Mary's telephone number is Mike's telephone number. I understand that Mike's telephone number is Mary's telephone number. Pat knew Mike's telephone number. I understand that Pat knew Mike's telephone number. Pat dialed Mike's telephone number. I understand that Pat dialed Mike's telephone number.

S.C. Shapiro Answering the Telephone Number Problem Did Pat dial Mary's telephone number? Yes, Pat dialed Mary's telephone number. Did Pat know Mary's telephone number? I don't know.

S.C. Shapiro Representing Propositions Propositions must be first-class entities of the domain Represented by terms.

S.C. Shapiro Discussing Propositions That Bill is sweet is Mary's favorite proposition. I understand that Mary's favorite proposition is that Bill is sweet. Mike believes Mary's favorite proposition. I understand that Mike believes that Bill is sweet.

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Logic for NLU & Commonsense Reasoning Either Pat is a man or Pat is a woman or Pat is a robot. I understand that Pat is a robot or Pat is a woman or Pat is a man. Pat is a woman. I understand that Pat is a woman. What is Pat? Pat is a woman and Pat is not a robot and Pat is not a man.

S.C. Shapiro Representation in FOPL? Man(Pat)  Woman(Pat)  Robot(Pat)

S.C. Shapiro Representation in FOPL? Man(Pat)  Woman(Pat)  Robot(Pat) but don’t want inclusive or

S.C. Shapiro Representation in FOPL? Man(Pat)  Woman(Pat)  Robot(Pat) but don’t want inclusive or Man(Pat) Woman(Pat) Robot(Pat) + + T T T F T So don’t want exclusive or either

S.C. Shapiro andor andor(i, j){P 1,..., P n } True iff at least i, and at most j of the P i are True

S.C. Shapiro thresh thresh(i, j){P 1,..., P n } True iff either fewer than i, or more than j of the P i are True Note: thresh(i, j)  ~andor(i, j)

S.C. Shapiro or-entailment {P 1,..., P n } v=> {Q 1,..., Q n } True iff for all i, jP i  Q j

S.C. Shapiro and-entailment {P 1,..., P n } &=> {Q 1,..., Q n } True iff for all j P 1 & … & P n  Q j

S.C. Shapiro Numerical entailment {P 1,..., P n } i=> {Q 1,..., Q n } True iff for all j andor(i, n){P 1, …, P n }  Q j

S.C. Shapiro Universal Quantifier all(ū)({R 1 (ū),..., R n (ū)} &=> {C 1 (ū),..., C m (ū)}) Every ā that satisfies R 1 (ū)&…& R n (ū) also satisfies C 1 (ū),..., C m (ū)})

S.C. Shapiro Numerical Quantifiers nexists(i,j,k)(x) ({P 1 (x),..., P n (x)}: {Q(x)})} There are k individuals that satisfy P 1 (x) ...  P n (x) and, of them, at least i and at most j also satisfy Q(x)

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro MENTAL ACTS Believe(proposition) Disbelieve(proposition)

S.C. Shapiro Act Selection Do-One({act1... actn}) Snif(if(condition, act),... if(condition, act) [else(act)])

S.C. Shapiro Act Iteration Do-All({act1... actn}) Sniterate(if(condition, act),... if(condition, act), [else(act)]) Snsequence(act1,..., actn) Cascade(act1,..., actn) P-Do-All({act1,..., act2})

S.C. Shapiro Entity Iteration WithSome(var, suchthat, do, [else]) WithAll(var, suchthat, do, [else]) WithSome+(var, suchthat, do, [else]) WithNew(vars, thatare, suchthat, do, [else])

S.C. Shapiro Proposition/Act Transformers Achieve(proposition) ActPlan(act, plan) GoalPlan(proposition, act) Precondition(act, proposition) Effect(act, proposition) WhenDo(proposition, act) WheneverDo(proposition, act) IfDo(proposition, act)

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Conditional Plans If a block is on a support then a plan to achieve that the support is clear is to pick up the block and then put the block on the table. all(x, y) ({Block(x), Support(y), On(x, y)} &=> {GoalPlan(Clear(y), Snsequence(Pickup(x), Put(x, Table)))}) (STRIPS-like representation: No times)

S.C. Shapiro Use of Conditional Plan GoalPlan(Clear(B), Snsequence(Pickup(A), Put(A, Table))) Remember (cache) derived propositions.

S.C. Shapiro Use of Conditional Plan GoalPlan(Clear(B), Snsequence(Pickup(A), Put(A, Table))) ??? SNeBR to the rescue!

S.C. Shapiro A FEVAHR Acting Rule all(a, o) ({Agent(a), Thing(o)} &=> {Precondition(Follow(a, o), Near(a, o)), GoalPlan(Near(a, o), Goto(a, o)), Precondition(Goto(a, o), Lookat(a, o)), ActPlan(Lookat(a, o), Find(a, o))}) Uses a temporal model.

S.C. Shapiro Acting According to the Rule

S.C. Shapiro Acting According to the Rule I found a red robot. I am looking at a red robot. Follow a red robot.

S.C. Shapiro Acting According to the Rule I went to a red robot. I am near a red robot. I am following a red robot. I found a red robot. I am looking at a red robot. Follow a red robot.

S.C. Shapiro A Plan for Blowing up UXOs all(a)(Agent(a) => ActPlan(Blowup(a, UXOs), Act(a, Cascade(SearchforUxo(a), WithSome+(obj, Near(a, obj), WithNew({ch ex}, {Charge(ch), Explosion(ex)}, Possess(a, ch), Cascade(Place(a, ch, obj), Hide(a), Waitfor(a, ex), SearchforUxo(a))), goto(a, SafeZone))))))

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Representation and Use of Indexicals Words whose meanings are determined by occasion of use E.g. I, you, now, then, here, there Deictic Center *I : SNePS term representing Cassie *YOU: person Cassie is talking with *NOW: current time.

S.C. Shapiro Analysis of Indexicals (in input) First person pronouns: *YOU Second person pronouns: *I “here”: location of *YOU Present/Past relative to *NOW.

S.C. Shapiro Generation of Indexicals *I : First person pronouns *YOU : Second person pronouns *NOW : used to determine tense and aspect.

S.C. Shapiro Come here. Use of Indexicals 1

S.C. Shapiro Come here. I came to you, Stu. I am near you. Use of Indexicals 2

S.C. Shapiro Who am I? Your name is ‘Stu’ and you are a person. Who have you talked to? I am talking to you. Talk to Bill. I am talking to you, Bill. Come here. Use of Indexicals 3

S.C. Shapiro Come here. I found you. I am looking at you. Use of Indexicals 4

S.C. Shapiro Come here. I came to you. I am near you. I found you. I am looking at you. Use of Indexicals 5

S.C. Shapiro Who am I? I talked to Stu and I am talking to you. Your name is ‘Bill’ and you are a person. Who are you? I am the FEVAHR and my name is ‘Cassie’. Who have you talked to? Use of Indexicals 6

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro A Personal Sense of Time *NOW contains SNePS term representing current time. *NOW moves when Cassie acts or perceives a change of state.

S.C. Shapiro B6 Representation of Time find lex action object B1 ! agent act event time NOW !! beforeafterbeforeafter ????????????? I

S.C. Shapiro Movement of Time t1 t2! beforeafter t3! beforeafter NOW

S.C. Shapiro Performing a Punctual Act t1 t3! beforeafter NOW t2! beforeafter ! time event

S.C. Shapiro Performing a Durative Act t1 NOW ! beforeafter t2 ! time event NOW t3 ! supint subint

S.C. Shapiro The Pacemaker PML process periodically increments variable COUNT. *COUNT = some PML integer. Reset to 0 when NOW moves. Provides bodily “feel” of passing time.

S.C. Shapiro Quantizing Time Cannot conceptualize fine distinctions in time intervals. So quantize, e.g. into half orders of magnitude (Hobbs, 2000).

S.C. Shapiro Movement of Time with Pacemaker NOW COUNTn hom 0 KL PML t1 t2 q ! beforeafter time duration !

S.C. Shapiro The Problem of the Fleeting Now How can you reason about “now” if it never stands still?

S.C. Shapiro Fleeting Now Example 1 12:15:00: “Is John having lunch now?” 12:15:02: Agent walks to John’s office. 12:17:00: Agent sees John at his desk, eating. 12:19:00: Agent reports “yes”. Appropriate granularity.

S.C. Shapiro Fleeting Now Example 2 12:15:00: “Is John having lunch now?” Agent knows John is at home without a phone. Agent contemplates driving to John’s home. Don’t bother---inappropriate granularity.

S.C. Shapiro The Vagueness of “now” I’m now giving a talk. I’m now supervising PhD students. I’m now visiting Paris. I’m now living in Buffalo. The agent is now walking to John’s office. The agent is now seeing if John is eating lunch. Multiple now’s at different granularities.

S.C. Shapiro NOW-MTF NOW Semi-lattice of times, all of which contain *NOW, any of which could be meant by “now” Finite---only conceptualized times of conceptualized states Maximal Temporal Frame based on *NOW

S.C. Shapiro Moving NOW with MTF NOW

S.C. Shapiro Typical Durations “If the walk light is on now, cross the street.” Relevant duration is typical duration of walk lights. “Is John having lunch now?” Relevant duration is typical duration of lunch. Use quantized typical durations when updating NOW-MTFs.

S.C. Shapiro Using Appropriate Granularity NOW Lunch time Lunch? Lunch! Yes!

S.C. Shapiro Outline Introduction Intensional Representation & Propositions SNePS Connectives and Quantifiers SNeRE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary

S.C. Shapiro Goal A computational cognitive agent/robot That can communicate in natural language.

S.C. Shapiro Intensional Representation & Propositions SNePS terms represent mental entities. May assert that two entities are coreferential. Relations/acts may be declared transparent. Propositions are first-class entities.

S.C. Shapiro SNePS Connectives and Quantifiers Designed logical connectives and rules of inference More appropriate for NLU and Commonsense reasoning than in standard FOPC.

S.C. Shapiro SNeRE Acting Constructs Separate, but Coordinated Syntax and Semantics For Acting and for Reasoning

S.C. Shapiro Representation and Use of Indexicals Use of Deictic Center for parser to interpret indexicals as current referents And for generator to generate indexicals from current referents.

S.C. Shapiro A Personal Sense of Time *NOW is current time. Updated when Cassie acts or perceives a change of state. Points into MTF to support vagueness of “now”.

S.C. Shapiro For More Information Personnel Manual Tutorial Bibliography ftp’able SNePS source code etc.