1. cse@buffalo S.C. Shapiro Development of a Cognitive Agent Stuart C. Shapiro Department of Computer Science and Engineering and Center for Cognitive Science State University of New York at Buffalo shapiro@cse.buffalo.edu

2. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

3. cse@buffalo S.C. Shapiro Goal A computational cognitive agent that can: –Understand and communicate in English; –Discuss specific, generic, and “rule-like” information; –Reason; –Discuss acts and plans; –Sense; –Act; –Remember and report what it has sensed and done.

4. cse@buffalo S.C. Shapiro Embodied Cassie A computational cognitive agent –Embodied in hardware –or Software-Simulated –Based on SNePS and GLAIR.

5. cse@buffalo S.C. Shapiro SNePS Knowledge Representation and Reasoning –Propositions as Terms SNIP: SNePS Inference Package –Bi-Directional Inference SNeBR: SNePS Belief Revision SNeRE: SNePS Rational Engine Interface Languages –SNePSUL: Lisp-Like –SNePSLOG: Logic-Like –GATN for Fragments of English.

6. cse@buffalo S.C. Shapiro GLAIR Architecture Knowledge Level Perceptuo-Motor Level Sensory-Actuator Level NL Vision Sonar Motion Proprioception Grounded Layered Architecture with Integrated Reasoning SNePS

7. cse@buffalo S.C. Shapiro Interaction with Cassie English (Statement, Question, Command) (Current) Set of Beliefs [SNePS] (Updated) Set of Beliefs [SNePS] Actions [SNeRE] (New Belief) [SNePS] English sentence expressing new belief answering question reporting actions Answer [SNIP] GATN Parser GATN Generator Reasoning Clarification Dialogue Looking in World Reasoning

8. cse@buffalo S.C. Shapiro Cassie, the FEVAHR

9. cse@buffalo S.C. Shapiro Cassie in the Lab

10. cse@buffalo S.C. Shapiro Garnet Simulation World

11. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

12. cse@buffalo S.C. Shapiro Entities, Terms, Symbols, Objects Cassie’s mental entity: a person named Bill SNePS term: B5 Object in world:

13. cse@buffalo S.C. Shapiro Intensional Representation Intensional entities are distinct even if coreferential. “The morning star is the evening star.” “George IV wondered if Scott was the author of Waverly.”

14. cse@buffalo S.C. Shapiro McCarthy’s Telephone Number Problem Mary's telephone number is Mike's telephone number. I understand that Mike's telephone number is Mary's telephone number. Pat knew Mike's telephone number. I understand that Pat knew Mike's telephone number. Pat dialed Mike's telephone number. I understand that Pat dialed Mike's telephone number.

15. cse@buffalo S.C. Shapiro Answering the Telephone Number Problem Did Pat dial Mary's telephone number? Yes, Pat dialed Mary's telephone number. Did Pat know Mary's telephone number? I don't know.

16. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

17. cse@buffalo S.C. Shapiro Logic for NLU & Commonsense Reasoning Either Pat is a man or Pat is a woman or Pat is a robot. I understand that Pat is a robot or Pat is a woman or Pat is a man. Pat is a woman. I understand that Pat is a woman. What is Pat? Pat is a woman and Pat is not a robot and Pat is not a man.

18. cse@buffalo S.C. Shapiro Representation in FOPL? Man(Pat)  Woman(Pat)  Robot(Pat) but don’t want inclusive or Man(Pat) Woman(Pat) Robot(Pat) + + T T T F T So don’t want exclusive or either

19. cse@buffalo S.C. Shapiro andor andor(i, j){P i,..., P n } True iff at least i, and at most j of the P i are True

20. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

21. cse@buffalo S.C. Shapiro Discussing Propositions That Bill is sweet is Mary's favorite proposition. I understand that Mary's favorite proposition is that Bill is sweet. Mike believes Mary's favorite proposition. I understand that Mike believes that Bill is sweet.

22. cse@buffalo S.C. Shapiro Discussing Sentences Hamlet said `To be or not to be'. I understand that Hamlet said `To be or not To be'. Polonius said `Neither a borrower nor a lender be'. I understand that Polonius said `Neither a borrower nor a lender be'. Who said `To be or not to be'? Hamlet said `To be or not To be'.

23. cse@buffalo S.C. Shapiro Representing Propositions and Sentences Propositions and sentences must be First-class entities of the domain Represented by terms

24. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

25. cse@buffalo S.C. Shapiro Symbol Grounding by Perception and Action Agent can demonstrate understanding –Of properties and descriptions By locating correct objects in the world –Of verbs By acting properly in the world

26. cse@buffalo S.C. Shapiro Symbol-Grounding 1 Go to the green robot.

27. cse@buffalo S.C. Shapiro Go to the green robot. I went to the green robot. I am near the green robot. Symbol-Grounding 2

28. cse@buffalo S.C. Shapiro Alignment Terms at GLAIR KR level Aligned with Descriptions and procedures At GLAIR PM level

29. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

30. cse@buffalo S.C. Shapiro Representation and Use of Indexicals Words whose meanings are determined by occasion of use E.g. I, you, now, then, here, there Deictic Center *I : SNePS term representing Cassie *YOU: person Cassie is talking with *NOW: current time.

31. cse@buffalo S.C. Shapiro Analysis of Indexicals (in input) First person pronouns: *YOU Second person pronouns: *I “here”: location of *YOU.

32. cse@buffalo S.C. Shapiro Generation of Indexicals *I : First person pronouns *YOU : Second person pronouns *NOW : used to determine tense.

33. cse@buffalo S.C. Shapiro Come here. Use of Indexicals 1

34. cse@buffalo S.C. Shapiro Come here. I came to you, Stu. I am near you. Use of Indexicals 2

35. cse@buffalo S.C. Shapiro Who am I? Your name is ‘Stu’ and you are a person. Who have you talked to? I am talking to you. Talk to Bill. I am talking to you, Bill. Come here. Use of Indexicals 3

36. cse@buffalo S.C. Shapiro Come here. I found you. I am looking at you. Use of Indexicals 4

37. cse@buffalo S.C. Shapiro Come here. I came to you. I am near you. I found you. I am looking at you. Use of Indexicals 5

38. cse@buffalo S.C. Shapiro Who am I? I talked to Stu and I am talking to you. Your name is ‘Bill’ and you are a person. Who are you? I am the FEVAHR and my name is ‘Cassie’. Who have you talked to? Use of Indexicals 6

39. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

40. cse@buffalo S.C. Shapiro A Personal Sense of Time *NOW contains SNePS term representing current time. *NOW moves when Cassie acts.

41. cse@buffalo S.C. Shapiro B6 Representation of Time find lex action object B1 ! agent act event time NOW !! beforeafterbeforeafter ????????????? I

42. cse@buffalo S.C. Shapiro Movement of Time t1 t2! beforeafter t3! beforeafter NOW

43. cse@buffalo S.C. Shapiro Performing a Punctual Act t1 t3! beforeafter NOW t2! beforeafter ! time event

44. cse@buffalo S.C. Shapiro Performing a Durative Act t1 NOW ! beforeafter t2 ! time event NOW t3 ! supint subint

45. cse@buffalo S.C. Shapiro Keeping Track of Time 1 Who are you looking at? I am looking at you, Stu. Come here.

46. cse@buffalo S.C. Shapiro Keeping Track of Time 2 Who are you looking at? I am looking at you. Come here. I came to you. I am near you. Who are you looking at? I am looking at you.

47. cse@buffalo S.C. Shapiro Keeping Track of Time 3 Find Bill. I found Bill. I am looking at Bill. Who are you looking at? I looked at you and I am looking at Bill. Who are you talking to? I am talking to you.

48. cse@buffalo S.C. Shapiro Keeping Track of Time 4 Follow a red robot. I found a red robot. I am looking at a red robot.

49. cse@buffalo S.C. Shapiro Keeping Track of Time 5 I went to a red robot. I am near a red robot. I am following a red robot. Follow a red robot. I found a red robot. I am looking at a red robot.

50. cse@buffalo S.C. Shapiro Keeping Track of Time 6 Who are you talking to? I am talking to you. Who am I? Your name is ‘Stu’ and you are a person. Stop. I stopped.

51. cse@buffalo S.C. Shapiro Keeping Track of Time 7 Who are you looking at? I looked at you and I looked at Bill and I looked at a red robot. Who are you following? I followed a red robot. Who are you talking to? I am talking to you.

52. cse@buffalo S.C. Shapiro Outline Introduction Intensional Representation Logic for NLU and Commonsense Reasoning Discussing Propositions and Sentences Symbol-Grounding by Perception and Action Representation and Use of Indexicals A Personal Sense of Time Summary

53. cse@buffalo S.C. Shapiro Goal A computational cognitive agent/robot That can communicate in natural language.

54. cse@buffalo S.C. Shapiro Intensional Representation SNePS terms represent mental entities. May assert that two entities are coreferential. Relations/acts may be declared transparent.

55. cse@buffalo S.C. Shapiro Logic for NLU and Commonsense Reasoning Designed logical connectives and rules of inference More appropriate than in standard FOPC.

56. cse@buffalo S.C. Shapiro Discussing Propositions and Sentences Propositions and sentences are first-class entities.

57. cse@buffalo S.C. Shapiro Symbol-Grounding by Perception and Action Use of GLAIR architecture to connect entities with descriptions/functions used by sensors and effectors.

58. cse@buffalo S.C. Shapiro Representation and Use of Indexicals Use of Deictic Center for parser to interpret indexicals as current referents And for generator to generate indexicals from current referents.

59. cse@buffalo S.C. Shapiro A Personal Sense of Time *NOW is current time. Updated when Cassie acts.

60. cse@buffalo S.C. Shapiro For More Information Personnel Manual Tutorial Bibliography ftp’able SNePS source code etc. http://www.cse.buffalo.edu/sneps/