1. Unified Cognitive Science Neurobiology Psychology Computer Science Linguistics Philosophy Social Sciences Experience Take all the Findings and Constraints Seriously

2. What are schemas? Regularities in our perceptual, motor and cognitive systems Structure our experiences and interactions with the world. May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience)

3. Basis of Image schemas Perceptual systems Motor routines Social Cognition Image Schema properties depend on Neural circuits Interactions with the world

4. Spatial schemas TR/LM relation Boundaries, bounded region Topological relations Orientational Axes Proximal/Distal

5. Trajector/Landmark Schema Roles: Trajector (TR) – object being located Landmark (LM) – reference object TR and LM may share a location (at)

6. TR/LM -- asymmetry The cup is on the table ?The table is under the cup. The skateboard is next to the post. ?The post is next to the skateboard.

7. Topological Relations Separation

8. Topological Relations Separation Contact

9. Topological Relations Separation Contact Coincidence: - Overlap - Inclusion

10. Orientation Vertical axis -- up/down up down above below upright

11. Orientation Horizontal plane – Two axes:

12. Language and Frames of Reference There seem to be three prototypical frames of reference in language (Levinson) Intrinsic Relative Absolute

13. Intrinsic frame of reference front back right left

14. Relative frame of reference front back left?? right??

15. Absolute frame of reference north west south east

16. semantic schema Container roles: interior exterior portal boundary Representing image schemas Interior Exterior Boundary Portal Source Path Goal Trajector These are abstractions over sensorimotor experiences. semantic schema Source-Path-Goal roles: source path goal trajector

17. Language and Spatial Schemas People say that they look up to some people, but look down on others because those we deem worthy of respect are somehow “above” us, and those we deem unworthy are somehow “beneath” us. But why does respect run along a vertical axis (or any spatial axis, for that matter)? Much of our language is rich with such spatial talk. Concrete actions such as a push or a lift clearly imply a vertical or horizontal motion, but so too can more abstract concepts. Metaphors: Arguments can go “back and forth,” and hopes can get “too high.”

18. Regier Model Lecture Jerome A. Feldman March 4, 2008 With help from Matt Gedigian

19. Neural Theory of Language

20. Language Development in Children 0-3 mo: prefers sounds in native language 3-6 mo: imitation of vowel sounds only 6-8 mo: babbling in consonant-vowel segments 8-10 mo: word comprehension, starts to lose sensitivity to consonants outside native language 12-13 mo: word production (naming) 16-20 mo: word combinations, relational words (verbs, adj.) 24-36 mo: grammaticization, inflectional morphology 3 years – adulthood: vocab. growth, sentence-level grammar for discourse purposes

23. Trajector/Landmark Schema Roles: Trajector (TR) – object being located Landmark (LM) – reference object TR and LM may share a location (at)

24. TR/LM -- asymmetry The cup is on the table ?The table is under the cup. The skateboard is next to the post. ?The post is next to the skateboard.

26. Language and Frames of Reference There seem to be three prototypical frames of reference in language (Levinson) Intrinsic Relative Absolute

27. English ‘on’ 1. The computer is on the desk 2. The picture is on the wall 3. The projector is on the ceiling LM TR DN UP TR/LM, verticality, contact, support LM TR TR/LM, contact, attaching force LM TR TR/LM, contact, attaching force

29. Basis of Image Schemas Perceptual systems Motor routines Social Cognition Image Schema properties depend on Neural circuits Interactions with the world

30. Image schemas Trajector / Landmark (asymmetric) The bike is near the house ? The house is near the bike Boundary / Bounded Region bounded region has a closed boundary Topological Relations Separation, Contact, Overlap, Inclusion, Surround Orientation Vertical (up/down), Horizontal Absolute (E, S, W, N) LM TR bounded region boundary

31. Spatial schemas TR/LM relation Boundaries, bounded region Topological relations Orientational Axes Proximal/Distal

32. Regier’s Model Training input: configuration of TR/LM and the correct spatial relation term Learned behavior: input TR/LM, output spatial relation Learning System abovebelowleftrightinoutonoff Input: TR LM above

34. Issue #1: Implicit Negatives Children usually do not get explicit negatives But we won’t know when to stop generalizing if we don’t have negative evidence Yet spatial relation terms aren’t entirely mutually exclusive The same scene can often be described with two or more spatial relation terms (e.g. above and outside) How can we make the learning problem realistic yet learnable?

38. Dealing with Implicit Negatives Explicit positive for above Implicit negatives for below, left, right, etc in Regier: E = ½ ∑ i,p (( t i,p – o i,p ) * β i,p ) 2, where i is the node, p is the pattern, β i,p = 1 if explicit positive, β i,p < 1 if implicit negative

41. above – positive examples

42. above – negative examples

43. above – after training

44. above – test examples

47. Learning System dynamic relations (e.g. into) structured connectionist network (based on visual system)

57. Issue #2: Shift Invariance Backprop cannot handle shift invariance (it cannot generalize from 0011, 0110 to 1100) But the cup is on the table whether you see it right in the center or from the corner of your eyes (i.e. in different areas of the retina map) What structure can we utilize to make the input shift-invariant?

59. Topological Relations Separation Contact Coincidence: - Overlap - Inclusion - Encircle/surround

64. Limitations Scale Uniqueness/Plausibility Grammar Abstract Concepts Inference Representation

65. Demo of the Regier System on the English above

66. Language and Thought We know thought (our cognitive processes) constrains the way we learn and use language Does language also influence thought? Benjamin Whorf argues yes Psycholinguistics experiments have shown that linguistics categories influence thinking even in non-linguistics task Language Thought cognitive processes

67. Image schemas Trajector / Landmark (asymmetric) The bike is near the house ? The house is near the bike Boundary / Bounded Region a bounded region has a closed boundary Topological Relations Separation, Contact, Overlap, Inclusion, Surround Orientation Vertical (up/down), Horizontal (left/right, front/back) Absolute (E, S, W, N) LM TR bounded region boundary

68. More image schemas Proximal / Distal distance from center (near/far) Part / Whole top of the hill, cover of the magazine Container interior, exterior, boundary, portal Source-Path-Goal source, path, goal, trajector Force-Dynamics support, force SG P TR

69. English ‘on’ 1. The computer is on the desk 2. The picture is on the wall 3. The projector is on the ceiling LM TR DN UP TR/LM, verticality, contact, support LM TR TR/LM, contact, attaching force LM TR TR/LM, contact, attaching force

70. Language Development in Children 0-3 mo: prefers sounds in native language 3-6 mo: imitation of vowel sounds only 6-8 mo: babbling in consonant-vowel segments 8-10 mo: word comprehension, starts to lose sensitivity to consonants outside native language 12-13 mo: word production (naming) 16-20 mo: word combinations, relational words (verbs, adj.) 24-36 mo: grammaticization, inflectional morphology 3 years – adulthood: vocab. growth, sentence-level grammar for discourse purposes

71. 1;2.29don’t throw the bear. 1;10.11don’t throw them on the ground. 1;11.3Nomi don’t throw the books down. … what do you throw it into? what did you throw it into? 1;11.9they’re throwing this in here. throwing the thing. throwing the frisbee. … do you throw the frisbee? do you throw it? Development of throw 1;8.0 throw throw off 1;10.28 I throwded it. (= I fell) I throwded. (= I fell) 1;11.3 I throw it. I throw it ice. (= I throw the ice) throwing in. throwing. throw it. I throw it. … she’s throwing the frisbee. 1;11.12they’re throwing a ball. 2;0.3don’t throw it Nomi. Nomi stop throwing. well you really shouldn’t throw things Nomi you know. remember how we told you you shouldn’t throw things. throw frisbee. throwing ball. 2;0.2I throwed it. can I throw it? I throwed Georgie. could I throw that? 2;0.5throw it? you throw that? … 2;0.18gonna throw that? 2;1.17 throw it in the garbage. throw in there. 2;5.0 throw it in that. 2;11.12 I throwed it in the diaper pail.

72. Regier’s Model Training input: configuration of TR/LM and the correct spatial relation term Learned behavior: input TR/LM, output spatial relation Learning System abovebelowleftrightinoutonoff Input: TR LM above

73. Learning System We’ll look at the details next lecture dynamic relations (e.g. into) structured connectionist network (based on visual system)