1. Qualitative Spatial Reasoning in Interpreting Text and Narrative Ernest Davis New York University COSIT 2011

2. Thesis Simple natural language texts and narratives often raise problems in commonsense spatial knowledge and reasoning of surprising logical complexity and geometric richness.

3. Outline Examples: 5 literary texts, 6 (or 9) contrived texts –Fitting things into containers –Blocking, pursuing, escaping, and hiding –Household objects Summary Morals for research

4. Let Stalk Strine Afferbeck Lauder Usage of aorta Aorta have more busses. An aorta mikem smaller so they dont take up half the road. An aorta put more seats innem so you doan tefter stann all the time. An aorta have more room innem --- you carn tardly move innem air so crowded. Aorta do something about it.

5. Unusual logical form Forward inference to notice contradiction. Infer: The speaker is a fool. (Common in actual narratives, but unusual in inference systems).

6. Simple representation AreaOf(Bus) = NumberOfSeatsIn(Bus) * AreaOf(Seat) Infer: Area(Seat) NumberOfSeatsIn(Bus) Area(Bus)

7. More general and deeper argument The region occupied by a bus is (to simplify) the union of the regions occupied by the seats. Two seats on a bus occupy disjoint regions. The area of a collection of disjoint regions is the sum of the area of each region.

8. Infer If each seat in bus b1 corresponds to a seat in bus b2; each seat in b1 is smaller than the corresponding seat in b2; and there are more seats in b2 than b1 then b2 must be bigger than b1.

9. Expressive Representation. Given: Bus(b) PlaceOf(b) = s SeatsOf(b) PlaceOf(s). Bus(b) s1 s2 s1,s2 SeatsOf(b) DR(PlaceOf(s1),PlaceOf(s2)). CollectionOfRegions(c) [ r1,r2 c r1 r2 DR(r1,r2)] AreaOf( r c r) = r c AreaOf(r).

10. Infer: Bus(b1) Bus(b2) Injection(m,SeatsOf(b1),SeatsOf(b2)) [ s SeatsOf(b1) AreaOf(PlaceOf(s)) < AreaOf(PlaceOf(Apply(m,s)))] AreaOf(PlaceOf(b2)) > AreaOf(PlaceOf(b1)).

11. Disclaimer The logic is just a convenient notation. I am not saying that a reasoning program has to use a logical representation. I am saying that the reasoning here will pretty much have to carry out this inference.

12. Infer: Bus(b1) Bus(b2) Injection(m,SeatsOf(b1),SeatsOf(b2)) [ s SeatsOf(b1) AreaOf(PlaceOf(s)) < AreaOf(PlaceOf(Apply(m,s)))] AreaOf(PlaceOf(b2)) > AreaOf(PlaceOf(b1)). Representational issues: Sets of regions. Interaction of measure (Area) and mereology (DR, Union).

13. The Winograd Schema Challenge Hector Levesque Commonsense 2011 Winograd schema: A pair of sentences, differing in one or two words, with an ambiguity that is resolved oppositely. Winograd (1972): The city councilmen refused the demonstrators a permit because they feared violence vs.… because they advocated violence.

14. Winograd Schema Challenge Collecting corpus of schemas that are effortlessly disambiguated by human readers sound natural. cant be solved using selectional restrictions are not easily Googlable. Challenge for AI. Less far-reaching than the Turing Test, but less problematic.

15. The Winograd Schema Challenge Hector Levesque Commonsense 2011 The trophy does not fit in the suitcase because it is too large. The trophy does not fit in the suitcase because it is too small. Gloss The trophy is too large to fit as The trophy does not fit and no larger object fits, but some smaller object fits

16. Rule out incorrect interpretation for disambiguation Incorrect reading: The suitcase is too large. FitsIn(Trophy,Suitcase) [ s1 s1 Suitcase FitsIn(Trophy,s1)] [ s2 s2Suitcase FitsIn(Trophy,s2)]. Infer that this is impossible.

17. Representational issues FitsIn(Trophy,Suitcase) [ s1 s1 Suitcase FitsIn(Trophy,s1)] [ s2 s2Suitcase FitsIn(Trophy,s2)]. Interaction of subset with fit. Existential quantification over regions.

18. Books in boxes Each of my books fits in this box, but they do not all fit in the box. If I had more boxes like this, I could fit them all. a,c Arrangement(a,c) [ x c FeasibleShape(PlaceOf(x,a),x)] [ x1,x2 c x1x2 DR(PlaceOf(x1,a),PlaceOf(x2,a))]

19. M=My books. X = the inside of the box. b M a Arrangement(a,{b}) PlaceOf(b,a)X. a Arrangement(a, M) b M PlaceOf(b,a)X c,a [ y c Congruent(y,X)] Arrangement(a,M) b M y c PlaceOf(b,a)y.

20. Representational issues c,a [ y c Congruent(y,X)] RigidArrangement(a,M) b M y c PlaceOf(b,a)y. Arrangement of a collection of objects. 3-level quantifier alternation. (A natural formulation in terms of the actual boxes has a 5-level quantifier alternation.)

21. A Representational Challenge The equipment came out of the box, but now I cant fit it into the box. A common experience. Geometrically impossible. Geometric aspects. A reasoner that knows more about geometry than people might interpret it as the box.

22. Dog out of the Kitchen I tried to keep the dog out of the kitchen by putting a chair in the middle of the doorway, but it was too wide. What was too wide? Why in the middle? How did the dog get into the kitchen? Spatio-temporal reasoning.

23. The chair blocks the dog h Fixed(h,Frame,PosF) Fixed(h,Chair,PosC) Feasible(h) DR(PlaceOf(Dog,Start(h)),Kitchen) DR(PlaceOf(Dog,End(h)),Kitchen) Feasible(h) x,y,s Solid(x) Solid(y) xy State(s,h) DR(PlaceOf(x,s),PlaceOf(y,s))

24. No placement of the chair blocks the dog [ pc h Fixed(h,Frame,PosF) Fixed(h,Chair,pc) Feasible(h) DR(PlaceOf(Dog,Start(h)),Kitchen) DR(PlaceOf(Dog,End(h)),Kitchen)]

25. Central placement I put the chair in the center of the doorframe, because I know that, for any chair, if any placement blocks the dog, then there is a central placement that blocks the dog.

26. Representational issues Spatio-temporal reasoning. Alternating quantification over histories and object positions.

27. The Tale of Benjamin Bunny Beatrix Potter

30. Spatial Inferences for Narrative Coherence The cat does not initially see the rabbits, because they are behind her head. The cat cannot see the rabbits under the basket. No light can penetrate under the basket. The rabbits cannot get out without lifting the basket, which would disturb the cat.

31. A Man and his Dog Thomas Mann Many dogs, they say, are the death of a hare, a single dog cannot achieve it, even one much speedier and more enduring than Bashan. The hare can double and Bashan cannot --- and that is all there is to it. … The hare gives a quick, easy, almost malicious twitch at right angles to the course and Bashan shoots past from his rear … Before he can stop, turn around, and get going in the other direction, the hare has gained so much ground that it is out of sight.

33. Inferences Before the double, Bashan and the hare are running in the same direction. Bashan is close behind the hare (a second or two) The distance of the double is at least comparable to the width of Bashan, but not large enough that jumping forward that distance would allow the hare to escape.

34. The dog is faster than the hare, so could catch it if the hare didnt double. The hare can change velocity discontinuously. The dog has a limit on the rate of deceleration. A pack of dogs can catch the hare by surrounding it, so doubling does not help. The reader can visualize the scenario.

35. Representational issues Differential games Properties of the second derivative.

36. Household Objects Buttons and Cord No end of hard problems. Hard to separate physics from geometry The physics is often not easy.

37. Buttons I tried to put the button through the hole, but it was too [large/small]. I tried to put the button through the hole, but it had been sewn too tightly to the coat. I forgot that the top button was fastened, so when I took off the coat, it tore off. * I forgot that the top button was unfastened …

38. David Copperfield, Charles Dickens [Peggotty] gave [my head] a good squeeze. I know it was a good squeeze because, being very plump, whenever she made any little exertion after she was dressed, some of the buttons on the back of her gown flew off. And I recollect two bursting to the opposite side of the parlour while she was hugging me.

39. Inferences The gown is too tight. The burst buttons are behind Peggotty and in front of David. The gown will now gap in back until the buttons are fixed.

40. Difficulties Characterize the position of a fastened button – doable. Characterizing the motion involved in fastening a button – difficult. (Too small refers to the action, not to the static state.) Sufficient characterization of the physics of fabric (The gown is too tight) – difficult.

41. Cord The power cord on the laptop would not reach from the desk to the outlet, so I got an extension cord. Then my wife objected to having an electric cord across the center of the living room (Men!), so I laid it around the edge of the room, and hid it behind the furniture, but of course I had to get a much longer cord. Note the disambiguation of it, and the time sequence of laid and had to get.

42. Representational Issues Cord(c) FeasibleShape(r,c) LengthOf(AxisOf(r)) = CordLength(c) p FeasibleShape(p,C1) ^ Through(p,LivingRoom) ^ At(End1(p),Table) ^ At(End2(p),Outlet) Reasoning about arc lengths Quantifying over paths

43. Death of Bill Sykes Oliver Twist, Charles Dickens [H]e set his foot against the stack of chimneys, fastened one end of the rope tightly and firmly around it, and with the other made a strong running noose … He could let himself down by the cord to within a less distance of the ground than his own height, and had his knife ready in his hand to cut it then and drop. At the very instant when he brought the loop over his head previous to slipping it beneath his arm- pits … he lost his balance …

45. Inferences What is the cause of Sykes death? What would have happened if he had fallen Before attaching the rope to the chimneys? Before slipping the rope over his head? After getting the rope under his armpits? If the rope had been much shorter? 10 feet longer? If things had gone right, what would have been the final state of the rope?

46. Summary:Text as a source for qualitative reasoning problems Advantages: Natural, interesting Undeniable qualitative information Wide range of –Spatial relations –Logical forms –Directions of inference Less susceptible to Use simulation! and Use ML! than other applications

47. Disadvantages Finding or contriving texts takes work. Does not yield a well-defined class of problems. Some important spatial characteristics e.g. shape are entirely implicit. Hard to separate spatial from other issues Often hard to be sure of logical form Difficult to connect logical form to text

48. Morals for research WARNING! The following should not be taken as career advice, particularly if you are pursuing: A degree Tenure Funding (at least US)

49. Morals for research QSR research based on close reading of one or few texts. Look at small, logically complex, heterogeneous problems. We dont know how to evaluate these, empirically or theoretically. Table the issue of evaluation.

50. More morals Study common, geometrically complex, relations e.g. Blocks(c,s,x,k) The collection of objects c as placed in state s blocks object x from entering region k. Problem: These appear in many different variants. Problem: What inferences? Study common higher-level operators: Too φ.

51. More morals Get help from linguists and cognitive psychologists. Perhaps attempt a taxonomy of the QSR problems in text.

52. Final remark At this point, automated commonsense reasoning is high-risk, high-payoff, like fusion reactors or SETI. More fun than either of those, though.