1. Practical Applications of Temporal and Event Reasoning James Pustejovsky, Brandeis Graham Katz, Osnabrück Rob Gaizauskas, Sheffield ESSLLI 2003 Vienna, Austria August 25-29, 2003

2. Course Outline Monday- –Theoretical and Computational Motivations –Overview of Annotation Task –Events and Temporal Expressions Tuesday –Anchoring Events to Times –Relations between Events Wednesday –Syntax of TimeML Tags –Semantic Interpretations of TimeML –Relating Annotations –Temporal Closure Thursday –Automatic Identification of Expressions –TimeBank Corpus –TANGO –Automatic Link Construction Friday- –Outstanding Problems

3. Thursday Topics –Automatic Identification of Expressions –TimeBank Corpus –TANGO –Automatic Link Construction –Developmental Complexity Models of Narratives

4. Annotating the Corpus Distinguishing features of TimeML are: It builds on TIMEX2 (Ferro et al., 2001), but introduces new features such as temporal functions to allow intensionally specified expressions like three years ago. It identifies signals determining interpretation of temporal expressions, such as temporal prepositions (for, during) and temporal connectives (before, after). It identifies a wide range of classes of event expressions, such as tensed verbs (has left), stative adjectives (sunken), and event nominals (merger). It creates dependencies between events and times or other events, such as anchoring (John left on Monday), ordering (The party happened after midnight), and embedding (John said Mary left).

5. The Conceptual and Linguistic Basis TimeML presupposes the following temporal entities and relations. Events are taken to be situations that occur or happen, punctual or lasting for a period of time. They are generally expressed by means of tensed or untensed verbs, nominalisations, adjectives, predicative clauses, or prepositional phrases. Times may be either points, intervals, or durations. They may be referred to by fully specified or underspecified temporal expressions, or intensionally specified expressions. Relations can hold between events and events and times. They can be temporal, subordinate, or aspectual relations.

6. Annotating Events Events are marked up by annotating a representative of the event expression, usually the head of the verb phrase. The attributes of events are a unique identifier, the event class, tense, and aspect. Fully annotated example: All 75 passengers died See full TimeML spec for handling of events conveyed by nominalisations or stative adjectives.

7. Annotating Times Annotation of times designed to be as compatible with TIMEX2 time expression annotation guidelines as possible. Fully annotated example for a straightforward time expression: July 1966 Additional attributes are used to, e.g. anchor relative time expressions and supply functions for computing absolute time values (last week).

8. Annotating Signals The SIGNAL tag is used to annotate sections of text, typically function words, that indicate how temporal objects are to be related to each other. Also used to mark polarity indicators such as not, no, none, etc., as well as indicators of temporal quantification such as twice, three times, and so forth. Signals have only one attribute, a unique identifier. Fully annotated example: Two days the attack …

9. Annotating Relations (1) To annotate the different types of relations that can hold between events and events and times, the LINK tag has been introduced. There are three types of LINKs: TLINKs, SLINKs, and ALINKs, each of which has temporal implications. A TLINK or Temporal Link represents the temporal relationship holding between events or between an event and a time. It establishes a link between the involved entities making explicit whether their relationship is: before, after, includes, is_included, holds, simultaneous, immediately after, immediately before, identity, begins, ends, begun by, ended by.

10. Annotating Relations (2) An SLINK or Subordination Link is used for contexts introducing relations between two events, or an event and a signal. –SLINKs are of one of the following sorts: Modal, Factive, Counter-factive, Evidential, Negative evidential, Negative. An ALINK or Aspectual Link represents the relationship between an aspectual event and its argument event. –The aspectual relations encoded are: initiation, culmination, termination, continuation.

11. Annotating Relations (2) Annotated examples: TLINK: John taught on Monday SLINK: John said he taught ALINK: John started to read

12. The Corpus: Text Sources The 300 texts in the TIMEBANK corpus were chosen to cover a wide variety of media sources from the news domain: –DUC (TIPSTER) texts from the Document Understanding Conference corpus cover areas like biography, single and multiple events (for example dealing with news about earthquakes and Iraq). This covers 12% of the corpus; –Texts from the Automatic Content Extraction (ACE) program come from transcribed broadcast news (ABC, CNN, PRI, VOA) and newswire (AP, NYT). These comprise 17% and 16% of the corpus, respectively. –Propbank (Treebank2) texts are Wall Street Journal newswire texts, making up 55% of the corpus.

13. The Annotation Effort The annotation of each document involves: –an automatic pre-processing step in which some of the events and temporal, modal and negative signals are tagged; –a human annotation step which checks the output of the pre-processing step; introduces other signals and events, time expressions, and the appropriate links among them. The average time to annotate a document of 500 words by a trained annotator is 1 hour. The annotators came from a variety of backgrounds. –70% of the corpus annotated by TimeML developers; –30% annotated by students from Brandeis University.

14. The Annotation Tool (1) To help the annotators with the annotation effort, a modified version of the Alembic Workbench (Vilain and Day 1996) was developed. When a text is loaded into the tool: – the text is shown in one window with the results of the pre-processing shown via coloured tags. These tags can be edited or deleted, and new tags can be introduced. – links are shown in a second window These links can be created by selecting tags in the text window and inserting these into the link window.

15. The Annotation Tool (2)

16. The Annotation Tool (3)

17. EVENT TIMEX STATE Key:

18. O’SMACH, Cambodia (AP) - The top commander of a Cambodian resistance force said Thursday he has sent a team to recover the remains of a British mine removal expert kidnapped and presumed killed by Khmer Rouge guerrillas almost two years ago. February 19, 1998 saidsent recover kidnapped presumed killed Duration=almost two years Thursday DCT before Signal=ago relType=after before eventArg Is_included irrealis Cambodian British Within-sentence annotation time

19. Gen. Nhek Bunchhay, a loyalist of ousted Cambodian Prime Minister Prince Norodom Ranariddh, said in an interview with The Associated Press at his hilltop headquarters that he hopes to recover the remains of Christopher Howes within the next two weeks. DCT said before Is_included hopes recover irrealis the next two weeks Is_included loyalist Cambodian Signal=within ousted before Within-sentence annotation

20. Howes had been working for the Britain-based Mines Advisory Group when he was abducted with his Cambodian interpreter Houn Hourth in March 1994. There were many conflicting accounts of his fate. working abducted March 1994 DCT ibefore Signal=when Is_included Signal=in Within-sentence annotation

21. Howes’ team was clearing mines 17 kilometers (10 miles) from Angkor Wat, the fabled 11th Century temple that is Cambodia’s main tourist attraction, when it was attacked. DCT clearing attacked ibefore Signal=when Within-sentence annotation

22. saidsent recover kidnappedkilled Thursday DCT before after before Is_included irrealis Within-document annotation (four sentences) working abducted March 1994 ibefore Is_included ID clearing attacked ibefore Is_included said hopes ousted before Is_included recover the next two weeks before Is_included irrealis

23. TANGO Demo Performing link analysis on a text

24. Closure: lessons from TANGO Discovery aspects less important The spatial metaphor of TANGO guides the annotator to an event graph that requires less user prompting in order to get to a complete annotation. Closure makes a consistent and complete annotation possible Closure is still needed to infer implicit relations and to have prior choices of links restrict the relation type of other links.

25. Domains and Data Sets Document Collection (300): –ACE –DUC –PropBank (WSJ) Query Corpus Collection: –Excite query logs –MITRE Corpus –TREC8/9/10 –Queries from TIMEBANK

26. Corpus Statistics (1) The statistics collected so far give: –the proportion of tagged text in the corpus –the distribution of: event classes TIMEX3 types LINK types Information like this gives a useful starting point when analysing the mechanisms used to convey temporal information. For example, 62% of links were TLINKs, indicating the importance of this link type. Further analysis of the TLINK will reveal the proportion of explicitly expressed temporal relations (i.e. a signal is used) to implicitly expressed temporal relations (no signal is used).

27. Corpus Statistics (2) For example, here is the distribution of tag types: CountProportio n Tokens68555100% Tags (all 3 kinds) 1120616.3% Events757111.0% TIMEX314232.1% Signals22123.2%

28. The Question Corpus TimeML aims to contribute to Question Answering (QA) – temporal question answering in particular. Temporal questions can be broadly classified into two categories: –Questions that ask for a temporal expression as an answer, like When was Clinton president of the United States? When was Lord of the Rings: The Two Towers released? We call this type explicit. –Questions that either use temporal expression to ask for a non- temporal answer or that ask about the relations holding between events. Who was president of the United States in 1990? Did world steel output increase during the 1990s? We call this type of temporal question implicit.

29. The Question Corpus (2) To evaluate the usefulness of TimeML for (temporal) QA, a question corpus of 50 questions has been created. This corpus was annotated according to a specially developed annotation scheme. This scheme allows features such as: –the type of the expected answer –the volatility of the answer (i.e. how often it changes) to be annotated. The questions contained in the corpus cover both types mentioned above. Examples of questions in the corpus are: When did the war between Iran and Iraq end? When did John Sununu travel to a fundraiser for John Ashcroft? How many Tutsis were killed by Hutus in Rwanda in 1994? Who was Secretary of Defense during the Gulf War?

30. Conclusion There has as yet been no time to analyse the corpus –the statistics collected so far do not represent such an analysis, but only a very preliminary scoping. We anticipate that the corpus will allow a new range of explorations both theoretical and practical. For example: –Theoretical: can study to what extent temporal ordering of events is conveyed explicitly through signals, such as temporal subordinating conjunctions, versus implicitly through the lexical semantics of the verbs or nominalizations expressing the events. –Practical: can train and evaluate algorithms to determine event ordering and time-stamping, and explore their utility in QA.

31. Tempex –Wilson and Mani (2002) MITRE –Timex2 parsing –Direct Interpretation to ISO value

32. What is TempEx? Perl module that implements the TIDES Temporal Annotation Guidelines Handles many formats - Feb. 10, Feb. 10th, February Tenth Some parts of standard not fully implemented - Embedded Expressions: Two weeks ago tomorrow - Unknown Components: June 10 (VAL = XXXX0610) Some very small extensions - Easter gets an ALT_VAL

33. Sample Output POS Tags removed I got up early this morning. I ate lunch an hour and a half ago. In the future, I will know better. I went to Hong Kong the week of October third. I went to Hong Kong the third week of October. Reference Date: 02/16/2001 13:37:00

34. Performance Interannotator agreement TIMEXVALMOD Human x Human0.7890.8890.871 TempEx x Human0.6240.7050.301 Speed - 0.5Megabyte/Minute Demo: TempexTempex

35. TIMEX3 Parser Objects (T3PO) Automatic TimeML Markup Extends TIDES TIMEX2 annotation: –Broader Coverage of temporal expressions –Larger lexicon of temporal triggers Delays Computation of Temporal Math: –Annotation with Temporal Functions –Import Hobbs’ Semantic Web Temporal System Distinct Cascaded Processes: –TIMEX3 and signal recognizer; –Event Predicate recognizer –LINK creation transducer.

36. T3PO Overview  Preprocessing:  POS, Shallow Parsing  Three Finite State modules : – Temporal Expressions – Events – Signals – Links  Discourse Information

37. Temporal Expressions  Extension to Timex2 – Coverage – Absolute ISO Values – Signals  Functional Representation: – Anchor Resolution – Suite of Temporal Functions

38. Event Recognition  In Verbal uses VG chunks: – Encodes Tense and Aspect information  Nominal Events using: – Morphological information – POS ambiguity – Signals – Semantic Information

39. Link Recognition  Event -Timex Links Use of heuristics. Extra-sentential (Event-DCT Links)  Event-Event Links : – Intrasentential SLINKS (evidential) SLINKS (infinitivals) – Extrasentential

40. Preliminary Tests Estimation (6 documents with human annotated version) Precision Recall TASK 90%77%TIMEX RECOGNITION 81%69%TIMEX VALUE RECOGNITION 48%55%SIGNAL RECOGNITION 85%89 %EVENT RECOGNITION. 48%54%EVENT CLASS RECOGNITION 50%TIMEX-EVENT LINK RECOGNITION

41. Mani et al. (2003) A variety of theories have been proposed as to the roles of semantic and pragmatic knowledge in event ordering Very little prior work on corpus-based methods for event ordering They carried out a pilot experiment with 8 subjects who provided event-ordering judgments for 280 clause pairs. Results revealed that: –A. Narrative convention applied only 47% of the time in ordering events in 131 pairs of successive past-tense clauses –B. ~75% of clauses lack explicit time expressions Suggests that anchoring events only to explicit times wouldn’t be sufficient

42. Motivation Question Answering from News –When do particular events occur When did the war between Iran and Iraq end? –Which events occur in a temporal relation to a given event What is the largest U.S. military operation since Vietnam? Multi-Document News Summarization –Event chronologies (e.g., timelines) are used widely in everyday news –Need to know when events occur, to avoid inappropriate merging of distinct events

43. Problem Characteristics In news, events aren’t usually described in the (narrative) order in which they occur –Temporal structure dictated by perceived news value Latest news usually presented first –News sometimes expresses multiple viewpoints, with commentaries, eyewitness recapitulations, etc., Temporal ordering appears to involve a variety of knowledge sources –Tense & aspect Max entered the room. Mary stood up/was seated on the desk. –Temporal adverbials Simpson made the call at 3. Later, he was spotted driving towards Westwood. –Rhetorical relations and World Knowledge Narration Max stood up. John greeted him. Cause/Explanation Max fell. John pushed him. Background Boutros-Ghali Sunday opened a meeting in Nairobi of....He arrived in Nairobi from South Africa.

44. Event Ordering and Reference Time Reference Time (Reichenbach 47) – provides temporal anchoring for events u I had r mailed e the letter (when John came and told me the news). Past Perfect: e < r < u Past: e =r < u Movement of Reference Time depends on tense, aspect, rhetorical relations, world knowledge, etc. u1 John picked r1,e1 up the phone (at 3 pm) u2 He had r2 told e2 Mary he would call her Assuming r2 = e1 (stative), e2 < e1 (Hwang & Schubert 92) u1,u2 r1=3pm e1 r2 e2

45. Two Clause Interpretation Past2Past Max stood up. John greeted him –AFTER relation Max fell. John pushed him. –BEFORE relation Max entered the room. Mary was seated behind the desk. –Equal (SIMULTANEOUS or INCLUDE) relation Past2PastPerfect Max entered the room. He had drunk a lot of wine –BEFORE relation PastPerfect2Past Max had been in Boston. He arrived late. –AFTER relation

46. Factors That Determine Relation Aspect: –Progressive or not Order: –The iconic order in text Tempex: –The existence of a temporal expression Tense: –Past vs. Past Perfect Meaning: –Lexical or constructional semantics of the sentence.

47. Event Ordering: Human Experiment Foreign Minister John Chang confirmed to reporters that Lien, during a Sunday stopover in New York, had made a detour to a “third country'' with which Taiwan has no diplomatic ties and would not return to Taipei as scheduled on Monday. But Chang and other Taiwan spokesmen pointedly refused to confirm local media reports that Lien was in Europe, much less to confirm that he had flown to France. Since a civil war divided them in 1949, China has regarded Taiwan as a rebel province ineligible for sovereign foreign relations. In mid-1995, a furious Beijing downgraded ties with Washington and froze talks with Taiwan after President Lee Teng-hui made a private visit to the United States. ‘ meets’ and ‘during/includes’ not used

48. Results on Human Event Ordering Narrative convention applies in less than half of the Past to Past cases and less than two-thirds of the Past Perfect to Past cases While shallow features can be leveraged in ordering, meaning and commonsense knowledge also play a crucial role 5 subjects X 48 exs = 240 exs 131 109

49. Inter-annotator Agreement on Temporal Ordering Overall: 24/40 = 60% Removing Unclears: 24/33 = 72% Unclears Breakdown: –Clear: 1; POS error: 1; Not enough context: 5 Other Disagreements: –Polar Opposition: 4 (1 difficult) –Entirely Before vs Equal: (1) In an interview with Barbara Walters to be shown on ABC’s “Friday nights”, Shapiro said he tried on the gloves and realized they would never fit Simpson’s larger hands. –Entirely Before vs Upto: (2) They had contested the 1992 elections separately and won just six seats to 70 for MPRP. Based on 3 subjects on a common set of 40 examples Fine-grained decisions about temporal ordering, are difficult Subjects show an acceptable level of agreement on more coarse-grained ordering (collapsing Entirely Before and Upto)

50. Automatic Link Identification in Text Mani, Schiffman, and Zhang (2003)

51. Approach: Mixed-initiative Corpus Annotation 1.Automatic preprocessing –time expression flagging and evaluation (TempEx using TIDES TIMEX2 spec) –clause structure (Clause-IT) events identified with finite clause indices –lexical aspect (lexicon) –tense (part-of-speech and patterns) 2.Automatic computing of reference time value (tval) for each clause (given finding B above) –tval is either time value of explicit timex in clause, or, when timex is absent, an implicit time value inferred from context by a naïve algorithm Simpson made the call at 3. He had visited … 3.Human annotation –specify anchoring relation (AT, BEF, AFT, undef) of event wrt corrected tval 4.Automatic learning of anchoring rules 5.Automatic computation of temporal ordering

52. Time Expression and Clause Processing TIDES TIMEX2 Annotation Scheme The Foreign Minister told Thailand's Nation Newspaper Sunday Pol Pot had left Cambodia but was not in Thailand, ending credence to a claim last summer the aged and ailing former Khmer Rouge leader had fled to China. TIMEX2 Accuracy 5 annotators F-measure 193 TDT2 docs Extent Value Human Agreement.79.86 TempEx 1.03.76.82 Clause Tagging The United States unleashed what appeared to be its fiercest daylight strike on Afghanistan on Monday but the administration faced concern from Saudi Arabia and Pakistan over the bombardment to force Taliban leaders to hand over Saudi militant Osama bin Laden. CLAUSE-IT Tagger special-purpose finite-state grammars used with CASS to identify NPs, PPs, and VPs, and links between verbs and their subjects. proposed clause boundaries confirmed or adjusted using verb subcategorization information from Penn Treebank –e.g., a PP can be attached to a VP containing an object NP if the verb has been followed in the PTB by a NP and a PP headed by the current prep.

53. Computing Reference Times history_list := {doc_date} for each finite clause c do rtime = timex2(c) if rtime then tval(c) = rtime unless type(c, rel_clause) push(rtime, history_list) elsif reporting_verb(c) then tval(c) = doc_date elsif  j s.t. inside_quote(c, j) then tval(c) = tval(j) else tval(c) = last (history_list) Implicit reference time encoded in clause tval feature Explicit reference time A Naïve Algorithm For Computing tval (59% accurate)

54. Partially Ordering Links Machine-learnt rules used to generate anchor tuples – Timex2 sorting used to generate tval tuples – Some 280,000 federal workers have been furloughed … After breakfast with weekend participants, Clinton went to play 18 holes of golf with several friends despite fog and rain. The president and his family celebrated New Year's Eve at a dinner party sponsored by the Renaissance Weekend. Event Ordering Link Recall Link Precision F- measure.777 (562/723).737 (563/764).757 7 docs, 194 clauses, 723 human links r7=19960101 r11 e11 e7 r12=19951231NI e12

55. Mani et al. Results Introduces a corpus-based approach for anchoring and ordering events –Approach is motivated by a pilot experiment investigating human event ordering capabilities Uses clause tagging and shallow semantic tagging of tense, aspect, time expressions Achieves.84 accuracy in anchoring events and.75 F- measure in partially ordering them

56. Developmental Narrative Models –Use Developmental Studies to Model Event Narrative Structure –Take corpora from developmental models to train algorithms

57. Developmental Corpus: Level 1 David wants to buy a Christmas present for a very special person, his mother. David's father gives him $5.00 a week pocket money and David puts $2.00 a week into his bank account. After three months David takes $20.00 out of his bank account and goes to the shopping mall. He looks and looks for a perfect gift. Suddenly he sees a beautiful brooch in the shape of his favorite pet. He says to himself "Mother loves jewelry, and the brooch costs only $l7.00." He buys the brooch and takes it home. He wraps the present in Christmas paper and places it under the tree. He is very excited and he is looking forward to Christmas morning to see the joy on his mother's face. But when his mother opens the present she screams with fright because she sees a spider.

58. Event Ordering: Level 1 David wants to buy a Christmas present for a very special person, his mother. David's father gives him $5.00 a week pocket money and David puts $2.00 a week into his bank account. After three months David takes $20.00 out of his bank account and goes to the shopping mall. He looks and looks for a perfect gift. Suddenly he sees a beautiful brooch in the shape of his favourite pet. He says to himself "Mother loves jewelry, and the brooch costs only $l7.00." He buys the brooch and takes it home. He wraps the present in Christmas paper and places it under the tree. He is very excited and he is looking forward to Christmas morning to see the joy on his mother's face. But when his mother opens the present she screams with fright because she sees a spider. Present stative: want, give, put Take < go < look < See < say Present stative: love, cost Buy < take < wrap < place Present stative: be-excited, looking-forward < see Open < See < scream

59. Narrative Convention: Level 1 Strategies: - Scene setting with present tense - Narration with present tense 1. For a state sentence in present tense A, if there is a sentence in present tense, B, in the document, interpret T(B) = T(A). 2. For an action sentence in present tense, A, if there is a sentence in present tense, B, in the document, interpret T(B) < T(A).

60. Developmental Corpus: Level 2 Mrs Wilson and Mrs Smith are sisters. Mrs Wilson lives in a house in Duncan and Mrs Smith lives in a condominium in Victoria. One day Mrs Wilson visited her sister. When her sister answered the door Mrs Wilson saw tears in her eyes. "What's the matter?" she asked. Mrs Smith said "My cat Sammy died last night and I have no place to bury him". She began to cry again. Mrs Wilson was very sad because she knew her sister loved the cat very much. Suddenly Mrs. Wilson said "I can bury your cat in my garden in Duncan and you can come and visit him sometimes. Mrs. Smith stopped crying and the two sisters had tea together and a nice visit. It was now five o'clock and Mrs Wilson said it was time for her to go home. She put on her hat, coat and gloves and Mrs Smith put the dead Sammy into a shopping bag. Mrs Wilson took the shopping bag and walked to the bus stop. She waited a long time for the bus so she bought a newspaper. When the bus arrived she got on the bus, sat down and put the shopping bag on the floor beside her feet. She then began to read the newspaper. When the bus arrived at her bus stop she got off the bus and walked for about two minutes. Suddenly she remembered she left the shopping bag on the bus.

61. Event Ordering: Level 2 Mrs Wilson and Mrs Smith are sisters. Mrs Wilson lives in a house in Duncan and Mrs Smith lives in a condominium in Victoria. One day Mrs Wilson visited her sister. When her sister answered the door Mrs Wilson saw tears in her eyes. "What's the matter?" she asked. Mrs Smith said "My cat Sammy died last night and I have no place to bury him". She began to cry again. Mrs Wilson was very sad because she knew her sister loved the cat very much. Suddenly Mrs. Wilson said "I can bury your cat in my garden in Duncan and you can come and visit him sometimes. Mrs. Smith stopped crying and the two sisters had tea together and a nice visit. It was now five o'clock and Mrs Wilson said it was time for her to go home. She put on her hat, coat and gloves and Mrs Smith put the dead Sammy into a shopping bag. Mrs Wilson took the shopping bag and walked to the bus stop. She waited a long time for the bus so she bought a newspaper. When the bus arrived she got on the bus, sat down and put the shopping bag on the floor beside her feet. She then began to read the newspaper. When the bus arrived at her bus stop she got off the bus and walked for about two minutes. Suddenly she remembered she left the shopping bag on the bus. Present stative: be-sister, live-1, live-2 visit < answer, see-tears < ask [Present stative: be-the-matter ] < said [ > die (last night) Present stative: -have < bury ] begin ≤ cry Present stative: be-sad, BECAUSE know love ….

62. Narrative Convention: Level 2 Strategies: - Scene setting with present tense - Narration with past tense

63. Developmental Corpus: Level 3 One day Nasreddin borrowed a pot from his neighbour Ali. The next day he brought it back with another little pot inside. "That's not mine," said Ali. "Yes, it is," said Nasreddin. "While your pot was staying with me, it had a baby." Some time later Nasreddin asked Ali to lend him a pot again. Ali agreed, hoping that he would once again receive two pots in return. However, days passed and Nasreddin had still not returned the pot. Finally Ali lost patience and went to demand his property. "I am sorry," said Nasreddin. "I can't give you back your pot, since it has died." "Died!" screamed Ali, "how can a pot die?" "Well," said Nasreddin, "you believed me when I told you that your pot had had a baby."

64. Developmental Corpus: Level 3.5 One day, Nasreddin was up on the roof of his house, mending a hole in the tiles. He had nearly finished, and he was pleased with his work. Suddenly, he heard a voice below call "Hello!" When he looked down, Nasreddin saw an old man in dirty clothes standing below. "What do you want?" asked Nasreddin. "Come down and I'll tell you," called the man. Nasreddin was annoyed, but he was a polite man, so he put down his tools. Carefully, he climbed all the way down to the ground. "What do you want?" he asked, when he reached the ground. "Could you spare a little money for an old beggar?" asked the old man. Nasreddin thought for a minute. Then he said, "Come with me." He began climbing the ladder again. The old man followed him all the way to the top. When they were both sitting on the roof, Nasreddin turned to the beggar. "No," he said.

65. Developmental Corpus: Level 4 It was a cold night in September. The rain was drumming on the car roof as George and Marie Winston drove through the empty country roads towards the house of their friends, the Harrisons, where they were going to attend a party to celebrate the engagement of the Harrisons' daughter, Lisa. As they drove, they listened to the local radio station, which was playing classical music. They were about five miles from their destination when the music on the radio was interrupted by a news announcement: "The Cheshire police have issued a serious warning after a man escaped from Colford Mental Hospital earlier this evening. The man, John Downey, is a murderer who killed six people before he was captured two years ago. He is described as large, very strong and extremely dangerous. People in the Cheshire area are warned to keep their doors and windows locked, and to call the police immediately if they see anyone acting strangely." Marie shivered. "A crazy killer. And he's out there somewhere. That's scary." "Don't worry about it," said her husband. "We're nearly there now. Anyway, we have more important things to worry about. This car is losing power for some reason -- it must be that old problem with the carburetor. If it gets any worse, we'll have to stay at the Harrisons' tonight and get it fixed before we travel back tomorrow."

66. Conclusion and Discussion