1. SIG IL 2000 Evaluation of a Practical Interlingua for Task-Oriented Dialogue Lori Levin, Donna Gates, Alon Lavie, Fabio Pianesi, Dorcas Wallace, Taro Watanabe, Monika Woszczyna

2. SIG IL 2000 Interchange Format Design The CSTAR II Interchange Format was designed and developed by all of the CSTAR II partners: CMU, IRST, ETRI, UKA, CLIPS++ and ATR. www.c-star.org

3. SIG IL 2000 Expressivity vs Simplicity If it is not expressive enough, components of meaning will be lost. If it is not simple enough, it can’t be used reliably across sites. If it is not simple enough, it will not be quickly portable to new domains.

4. SIG IL 2000 Task Oriented Sentences Perform an action in the domain. Are not descriptive. Contain fixed expressions that cannot be translated literally. Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

5. SIG IL 2000 Domain Actions: Extended, Domain-Specific Speech Acts Examples: c:request-information+availability+room a:give-information+personal-data c:give-information+temporal+arrival Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

6. SIG IL 2000 Components of the Interchange Format a: speaker a: (agent) give-information speech act give-information +availability+room concept* +availability+room (room-type=(single & double), argument* (room-type=(single & double), time=md12) Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

7. SIG IL 2000 Examples  no that’s not necessary  c:negate  yes I am  c:affirm  and I was wondering what you have in the way of rooms available during that time  c:request-information+availability+room  my name is alex waibel  c:give-information+personal-data (person-name=(given-name=alex, family-name=waibel))  and how will you be paying for this  a:request-information+payment (method=question)  I have a mastercard  c:give-information+payment (method=mastercard) Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

8. SIG IL 2000 Not Covered or Not Represented in IF Relative clauses Comparatives (in general) Tense Number (but quantity is represented)

9. SIG IL 2000 Scope of the IF May 1999  Speech acts 54  Concepts 84  Arguments 118 Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

10. SIG IL 2000 Expressivity: Coverage Experiment Development data was tagged with interlingua representations by human experts. Sentences that are not intended to be covered by the interlingua (as judged by human experts) were given the tag “no-tag.” Test data was tagged by human experts.

11. SIG IL 2000 Coverage Experiment: Development and Test Data

12. SIG IL 2000 The Interchange Format Database 61.2.3 olang I lang I Prv IRST “telefono per prenotare delle stanze per quattro colleghi” 61.2.3 olang I lang E Prv IRST “I’m calling to book some rooms for four colleagues” 61.2.3 IF Prv IRST c:request-action+reservation+features+room (for-whom= (associate, quantity=4)) 61.2.3 comments: dial-oo5-spkB-roca0-02-3 d.u.sdu olang X lang Y Prv Z “sdu-in-language-Y on one line” d.u.sdu olang X lang E Prv Z “sdu-in-English on one line” d.u.sdu IF Prv Z dialogue-act-on-one-line d.u.asdu comments: your comments d.u.asdu comments: go here

13. SIG IL 2000 Coverage of Top 10 Dialogue Acts in Development Data

14. SIG IL 2000 Coverage of Top 10 Speech Acts in Development Data

15. SIG IL 2000 Coverage of Top 10 Dialogue Acts in Test Data

16. SIG IL 2000 Coverage of Top 10 Speech Acts in Test Data

17. SIG IL 2000 Simplicity: Consistency of Use Across Sites Successful international demo. After testing English-Italian and English- Korean, Italian-Korean worked without extra effort. Inter-coder agreement experiment Cross-site evaluation experiment

18. SIG IL 2000 Inter-coder Agreement Experiment 84 DA units from Japanese-English data Some dialogue fragments and some isolated sentences Coded at CMU and IRST Results reported in percent agreement

19. SIG IL 2000 Inter-Coder Agreement Resuts Speech Act 82.14 Concept List 88.00 Dialogue Act 65.48 Argument List 85.79

20. SIG IL 2000 Inter-Coder Agreement Error Analysis of 33 Sentences 6 are equivalent due to ambiguity in the IF specification. 16 are similar enough to produce output with equivalent meaning. –offer-search+availability: Let me check the availability –give-information+search+availability: I will check the availability 4 contain differences where the input sentence was ambiguous and taggers chose different meanings. –6 o’clock could be 6:00 or 18:00 5 contain errors by one or more taggers and would produce outputs with different meanings

21. SIG IL 2000 Cross-Site Evaluation Analysis and generation grammars were written at different sites (CMU and IRST). Analysis at CMU produces IF. IF is sent to IRST. Generation at IRST produces Italian sentences.

22. SIG IL 2000 Intra-Site Evaluation Analysis and generation are both performed at CMU by researchers in constant contact with each other. English-IF-English, English-German, and English-Japanese

23. SIG IL 2000 Cross Site Evaluation Data 130 utterances from a user study performed at CMU Speech input “Traveller” is a second time user. “Agent” is a system developer. Traveller and agent cannot see or hear each other. All communication is through English-IF- English paraphrase.

24. SIG IL 2000 Evaluation Scoring OK: meaning is preserved Perfect: meaning is preserved and the output is fluent Bad: meaning is not preserved Acceptable: Sum of Perfect and OK English-German was graded at CMU, IRST and CLIPS. English-IF-English was graded at CMU and CLIPS English-Japanese was graded at CMU. English-Italian was graded at IRST. English-French was graded at CLIPS

25. SIG IL 2000 End-to-End Evaluation Results

26. SIG IL 2000 End-to-End Evaluation Results

27. SIG IL 2000 Conclusions Coverage is surprisingly good for a certain type of data: role playing for flight reservations, hotel reservations, greetings, and payment. Cross-site evaluation is about as good as intra-site evaluation. Inter-coder agreement could be improved, but not all errors affect translation quality.

28. SIG IL 2000 Current Work Integrating the task-oriented interlingua with a more traditional frame-based interlingua for descriptive sentences. The NEPSOLE! Consortium: http://nespole.itc.it