1 Today’s Plan 900am–915am:Sort out questions regarding refund forms 915am–945am:Finalise today’s agenda 945am-1045am:Breakout Groups Session am–1115am:Coffee + recalibration at am–1215pm:Breakout Groups Session pm–145pm:Lunch 145pm–245pm:Report-backs 245pm–300pm:Wrap-up

2 Yesterday’s Shifting Sentiment #1

3 Yesterday’s Shifting Sentiment #2

4 Yesterday’s Main Themes #1 General Methodological Concerns: – Reasons to be cautious [Scott, Moore, Di Eugenio, McCoy, Green] Frameworks for Evaluation: – Frameworks for evaluation [Paris, Stent] – GENEVAL [Belz and Reiter] – Building on RAGS [Mellish + Scott] Resources: – A Wiki [Stent] – Shared data resources [Walker]

5 Yesterday’s Main Themes #2 Shared Tasks: – DUC or other existing STECs as a host [McKeown] – Referring Expression Generation [Gatt, Viethen] – Virtual Environments [Koller et al] – Question Qeneration [Rus + Graesser] – Textual Variation [McDonald]

6 Today’s Goals Observation: – Opinion is spread between those who favour a Shared Task (one or many) and those who are cautious Hypothesis #1: – Those who favour a Shared Task will push ahead anyway Hypothesis #2: – Those who are cautious are the best people to identify the relevant desiderata

7 Possible Working Groups Desiderata Development Evaluation Shared Resources Shared Task #1: Text to Text (perhaps via DUC?) Shared Task #2: Referring Expression Generation Shared Task #3: Data to Text Shared Task #4: Virtual Environments

8 Target Outcome A document How far can we get today? – A set of PowerPoint files that plans the sections of the document to one bullet point per subsubsection or paragraph – Report-back on planned content to the assembled group After the workshop: – Flesh out the text

9 Task 1: Desiderata Main Sections: – What’s unique about NLG Evaluation – Approaches and Frameworks – A ‘due diligence checklist’ —` a clear enumeration of the things you should consider if you are going to pursue a Shared Task in NLG

10 Tasks 2 through N: Shared Tasks Proposals Main Sections: – A definition of the Shared Task and its type – The aims of the Shared Task: what it will achieve – The subcommunity it seeks to engage, and how it will do this – Whether the task will involve evaluation and how – The resources required, and how they will be obtained – A plan of execution – [Later] How this Shared Task addresses the desiderata

11 Today’s Plan 900am–915am:Sort out questions regarding refund forms 915am–945am:Finalise today’s agenda 945am-1045am:Breakout Groups Session am–1115am:Coffee + recalibration 1115am–1215pm:Breakout Groups Session pm–115pm:Lunch 115pm–245pm:Report-backs 245pm–300pm:Wrap-up

12 Possible Working Groups Desiderata Development [5] Shared Task #1: Text to Text (perhaps via DUC?) [3] Shared Task #2: Referring Expression Generation [5] Shared Task #4: Virtual Environments [5]

13 New Thoughts and Considerations Is this working? Is the spec broken or can it be refined? What have you thought of that you think the others could benefit from in working out their plan?

14 Today’s Plan 900am–915am:Sort out questions regarding refund forms 915am–945am:Finalise today’s agenda 945am-1045am:Breakout Groups Session am–1115am:Coffee + recalibration 1115am–1215pm:Breakout Groups Session pm–115pm:Lunch 115pm–245pm:Report-backs 245pm–300pm:Wrap-up

15 Next Steps One or two people in each group commits to making sure there is follow-through Each group commits to writing up their section of the report and delivering a draft to MW + RD by June 1st. At time June 1st, all sections are redistributed to all workshop participants. All can send comments on any part to MW+RD by mid June. In mid-June, comments are collated and redistributed to authors of sections. Revised versions are returned to MW + RD by 15th July. Distributed to community for input to be received by mid August. Comments taken on board and final report made available to the community at end August.

16 The Responsible Adults Group 1: Donia Scott and Cécile Paris Group 2: Marilyn Walker and Amanda Stent Group 3: Albert Gatt Group 4: Johanna Moore and Alex Koller

17 Report-Backs Desiderata Development Shared Task #1: Text to Text Shared Task #2: Referring Expression Generation Shared Task #3: Virtual Environments