1. 1 Evaluation of Opinion Questions ä Session leaders: Ed Hovy, Kathy McKeown ä Topics ä Is evaluating opinion questions feasible at all? How can we construct a reliable evaluation? ä What are a subset of opinion questions that could be handled? ä What are possible methods for evaluating results? ä What should the form of the response be?

2. 2 Is evaluating opinions feasible? ä Preliminary results indicate yes ä Necessary: ä Multiple human judgments on a response ä Evaluation must be computed against multiple models ä Measure agreement between models ä Models of opinions and arguments need to inform evaluation design

3. 3 What subset of opinion questions should be addressed? ä Some characteristics ä Concrete statement, specific terms ä Is European criticism of President Bush and United States foreign policy justified? ä Question can be marked as seeking opinion ä (justified, think, opinion, success) ä Decided: No need for question analysis or for determining if the question is an opinion question ä Provide a fact, topic and find all opinions related to that fact ä Closer to what analysts do ä Avoids problem of disambiguating questions

4. 4 Measures for scoring responses Form of the response ä Input: proposition/topic ä Task: find the opinions and cluster them ä Expect that each cluster will represent a viewpoint (e.g., pro or con) ä Answer represents an objective view of multiple opinions from different sources, not a generated system opinion ä Form of answer: list of items/sentences with links to source documents ä Metrics used to score answer: ä number and validity of clusters, ä relevance of each item to input topic, ä lack of redundancy, ä Is the item an opinion? ä Subjective: ä Formative user studies ä Think-aloud protocols

5. 5 Discussed but left for the future ä Measure level of support for a viewpoint ä Label clusters (e.g., pro, con) ä Distinguish between opinions and facts ä Mixed response from analyst on need ä Provide justification of opinions by linking to supporting facts ä Analysts did not seem to feel a high priority ä Distinguish public and privately held opinions ä Identify deceptions ä What overt behavior results from opinions?