1 Evaluation of Opinion Questions ä Session leaders: Ed Hovy, Kathy McKeown ä Topics ä Is evaluating opinion questions feasible at all? How can we construct.

Slides:

Advertisements

Similar presentations

The Application of ECD to the Redesign of Advanced Placement Exams CCSSO June 2012 Maureen Ewing, Kristen Huff, Amy Hendrickson, Pamela Kaliski.

Advertisements

Extended-Response Questions

Measuring Complex Achievement: Essay Questions

OAA Extended Response/Short Answer

Copyright © 2014 McGraw-Hill Higher Education. All rights reserved. CHAPTER 3 SUPPLEMENT Process Mapping and Analysis McGraw-Hill/Irwin.

Describing Process Specifications and Structured Decisions Systems Analysis and Design, 7e Kendall & Kendall 9 © 2008 Pearson Prentice Hall.

Dd. This learning session will help the auditor: Design audit objectives understand why audit criteria are used in performance audits; learn how to develop.

Minnesota State Community and Technical College Critical Thinking Assignment Example and Assessment.

Argumentative Thesis Statements For use with Stepping Stone Argumentative Research Project.

 Essential Questions ◦ How do you create a strong justification relating directly to an ethical question?  Other questions to ponder ◦ How do you determine.

Other Measurement Validity Types. OverviewOverview l Face validity l Content validity l Criterion-related validity l Predictive validity l Concurrent.

INTERVIEWING SKILLS FOR EFFECTIVE PERFORMANCE APPRAISAL Ministry of Public Health and Sanitation Ministry of Medical Services 1.

Academic Writing: Writing in a critical way Dr. Tamara O’Connor Student Learning Development Student Counselling Service

6/5/2007SE Survival Exercise Recap1 Team Software Project (TSP) June 05, 2007 Planning, Quality, Risks.

Chapter 9 Describing Process Specifications and Structured Decisions

Institute for Social & Behavioral Research Institute for Social & Behavioral Research.

Midterm Review Evaluation & Research Concepts Proposals & Research Design Measurement Sampling Survey methods.

MCAT University Writing Center Format First draft timed writing  2 – 30 minutes essays  Expository response to specific topic (3 parts)

Kendall & KendallCopyright © 2014 Pearson Education, Inc. Publishing as Prentice Hall 9 Kendall & Kendall Systems Analysis and Design, 9e Process Specifications.

Chapter 4 – Strategic Job Analysis and Competency Modeling

The phases of research Dimitra Hartas. The phases of research Identify a research topic Formulate the research questions (rationale) Review relevant studies.

Argumentative writing / public speaking is based on evidence (facts and data) – not opinions! So carefully choose your sources!

Easy steps to writing THE ESSAY. Writing an essay means: Creating ideas from information Creating arguments from ideas Creating academic discourse to.

Nurturing Historical Thinking Document Analysis/Socratic Seminar Persistent Issue: What should society do to promote fairness and justice for people who.

DBQ’S MRS. ALLEY Lesson 9- Day 1. What is a DBQ?  A DBQ, document based question, is a question that focuses around one or more documents.  The documents.

Describing Process Specifications and Structured Decisions Systems Analysis and Design, 7e Kendall & Kendall 9 © 2008 Pearson Prentice Hall.

ASSESSMENT IN EDUCATION ASSESSMENT IN EDUCATION. Copyright Keith Morrison, 2004 ITEM TYPES IN A TEST Missing words and incomplete sentences Multiple choice.

Session 2 Traditional Assessments Session 2 Traditional Assessments.

The Sarbanes-Oxley Act of PricewaterhouseCoopers Introduction of Panel Members The Sarbanes-Oxley Act of 2002 Sample testing of controls Marcus.

*Erasmus University Rotterdam P.O. Box 1738, NL-3000 DR Rotterdam, the Netherlands † Teezir BV Wilhelminapark 46, NL-3581 NL, Utrecht, the Netherlands.

Answer Keys, Checklists, Rating Scales & Rubrics

Learning Targets for MP 4. Punctuation Marks TEK 7.20 B (ii) semicolons; colons: hyphens -

Analyzing Different Viewpoints Persuasive Writing Grade 4 – Language Arts Mr. Luvera.

Appraisal and Its Application to Counseling COUN 550 Saint Joseph College Ability, Intelligence, Aptitude and Achievement Testing For Class #12 Copyright.

HITIQA: Scenario Based Question Answering Tomek Strzalkowski, et al The State University of New York at Albany Paul Kantor, et al Rutgers University Boris.

Multiplication Facts X 3 = 2. 8 x 4 = 3. 7 x 2 =

Assessment Basics and Active Student Involvement Block II.

WALK IN WORK Get out your notebook and something to write with. Go to the writing section and label the next two pages “Research Questions and terms”

How To Write an AP European History Thesis Statement Mr. Ott – Park East

S ystems Analysis Laboratory Helsinki University of Technology 1 Decision Analysis Raimo P. Hämäläinen Systems Analysis Laboratory Helsinki University.

Team Exercise. 5/29/2007SE Survival Exercise2 SURVIVAL!

Issues/Research KNR 208. Topic vs Issue Topic – The subject of a discussion, speech Issue - Are matters of wide public concern arising out of complex.

Tracing and Evaluating Arguments Identifying and exploring how an argument is made in an essay, speech, or other text.

Chapter Twelve Copyright © 2006 McGraw-Hill/Irwin Attitude Scale Measurements Used In Survey Research.

Direction Words: Definitions. Copyright © Houghton Mifflin Company. All rights reserved.8 | 2 Prove or give reasons.

Do not on any account attempt to write on both sides of the paper at once. W.C.Sellar English Author, 20th Century.

Assessment and the Institutional Environment Context Institutiona l Mission vision and values Intended learning and Educational Experiences Impact Educational.

EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014.

Fact Finding (Capturing Requirements) Systems Development.

IS6146 Databases for Management Information Systems Lecture 12: Exam Revision Rob Gleasure robgleasure.com.

Checklists and Rating Scales Read my Jigsaw Chapter: Yes____ No____ (check one)

Academic Writing Fatima AlShaikh. A duty that you are assigned to perform or a task that is assigned or undertaken. For example: Research papers (most.

Dr Anie Attan 26 April 2017 Language Academy UTMJB

Information Storage and Retrieval Fall Lecture 1: Introduction and History.

Keys to creating a successful thesis statement

Keys to creating a successful thesis statement

Aim: What is a Persuasive essay and how can we compose one?

EDU 385 Session 8 Writing Selection items

Lecture 01: A Brief Summary

An Introduction to the Colorado Assessment Standards

FEASIBILITY STUDY Feasibility study is a means to check whether the proposed system is correct or not. The results of this study arte used to make decision.

School of EE and Computer Science

How does a Requirements Package Vary from Project to Project?

Part A. Identification and Evaluation of Sources

Changes in Individual and Team Performance Over Time:

Moral Reasoning 1.

Chapter 11 Describing Process Specifications and Structured Decisions

Subject Name: SOFTWARE ENGINEERING Subject Code:10IS51

MEASUREMENT AND QUESTIONNAIRE CONSTRUCTION:

Debating metal bonding

Presentation transcript:

1 Evaluation of Opinion Questions ä Session leaders: Ed Hovy, Kathy McKeown ä Topics ä Is evaluating opinion questions feasible at all? How can we construct a reliable evaluation? ä What are a subset of opinion questions that could be handled? ä What are possible methods for evaluating results? ä What should the form of the response be?

2 Is evaluating opinions feasible? ä Preliminary results indicate yes ä Necessary: ä Multiple human judgments on a response ä Evaluation must be computed against multiple models ä Measure agreement between models ä Models of opinions and arguments need to inform evaluation design

3 What subset of opinion questions should be addressed? ä Some characteristics ä Concrete statement, specific terms ä Is European criticism of President Bush and United States foreign policy justified? ä Question can be marked as seeking opinion ä (justified, think, opinion, success) ä Decided: No need for question analysis or for determining if the question is an opinion question ä Provide a fact, topic and find all opinions related to that fact ä Closer to what analysts do ä Avoids problem of disambiguating questions

4 Measures for scoring responses Form of the response ä Input: proposition/topic ä Task: find the opinions and cluster them ä Expect that each cluster will represent a viewpoint (e.g., pro or con) ä Answer represents an objective view of multiple opinions from different sources, not a generated system opinion ä Form of answer: list of items/sentences with links to source documents ä Metrics used to score answer: ä number and validity of clusters, ä relevance of each item to input topic, ä lack of redundancy, ä Is the item an opinion? ä Subjective: ä Formative user studies ä Think-aloud protocols

5 Discussed but left for the future ä Measure level of support for a viewpoint ä Label clusters (e.g., pro, con) ä Distinguish between opinions and facts ä Mixed response from analyst on need ä Provide justification of opinions by linking to supporting facts ä Analysts did not seem to feel a high priority ä Distinguish public and privately held opinions ä Identify deceptions ä What overt behavior results from opinions?