Presentation is loading. Please wait.

Presentation is loading. Please wait.

Both methods give significant improvements in accuracy over two strong baselines: Above curves were generated using rationales from a single annotator,

Similar presentations


Presentation on theme: "Both methods give significant improvements in accuracy over two strong baselines: Above curves were generated using rationales from a single annotator,"— Presentation transcript:

1 Both methods give significant improvements in accuracy over two strong baselines: Above curves were generated using rationales from a single annotator, A0. What about other annotators? We solicited rationales from several other annotators (on 100 documents) and saw similar improvements in accuracy: Tags ( ) Words ( ) Armageddon This disaster flick is a disaster alright. Directed by Tony Scott (Top Gun), it's the story of an asteroid the size of Texas caught on a collision course with Earth. After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a comet shower, NASA detects said asteroid and go into a frenzy. They hire the world's best oil driller (Bruce Willis), and send him and his crew up into space to fix our global problem. The action scenes are over the top and too ludicrous for words. So much so, I had to sigh and hit my head with my notebook a couple of times. Also, to see a wonderful actor like Billy Bob Thornton in a film like this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep Impact. Bottom line is, Armageddon is a failure. يوم الحساب هذا الفيلم الكارثي هو بحق فيلم كارثي. الفيلم، من إخراج توني سكوت (مخرج فيلم توب جن)، يروي قصة كويكب بحجم ولاية تكساس في طريقه للاصطدام بالكرة الأرضية. بعد مشهد افتتاحي رائع، تدمر فيه سفينة فضاء أمريكية، إضافة إلى مدينة نيويورك، تكتشف ناسا الكويكب آنف الذكر ويجن جنونها. فيستعينون بأفضل منقب عن النفط في العالم (بروس ويليس)، ويرسلونه وطاقمه إلى الفضاء لإصلاح مشكلتنا العالمية هذه. مشاهد الإثارة والآكشن مبالغ بها ويصعب وصف سخفها بأي كلمات، إلى حد وجدت نفسي فيه أتنهد، بل وضربت جبهتي بمدونتي بضع مرات. و أيضا فإن رؤية ممثل مثل بيلي بوب ثورنتون في فيلم كهذا هو بحق مضيعة لمواهبه. السبب الوحيد لصنع هذا الفيلم هو محاولة التفوق على فيلم ديب امباكت بطريقة ما. خلاصة الأمر، يوم الحساب فيلم فاشل. Omar F. Zaidan Jason Eisner Christine D. Piatko Machine Learning with Annotator Rationales to Reduce Annotation Cost Task: classification of movie reviews into positive and negative reviews. Typically, an annotator is given a set of unannotated documents, and annotates the correct class for each one (  ). This classification task is actually hard for a machine learner. Place yourself in its position, and imagine the task in Arabic instead (  ). If you are not fluent in Arabic, class data alone might not be very helpful. This is truly the situation from a computer’s point of view! But what if, in addition to class, you were also told which segments of the text actually support that class (  )? That should make it easier for you to learn the true model. Presumably, the same is true for a machine learner as well. Proposal: we wish to better utilize annotators by having them tell us more about their classification process. We propose annotators indicate not only what the correct answers are, but also provide hints about why. We propose that they should highlight relevant portions of the example, such as substrings (  ), that help to justify their annotations. We call such hints rationales. We have collected rationales for the movie review dataset of [PL04], and have developed two methods that use the rationales during training. One is a discriminative method [ZEP07] and one is generative [ZE08]. Both methods yield significant accuracy improvements… From the rationale annotations on a positive example x i, we construct several “not-quite-as- positive” contrast examples v ij. Each contrast v ij is obtained by starting with the original and “masking out” a rationale substring: The intuition: a correct model should be less sure of a positive classification on the contrast example v ij than on the original example x i, because v ij lacks evidence the annotator found significant. We express our intuition as additional constraints on an SVM-like model: we want (for each j ) to have w · x i – w · v ij ≥ μ, where μ ≥ 0 controls the size of a margin between original and contrast examples: What this means in practice Standard SVM cares about this margin Modified SVM cares about both margins We typically choose parameters that explain class labels y of training data. With rationales, we propose that parameters be chosen to explain rationale data r in addition to class labels y. For instance, a conditional log-linear model: would have a parameter vector chosen so that: The second factor corresponds to a model for r. Its parameters,, capture how the true influence the annotator. To do this, we encode the rationales as a tag sequence and model it using a CRF: The first-order “emission” features of relate the tag r m to (x m,y, m ), whereas the second-order “transition” features of relate the tag r m to r m–1. What this means in practice Learned jointly i.e. try to model class labels & rationales well … r1r1 x1x1 r2r2 x2x2 rMrM xMxM r3r3 x3x3 … … ! " ( ),. a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe cream debbie. debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl. good hand hanging having he him his i ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie. mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup starts steve story take. the then think thinking this though to tommy trees tries truck uncle up well when with writing year you you've. ! “ ( ),. a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe cream debbie. debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl good hand hanging having he him his I ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie. mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup starts steve story take. the then think thinking this though to tommy trees tries truck uncle up well when with writing year you you've. +1 Our model: –1 Standard model: Training Set Size (Documents) Accuracy (%) Significantly different Generative Approach Discriminative Approach SVM Baseline Log-Linear Baseline SVM baseline Discriminative Method A0A3A4A5 Log-linear baseline Generative Method References: [PL04] B. Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proc. ACL, pages 271–278, [ZEP07] O. Zaidan, J. Eisner, and C. Piatko. Using “annotator rationales” to improve machine learning for text categorization. Proc. NAACL HLT, pages 260–267, [ZE08] O. Zaidan and J. Eisner. Modeling annotators: A generative approach to learning from annotator rationales. Proc. EMNLP, pages 31–40, يوم الحساب هذا الفيلم الكارثي هو بحق فيلم كارثي. الفيلم، من إخراج توني سكوت (مخرج فيلم توب جن)، يروي قصة كويكب بحجم ولاية تكساس في طريقه للاصطدام بالكرة الأرضية. بعد مشهد افتتاحي رائع، تدمر فيه سفينة فضاء أمريكية، إضافة إلى مدينة نيويورك، تكتشف ناسا الكويكب آنف الذكر ويجن جنونها. فيستعينون بأفضل منقب عن النفط في العالم (بروس ويليس)، ويرسلونه وطاقمه إلى الفضاء لإصلاح مشكلتنا العالمية هذه. مشاهد الإثارة والآكشن مبالغ بها ويصعب وصف سخفها بأي كلمات، إلى حد وجدت نفسي فيه أتنهد، بل وضربت جبهتي بمدونتي بضع مرات. و أيضا فإن رؤية ممثل مثل بيلي بوب ثورنتون في فيلم كهذا هو بحق مضيعة لمواهبه. السبب الوحيد لصنع هذا الفيلم هو محاولة التفوق على فيلم ديب امباكت بطريقة ما. خلاصة الأمر، يوم الحساب فيلم فاشل. Armageddon This disaster flick is a disaster alright. Directed by Tony Scott (Top Gun), it's the story of an asteroid the size of Texas caught on a collision course with Earth. After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a comet shower, NASA detects said asteroid and go into a frenzy. They hire the world's best oil driller (Bruce Willis), and send him and his crew up into space to fix our global problem. The action scenes are over the top and too ludicrous for words. So much so, I had to sigh and hit my head with my notebook a couple of times. Also, to see a wonderful actor like Billy Bob Thornton in a film like this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep Impact. Bottom line is, Armageddon is a failure. Original Example ( x i ) Contrast Examples ( v ij ) Take-home Message:  Annotators are underutilized! Richer annotations, such as rationales, can aid machine learning.  Existing machine learning methods can be modified to exploit rationales.  Remember, at test time: - No change to decision rule. - No new features. - No need for rationales. Improvements due solely to better-learned w /  Doing an annotation project? Collect rationales! Even a small number could help. Discriminative ApproachGenerative Approach What is B 10 ? Classifying pos_987.txt (correct class: +1 ): Original Contrast O O O O I I I I I O the prince of egypt succeeds where other movies failed. (Plus several other g features…)

2 Armageddon This disaster flick is a disaster alright. Directed by Tony Scott (Top Gun), it's the story of an asteroid the size of Texas caught on a collision course with Earth. After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a comet shower, NASA detects said asteroid and go into a frenzy. They hire the world's best oil driller (Bruce Willis), and send him and his crew up into space to fix our global problem. The action scenes are over the top and too ludicrous for words. So much so, I had to sigh and hit my head with my notebook a couple of times. Also, to see a wonderful actor like Billy Bob Thornton in a film like this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep Impact. Bottom line is, Armageddon is a failure. يوم الحساب هذا الفيلم الكارثي هو بحق فيلم كارثي. الفيلم، من إخراج توني سكوت (مخرج فيلم توب جن)، يروي قصة كويكب بحجم ولاية تكساس في طريقه للاصطدام بالكرة الأرضية. بعد مشهد افتتاحي رائع، تدمر فيه سفينة فضاء أمريكية، إضافة إلى مدينة نيويورك، تكتشف ناسا الكويكب آنف الذكر ويجن جنونها. فيستعينون بأفضل منقب عن النفط في العالم (بروس ويليس)، ويرسلونه وطاقمه إلى الفضاء لإصلاح مشكلتنا العالمية هذه. مشاهد الإثارة والآكشن مبالغ بها ويصعب وصف سخفها بأي كلمات، إلى حد وجدت نفسي فيه أتنهد، بل وضربت جبهتي بمدونتي بضع مرات. و أيضا فإن رؤية ممثل مثل بيلي بوب ثورنتون في فيلم كهذا هو بحق مضيعة لمواهبه. السبب الوحيد لصنع هذا الفيلم هو محاولة التفوق على فيلم ديب امباكت بطريقة ما. خلاصة الأمر، يوم الحساب فيلم فاشل. Armageddon This disaster flick is a disaster alright. Directed by Tony Scott (Top Gun), it's the story of an asteroid the size of Texas caught on a collision course with Earth. After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a comet shower, NASA detects said asteroid and go into a frenzy. They hire the world's best oil driller (Bruce Willis), and send him and his crew up into space to fix our global problem. The action scenes are over the top and too ludicrous for words. So much so, I had to sigh and hit my head with my notebook a couple of times. Also, to see a wonderful actor like Billy Bob Thornton in a film like this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep Impact. Bottom line is, Armageddon is a failure. يوم الحساب هذا الفيلم الكارثي هو بحق فيلم كارثي. الفيلم، من إخراج توني سكوت (مخرج فيلم توب جن)، يروي قصة كويكب بحجم ولاية تكساس في طريقه للاصطدام بالكرة الأرضية. بعد مشهد افتتاحي رائع، تدمر فيه سفينة فضاء أمريكية، إضافة إلى مدينة نيويورك، تكتشف ناسا الكويكب آنف الذكر ويجن جنونها. فيستعينون بأفضل منقب عن النفط في العالم (بروس ويليس)، ويرسلونه وطاقمه إلى الفضاء لإصلاح مشكلتنا العالمية هذه. مشاهد الإثارة والآكشن مبالغ بها ويصعب وصف سخفها بأي كلمات، إلى حد وجدت نفسي فيه أتنهد، بل وضربت جبهتي بمدونتي بضع مرات. و أيضا فإن رؤية ممثل مثل بيلي بوب ثورنتون في فيلم كهذا هو بحق مضيعة لمواهبه. السبب الوحيد لصنع هذا الفيلم هو محاولة التفوق على فيلم ديب امباكت بطريقة ما. خلاصة الأمر، يوم الحساب فيلم فاشل.


Download ppt "Both methods give significant improvements in accuracy over two strong baselines: Above curves were generated using rationales from a single annotator,"

Similar presentations


Ads by Google