1. MEANT: semi-automatic metric for evaluating for MT evaluation via semantic frames an asembling of ACL11,IJCAI11,SSST11 Chi-kiu Lo & Dekai Wu Presented by SUN Jun

2. MT’s often bad MT3: So far, the sale in the mainland of China for nearly two months of SK – II line of products MT1: So far, nearly two months sk –ii the sale of products in the mainland of China to resume sales. MT2: So far, in the mainland of China to stop selling nearly two months of SK – 2 products sales resumed. Ref: Until after their sales had ceased in mainland China for almost tow months, sales of the complete range of SK – II products have now been resumed. BLEU: 0.124 BLEU: 0.012 BLEU: 0.013

3. Metrics besides BLEU have problems Lexical similarity based metrics (eg. NIST, METEOR) – Good at capturing fluency – Correlate poorly with human judgment on adequacy Syntax based (eg. STM, Liu and Gildea, 2005) – Much better at capturing grammaticality – Still more fluency oriented than adequacy-oriented Non-automatic metrics (eg. HTER) – Use human annotators to solve non-trivial problem of finding min edit distance to evaluate adequacy – Human-training & Labor intensive

4. ＭＥＡＮＴ： SRL for MT evaluation Intuition behind the idea: – Useful translation help users accurately understand the basic event structure of source utterances—“ who did what to whom, when, where and why”. Hypothesis of the work: – MT utility can best be evaluated via SRL – Better than: N-gram based metrics like BLEU (adequacy) Human training intensive metrics like HTER (time cost) Complex aggregate metrics like ULC (representation transparency)

5. Q. Do PRED & ARG j correlate to human adequacy judgments? N-gramMatching #Syntax- subtree Matching #SRLMatching # 1-gram151-level34Predicate0 2-gram42-level8 3-gram33-level2 4-gram14-level0

6. Q. Do PRED & ARG j correlate to human adequacy judgments? N-gramMatching #Syntax- subtree Matching #SRLMatching # 1-gram15 1-level3534Predicate20 2-gram442-level68Argument10 3-gram133-level12 4-gram014-level00

7. Experimental settings Exp settings 1 -- Corpus – ACL11: draw 40 sentences from Newswire datasets in GALE P2.5 (with SRL in ref/src, 3- output) – IJCAI11: draw 40, draw 35 from previous data set and draw 39 from broadcast news WMT2010- MetricsMaTr

8. Experimental settings Exp settings 2 – Annotation of SRL on MT reference and output – SRL: Propbank style

9. Experimental settings Exp settings 3 –SRL evaluation as MT evaluation – Correct, incorrect, partial (predicate & argument) Partial: part of the meaning is correctly translated Extra meaning in a role filler is not penalized unless it belongs in another role Incorrectly translated predicate means the entire frame is wrong (no count of arguments)

10. Experimental settings Exp settings 3 –SRL evaluation as MT evaluation – F-measure Based scores – weights tuned by confusion Matrix on dev

11. Experimental settings Exp settings 4 – Evaluation of evaluation – WMT and NIST MetricsMaTr (2010) – Kendall’s τ rank correlation coefficient evaluate the correlation of the proposed metric with human judgments on translation adequacy ranking. A higher value for τ indicates more similarity to the ranking by the evaluation metric to the human judgment. The range of possible values of correlation coefficient is [- 1,1], where 1 means the systems are ranked

12. Observations HMEANT vs other metric

13. Observations HMEANT on CV data

14. Observations HMEANT annotated via Mono vs Bi-lingual Error analysis: annotators drop parts of the meaning in the translation when trying to align them to the source input

15. Observations HMEANT vs MEANT (automatic SRL) – SRL tool: ASSERT, 87% (Pradhan et al. 2004) 80 %

16. Q2: Impact of each individual semantic role to the metric’s correlation A preliminary exp – For each ARGj, PRED, we manually compared each English MT output against its reference translation. Using the counts thus obtained, we computed the precision, recall, and f-score for PRED and each ARGj type.

17. IJCAI 11: evaluation the individual impact The preliminary exp suggest effectiveness – Propose metrics for evaluating individual impact

18. IJCAI 11: evaluation the individual impact The preliminary exp suggest effectiveness

19. IJCAI 11: evaluation the individual impact Results

20. IJCAI 11: evaluation the individual impact Results2 Automatic SRL tool 76-93%

21. Q: Can be Even more Accurate? SSST11...

22. Conclusion ACL11 – Bring MEANT, HMEANT – HMEANT correlates well to human judges, as well as more expensive HTER – Automatic SRL yields 80% correlations IJCAI11 – Study impact of each individual semantic roles SSST11 – Propose Length based weighting scheme to evaluate contribution of each semantic frame

23. END