1. ResPubliQA IR baselines and UNED participation Álvaro Rodrigo Joaquín Pérez Anselmo Peñas Guillermo Garrido Lourdes Araujo nlp.uned.es

2. IR Baselines for ResPubliQA Paragraph retrieval, high precision, P@1 Don’t use weighting schemas tuned for document retrieval Reduce the effect of term frequency Reduce the effect of normalization by document length There are some paragraphs with a couple of words

3. IR Baselines for ResPubliQA Okapi BM25 k 1  [0,∞) b  [0,1] k 1 =0.1, minimize the effect of term frequency b=0.6, give some value to paragraphs longer than the average Empirical tuning over the English development collection

4. IR Baselines for ResPubliQA SystemBGDEENESFRITPTRO icia092 0.68 nlel092 0.47 uned092 0.610.41 uned091 0.60.41 icia091 0.58 nlel091 0.580.35 0.52 uaic092 0.54 0.47 loga091 0.44 loga092 0.44 elix092 0.48 uaic091 0.44 0.45 elix091 0.42 mira091 0.32 mira092 0.29 iles091 0.28 syna091 0.28 0.23 isik091 0.25 iiit091 0.2 base0920.38 0.530.40.450.420.490.44 base0910.380.350.510.330.39 0.460.37 Better with Stemming Systems over Baselines Systems under Baselines

5. UNED system Question Paragraph Retrieval BM25 b=0.6 k=0.1 Paragraph Validation 100 candidates NER Acronyms Expected Answer Type Accepted candidates Paragraph Selection Answer N-gram based similarity + Lexical entailment

6. Paragraph Validation Expected Answer Type matching Two classifications Coarse: named, numeric, other Fine: location, organization, person, definition, count, time, other No much difference in performace NER Only accept paragraphs with all NE of the question Too restrictive, bad performance

7. Paragraph Validation Acronym checking Applied to definitions What does NATO stand for? Good performance in Spanish but not in English If no paragraph is accepted by the validation step -> NoA Bad performance

8. Paragraph Selection N-gram based similarity Setting 1 Count overlapping of word sequences (1-5) Setting 2 + Consider lexical entailment in the matching Good performance Promising results adding lexical entailment

9. Results Systemc@1Accuracy#R#W#NoA#NoA R #NoA W #NoA empty combination0.9 451490000 uned092enen0.61 288184281512 1 uned091enen0.60.59282190281513 0 nlel091enen0.580.57287211200 2 uaic092enen0.540.52243204531835 0 base092enen0.53 263236110 0 base091enen0.51 256243101 0 elix092enen0.48 240260000 0 uaic091enen0.440.42200253471136 0 elix091enen0.42 211289000 0

10. Conclusion Good IR performance Significant improvement in English N-gram based selection Not in Spanish We need to improve paragraph validation