1. Overview of the merger prototype

2. Overview Backgrounds: The MUMIS project Cross document annotation merging Alignment of parallel fragments Unification of aligned fragments Clean up unified fragments Reasoning Evaluation Future work & Conclusions

3. The MUMIS project Semantic access to a multimedia database.

4. The MUMIS project Semantic access to a multimedia database. Subject: Soccer

5. The MUMIS project Semantic access to a multimedia database. Corpus: Video recordings of matches, formal texts, ‘ticker’ texts.

6. The MUMIS project Semantic access to a multimedia database. Approach: Extract knowledge from textual sources Align this (time based) knowledge with video Do retrieval on annotation, returning corresponding video fragments to user

7. The MUMIS project Semantic access to a multimedia database. Main subject of this presentation: Merging the annotations resulting from separate texts into one cross-document annotation.

8. Merging Intention of merging: - start with various texts - annotate each text individually - combine annotations Example match: Netherlands – Yugoslavia (European Championship 2000)

9. Two types of text in merger: Formal texts Ticker texts

10. Example formal text Netherlands-Yugoslavia Final score: 6-1 Referee: Garcia Aranda Goals: 24' Patrick Kluivert 90' Marc Overmars 91' Savo Milosevic Substitutions: 53' out : Nisa Saveljic in : Jovan Stankovic 58' out : Patrick Kluivert in : Roy Makaay Yellow Cards: Paul Bosvelt

11. Example ticker text (BBC) 19 mins: Bergkamp scuffs his left-foot shot but still forces Kralj into a diving save low down to his left. 20 mins: Edgar Davids wastes the best chance of the game so far when he blazes over with just the goalkeeper to beat after being put through by Bergkamp. 24 mins: Kluivert puts Holland in front after latching onto a wonderful chip from Bergkamp and then planting a right- foot shot past Kralj from eight yards. 25 mins: Boudewijn Zenden comes close to doubling Holland's lead when he fires in low, right-foot shot which Kralj just about hangs onto.

12. Example of parallel fragments BBC - 15: Van der Sar pulls of great save to block Mijatovic's shot after Savo Milosevic has cut through the Dutch defence like a knife. Guardian - 17: Mijatovic, played in with a quick square ball from Milosevic, finds himself one-on-one with van der Sar 10 yards out. He picks his spot, but unfortunately for Mijatovic, it's the spot occupied by van der Sar. A great save and Yugoslavia should be one-nil up. Kickers 15: Milosevic auf Mijatovic, doch der Stuermer vom AC Florenz scheitert aus 12 Metern freistehend an van der Sar. WEBTEC 15: Milosevic filtreert door de Nederlandse defensie door één beweging en legt af voor Mijatovic. Deze laatste trapt op van der Sar.

13. Example of parallel fragments BBC - 15: Van der Sar pulls of great save to block Mijatovic's shot after Savo Milosevic has cut through the Dutch defence like a knife. Guardian - 17: Mijatovic, played in with a quick square ball from Milosevic, finds himself one-on-one with van der Sar 10 yards out. He picks his spot, but unfortunately for Mijatovic, it's the spot occupied by van der Sar. A great save and Yugoslavia should be one-nil up. Kickers 15: Milosevic auf Mijatovic, doch der Stuermer vom AC Florenz scheitert aus 12 Metern freistehend an van der Sar. WEBTEC 15: Milosevic filtreert door de Nederlandse defensie door één beweging en legt af voor Mijatovic. Deze laatste trapt op van der Sar.

14. Merging process: overview 2 document alignment N-document alignment Unification of events from separate sources Special situations

15. Merging process: 2-document alignment Step 1 of the merging process: merge annotations of 2 texts

16. Merging process: 2-document alignment Source ASource B

17. Merging process: 2-document alignment The strongest binding is selected, ruling out certain other bindings.

18. Merging process: 2-document alignment The strongest binding is selected, ruling out certain other bindings.

19. Merging process: 2-document alignment The strongest binding is selected, ruling out certain other bindings.

20. Merging process: 2-document alignment The strongest binding is selected, ruling out certain other bindings.

21. Merging process: N-document Given the 2-document alignments for each pair of sources, find the n-document alignment where all fragments describing same scene in all separate sources are aligned.

22. Merging process: N-document

25. Merging process: Unification of scenes

26. Merging and reasoning: types of rules Within events or scenes: Player1 and Player2 will not be the same person, a player performing a save will not score a goal in the same scene, etc. Role of teams and events: offensive vs. defensive Combinations of events that probably have the same player: ShotOnGoal+Goal, Penalty+HitThePost Terminology of authors may vary: Cross—Pass, Save—Clearance

27. Merging and reasoning: example rules

29. Reasoning: mistakes in IE Sometimes the information extraction component makes mistakes. Example rules have been applied to solve some of these.

30. Reasoning: mistakes in IE Fix: The goal made by Kralj (Yugoslavian keeper) is removed

31. Evaluation: What do we want to know? Quality of the merger in itself The advantages and disadvantages of merging

32. Evaluation: Quality of the merger Quality of alignments Quality of unification The effect of the quality of the original information extraction on both

33. Evaluation: Approach Create gold standard annotations for single sources Create gold standard merged annotation of all sources Run merger in different conditions Compare everything with everything

34. Evaluation: Results Alignments based on machine IE Version 1Version 2Version 3 Manual210 Automatic104187189 Overlap82172 Precision78.892.091.0 Recall39.081.9

35. Evaluation: Results Alignments based on manual IE Version 1 Manual210 Automatic188 Overlap174 Precision92.6 Recall82.9

36. Evaluation: Conclusions Quality of alignments is pretty good. Better IE improves alignments. Low quality IE does not degrade alignments too much.

37. MORE TO COME….

38. ----- Extra Sheets -----

40. Extra example – 15 th min.

41. Extra example – graph

42. Extra example – unification

43. Extra example – the source BBC - 15: Van der Sar pulls of … Milosevic has cut … Guardian - 17: Mijatovic, played in with a quick square ball from Milosevic, finds himself one-on-one with van der Sar 10 yards out. He picks his spot, but unfortunately for Mijatovic, it's the spot occupied by van der Sar. A great save and Yugoslavia should be one-nil up. Kickers 15: Milosevic auf Mijatovic, doch der Stuermer vom AC Florenz scheitert aus 12 Metern freistehend an van der Sar. WEBTEC 15: Milosevic filtreert door de Nederlandse defensie door één beweging en legt af voor Mijatovic. Deze laatste trapt op van der Sar. Pass Milosevic ShotOnGoal Mijatovic Save Van der Sar

45. Reasoning: incomplete graphs

54. Reasoning: own goal

56. Reordering Observation from corpus: Scenes in correct order Events within scenes often in wrong order

57. Reordering Manual annotation of several matches Pass, Shot-on-goal, Goal Pass, Shot-on-goal, Save Shot-on-goal, Hitting-the-post Foul, Free-kick, Shot-on-goal, Corner

58. Reordering

59. Not fully implemented yet in the merger.