1 Language Technologies (2) Valentin Tablan University of Sheffield, UK ACAI 05 ADVANCED COURSE ON KNOWLEDGE DISCOVERY

2 Overview Examples of HLT for the Semantic Web in use Work in context of EU SEKT and PrestoSpace projects Mixed Initiative Information Extraction RichNews (automated annotation of news programs)

3 Mixed Initiative IE -Using Machine Learning for Information Extraction -Human annotator and the system can take the initiative -HA provides some bootstrap examples -MI Engine learns and starts suggesting annotations -HA corrects these annotations -And so on…

4 What is Mixed Initiative IE? Also known as adaptive IE Not active learning ! System selects the next document to be annotated by the user. Improves the performances Active learning is not a part of MI API but will use it

5 Requirements A MI engine must : Work as a background task Suggest annotations only when a given performance level is reached Be easily usable for a non – expert user Fined grained parameters for experts

6 OBIE example Find instances in a document of entities and relations from an ontology Usable by a non-expert end user No learning corpus available Quick adaptation to a new ontology

7 Specifics of the MI API Train a statistical model Use several ML algorithms SVM – Decision Trees – Neural Nets – etc … Compare the ML models and use the one which performs the best at time t Combine the ML models

8 Expected behaviour Performance Time – Size of learning corpus Engine 2 Engine 3 Engine 1 MI Engine Minimal performance level tolerated

9 Limitations of the ML API Configure a file per engine > not suitable for a non expert Set the class definition in the file > problem for OBIE : ontology is not NE > dynamic settings Engine characteristics : binary, numeric, nominal > uniform declaration, automatic conversion Operate on tokens > cannot annotate spans One class per engine > how to set several possible values for an entity with a binary engine?

10 Meta Engine Combines several instances of ‘simple’ engines (has to be same engine type e.g Maxent) Accepts rich descriptions of class & attributes Converts into suitable format for ‘simple’ engine Merges results of embedded engines Behaves like a simple engine Hides the dirty job

11 MI API Architecture Mixed Initiative Engine Mixed Initiative API GUI / client code Meta Engine Orchestrator Evaluation Module DataSet

12 Data Set Information stored as examples No documents Used by Meta Engines Possibly converted to a native Data Set format (e.g. SVMlight) Possibly reuse an existing implementation (WEKA, Yale, …)

13 MI API Architecture Mixed Initiative Engine Mixed Initiative API GUI / client code Meta Engine Orchestrator Evaluation Module DataSet

14 Evaluation module Operate on Data Set Choice for corpus splitting (has K-fold cross validation) Different evaluation metrics

15 MI API Architecture Mixed Initiative Engine Mixed Initiative API GUI / client code Meta Engine Orchestrator Evaluation Module DataSet

16 Orchestrator Core of a MI Engine Manages the Meta Engines Uses the Data Set and Evaluation Module Return information about the Meta Engines confusion matrix – performances – etc … Combines the ML models Convert from / to annotations

17 Allegory : MI Engine = Orchestra Music School ME Teacher Orchestra Conductor ME 1- Learn one/some/all instruments (entities)? 2- Exams for all at the same time ? 3- Good enough and better than existing orchestra? 1- Combine their skills 2- Play for an audience

18 Summary of MI IE Required component for Ontology based Information Extraction State-of-the-art functionalities Reach high performance level by combining classification algorithms

19 RichNews RichNews aims to automate the annotation of news programs Start from recordings of broadcasts. Produces annotations that can be included in a semantic repository (i.e. KIM/Sesame) Works for English but most processing resources can be adapted for other languages.

20 Key Problems Speech recognition produces poor quality transcripts with many mistakes. A news broadcast contains several stories. How do we work out where one starts and another one stops? How can we make a summary or headline for each story from a poor quality transcript? How can we work out what kind of news it reports?

21 Augmented Television News New Broadcasts are often augmented with textual content. –Usually only limited content is available. –The TV company controls content production. Rich News finds content automatically. –Developed on BBC news. –Recordings of broadcasts go in one end. –Relevant news web pages are associated with the stories in the broadcasts fully automatically.

22 Semantic Indexing of News Systems already exist that can index news broadcasts in terms of ‘named entities’ that they refer to. –e.g. Mark Maybury’s Broadcast News Navigator. –Entities such as cities, people, organizations are marked as such. Rich News can improve annotation: –Annotation is in terms of an ontology. –Uses Automatic Speech Recognition, so can be applied when no subtitles are available. –Web pages are used to help find named entities.

23 Using ASR Transcripts ASR is performed by the THISL system. Based on ABBOT connectionist speech recognizer. Optimized specifically for use on BBC news broadcasts. Average word error rate of 29%. Error rate of up to 90% for out of studio recordings.

24 SA General Architecture Source Extractor... Media Object Information Source IE Semantic Index Multi-source IE Merger (?) Story Segmentation Source Detection

25 Multi-source IESource Detection RichNews Architecture... Media Object Story Segmenter Story 1 Story 2 Story N ASR Transcript ASR IE System Web Miner Related Web Pages KIM Ontological IE System Entity 2 Instance Semantic Index THISL ASR GATE/ELAN Manual Annotation (optional)...

26 Topical Segmentation Uses C99 segmenter: Removes common words from the ASR transcripts. Stems the other words to get their roots. Then looks to see in which parts of the transcripts the same words tend to occur.  These parts will probably report the same story.

27 Key Phrase Extraction Term frequency inverse document frequency (TF.IDF): Chooses sequences of words that tend to occur more frequently in the story than they do in the language as a whole. Any sequence of up to three words can be a phrase. Up to four phrases extracted per story.

28 Web Search The Key-phrases are used to search on the BBC, and the Times, Guardian and Telegraph newspaper websites for web pages reporting each story in the broadcast. Searches are restricted to the day of broadcast, or the day after. Searches are repeated using different combinations of the extracted key-phrases. The text of the returned web pages is compared to the text of the transcript to find matching stories.

29 Evaluation Success in finding matching web pages was investigated. Evaluation based on 66 news stories from 9 half-hour news broadcasts. Web pages were found for 40% of stories. 7% of pages reported a closely related story, instead of that in the broadcast. Results are based on earlier version of the system, only using BBC web pages.

30 Using the Web Pages Web pages can be made available to the viewer as additional content. The web pages contain: A headline, summary and section for each story. High quality text that is readable, and contains correctly spelt proper names. They give more in depth coverage of the stories. Web pages could be included in the broadcast by the TV company. Or discovered by a device in viewers’ homes.

31 Semantic Annotation KIM can semantically annotate the text derived from the web pages: KIM will identify people, organizations, locations etc. KIM performs well on the web page text, but very poorly when run on the transcripts directly. This allows for semantic ontology-aided searches for stories about particular people or locations etcetera. So we could search for people called Sydney, which would be difficult with a text-based search.

32 Search for Entities

33 Story Retrieval

34 Summary of RichNews Rich News can automatically segment, describe and classify news broadcasts: Requires an on-line textual source that closely parallels the broadcasts. High precision, moderate recall (so far). Easy to adapt to other languages.