Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2011 IBM Corporation Semantic Technology in Watson Chris Welty IBM Research ibmwatson.com.

Similar presentations


Presentation on theme: "© 2011 IBM Corporation Semantic Technology in Watson Chris Welty IBM Research ibmwatson.com."— Presentation transcript:

1 © 2011 IBM Corporation Semantic Technology in Watson Chris Welty IBM Research ibmwatson.com

2 © 2011 IBM Corporation What is Watson? Open Domain Question-Answering Machine Given –Rich Natural Language Questions –Over a Broad Domain of Knowledge Delivers –Precise Answers: Determine what is being asked & give precise response –Accurate Confidences: Determine likelihood answer is correct –Consumable Justifications: Explain why the answer is right –Fast Response Time: Precision & Confidence in <3 seconds –At the level of human experts –Proved its mettle in a televised match –Won a 2-game Jeopardy match against the all-time winners –viewed by over 50,000,000 2

3 © 2011 IBM Corporation $200 If you are looking at the wainscoating, you are looking in this direction. $1000 The first person mentioned by name in The Man in the Iron Mask is this hero of a previous book by the same author. 3 The Jeopardy! Challenge H ard for humans, hard for machines Broad/Open Domain Complex Language High Precision AccurateConfidence HighSpeed $600 In cell division, mitosis splits the nucleus & cytokinesis splits this liquid cushioning the nucleus $800 The conspirators against this man were wounded by each other while they stabbed at him But hard for different reasons. For people, the challenge is knowing the answer For machines, the challenge is understanding the question What is down? Who is DArtagnan? What is cytoplasm? Who is Julius Caesar?

4 © 2011 IBM Corporation The Watson Enablers J!-Archive: the complete 30-year history of the Jeopardy! show, every question, every answer (200,000 questions) Wikipedia: 92% of Jeopardy questions have answers in Wikipedia. 81% of the answers are Wikipedia titles Metric: after a 4-month study of the problem we devised a satisfactory metric that we strongly believed was indicative of our chances of winning a game Machine Learning, ensemble methods: The emergence, availability, and stability of software for ML Semantic Web, linked data …and now we realize –Jeopardy! questions have only one correct answer –Almost every answer is expressed in one sentence

5 © 2011 IBM Corporation The Winners Cloud W hat It Takes to compete against Top Human Jeopardy! Players Winning Human Performance 2007 QA Computer System Grand Champion Human Performance Each dot – actual historical human Jeopardy! games More Confident Less Confident

6 © 2011 IBM Corporation 2007 TREC QA System In 2007, we committed to making a Huge Leap! More Confident Less Confident Each dot – actual historical human Jeopardy! games Computers? Not So Good. Winning Human Performance Grand Champion Human Performance The Winners Cloud W hat It Takes to compete against Top Human Jeopardy! Players

7 © 2011 IBM Corporation Baseline 12/2007 8/2008 5/ / / /2008 Steady Progress 5/2008 4/2010 Precision.87!

8 © 2011 IBM Corporation In May 1898 Portugal celebrated the 400th anniversary of this explorers arrival in India. On the 27 th of May 1498, Vasco da Gama landed in Kappad Beach DO: Find relevant passages that may contain an answer DO: Match or align the passage with the question using diverse evidence DONT: Translate the question into a formal query and look up the answer In May, Gary arrived in India after he celebrated his anniversary in Portugal. Celebration(e), date(e,1898), celebrationOf(e,e1), location(e, Portugal), date(e1, dateOf(e) – 400)), arrival(e1), location(e1,India), particpantOf(e1,?x). Location(e2, Kappad Beach), Date(e2, 1498), landing(e2), particpantOf(e1,Vasco). What we do & dont

9 © 2011 IBM Corporation Matching Keyword Evidence celebrated India In May th anniversary arrival in Portugal India In May Gary explorer celebrated anniversary in Portugal Keyword Matching 9 arrived in In May, Gary arrived in India after he celebrated his anniversary in Portugal. In May 1898 Portugal celebrated the 400th anniversary of this explorers arrival in India. Evidence suggests Gary is the answer BUT the system must learn that keyword matching may be weak relative to other types of evidence

10 © 2011 IBM Corporation On 27th May 1498, Vasco da Gama landed in Kappad Beach celebrated May th anniversary arrival in In May 1898 Portugal celebrated the 400th anniversary of this explorers arrival in India. Portugal landed in27th May 1498 Vasco da Gama Temporal Reasoning Passage Paraphrasing GeoSpatial Reasoning explorer On 27th May 1498, Vasco da Gama landed in Kappad Beach On the 27 th of May 1498, Vasco da Gama landed in Kappad Beach Kappad Beach Para- phrase s Geo- KB Date Math 10 India Stronger evidence can be much harder to find and score. The evidence is still not 100% certain. Search Far and Wide Explore many hypotheses Find Judge Evidence Many inference algorithms Search Far and Wide Explore many hypotheses Find Judge Evidence Many inference algorithms Matching Deeper Evidence

11 © 2011 IBM Corporation Answer & Confidence Question Evidence Sources Models Primary Search Candidate Answer Generation Hypothesis Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Question & Topic Analysis Evidence Retrieval Evidence Scoring Question Decomposition Question Decomposition Synthesis Watsons Architecture

12 © 2011 IBM Corporation Model Selection Q: Annexation of this in Use different feature weights Answer & Confidence Question Evidence Sources Models Primary Search Candidate Answer Generation Hypothesis Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Question & Topic Analysis Evidence Retrieval Evidence Scoring Question Decomposition Question Decomposition Synthesis Using Structured Data and Inference LAT Inference Q: Annexation of this in (Using PRISMATIC) this Region Relation Detection and Scoring Using Structured KBs Q: This 1997 Titanic hero.. matches Answer Typing (Type Coercion) LAT: Scottish Inventor Answer: James Watt Anti-Type Coercion LAT: Country Candidate: Einstein Spatial Reasoning Containment (This African country..) Relative direction (This sea east of Florida..) Border (This state bordering the Great Lakes..) Relative location (bldg. near Times Square..) Numeric Properties: area/population/height (This sea, largest in area,..) Temporal Reasoning Lifespan, Duration Answer In Clue Q: In 2003, Big Blue acquired this company.. Downweigh IBM Evidence Diffusion Q: Sunan Intl. Airport is in this country Diffuse evidence from (Pyongyang ->> N Korea)

13 © 2011 IBM Corporation Model Selection Q: Annexation of this in Use different feature weights Answer & Confidence Question Evidence Sources Models Primary Search Candidate Answer Generation Hypothesis Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Question & Topic Analysis Evidence Retrieval Evidence Scoring Question Decomposition Question Decomposition Synthesis LAT Inference Q: Annexation of this in (Using PRISMATIC) this Region Relation Detection and Scoring Using Structured KBs Q: This 1997 Titanic hero.. matches Spatial Reasoning Containment (This African country..) Relative direction (This sea east of Florida..) Border (This state bordering the Great Lakes..) Relative location (bldg. near Times Square..) Numeric Properties: area/population/height (This sea, largest in area,..) Temporal Reasoning Lifespan, Duration Answer In Clue Q: In 2003, Big Blue acquired this company.. Downweigh IBM Evidence Diffusion Q: Sunan Intl. Airport is in this country Diffuse evidence from (Pyongyang ->> N Korea) Using Structured Data and Inference Answer Typing (Type Coercion) LAT: Scottish Inventor Answer: James Watt Anti-Type Coercion LAT: Country Candidate: Einstein

14 © 2011 IBM Corporation Role of Answer Typing in QA Type Information - a crucial hint to get the correct answer ASTRONOMY: In 1610 Galileo named the moons of this planet for the Medici brothers Telescope Giovanni Medici Sidereus Nuncius Jupiter Ganymede Telescope (Instrument) Telescope (Instrument) Giovanni Medici (Person) Giovanni Medici (Person) Sidereus Nuncius (Book) Sidereus Nuncius (Book) Jupiter (Planet) Jupiter (Planet) Ganymede (Moon) Ganymede (Moon) Terms Associated with Clue Context (e.g. via Keyword Search)

15 © 2011 IBM Corporation Broad Domain Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text. We do NOT attempt to anticipate all questions and build databases. We do NOT try to build a formal model of the world We do NOT try to build a formal model of the world

16 © 2011 IBM Corporation JFK (Cand) Problem: Compute type match for candidate w.r.t. LAT – Both candidate and LAT expressed as Strings – 4 Steps: EDM (Entity Disambiguation and Matching), PDM (Predicate Disambiguation and Matching), TR (Type Retrieval), TA (Type Alignment) TyCor Framework EDM: Candidate Instance Wikipedia:John_F_Kennedy_International (0.7) TR: Instance Type PDM: LAT Type TA: Compare LAT-type and Instance-type facility (LAT) WN:Airport (1.0) Airport is-a Facility (1.0) TyCor Match ! (0.63) WN:Facility (0.9)

17 © 2011 IBM Corporation EDM Issue 1 (Entity Matching): Many different ways to refer to the same entity (spellings, aliases, nicknames, abbreviations) Issue 2 (Entity Disambig): Sense Disambiguation depends on context Entity Disambiguation and Matching Problem Fundamental Task in NLP: Map entity string to source identifier (dbpedia URI, UMLS cui, etc.) Flight took off from JFK… JFK was assassinated… Film critics loved JFK… Lincoln Abe Lincoln President Lincoln

18 © 2011 IBM Corporation More on EDM For Matching (aka Identification) Wikipedia redirects (Myanmar ->> Burma) Synonyms / aliases extracted from text IBMs distinctive culture and product branding has given it the nickname Big Blue Anchor-Hyperlink Data Newton Heath LYR Football Club MUFC Man Utd. Manchester United FC Wikipedia Page For Disambiguation Wikipedia Disambiguation Pages (wide coverage) -~150K disambiguation pages in E.g. Java has >50 different senses spanning >20 Distinct Types Do not need WordNet Synsets for Instances (poor coverage) Resources Used In Watson: 16 links 276 links 1855 links

19 © 2011 IBM Corporation EDM Approaches Basic EDM Algorithm (2010) –Input: Entity String –Output: Set of Wikipedia URIs ranked along following criteria 1.Exact Match to WP Page Title 2.Redirect to WP Page 3.{Disambiguations, Link Anchors, Mined Synonym Lists} Entities in (3) sorted by popularity E.g. Emerson Relevance Context-Sensitive EDM (2011) Use context to disambiguate entities -Question text (BoW) as context for question entities, match against: -Wikipedia page text for dbPedia entities (BoW) -Label, variant, and relational neighbors for Freebase, GeoNames - Look at relations involving entity (JFKs brother Teddy -> JFK hasBrother TeddyKennedy) - Train & Evaluate against wikiPedia link-anchor text Note: component level improvements reach a effort/effect saturation. This did not improve Jeopardy! performance and was resource-intensive – not used in J! Watson F1:.83 F1:.89

20 © 2011 IBM Corporation Type Retrieval Obtain Types for Instances returned by EDM Taxonomies Used In Watson: – WordNet – Yago Ontology (from DBpedia) – Wikipedia Categories – Auto-Mined Types from Text (Prismatic) RECALLPRECISION Interesting Points – Type Systems are linked Yago WordNet – Wiki-Categories contain extra information (modifiers) Einstein : German-Inventor, Swiss-Vegetarian, Patent-Examiner – Automatically Mined Types reflect real world usage Fluid -is-a- Liquid (strictly speaking incorrect)

21 © 2011 IBM Corporation PDM Predicate Disambiguation and Matching (PDM) Problem (basically WSD) – LAT: star Similar in principle to EDM – EDM – map named entity instance – PDM – map generic noun class/type In the northern hemisphere, latitude is equal to the angle above the horizon of this star, Alpha Ursae Minoris This star of "The Practice" played Clint Eastwood's Secret Service partner in the film "In the Line of Fire" PDM in Watson: – Lookup LAT in WordNet: Order concepts based on sense ranks – Special Case: Domain-specific PDM (manually specified) Mapped top-200 LATs in Jeopardy! to specific concept senses

22 © 2011 IBM Corporation Broad Domain Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text. We do NOT attempt to anticipate all questions and build databases. We do NOT try to build a formal model of the world We do NOT try to build a formal model of the world

23 © 2011 IBM Corporation TA Type Matching/Alignment Problem – Compare candidate types with LAT types – Produce a score depending degree of Match Various Types of Match Considered LAT-Type: Candidate-Type: Airport LAT-Type: Air Field Subclass Match (1.0) Sibling Match (0.5) Candidate-Type: Aerodrome Deep LCA Match (0.25) LAT-Type: TrainStation Disjoint Types (-1.0) Airfield Facility

24 © 2011 IBM Corporation Putting it all together TyCor Score = EDM * TR * PDM * TA An-TyCor - When TA score is -1 (Disjoint Types) AnTyCor Feature added to model - Strong negative signal against candidate - Helps rules out candidates of wrong type (e.g. LAT: Country, Candidate: Einstein) Multiple LATs - When multiple LATs in question with confidences: (L1, L2..Ln) - Final TyCor Score (weighted-sum) = (L1 * Tyc1) + (L2 * Tyc2) +.. (Ln * Tycn) Intermediate Failure - If any step fails, Tycor Score = 0 (consider smoothing) - Expose which step failed to final model (EDM-Failure, PDM-Failure…) TyCor Algorithm Suite in J! Watson -14 TyCors Developed - All TyCors follow 4 steps - Each TyCor score is a separate feature in model - Model learns weights on diff. TyCors: balances/combines type information - Trained on QA ground truth, not type ground truth

25 © 2011 IBM Corporation TyCor Impact on Jeopardy! clues + ~10%

26 © 2011 IBM Corporation Coverage Issues Knowledge-bases (e.g. dbPeda, Freebase) are imperfect –80% of Jeopardy answers were in dbPedia –15% of the name variants were missing –Errors found in 10% of entries (names, improper classification, incorrect relations) –Typing relations useful in 85% of Jeopardy! questions –all other relations combined to impact fewer than 15% of questions The fundamental barrier: –Given a knowledge graph triple P(triple is useful) = P(X is mentioned in a q) * P(one of Xs known variants was used in q) * P(R is useful information for answering q) * P(Y is the answer or leads to the answer) * P(one of Ys known variants is used in answer evidence) –Given a q about some X What nation borders S. Korea P(there is a useful triple) = P(X is in the graph) * P(X variant used in q is known) * P(the q requires a relation R that is in the graph) * P(R can be recognized in q) * P(X is connected by R to Y) * P(Y is useful for answering q) * P(one of Ys known variants is used in answer evidence We did not fix any sources, we needed accurate models of impact on Jeopardy! EDM is the gateway to exploiting structured sources For Medical QA, EDM is a major bottleneck XY R


Download ppt "© 2011 IBM Corporation Semantic Technology in Watson Chris Welty IBM Research ibmwatson.com."

Similar presentations


Ads by Google