1. Watson Robert Yates Watson Core Development

2.  A brief History of Watson  What is it good for?  How does it work?  Current Focus Agenda

3. A Brief History of Watson

4. Deep Blue Ended in 1997 Looking for a new research challenge 2004, IBM Research manager Charles Lickel, over dinner with coworkers, noticed that the restaurant they were in had fallen silent Ken Jennings Started in Research in 2005 Won Jeopardy In Feb 2011 SW Division in Aug 2011

6. What is it good for?

7. 7 Watson’s real value proposition: Efficient decision support over unstructured (and structured) content Unstructured Data  Broad, rich in context  Rapidly growing, current  Invaluable yet under utilized SQL/XQuery Existing BI Inference/Rules Structured Data  Precise, explicit  Narrow, expensive Jeopardy! Challenge Deeper Understanding but Brittle High Precision at High Cost Narrow Limited Coverage Shallow Understanding Low Precision Broad Coverage Deeper Understanding, Higher Precision and Broader, Timely Coverage at lower costs Key Word Search Relevance Ranking Open-Domain Question-Answering

8. How does it work?

9. Learning To Rank Systems  Training data is used by a learning algorithm to produce a ranking model which computes relevance of responses for actual queries.  Typically, users expect a search query to complete in a short time, which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used. First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation. In the second phase, a more accurate but computationally expensive machine-learned model is used to re- rank these documents.

10. Learning To Rank Systems  Training data is used by a learning algorithm to produce a ranking model which computes relevance of responses for actual queries.  Typically, users expect a search query to complete in a short time, which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used. First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation. In the second phase, a more accurate but computationally expensive machine-learned model is used to re- rank these documents.

11. Learning To Rank (Basic Architecture) Final Merging Ranking Hypothesis Evidence Scoring Search Results Trained Models Hypothesis Evidence Scoring Hypothesis Evidence Scoring Keywords

12. Scoring Learning To Rank (Watson) Final Merging Ranking Scoring Question Answer, Confidence, Evidence Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring

13. Who is the 44 th President of the United States ?

14. Question Analysis Who is the 44 th President of the United States?

15. Question Analysis Who is the 44 th President of the United States? 'Who' is the '44 th' 'President' of the 'United States'? Lexical Answer Type → Person Focus Can be replaced by the correct answer to make a true statement Keywords

16. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

17. Who is the 44 th President of the United States? Question Primary Search Question Analysis Keywords: 44 th President United States

18. Who is the 44 th President of the United States? Question Primary Search Question Analysis Keywords: 44 th President United States

19. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

20. Question Candidate Answer Generation Primary Search Question Analysis Who is the 44 th President of the United States?

21. Question Candidate Answer Generation Primary Search Question Analysis Barack Obama George W. Bush Harvard Law School Illinois Who is the 44 th President of the United States?

22. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

23. Answer Scoring Answer Scoring Question Contextual Answer Scoring Answer Scoring SearchQuestion Analysis Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush Who is the 44 th President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Answer Scoring Answer Scoring

24. Answer Scoring Answer Scoring Question Contextual Answer Scoring Answer Scoring SearchQuestion Analysis Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush Who is the 44 th President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Answer Scoring Answer Scoring Who is the 44 th President of the United States? → Person

25. Answer Scoring Answer Scoring Question Contextual Answer Scoring Answer Scoring SearchQuestion Analysis Who is the 44 th President of the United States? Barack Obama Who is the 44 th President of the United States? George W. Bush Who is the 44 th President of the United States? Harvard Law School Who is the 44 th President of the United States? Illinois Answer Scoring Answer Scoring Who is the 44 th President of the United States? → Person Is Barack Obama a Person?.90 Is George W. Bush a Person?.90 Is Harvard Law School a Person?.10 Is Illinois a Person?.15

26. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

27. Scoring Question Contextual Answer Scoring SearchQuestion Analysis Contextual Answer Scoring Contextual Answer Scoring Who is the 44 th President of the United States?

28. Scoring Question Contextual Answer Scoring SearchQuestion Analysis Contextual Answer Scoring Contextual Answer Scoring Who is the 44 th President of the United States? Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States

29. Scoring Question Contextual Answer Scoring SearchQuestion Analysis Contextual Answer Scoring Contextual Answer Scoring Who is the 44 th President of the United States? Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States

30. Scoring Question Contextual Answer Scoring SearchQuestion Analysis Contextual Answer Scoring Contextual Answer Scoring Who is the 44 th President of the United States? Barack Obama is the 44 th President of the United States George W. Bush is the 44 th President of the United States Harvard Law School is the 44 th President of the United States Illinois is the 44 th President of the United States Barack Hussein Obama II (i/bə ˈ r ɑː k hu ːˈ se ɪ n o ʊˈ b ɑː mə/; born August 4, 1961) is the 44th and current President of the United States.i/bə ˈ r ɑː khu ːˈ se ɪ no ʊˈ b ɑː mə/44thcurrentPresident of the United States George Walker Bush (born July 6, 1946) is an American politician who served as the 43rd President of the United States from 2001 to 2009 and the 46th Governor of Texas from 1995 to 2000. Barack Obama.95 George W. Bush.80 Harvard Law School.05 Illinois.10

31. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring Evidence Retrieval Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

32. Scoring Final Merging Ranking Scoring Question Trained Models Evidence Retrieval Scoring SearchQuestion Analysis Who is the 44 th President of the United States? Candidate AnswerAnswer Scoring Contextual Answer Scoring Confidence Barack Obama 0.90.95 George W. Bush 0.900.80.65 Harvard Law School 0.100.05.05 Illinois 0.150.10.10

33. Scoring Final Merging Ranking Scoring Question Trained Models Evidence Retrieval Scoring SearchQuestion Analysis Who is the 44 th President of the United States? Candidate AnswerAnswer Scoring Contextual Answer Scoring Confidence Barack Obama 0.90.95 George W. Bush 0.900.80.65 Harvard Law School 0.100.05.05 Illinois 0.150.10.10

34. Scoring Final Merging Ranking Scoring Question Trained Models Candidate Answer Generation Primary Search Contextual Answer Scoring Answer Scoring SearchQuestion Analysis Wikipedia etc. Contextual Answer Scoring Answer Scoring Contextual Answer Scoring Answer Scoring Who is the 44 th President of the United States? Answer, Confidence, Evidence

35. IEEE Xplore Library: “This is Watson” http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717 Recommended –“This is Watson”, For all: at least read this thoroughly. – “Question Analysis, How Watson Reads a Clue” –“Finding needles in the haystack, Search and Candidate Generation” –“Automatic Knowledge Extraction from Documents” –“Structured Data and Inference in DeepQA” –“A framework for merging and ranking of answers in DeepQA” Read the others to broaden your knowledge 35

36. Current Focus

37. Current Customer Engagements Medicine – Challenge: Body of Knowledge doubles every 5 years – Best Practice / Evidence Based Medicine – Oncology Treatments Finance

38. Evidence Based Medicine http://www.anthem.com/medicalpolicies/policies/mp_pw_a053323.htm Varicose Veins – Medically Necessary: – Endoluminal radiofrequency ablation or endoluminal laser ablation, of the greater saphenous vein (GSV) or lesser saphenous veins (LSV) is medically necessary when the following criteria are met: Saphenofemoral or saphenopopliteal junction incompetence (i.e., reflux) is demonstrated by Doppler or duplex ultrasound scanning; and One or more of the following criteria (a, b, or c) are met: a)Symptoms of venous insufficiency or recurrent thrombophlebitis (including but not limited to: aching, burning, itching, cramping, or swelling during activity or after prolonged sitting) which: i.are interfering with activities of daily living; and ii.persist despite appropriate non-surgical management; and iii.persist despite a trial of properly fitted gradient compression stockings for at least six weeks; or b)There is ulceration secondary to stasis dermatitis; or c)There is hemorrhage from a superficial varicosity.

39. Treatment Example (Not Oncology) A 44-year-old businessman presents to a physician because of a markedly inflamed and painful right great toe. He states that he just returned from a convention, and noticed increasing pain in his right foot during his plane trip home. Physical examination is remarkable for swelling and erythema of the right great toe as well as small nodules on the patient's external ear. Aspiration of the metatarsal- phalangeal joint of the affected toe demonstrates needle- shaped negatively birefringent crystals. What’s wrong with this patient? How do you treat it?

40. Questions?