1. Project Halo Towards a Digital Aristotle Noah S. Friedland, Paul G. Allen, Gavin Matthews, Michael Witbrock, David Baxter, Jon Curtis, Blake Shepard, Pierluigi Miraglia, Jurgen Angele, Steffen Staab, Eddie Moench, Henrik Oppermann, Dirk Wenke, David Israel, Vinay Chaudhri, Bruce Porter, Ken Barker, James Fan, Shaw Yi Chaw, Peter Yeh, Dan Tecuci, and Peter Clark Presented by Jacob Halvorson

2. Aristotle? Real person who lived from 384-322BC Known for: –The depth and scope of his knowledge Wide range of topics –Medicine –Philosophy –Physics –Biology –Ability to explain this knowledge to others

3. What is Project Halo? Goal: Create an application that will encompass much of the world’s scientific knowledge and be capable of applying sophisticated problem solving to answer novel questions. Roles (envisioned): Tutor to instruct students in the sciences. Interdisciplinary research assistant to help scientists. Sponsored by Vulcan, Inc.

4. Why is a program like Digital Aristotle important? Too much knowledge in the world for a single person to assimilate. –This forces people to become more specialized, thus defining their own restrictive “microworld”. Even these microworlds are too big. –MEDLINE 12 million publications with 2,000 added daily.

5. Don’t we have something like this already? Voorheese –Retrieval of simple facts from an “answer” database. Knowledge-based expert systems –Retrieval of answers that aren’t in a database. –Digital Aristotle fits in this category

6. How Digital Aristotle differs from other knowledge-based expert systems 1Speed and ease of knowledge formulation - Little or no help from knowledge engineers - *Other expert systems required years to perfect and highly skilled knowledge engineers to craft them 2Coverage - Encompass much of the world’s scientific knowledge 3Reasoning Techniques - Multiple technologies and problem solving methods 4Explanations - Appropriate to the domain and user’s level of expertise

7. Where to Start: The Halo Pilot Three teams contracted to participate in evaluation –SRI International Boeing Phantom Works and Univ. of Texas backing –Cycorp –Ontoprise Goal: –Determine the current state of knowledge representation & reasoning (KR&R) by mastering a 70- page subset of introductory college-level AP Chemistry. –*Secret goal: Set the bar so high that the current weaknesses of KR&R would be exposed Four months to create formal encodings Six months total.

8. 70-page AP Chemistry Overview Self-contained, no reasoning with uncertainty, no diagrams Large enough for complex inference –Nearly 100 distinct chemistry laws Small enough to be represented quickly

9. The three team’s technology All three teams needed to address –Knowledge formation All built knowledge bases in a formal language and had knowledge engineers encode. –Question answering All used automated deductive inference to answer questions. –Explanation generation All were different

10. Knowledge Formation – Ontoprise team Ontoprise encoded knowledge in three phases 1Encode knowledge into the ontology and rules without considering sample questions -Tested with questions from textbook 2Tested with questions from Vulcan - Refined knowledge base until 70% coverage - Coded explanation rules 3Refined encoding of knowledge base and explanation rules

11. Knowledge Formation - Cycorp Cycorp encoded knowledge in two phases 1Concentrated on representing the basic concepts and principles 2Shift over to a question-driven approach. *Avoid overfitting the knowledge to the specifics of the sample questions available.

12. Knowledge Formation - SRI Question-driven –Started with 50 sample questions Worked backwards to determine what knowledge was needed to solve them. –Found additional questions and continued Combined team of knowledge engineers and four chemistry domain experts.

13. Cycorp and SRI had preexisting knowledge based content. Ontoprise started from scratch.

14. Explanation Generation – Ontoprise team Used metainferencing –While processing a query, a log file of the proof tree is created. –The log file is used to create English answer justifications. Running short on time, they mostly used template matching.

15. Explanation of the Ka value of a substance given its quantity in moles (0.2) and its pH (3.0)

16. Explanation Generation - SRI The knowledge engineer specifies what text to display: –When a rule is invoked (“entry text”) –When the rule has been successfully applied (“exit text”) –A list of any other facts that should be explained in support of the current rule (“dependent facts”)

17. Explanation generated for the computation of concentration of ions in NaOH

18. Explanation Generation - Cycorp Cycorp already had a program that was capable of providing natural language explanations in any detail. –Much of the effort was spent on strengthening of the explanation filters Output errs on the side of verbosity. –The English is built up compositionally by automated techniques rather than handcrafted. Exhibits clumsiness of expression.

19. Evaluation of the three systems After four months, an exam was given to all three systems. –100 AP-style English questions (total score: 1008 points) 50 multiple choice Two sets of 25 multipart questions –Detailed answer »Fill in the blank and short essay –Free-form answer »Qualitative, comprehension questions (somewhat common sense questions) »Somewhat beyond the scope of the defined syllabus –Graded by 3 chemistry professors (336 points per prof) Correctness (168 points) Quality of explanation (168 points) –**Input could be in any form

20. Processing Time Ontoprise –2 hours SRI –5 hours Cycorp –12 hours

21. Exam Examples

22. Results All three systems scored above 40% Cycorp’s program looked for provably wrong answers if the correct answer couldn’t be found immediately

23. Results - Multiple Choice Cycorp’s program looked for provably wrong answers if the correct answer couldn’t be found immediately. No answer = no justification Incorrect answers = unconvincing justification SRI was the winner of multiple choice –Best answers and justification Cycorp was the loser –Generative-English was least comprehensible

24. Results – Detailed Answer Cycorp appears to be the best –It wasn’t penalized by going through all the answers like multiple choice.

25. Results – Free-Form SRI & Cycorp were expected to do well SRI did much better than others

26. Second Exam All three teams did their own modifications after the test and ran the challenges again Ontoprise –9 minutes (2 hours previously) SRI –30 minutes (5 hours previously) Cycorp –27 hours (12 hours previously)

27. Failure Analysis (What we’ve learned) Modeling –Incorrect knowledge was represented and captured at the wrong level of abstraction Solution: Domain experts Answer Justification –Answers don’t matter if they can’t be explained Perform metareasoning over the proof tree Scalability for Speed and Reuse –How to manage trade-off between expressiveness and tractability

28. Ontoprise Output Example

29. SRI Output Example

30. Cycorp Output Example

31. Where To Go From Here: Phase Two Goal: Domain expert uses an existing document such as a textbook as the basis for the formulation of a knowledge module. –30 month phase –Three stages 6 month analysis-driven design process stage 15 month implementation stage 9 month refinement stage *Get rid of knowledge engineers