Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International.

Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International Vinay Chaudhri, David Israel, Sunil Mishra Boeing Phantom Works Peter Clark * Funding for various parts of this research was provided by DARPA, Vulcan, Inc, Boeing, and the US Army.

Knowledge Systems Knowledge Systems use formal representations of knowledge to answer unanticipated questions with coherent explanations Knowledge System = KB + Q/A + Explanation Generator + Knowledge Acq. tools

Advances over Expert Systems Coverage of domain, not domain task Various modes of reasoning, well integrated Domain level explanation Rapid construction U Just how advanced are they?

Project Halo* Long term: build a Knowledge System encompassing much of the world’s scientific knowledge Short term: assess current technologies Use a portion of the Advance Placement (AP) chemistry exam as a metric * Full support for Project Halo was provided by Vulcan Inc, Seattle, WA

Challenges Systems must be robust in the face of widely varying, unanticipated questions. Explanations are as important as correctness. Hard-ball evaluation, aimed to expose weaknesses. New domain and short development time require using off-the-shelf KR&R methods and systems. It was not clear at the outset that these challenges could be met.

Example Questions The spectator ions in the reaction of barium nitrate with sodium sulfate are what? (choices) Although nitric acid and phosphoric acid have very different properties as pure substances, their aqueous solutions possess many common properties. List some general properties of these solutions and explain their common behavior in terms of the species present. Explain why a solution of HClO 4 and NaClO 4 cannot act as a buffer solution. Sodium azide is used in air bags to rapidly produce gas to inflate the bag. The products of the decomposition reaction are what? (choices) Questions were manually encoded in our formal language

Because Questions Vary Widely… … we can not anticipate the questions, or even the type of questions, so a retrieval method won’t do. A custom inference method won’t do. The system must be capable of using its knowledge in unanticipated ways.

An Example Explanation What are the products of the given decomposition reaction? By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction. In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0) Therefore, the reaction causes a change in oxidation state. Therefore, this is an oxidation reduction reaction. By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated. Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction. In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements. In this reaction, NaN 3 reacts to produce Na and N 2. The products of the decomposition reaction are: (d) Sodium and nitrogen-g

Our KR&R System * KM: KRL-like frame system with FOL semantics. …able to represent: –classes, instances, prototypes –defaults, fluents, constraints –(hypothetical) situations –actions (pre-, post-, and during- conditions) …and reason about: –inheritance with exceptions –constraints –automatic classification (given a partial description of an instance, determine the classes to which it belongs) –temporal projection (“my car is where I left it”) –effects of actions KM answers questions by interleaving two types of inference: –Automatic classification –Backward chaining * Details: AAAI’97

What are the products of the given decomposition reaction? By definition, oxidation-reduction reactions occur when electrons are transferred from the atom that is oxidized to the atom that is reduced. We need to look for changes in the oxidation states of the elements in the reaction. In the reactants, the oxidation state(s) of the element Na is/are (1). In the product, the oxidation state(s) is/are (0) Therefore, the reaction causes a change in oxidation state. Therefore, this is an oxidation reduction reaction. By definition, a Binary Ionic-Compound Decomposition Reaction occurs when a binary ionic compound is heated. Therefore, this reaction is a Binary-Ionic Compound Decomposition reaction. In general, a Binary Ionic-Compound Decomposition Reaction converts a binary ionic-compound into basic elements. In this reaction, NaN 3 reacts to produce Na and N 2. The products of the decomposition reaction are: (d) Sodium and nitrogen-g Oxidation Reduction of a Binary Ionic Compound (Sodium Azide)  the reaction classified as Binary-Ionic Compound Decomposition  Backward chaining to compute the products of such reactions

Structure of the Knowledge Base* Two principal types of chemistry knowledge: –terms, e.g. “binary ionic compound” –laws, e.g. problem-solving method for computing products of reactions of binary-ionic compounds Terms are encoded as definitions to enable automatic classification. Laws are encoded as rules to enable backward chaining. * Details: KR’04 (Barker, et.al.)

The Content of a Chemistry Law Concentration of Solute Law Context: a mixture M such that: volume(M) = V liters has-part(M) includes Chemical C such that: quantity(C) = Q moles concentration(C) = Conc molar Input: V, Q Output: Conc Method: Conc ← Q/V The conditions under which the law applies The subset of variables that must be bound The subset of variables that will be bound The axioms used to compute values for output variables

Knowledge Engineering Methodology Knowledge base built in 4 months: – Ontological engineering (4 person-months): designed representations, including structure of terms, laws, reactions, solutions, etc. –Knowledge capture (6 person-months): consolidated 70 textbook pages into 35 pages of terms and laws –Knowledge encoding (15 person-months): coded in KM 500 types and relations, 150 chemistry laws and 65 terms. Compiled a large test suite which was run daily –Explanation engineering (3 person-months): augmented the representations of terms and laws with templates

Results of Project Halo* After 4 month development effort, the knowledge systems were sequestered and given a test: –165 novel questions: 50 multiple choice; 115 free form response –Questions translated from English to formal language by each team, then assessed for fidelity by Vulcan and team representatives * Details: AI Magazine (Winter 2004) www.projecthalo.com : systems, Q/A, and analyses

Correctness Our system’s correctness score corresponds to an AP score of 3 – high enough for credit at UCSD, UIUC, and many other universities. We’ve predicted scoring 85% after a 3 month follow-on project.

Explanation Quality

Error Analysis * We analyzed every point lost. Most deductions were due to errors in domain modeling — mistakes that domain experts would not make. (More later) Some errors were caused by technology problems. Details: KR’04 (Friedland, et.al.)

Problems Due to KR&R Technology Explanations too verbose: e.g. passages repeated multiple times with only small variations – graders expected a general statement that covered them all. Requires explanation planning Questions that require reasoning about our representations: –Calculate the pH of a particular substance. Explain why the result is unreasonable. –Explain the difference between the subscript “3” and the coefficient “3” in 3HNO 3. –Explain when and why it’s OK for a particular chemistry method to use an equation that only approximates the true answer.

Reasoning about Relevance Hydrofluoric acid is a weak acid, Ka = 6.8 x 10 -4, and yet it is considered to be a very reactive compound. For example, HF dissolves glass. The major reason it is considered highly reactive is: (a) It is an acid. (b) It forms H3O+. (c) It dissociates. (d) It readily forms very stable fluoride compounds. (e) It is a weak electrolyte. All five statements are true. The question requires that the system reason about which of the multiple true statements is most relevant to the claim.

Bottom Line Halo I was a rigorous evaluation of current Knowledge System technology. In general, the systems were more capable than Vulcan expected. The major hurdles to building a Knowledge System for science are errors (in domain modeling) and cost ($10K/page).

Halo – Phase II Knowledge Systems built by domain experts, not knowledge engineers 3 test domains: chemistry, physics, biology Questions posed by (a different set of) domain experts, in English 2 teams, expanded with HCI experts It’s not clear that these challenges can be met.

Requirements Domain experts build knowledge systems –with no help from knowledge engineers –working with familiar concepts and without writing axioms –with little more effort than writing technical papers scientific codes Domain experts pose questions – in English – without knowing the ontology* * Details: AAAI’04 Controlled Language (Boeing)

(One part of) Our Approach Observation: Even domain-specific concepts are defined in terms of common abstractions. Approach: We build a library of reusable, common abstractions (“components”) and a set of relations for composing them together. Then, KB’s are built by instantiating and assembling them.

Component Library Common events, entities, and roles Currently about 800 components; projected to be about 3000 Relatively general and atomic. Include enter, but probably not burgle, and definitely not endocytosis. Every component richly axiomatized

Library Construction draw from related work –ontology design/knowledge engineering –linguistics semantic primitives case theory, discourse analysis, NP semantics draw from English lexical resources –dictionaries, thesauri, word lists –WordNet, Roget, LDOCE, COBUILD, etc.

Contents actions — things that happen, change states –Enter, Copy, Replace, Transfer, etc. states — relatively temporally stable events –Be-Closed, Be-Attached-To, Be-Confined, etc. entities — things that are –Substance, Place, Object, etc. roles — things that are, but only in the context of things that happen –Container, Catalyst, Barrier, Vehicle, etc.

Library Contents relations between events, entities, roles –agent, donor, object, recipient, result, etc. –content, part, material, possession, etc. –causes, defeats, enables, prevents, etc. –purpose, plays, etc. properties between events/entities and values –rate, frequency, intensity, direction, etc. –size, color, integrity, shape, etc.

Richly Axiomatized Components Knowledge about Enter –before the Enter, the object is outside some container –instances of Enter inherit axioms from Move, such as: the action changes the location of the object of the Move –during the Enter, the object passes through a portal of the container –if the portal has a covering, it must be open; and unless it is known to be closed, assume that it’s open –after the Enter, the object is inside that container and contained by it –the resulting Be-Contained state persists until the object Exits the container –etc.

Searching the Library (and expanding its coverage) components are linked to WordNet synsets –given a word, find similar components –climb the WordNet hypernym tree with search terms; rank the results by distance and specificity –assemble: Attach, Come-Together mend: Repair infiltrate: Enter, Traverse, Penetrate, Move-Into gum up: Block, Obstruct busted: Be-Broken, Be-Ruined

Using the Component Library for NL Interpretation: A Sample Physics Question “A person throws a ball toward the door of a room. The ball passes through the door and into the room…”

Component Library Adds…

The Ball and the Person must be together at t 0 The Ball must be held by the Person at t 0 The Ball must not be otherwise Restrained The path to the Door must not be Blocked At t 1 the Ball is at the Door and no longer at the Person At t 1 the Ball is outside the Room but not shut out of it The Door must not be Closed The Ball passes through the Doorway At t 2 the Ball is inside the Room and no longer outside At t 2 the Room has the Ball among its contents Blocked paths can be unblocked Closed Doors can be Opened …

Using the Clib for K. Engineering Details: KCap’01, IAAI’02

location

Results * With 1 week of training, each biologist encoded about 10 pages of a dense biology text in about 3 weeks. Biologists asked for only one new concept in the Component Library: Copy In retrospect, the Q/A task was too easy. For evaluating the Component Library, perhaps knowledge engineering is too easy. * Details: Paul Cohen, et.al.,

“Understanding” evaluated by ability to answer questions and solve problems. There’s a significant gap between the content of NL utterances and the knowledge required for reasoning. Can the Component Library fill the gap? Using the Component Library for NL Understanding

Study 1: Is Broad, General Knowledge Useful? * Task: interpretation of noun compounds, e.g. “animal virus” Setup –Search KB for relationship between noun modifier and head noun –KB is the merge of the Component Library and WordNet1.6 –Measure performance with full KB, then with successive levels of the KB ablated –Noun compounds from three corpora: biology, engine repair, sparc station Results –Ablating the top 3-5 levels significantly degraded performance –Ablating levels 6 and below had little effect, even though they have more axioms * Details: IJCAI’03

Results biologyengineSparc

Study 2: Can we Build Knowledge-based NL Systems that are Robust? * Task: interpretation of dialogues, e.g. –Can you please purchase the following items, for a Dell Inspiron 8100 Notebook: 256 MB Memory Module and a Lithium-Ion Battery. The total price should be about $400. Order from the Dell web site. You can use my service tag to find the parts faster. Please pay for this using my startup account. Requires incorporating background knowledge into the LF of the dialogue, e.g. to resolve indirect anaphora * Details: AAAI’05 (submitted)

Matching LF’s with Background Knowledge Ontological mismatches abound – e.g. computer 3GHz computer 3GHz speed CPU has-part LF: KB:

Rules that Enable Matching Transfers through rules XYZ r1r2 X r1 Z For r2 in four classes of relations: partonomy causality spatial temporal Systematic generation of about 200 rules Made possible by the small size and clear semantics of the Component Library

Experimental Setup Corpus of 75 Purchase Requests LF’s produced manually MUC template-filling task –Human: “perfect” precision and recall –System with matching knowledge: Precision and recall about 80% –System without matching knowledge: Precision and recall about 50% –LaSIE-II DI (Gaizauskas, Wilks, et. al.): Precision and recall about 35% Next: how impoverished can the input be?

My Dream Future Knowledge Systems will be built by the distributed community of active scientists, as part of their computational infrastructure, to answer questions and organize/search digital libraries. Far more profound will be their contribution to knowledge distribution and discovery. Although the Component Library was built for knowledge engineering, it will increasingly be used for other tasks, such as NL understanding and machine learning.

LEFT OVERS

A Simple Example When 70 ml of 3.0-Molar Na 2 CO 3 is added to 30 ml of 1.0-Molar NaHCO 3 the resulting concentration of Na + is: a)2.0 M b)2.4 M c)4.0 M d)4.5 M e)7.0 M

Question Representation volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M0.07 lit NaHCO 3 0.03 lit volume 1.0 M conc.base conc. result has-part conc. Question 26 context ?? output

Background Knowledge Chemistry laws: 1.Concentration of a solute 2.Composition of strong electrolyte solutions 3.Conservation of mass 4.Conservation of volume etc.

Law 1: Concentration of a Solute The concentration of a chemical in a mixture is the quantity of the chemical divided by the volume of the mixture. Divide the quantity by the volume: / = X *molar Therefore, the concentration of in = X *molar Explanation Template Mixture volume conc. Volume *liters Concentration *molar has-part Chemical Quantity *moles quantity Compute-Concentration Method context input output Note: when this law is applied, the quantities are automatically converted to the units- of-measurement specified here

Law 1’: Quantity of a Solute Law 1 (on the previous slide) computed: Concentration = quantity / volume Of course, a slight variant computes: Quantity = concentration * volume Currently, we code this variant as a separate law (call it 1’) because it has a slightly different explanation template

Law 2: Composition of Strong Electrolytes Strong Electrolyte Anion has-part Quantity *moles quantity Quantity *moles quantity Cation Quantity *moles quantity Compute-Ions-in-Strong-Electrolyte context input output

Law 3: Conservation of Mass Conservation of Mass context input output Mix Chemical 1 Chemical n Chemical raw-material result … Quantity *moles Quantity *moles quantity Chemical has-part Quantity *moles quantity part-of By the Law of Conservation of Mass, the quantity of a chemical in a mixture is the sum of the quantities of that chemical in the parts of the mix. The quantity of in is *moles … The quantity of in is *moles Therefore, the quantity of = X *moles Explanation Template

Law 4: Conservation of Volume Mix Chemical 1 Chemical n Mixture raw-material result … Volume *liter Volume *liter volume Volume *liter volume Conservation of Volume context input output By the Law of Conservation of Volume, the volume of a mixture is the sum of the volumes of the parts mixed. The sum of, …, and = *liter Therefore, the volume of = *liter Explanation Template

Step 1: Reclassify Terms volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M0.07 lit NaHCO 3 0.03 lit volume 1.0 M conc.base conc. result has-part Strong Electrolyte Solution superclass Strong Electrolyte superclass chemical superclass

Step 2: Use Law 1 to Compute Concentration Mixture volume conc. Volume *liters Concentration *molar has-part Chemical Quantity *moles quantity Law 1 conc. ?? *molar volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M0.07 lit NaHCO 3 0.03 lit volume 1.0 M conc. base conc. result has-part ?? *liters volume ?? *moles quantity

The Search is non-deterministic Multiple laws might be used to compute a value for any property. For example, here’s another way to compute concentration:  pH = - log [H + ], where [H + ] is the concentration of H + Since this applies only to H +, this search path ends quickly

Step 3: Use Law 4 to Compute Volume Mix Chemical raw-material result … Volume *liter Volume *liter volume Volume *liter volume Law 4.1 conc. ?? *molar volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M0.07 lit NaHCO 3 0.03 lit volume 1.0 M conc. base conc. result has-part ?? *liters volume ?? *moles quantity

Step 4: Use Law 3 to Compute Quantity volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Mix Chemical raw-material result … Quantity *moles Quantity *moles quantity Chemical has-part ?? *moles quantity part-of Law 3 Na + ?? *moles ?? *moles has-part quantity

Step 5: Use Law 2 to Compute Quantity of Ionic Parts ?? *moles quantity Strong Electrolyte Anion has-part Quantity *moles quantity Quantity *moles quantity Cation Quantity *moles quantity Law 2 volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Na + ?? *moles ?? *moles has-part quantity

Step 6: Use Law 1’ to Compute Quantity ?? *moles quantity Mixture volume conc. Volume *liters Concentration *molar has-part Chemical Quantity *moles quantity Law 1’.21 volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Na + ?? *moles ?? *moles has-part quantity

Step 7: Wind out of Law 2 from step 5 Strong Electrolyte Anion has-part Quantity *moles quantity Quantity *moles quantity Cation Quantity *moles quantity Law 2.42.21 *moles quantity volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Na + ?? *moles ?? *moles has-part quantity

Step 8-10: Similar to steps 5-7.03.21 *moles quantity volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Na + ?? *moles.42 *moles has-part quantity

Step 11: Wind out of Law 3 from Step 4 Mix Chemical raw-material result … Quantity *moles Quantity *moles quantity Chemical has-part ?? *moles quantity part-of Law 3.45.21 *moles quantity volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume ?? *moles quantity Na +.03 *moles.42 *moles has-part quantity

Step 12: Wind out of Law 1 from Step 2 Mixture volume conc. Volume *liters Concentration *molar has-part Chemical Quantity *moles quantity Law 1.21 *moles quantity volume Mix Aqueous Solution Mixture Na + raw material Na 2 CO 3 3.0 M 0.07 liters 0.03 liters volume 1.0 M conc. base NaHCO 3 base conc. result has-part conc. ?? *molar.1 * liters volume.45 *moles quantity Na +.03 *moles.42 *moles has-part quantity 4.5

Answer and Explanation When 70 ml of 3.0-Molar Na2CO3 is added to 30 ml of 1.0-Molar NaHCO3, what is the resulting concentration of Na+?. The concentration of a chemical in a mixture is the quantity of the chemical divided by the volume of the mixture. By the Law of Conservation of Mass, the quantity of a chemical in a mixture is the sum of the quantities of that chemical in the parts of the mix. In the Na2CO3 strong-electrolyte-solution and the NaHCO3 strong-electrolyte-solution : In the Na2CO3 : Multiply the concentration and the volume: 3 molar * 70 milliliter = 0.21 mole. The quantity of Na+ in the NA2CO3 solution is 0.42 mole. In the NaHCO3 : Multiply the concentration and the volume: 1 molar * 30 milliliter = 0.03 mole. The quantity of Na+ in the Na2CO3 strong-electrolyte-solution and the NaHCO3 strong-electrolyte-solution is 0.45 mole. Therefore, the quantity of Na+ = 0.45 mole. By the Law of Conservation of Volume, the volume of a mixture is the sum of the volumes of the parts mixed. The sum of 70 milliliter and 30 milliliter = 0.10 liter. Therefore, the volume of the strong-electrolyte-solution strong-electrolyte-solution mixture = 0.10 liter. Divide the quantity by the volume:. 0.45 mole / 0.10 liter = 4.50 molar. Therefore, the concentration of Na+ in the Na2CO3 and NaHCO3 mixture = 4.50 molar. When 70 ml of 3.0-Molar Na2CO3 is added to 30 ml of 1.0-Molar NaHCO3, the resulting concentration of Na+ is 4.50 molar

BioremediationAmount OilFertilizer GetApply Break Down Absorb MicrobesScript Bio- technologist Soil Rate environment contains Q+ I- Q- I- amount product absorbed then agent patientagent script pollutant se rate agent then product se patient remediator amount Building a Representation Compositionally

BioremediationAmount OilFertilizer GetApply Break Down Absorb MicrobesScript Bio- technologist Soil Rate environment contains Q+ I- Q- I- amount product absorbed then agent patientagent script pollutant se rate agent then product se patient remediator Conversion Amount Substance Rate Q+ I- Q- I- amountraw- materials rate product Substance amount An underlying abstraction...

BioremediationAmount OilFertilizer GetApply Break Down Absorb MicrobesScript Bio- technologist Soil Rate environment contains Q+ I- Q- I- amount product absorbed then agent patientagent script pollutant se rate agent then product se remediator amount Digest Substance Break Down Absorb AgentScript absorbed agent script food se then se patient eater agent Another abstraction... patient

BioremediationAmount Oil Fertilizer Break Down Absorb Bio- technologist Soil Rate environment contains Q+ I- Q- I- amount product absorbed then agent pollutant se rate agent GetApply MicrobesScript patient script then product se remediator amount TreatmentAgent Another abstraction... patient GetApply substanceScript patient script then substance patient se

Examples of Concepts Described Compositionally a Fuel-Cell is a Producer of Electricity a Bulb is an Electrical Resistor that Produces Light a Camera is an Image Recording Device a Wire is a Conduit of Electricity

Knowledge-Engineer Flaws they are less concerned with the fidelity of the representations they lack the knowledge to simplify and abstract the knowledge thoughtfully they operate with sentence-level facts rather than domain-level theories

Reasoning about Relevance An instructor handed you a bottle and indicated that the contents were one of the following: silver acetate, calcium acetate, and lead nitrate. The instructor asked you to identify the contents of the bottle. Using only solubility as a guide, indicate what chemicals you would use to test the unknown, and what the expected results would be for each of the possibilities. The question requires the system to choose some chemical to use in testing the unknown chemical. In order to avoid trial and error involving all known chemicals (which is actually what our system tried to do), you would need to reduce the search space to only those chemicals that are likely to be useful in the test.

Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International.

Similar presentations

Presentation on theme: "Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International.

Similar presentations

Presentation on theme: "Knowledge Systems and Project Halo* University of Texas at Austin Ken Barker, Shaw Yi Chaw, James Fan, Bruce Porter, Dan Tecuci, Peter Yeh SRI International."— Presentation transcript:

Similar presentations

About project

Feedback