1. 1 CHAPTER 11 Knowledge Acquisition and Validation －郭建甫＆楊善淵 Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

2. 2 Opening Vignette: AE Improves approval selection with Machine Learning n An automated statistical method – whether a loan should be made. n 85-90% decisions was provided. n The other 10-15% had to be handled manually— 50% would default. n They solved the problem with an automated approach called machine learning.

3. 3 Opening Vignette: AE Improves approval selection with Machine Learning n The computer “learned” by examining 1,014 historical loan applications. n Rule induction ‧ was used to create a decision tree correctly predicted the answer, 70% of the gray-area applications than 50% by the loan officers. ‧ Explain why the applications are being rejected.

4. 4 n A powerful and valuable tools n “machine learning”. n Increases the productivity of decision making and the accuracy of the prediction. n Rely on knowledge acquired directly from human experts. Opening Vignette: AE Improves approval selection with Machine Learning

5. 5 Knowledge Engineering － definition Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ n Narrow perspective: Deals with knowledge acquisition, representation, validation, inferencing, explanation and maintenance n Wide perspective: KE describes the entire process of developing and maintaining AI systems n We use the Narrow Definition

6. 6 Knowledge Engineering Process Activities n Knowledge Acquisition - From human experts, books, documents, sensors, or computer files. n Knowledge Validation n Knowledge Representation n Inferencing n Explanation and Justification Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

7. 7 Knowledge Engineering Process (Figure 11.1) Knowledge validation (test cases) Knowledge Representation Knowledge Acquisition Encoding Inferencing Sources of knowledge (experts, others) Explanation justification Knowledge base Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

8. 8 Knowledge Sources n Sources of knowledge — Books, films, computer databases, pictures, maps, flow diagrams, stories, sensors, songs, observed behavior… n Documented (books, manuals, etc.) n Undocumented (in people's minds) – From people – From machines n Knowledge Acquisition from Databases n Knowledge Acquisition Via the Internet Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

9. 9 Knowledge Levels n Shallow knowledge (surface) – deal with very specific situations. – EX. If gasoline tank is empty, then car will not start. – IF-THEN ruls. – Have little to do with problem solving. – Is insufficient in describing complex situations. Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

10. 10 Knowledge Levels n Deep knowledge - Computerized representation - Special knowledge representation methods to allow the implementation of deeper-level reasoning - Represent objects and processes of the domain of expertise - Relationships

11. 11 Major Categories of Knowledge n Declarative Knowledge n Procedural Knowledge n Metaknowledge Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

12. 12 Declarative Knowledge n Descriptive representation. n It tells us facts: what things are. n Truth and associations. n Important in the initial stage of knowledge acquisition Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

13. 13 Procedural Knowledge n The manner under different sets of circumstances. n Step-by-step sequences and how-to n Include explanations n How to use declarative knowledge and how to make inferences. Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

14. 14 Metaknowledge n Knowledge about Knowledge n In ES, Metaknowledge refers to knowledge about the operation of knowledge-based systems n Its reasoning capabilities Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

15. 15 Knowledge Acquisition Difficulties Problems in Transferring Knowledge n Expressing Knowledge n Transfer to a Machine n Number of Participants n Structuring Knowledge Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

16. 16 Expressing the knowledge n A human expert solve a problem. –Input information about the external world into the brain. –Uses an inductive, deductive, other problem-solving approach on the collected information. –Output a recommendation on how to solve the problem. n This process is internal. n The expert have to be introspective about his decision-making process. n The expert is often unaware of the detailed process uses to arrive at a conclusion.

17. 17

18. 18

19. 19

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. 26

27. 27

28. 28

29. 29

30. 30

31. 31

32. 32

33. 33

34. 34

35. 35 Transfer to a machine n In a organized particular manner. n Requires expressed explicitly, a lower, detailed level knowledge than humans use. n There is a mismatch between computers and experts — Human does not remember all the detail.

36. 36 Number of participants n A normal transfer of knowledge process, there are two participants. n In ES, there can be four participants : expert, knowledge engineer, system designer, user. n They have different backgrounds, use different terminology, possess different skills and knowledge.

37. 37 Structuring the knowledge n In ES it is necessary to elicit not only knowledge but also its structure. We must represent the knowledge in a structured way. (e.g., as rules).

38. 38 Required Knowledge Engineer Skills n Computer skills n Tolerance and ambivalence n Effective communication abilities n Broad educational background n Advanced, socially sophisticated verbal skills n Fast-learning capabilities (of different domains) n Must understand organizations and individuals n Wide experience in knowledge engineering n Intelligence n Empathy and patience n Persistence n Logical thinking n Versatility and inventiveness n Self-confidence Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

39. 39 Knowledge Acquisition Methods: An Overview Knowledge elicitation n Manual n Semiautomatic n Automatic (Computer Aided) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

40. 40 Manual Methods n Interviewing n Tracking the Reasoning Process n Observing n Manual methods: slow, expensive and sometimes inaccurate Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ Elicitation Knowledge base Documented knowledge Experts Coding Knowledge engineer (Figure 11.4)

41. 41 Semiautomatic Methods n Support Experts Directly n Help Knowledge Engineers Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ Knowledge base Knowledge engineer Expert Coding Computer-aided (interactive) interviewing (Figure 11.5)

42. 42 Automatic Methods n Expert’s and/or the knowledge engineer’s roles are minimized (or eliminated) n Induction Method Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ Knowledge base Case histories and examples Induction system (Figure 11.6)

43. 43 Manual Method - Interviews n Most Common Knowledge Acquisition: Face-to-face interviews – Direct dialog between the expert and the knowledge engineer. – Conventional instruments to record and is subsequently transcribed, analyzed, and coded. n Interview Types – Unstructured (informal) – Semi-structured – Structured Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

44. 44 Unstructured Interviews n Most Common Variations – Talkthrough— talk to the engineer – Teachthrough— teach the engineer – Readthrough— instruct the engineer to read Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

45. 45 Issues about Unstructured Interviews n Seldom provides complete or well- organized descriptions of cognitive processes because – Domains are complex – Experts difficult to express important knowledge – Data acquired are often unrelated, exist at varying levels of complexity. – Few knowledge engineers can conduct an efficient unstructured interview Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

46. 46 Structured Interviews n Systematic goal-oriented process n Forces an organized communication between the knowledge engineer and the expert n Interpersonal communication and analytical skills are important Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

47. 47 Procedures for structured interviews n KE studies available material on the domain n KE identifies targets for the questions to be asked during the knowledge acquisition session. n KE formally schedules and plans the structured interviews (use a form).

48. 48 Recommendation Before interviews the expert 1. Interview a less knowledgeable expert -Helps the knowledge engineer 2. Read about the problem 3. Interview the expert Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

49. 49 Tracking Methods n Track the reasoning process of an expert n To find what information is being used and how it is being used. n Most common formal method: Protocol Analysis Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

50. 50 Protocol Analysis n Protocol : a record or documentation of the expert's step-by-step information processing and decision-making behavior n The expert performs a real task and verbalizes his/her thought process (think aloud) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

51. 51 Observations and Other Manual Methods n Observe the Expert Work – The most obvious and straightforward approach. – Expert may be multi-tasked. – If recordings or videotapes are made, that costs much time and money for transcribing knowledge. – Can be view as two types of protocols: motor and eye movement. They are expensive and time-consuming. Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

52. 52 Other Manual Methods n Case analysis n Critical incident analysis n Discussions with the users n Commentaries n Conceptual graphs and models n Brainstorming n Prototyping n Multidimensional scaling n Johnson's hierarchical clustering n Performance review Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

53. 53 Expert-driven Methods n Knowledge Engineers Typically – Lack Knowledge About the Domain – Expensive – May Have Problems Communicating With Experts n Knowledge Acquisition May be Slow, Expensive and Unreliable n Can Experts Be Their Own Knowledge Engineers? Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

54. 54 Approaches to Expert-Driven Systems n Manual n Computer-Aided (Semiautomatic) Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

55. 55 n A self administered questionnaire - open-ended - high level concepts catch An organized report - an introductory lecture Manual Method: Expert's Self-reports

56. 56 Issue about Expert's Self-reports 1. Requires the expert to act as knowledge engineer 2. Reports are biased 3. Experts often describe new and untested ideas and strategies 4. Experts lose interest rapidly 5. Experts must be proficient in flowcharting 6. Experts may forget certain knowledge 7. Experts are likely to be vague Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

57. 57 Benefits of Expert's Self-reports n May provide useful preliminary knowledge discovery and acquisition n Computer support can eliminate some limitations Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

58. 58 Computer-aided Approaches n To reduce or eliminate the potential problems : bias and ambiguity. n Other – New machine learning methods to induce decision trees and rules – Tools based on repertory grid analysis Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. Aronson 6th ed, Copyright 2001, Prentice Hall, Upper Saddle River, NJ

59. 59 11.10 Repertory Grid Analysis (RGA) n Techniques, derived from psychology n Use the classification interview n Primary Method: Repertory Grid Analysis (RGA)

60. 60 The Grid n Based on Kelly's model of human thinking: Personal Construct Theory (PCT) n Each person is a "personal scientist" seeking to predict and control events by –Forming Theories –Testing Hypotheses –Analyzing Results of Experiments n Knowledge and perceptions about a domain are classified by each individual as a personal, perceptual model n Each individual anticipates and then acts

61. 61 How RGA Works 1. The expert identifies the important objects in the domain of expertise (interview) 2. The expert identifies the important attributes 3. For each attribute, the expert is asked to establish a bipolar scale with distinguishable characteristics and their opposites 4. The interviewer picks any three of the objects and asks: What attributes distinguish any two of these objects from the third? Translate answers on a scale of 1-3 (or 1-5)

62. 62

63. 63

64. 64 n Expert may change the ratings inside box n Can use the grid for recommendations How RGA Works (Cont.)

65. 65 RGA in Expert Systems - Tools n Expertise Transfer System (ETS) –A computer program help expert build expert system –Uncover vocabulary conclusions, problem-solving trait, trait structure,trait weights, and inconsistencies. –Construct prototypes rapidly (less than 2 hours) n AQUINAS –A very complex tool –Allow expert to structure knowledge in hierarchies –Provide guidance in the knowledge acquisition process n OTHER RGA TOOLS –Ex. PCGRID, WebGrid, Circumgrids

66. 66 11.11 Supporting The Knowledge Engineer n Knowledge Acquisition Aids n Special Languages n Editors and Interfaces n Explanation Facility n Revision of the Knowledge Base n Pictorial Knowledge Acquisition (PIKA)

67. 67 n Integrated Knowledge Acquisition Aids –KADS (Knowledge Acquisition and Documentation System) n Front-end Tools n Knowledge Analysis Tool (KAT) Supporting The Knowledge Engineer (cont.)

68. 68 11.12 Machine Learning n Manual and semiautomatic elicitation methods: slow and expensive n Other Deficiencies n There is a gap between verbal reports and mental behavior n Sometimes experts cannot describe their decision making process n System quality depends too much on the quality of the expert and the knowledge engineer n The expert does not understand ES technology n The knowledge engineer may not understand the business problem n It is difficult to validate acquired knowledge

69. 69 Knowledge Acquisition Method’s Objectives n Increase the productivity of knowledge engineering n Reduce the required knowledge engineer’s skill level n Eliminate the need for an expert n Eliminate the need for a knowledge engineer n Increase the quality of the acquired knowledge

70. 70 Automated Knowledge Acquisition (Machine Learning) n Rule Induction n Case-based Reasoning n Neural Computing n Intelligent Agents

71. 71 Automated Rule Induction n Induction: Process of Reasoning from Specific to General n In ES: Rules Generated by a Computer Program from Cases n Interactive Induction (ex.ACQUIRE)

72. 72

73. 73 Case-based Reasoning (CBR) n An approach for Building ES by Accessing Problem-solving Experiences for Inferring Solutions for Solving Future Problems n Cases and Resolutions Constitute a Knowledge Base

74. 74 Neural Computing n Fairly Narrow Domains with Pattern Recognition n Requires a Large Volume of Historical Cases

75. 75 Intelligent Agents for Knowledge Acquisition Led to n KQML (Knowledge Query and Manipulation Language) for Knowledge Sharing n KIF, Knowledge Interchange Format (Among Disparate Programs)

76. 76 11.13 Selecting an Appropriate Knowledge Acquisition Method Ideal Knowledge Acquisition System Objectives n Direct interaction with the expert without a knowledge engineer n Applicability to virtually unlimited problem domains n Tutorial capabilities n Ability to analyze work in progress to detect inconsistencies and gaps in knowledge n Ability to incorporate multiple knowledge sources n An user friendly interface n Easy interface with different expert system tools n Hybrid Acquisition - Another Approach

77. 77 11.14 Knowledge Acquisition from Multiple Experts Major Purposes of Using Multiple Experts n Better understand the knowledge domain n Improve knowledge base validity, consistency, completeness, accuracy and relevancy n Provide better productivity n Identify incorrect results more easily n Address broader domains n To handle more complex problems and combine the strengths of different reasoning approaches

78. 78 Knowledge Acquisition from Multiple Experts (cont.) Benefits And Problems With Multiple Experts (P.469) Benefits And Problems With Multiple Experts (P.469) Benefits Problems 1.On the average, fewer mistakes by a group of experts than by a single expert 2.Several experts in a group eliminate the need for using a world-class expert 3.Wider domain than a single expert ’ s 4.Synthesis of expertise 5.Enhanced quality from synergy among experts 1.Group thinking phenomena 2.Fear on the part of some domain experts of senior experts 3.Compromising solutions generated by a group with conflicting opinions 4.Wasted time in group meeting 5.Difficulties in scheduling the experts 6.Dominating experts (not let others speak )

79. 79 Handling Multiple Expertise n Blend several lines of reasoning through consensus methods n Use an analytical approach n Select one of several distinct lines of reasoning n Automate the process n Decompose the knowledge acquired into specialized knowledge sources

80. 80 n Evaluation –Assess an expert system's overall value –Analyze whether the system would be usable, efficient and cost-effective Validation –Deals with the performance of the system (compared to the expert's) –Was the “right” system built (acceptable level of accuracy?) Verification –Was the system built "right"? –Was the system correctly implemented to specifications? 11.15 Validation and Verification of the Knowledge

81. 81 Dynamic Activities n Activities must be repeated each time the prototype is changed For the Knowledge Base –Must have the right knowledge base –Must be constructed properly (verification)

82. 82 A Method to Validate an ES n Test 1. The extent to which the system and the expert decisions agree 2. The inputs and processes used by an expert compared to the machine 3. The difference between expert and novice decisions (Sturman and Milkovich [1995])

83. 83 11.16 Analyzing, Coding, Documenting, and Diagramming Method of Acquisition and Representation 1. Transcription 2. Phrase Indexing 3. Knowledge Coding 4. Documentation (Wolfgram et al. [1987])

84. 84 Knowledge Diagramming n Hierarchical, top-down description of the type of knowledge that describes facts and reasoning strategies in ES n Types –Objects –Events –Performance –Metaknowledge n Describes the linkages and interactions among knowledge types n Supports the analysis and planning of subsequent acquisitions n Useful in analyzing acquired knowledge

85. 85 11.17 Numeric and Documented Knowledge Acquisition n Acquisition of Numeric Knowledge –Special approach needed to capture numeric knowledge – Acquisition of Documented Knowledge –Major Advantage: No Expert –To Handle a Large or Complex Amount of Information –New Field: New Methods That Interpret Meaning to Determine – Rules – Other Knowledge Forms (Frames for Case- Based Reasoning)

86. 86 11.18 Knowledge Acquisition and the Internet/Intranet n Hypermedia (Web) to Represent Expertise Naturally n Natural Links can be Created in the Knowledge n HypEs is an architecture for the development of media rich expert system

87. 87 The Internet/Intranet for Knowledge Acquisition n Electronic Interviewing n Experts can Validate and Maintain Knowledge Bases from a distance n Documented Knowledge can be accessed n Identifying relevant knowledge n Many Web Search Engines have intelligent agents n Data Fusion Agent for multiple Web searches and organizing n Automated Collaborative Filtering (ACF) statistically matches peoples’ evaluations of a set of objects

88. 88 11.19 Induction Table Example n Induction tables (knowledge maps) focus the knowledge acquisition process n Choosing a hospital clinic facility site

89. 89

90. 90 THE END