1. Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California USA http://cll.stanford.edu/ Transfer of Knowledge in Cognitive Systems Thanks to D. Aha, D. Choi, K. Forbus, J. Laird, S. Rogers, and T. Senator for useful dicsussions. This talk reports research funded by a grant from DARPA and AFRL, which are not responsible for its contents.

2. general learning in multiple domains Generality in Learning training items test items Humans exhibit general intelligence by their ability to learn in many domains.

3. general learning in multiple domains transfer of learning across domains Generality and Transfer in Learning training items test items training items test items Humans exhibit general intelligence by their ability to learn in many domains. Humans are also able to utilize knowledge learned in one domain in other domains.

4. experience performance A learner exhibits transfer of learning from task/domain A to task/domain B when, after it has trained on A, it shows improved behavior on B. learning curve for task A experience performance experience performance better intercept on task B better asymptote on task B faster learning rate on task B What is Transfer? experience w/training on A w/o training on A w/training on A w/o training on A w/training on A w/o training on Aperformance

5. What is Transfer? Transfer is a sequential phenomenon that occurs in settings which involve on-line learning. Transfer is a sequential phenomenon that occurs in settings which involve on-line learning. Thus, multi-task learning does not involve transfer. Thus, multi-task learning does not involve transfer. Transfer involves the reuse of knowledge structures. Transfer involves the reuse of knowledge structures. Thus, it requires more than purely statistical learning. Thus, it requires more than purely statistical learning. Transfer can lead to improved behavior (positive transfer). Transfer can lead to improved behavior (positive transfer). But it can also produce worse behavior (negative transfer). But it can also produce worse behavior (negative transfer). Transfer influences learning but is not a form of learning. Transfer influences learning but is not a form of learning. Thus, transfer learning is an oxymoron, much like the phrase learning performance. Thus, transfer learning is an oxymoron, much like the phrase learning performance.

6. Roots of Transfer in Psychology The notion of transfer comes from psychology, where it has been studied for over a hundred years: benefits of Latin (Thorndike & Woodworth, 1901) benefits of Latin (Thorndike & Woodworth, 1901) puzzle solving (Luchins & Luchins, 1970) puzzle solving (Luchins & Luchins, 1970) operating devices (Kieras & Bovair, 1986) operating devices (Kieras & Bovair, 1986) using text editors (Singley & Anderson, 1988) using text editors (Singley & Anderson, 1988) analogical reasoning (Gick & Holyoak, 1983) analogical reasoning (Gick & Holyoak, 1983) Some recent studies have included computational models that predict the transfer observed under different conditions.

7. Inference tasks that require multi-step reasoning to obtain an answer, such as solving physics word problems and aptitude/achievement tests. Classification tasks that involve assigning items to categories, such as recognizing types of vehicles or detecting spam. These are not very interesting. Domain Classes that Exhibit Transfer Procedural tasks that involve execution of routinized skills, both cognitive (e.g., multi-column arithmetic) and sensori-motor (e.g., flying an aircraft). Problem-solving tasks that benefit from strategic choices and heuristic search, such as complex strategy games. 654 - 321 456 - 237 821 - 549 940 - 738 601 - 459 400 - 321 From: tsenator@darpa.mil To: langley@csli.stanford.edu Subject: site visit next week Date: Nov 14, 2004 Pat – I am looking forward to hearing about your progress over the past year during my site visit next week. - Ted From: noname@somewhere.com To: langley@csli.stanford.edu Subject: special offer!!! Date: Nov 14, 2004 One week only! Buy v*i*a*g*r*a at half the price available in stores. Go now to http://special.deals.com A block sits on an inclined plane but is connected to a weight by a string through a pulley. If the angle of the plane is 30 degrees and... Which ladder is safer to climb on? Which jump should red make? What should the blue team do? What are the problem answers? What path should the plane take? Which is an emergency vehicle? Which email is spam?

8. The degree of transfer depends on the structure shared with the training tasks. Transfer requires the ability to compose these knowledge elements dynamically. Transfer requires that knowledge be represented in a modular fashion. Claims About Transfer Transfer across domains requires abstract relations among representations.

9. Dimensions of Knowledge Transfer Difference in Content Difference in Representation 0 0 Memorization Different Representations (e.g., most cross-domain transfer) Similar Representations (e.g., within- domain transfer) Knowledge Reuse First- Principles Reasoning Isomorphism We have already solved these problems. We know the solution to a similar problem with a different representation, possibly from another domain. We have not solved this before, but we know other pertinent information about this domain that uses the same representation. We have not solved similar problems, and are not familiar with this domain and problem representation. Knowledge transfer complexity is determined primarily by differences in the knowledge content and representation between the source and target problems. Problem Solver

10. Memorization target items source items E.g., solving the same geometry problems on a homework assignment as were presented in class. This is not very interesting. Improvement in which the transfer tasks are the same as those encountered during training.

11. Within-Domain Lateral Transfer target items source items E.g., solving new physics problems that involve some of the same principles but that also introduce new ones. Improvement on related tasks of similar difficulty within the same domain that share goals, initial state, or other structure.

12. Within-Domain Vertical Transfer target items source items E.g., solving new physics problems that involve the same principles but that also require more reasoning steps. Improvement on related tasks of greater difficulty within the same domain that build on results from training items.

13. Cross-Domain Lateral Transfer target items source items E.g., solving problems about electric circuits that involve some of the same principles as problems in fluid flow but that also introduce new ones. Improvement on related tasks of similar difficulty in a different domain that shares either higher-level or lower-level structures.

14. Cross-Domain Vertical Transfer target items source items E.g., solving physics problems that require mastery of geometry and algebra or applying abstract thermodynamic principles to a new domain. Improvement on related tasks of greater difficulty in a different domain that share higher-level or lower-level structures.

15. 11 12 3 45 6 12 78 9 10 Methods for cumulative learning of hierarchical skills and concepts define new cognitive structures in terms of structures learned on earlier tasks. 13 11 12 3 4 6 12 5 8 9 10 12 3 45 6 78 9 This approach is well suited to support vertical transfer to new tasks of ever increasing complexity. Learning can operate on problem-solving traces, observations of another agents behavior, and even on direct instructions. Approaches to Transfer: Cumulative Learning

16. Methods for analogical reasoning store cognitive structures that encode relations in training problems. Additional relations are then inferred based on elements in the retrieved problem. Analogical reasoning can operate over any stored relational structure, but must map training elements to transfer elements, which can benefit from knowledge. This approach is well suited for lateral transfer to tasks of similar difficulty. Approaches to Transfer: Analogical Reasoning Upon encountering a new problem, they retrieve stored experiences with similar relational structure.

17. Approaches to Transfer: Mapping Representations Mapping Process Source domain: Electricity Target domain: Fluid Flow Transfer of learned knowledge across domains may require mapping between their representations of shared content. Electrical Resistance R Voltage Drop V1V1V1V1 I V2V2V2V2 I = V 1 - V 2 R Knowledge: Ohms law Pressure Drop F = P 1 - P 2 R Knowledge: Poiseuilles law P1P1P1P1 P2P2P2P2 Resistance to Flow R F Q: If P 1 =3, P 2 =2, and R=2, then what force F is being applied, assuming we only know Ohms law for electric currents?

18. Experimental Studies of Transfer Compare results from transfer and control conditions Transfer condition Control condition Train on items from source domain Test and train on target domain items Present no items from source domain Test and train on target domain items

19. Dependent Variables in Transfer Studies Dependent variables for transfer experiments should include: Initial performance on the transfer tasks Initial performance on the transfer tasks Asymptotic performance on the transfer tasks Asymptotic performance on the transfer tasks Rate of improvement on the transfer tasks Rate of improvement on the transfer tasks These require collecting learning curves over a series of tasks. Such second-order variables build on basic metrics such as: Accuracy of response or solutions to tasks Accuracy of response or solutions to tasks Speed or efficiency of solutions to tasks Speed or efficiency of solutions to tasks Quality or utility of solutions to tasks Quality or utility of solutions to tasks Different basic measures are appropriate for different domains.

20. Transfer in Urban Combat Urban Combat is a first-person shooter game developed at UT Arlington that we are using as a testbed to study approaches to the transfer of learned knowledge.

21. The I CARUS Architecture We are studying transfer in I CARUS, an architecture that incorporates some key assumptions from theories of human cognition: These claims give I CARUS much in common with other cognitive architectures like Soar and ACT-R. 1.Short-term memories are distinct from long-term stores 2.Memories contain modular elements cast as list structures 3.Long-term structures are accessed through pattern matching 4.Cognition occurs in retrieval/selection/action cycles 5.Performance and learning compose elements in memory

22. I CARUS Functional Processes Long-TermConceptualMemory Short-TermBeliefMemory Short-Term Goal Memory ConceptualInference SkillExecution Perception Environment PerceptualBuffer Problem Solving Skill Learning MotorBuffer Skill Retrieval and Selection Long-Term Skill Memory

23. Representing Long-Term Structures Conceptual clauses: A set of relational inference rules with perceived objects or defined concepts in their antecedents; Conceptual clauses: A set of relational inference rules with perceived objects or defined concepts in their antecedents; Skill clauses: A set of executable skills that specify: Skill clauses: A set of executable skills that specify: a head that indicates a goal the skill achieves; a head that indicates a goal the skill achieves; a single (typically defined) precondition; a single (typically defined) precondition; a set of ordered subgoals or actions for achieving the goal. a set of ordered subgoals or actions for achieving the goal. These define a specialized class of hierarchical task networks in a syntax very similar to Nau et al.s SHOP2 formalism. Beliefs, goals, and intentions are instances of these structures. ICARUS encodes two forms of general long-term knowledge:

24. Primitive Concepts for Urban Combat ((stopped ?self) :percepts((self ?self xvel ?xvel yvel ?yvel)) :percepts((self ?self xvel ?xvel yvel ?yvel)) :tests ((< (+ (* ?xvel ?xvel) (* ?yvel ?yvel)) 1))) :tests ((< (+ (* ?xvel ?xvel) (* ?yvel ?yvel)) 1))) ((in-region ?self ?region) ((in-region ?self ?region) :percepts((self ?self region ?region))) :percepts((self ?self region ?region))) ((connected-region ?target ?gateway) ((connected-region ?target ?gateway) :percepts ((gateway ?gateway region ?target))) :percepts ((gateway ?gateway region ?target))) ((blocked-gateway ?gateway) ((blocked-gateway ?gateway) :percepts((gateway ?gateway visible1 ?v1 visible2 ?v2)) :percepts((gateway ?gateway visible1 ?v1 visible2 ?v2)) :tests((and (equal ?v1 'B) (equal ?v2 'B))) ) :tests((and (equal ?v1 'B) (equal ?v2 'B))) ) ((first-side-blocked-gateway ?gateway) ((first-side-blocked-gateway ?gateway) :percepts((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :percepts((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :tests ((equal ?type 'WALK) (equal ?v1 'B) (equal ?v2 'C))) :tests ((equal ?type 'WALK) (equal ?v1 'B) (equal ?v2 'C))) ((flag-captured ?self flag1) ((flag-captured ?self flag1) :percepts ((self ?self holding ?flag1) (entity ?flag1) :percepts ((self ?self holding ?flag1) (entity ?flag1) :tests ((not (equal ?flag NIL)))) :tests ((not (equal ?flag NIL))))

25. ((not-stopped ?self) :percepts ((self ?self)) :percepts ((self ?self)) :relations ((not (stopped ?self)))) :relations ((not (stopped ?self)))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :percepts ((gateway ?gateway type ?type visible1 ?v1 visible2 ?v2)) :relations ((not-stopped ?self)) :relations ((not-stopped ?self)) :tests ((equal ?type 'WALK) (equal ?v1 'C) (equal ?v2 'C))) :tests ((equal ?type 'WALK) (equal ?v1 'C) (equal ?v2 'C))) ((stopped-in-region ?self ?region) ((stopped-in-region ?self ?region) :percepts ((self ?self)) :percepts ((self ?self)) :relations ((in-region ?self ?region)(stopped ?self))) :relations ((in-region ?self ?region)(stopped ?self))) ((crossable-region ?target) ((crossable-region ?target) :percepts ((self ?self) (region ?target)) :percepts ((self ?self) (region ?target)) :relations ((connected-region ?target ?gateway) (clear-gateway ?gateway))) :relations ((connected-region ?target ?gateway) (clear-gateway ?gateway))) ((in-region-able ?self ?current ?region) ((in-region-able ?self ?current ?region) :percepts ((self ?self) (region ?current) (region ?region)) :percepts ((self ?self) (region ?current) (region ?region)) :relations ((crossable-region ?region) (in-region ?self ?current))) :relations ((crossable-region ?region) (in-region ?self ?current))) Nonprimitive Urban Combat Concepts

26. ((in-region ?self ?region) :percepts((self ?self) (region ?current) (region ?region) :percepts((self ?self) (region ?current) (region ?region) (gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 (gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 dist2 ?dist2 angle2 ?angle2)) dist2 ?dist2 angle2 ?angle2)) :start((in-region-able ?self ?current ?region)) :start((in-region-able ?self ?current ?region)) :actions((*move-toward (max ?dist1 ?dist2) (mid-direction ?angle1 ?angle2)))) :actions((*move-toward (max ?dist1 ?dist2) (mid-direction ?angle1 ?angle2)))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((self ?self) (gateway ?gateway type WALK)) :percepts ((self ?self) (gateway ?gateway type WALK)) :start ((stopped ?self)) :start ((stopped ?self)) :actions((*move-toward 50 0))) :actions((*move-toward 50 0))) ((clear-gateway ?gateway) ((clear-gateway ?gateway) :percepts ((gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 :percepts ((gateway ?gateway region ?region dist1 ?dist1 angle1 ?angle1 visible1 ?v1dist2 ?dist2 angle2 ?angle2 visible2 ?v2)) visible1 ?v1dist2 ?dist2 angle2 ?angle2 visible2 ?v2)) :start ((first-side-blocked-gateway ?gateway)) :start ((first-side-blocked-gateway ?gateway)) :actions ((*move-toward ?dist2 ?angle2))) :actions ((*move-toward ?dist2 ?angle2))) ((flag-captured ?self ?flag) ((flag-captured ?self ?flag) :percepts ((self ?self) (entity ?flag dist ?dist angle ?angle)) :percepts ((self ?self) (entity ?flag dist ?dist angle ?angle)) :start ((in-region ?self region107)) :start ((in-region ?self region107)) :actions ((*move-toward ?dist ?angle))) :actions ((*move-toward ?dist ?angle))) Primitive Skills for Urban Combat

27. ((crossable-region ?target) :percepts ((region ?target) (gateway ?gateway)) :percepts ((region ?target) (gateway ?gateway)) :start ((connected-region ?target ?gateway)) :start ((connected-region ?target ?gateway)) :subgoals ((clear-gateway ?gateway))) :subgoals ((clear-gateway ?gateway))) ((in-region-able ?self ?current ?region) ((in-region-able ?self ?current ?region) :percepts ((self ?self) (region ?current) (region ?region)) :percepts ((self ?self) (region ?current) (region ?region)) :subgoals ((in-region ?self ?current) (crossable-region ?region))) :subgoals ((in-region ?self ?current) (crossable-region ?region))) ((stopped-in-region ?self ?region) ((stopped-in-region ?self ?region) :percepts ((self ?self) (region ?region)) :percepts ((self ?self) (region ?region)) :subgoals ((in-region ?self ?region) (stopped ?self))) :subgoals ((in-region ?self ?region) (stopped ?self))) ((stopped-in-region ?self ?current) ((stopped-in-region ?self ?current) :percepts ((self ?self)) :percepts ((self ?self)) :start((in-region ?self ?current)) :start((in-region ?self ?current)) :subgoals ((stopped ?self))) :subgoals ((stopped ?self))) ((flag-captured ?self ?flag) ((flag-captured ?self ?flag) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region ?self region107) (flag-captured ?self ?flag))) :subgoals ((in-region ?self region107) (flag-captured ?self ?flag))) Nonprimitive Skills for Urban Combat

28. ((in-region ?self region105) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region115 region105) (in-region ?self region105))) :subgoals ((in-region-able ?self region115 region105) (in-region ?self region105))) ((in-region ?self region114) ((in-region ?self region114) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region105 region114) (in-region ?self region114))) :subgoals ((in-region-able ?self region105 region114) (in-region ?self region114))) ((in-region ?self region110) ((in-region ?self region110) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region114 region110) (in-region ?self region110))) :subgoals ((in-region-able ?self region114 region110) (in-region ?self region110))) ((in-region ?self region116) ((in-region ?self region116) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region110 region116) (in-region ?self region116))) :subgoals ((in-region-able ?self region110 region116) (in-region ?self region116))) ((in-region ?self region107) ((in-region ?self region107) :percepts ((self ?self)) :percepts ((self ?self)) :subgoals ((in-region-able ?self region116 region107) (in-region ?self region107))) :subgoals ((in-region-able ?self region116 region107) (in-region ?self region107))) Route Knowledge for Urban Combat

29. Hierarchical Structure of Long-Term Memory concepts skills Each concept is defined in terms of other concepts and/or percepts. Each skill is defined in terms of other skills, concepts, and percepts. I CARUS organizes both concepts and skills in a hierarchical manner.

30. Hierarchical Structure of Long-Term Memory conceptsskills For example, the skill highlighted here refers directly to the highlighted concepts. I CARUS interleaves its long-term memories for concepts and skills.

31. Basic I CARUS Processes concepts skills Concepts are matched bottom up, starting from percepts. Skill paths are matched top down, starting from intentions. I CARUS matches patterns to recognize concepts and select skills.

32. Transfer in I CARUS What forms of knowledge does I CARUS transfer? What forms of knowledge does I CARUS transfer? Hierarchical/relational skill and concept clauses Hierarchical/relational skill and concept clauses Where does the transferred knowledge originate? Where does the transferred knowledge originate? It comes from experience on source problems and background knowledge It comes from experience on source problems and background knowledge How does I CARUS know what to transfer? How does I CARUS know what to transfer? Skills are indexed by goals they achieve, with preference for more recently learned structures Skills are indexed by goals they achieve, with preference for more recently learned structures

33. A Transfer Scenario from Urban Combat Target Problem Source Problem Here the first part of the source route transfers to the target, but the second part must be learned to solve the new task.

34. Source Target Shared structures Structures Transferred in Scenario

35. We have also tested ICARUS in domains like FreeCell solitaire. Other Transfer Results with I CARUS Experiments suggest that learned knowledge transfers well here.

36. Key Ideas about Transfer in I CARUS The most important transfer concerns goal-directed behavior that involves sequential actions aimed toward an objective. The most important transfer concerns goal-directed behavior that involves sequential actions aimed toward an objective. Transfer mainly involves the reuse of knowledge structures. Transfer mainly involves the reuse of knowledge structures. Organizing structures in a hierarchy aids reuse and transfer. Organizing structures in a hierarchy aids reuse and transfer. Indexing skills by goals they achieve determines relevance. Indexing skills by goals they achieve determines relevance. One can learn hierarchical, relational, goal-directed skills by analyzing traces of expert behavior and problem solving. One can learn hierarchical, relational, goal-directed skills by analyzing traces of expert behavior and problem solving. Skill learning can build upon structures acquired earlier. Skill learning can build upon structures acquired earlier. Successful transfer benefits from knowledge-based inference to recognize equivalent situations. Successful transfer benefits from knowledge-based inference to recognize equivalent situations.

37. General Game Playing GGP is a Web-based software environment developed at Stanford that supports: logical specification of many different games in terms of: logical specification of many different games in terms of: relational descriptions of states relational descriptions of states legal moves and their effects legal moves and their effects goal relations and their payoffs goal relations and their payoffs management of matches between automated players management of matches between automated players competitions that involve many players and games competitions that involve many players and games The GGP framework (http://games.stanford.edu) encourages research on systems that exhibit general intelligence. This summer, AAAI will host its second GGP competition.

38. Transfer in General Game Playing GGP is especially appropriate for research on transfer because: one can specify game environments and rules precisely one can specify game environments and rules precisely one can formalize relationships between different games one can formalize relationships between different games to support different types/levels of transfer to support different types/levels of transfer including ones that involve representational shifts including ones that involve representational shifts the game manager eases running of experiments the game manager eases running of experiments We are using GGP to evaluate methods for transferring learned knowledge to new but related settings.

39. A GGP Scenario for Extrapolation Transfer Source Target Goal:White king and rook in castle formation Rules: Standard Chess Concept s: castle-blocked(?x,?y)castle-threatened(?x,?y)king-in-check(?player,?x,?y)...

40. Open Research Problems Goal transfer across tasks with distinct but related objectives Goal transfer across tasks with distinct but related objectives Negative transfer minimizing use of inappropriate knowledge Negative transfer minimizing use of inappropriate knowledge Context handling avoiding catastrophic interference Context handling avoiding catastrophic interference Representation mapping Representation mapping Lateral Deep analogy that involves partial isomorphisms Lateral Deep analogy that involves partial isomorphisms Vertical Bootstrapped learning that builds on lower levels Vertical Bootstrapped learning that builds on lower levels There remain many research issues that we must still address: These challenges should keep our field occupied for some time.

41. Closing Remarks involves the sequential reuse of knowledge structures involves the sequential reuse of knowledge structures takes many forms depending on source/target relationships takes many forms depending on source/target relationships has been repeatedly examined within psychology/education has been repeatedly examined within psychology/education has received little attention in AI and machine learning has received little attention in AI and machine learning requires a fairly sophisticated experimental method requires a fairly sophisticated experimental method Transfer of learned knowledge is an important capability that: Transfer originated in psychology, and it is best studied in the context of cognitive architectures, which have similar roots.

42. End of Presentation