1. Center for the Study of Language and Information Stanford University, Stanford, California March 20-21, 2004 Symposium on Reasoning and Learning in Cognitive Systems The views contained in these slides are the authors and do not represent official policies, either Expressed or implied, of the Defense Advanced Research Projects Agency or the DoD.

2. Motivation for the Symposium Reasoning and learning are both central aspects of intelligence, Reasoning and learning are both central aspects of intelligence, but the two research groups have become nearly disjoint. but the two research groups have become nearly disjoint. There exist substantial results on reasoning and learning, There exist substantial results on reasoning and learning, but many have forgotten or never learned about them. but many have forgotten or never learned about them. There are growing needs for integrated intelligent systems, There are growing needs for integrated intelligent systems, but research focuses primarily on component technologies. but research focuses primarily on component technologies. DARPA now wants cognitive systems that reason and learn. DARPA now wants cognitive systems that reason and learn. A number of factors encouraged us to organize this symposium: We hope this meeting can help build a community of researchers that can respond to these problems and opportunities.

3. Elements of Machine Learning environment knowledge learning element performance element

4. Learning to Improve Reasoning Given: Initial knowledge elements for a particular domain; Given: Initial knowledge elements for a particular domain; Given: A performance system that can compose these elements dynamically to solve problems or accomplish goals; Given: A performance system that can compose these elements dynamically to solve problems or accomplish goals; Given: Traces of the performance systems behavior or advice about how to solve problems in the domain; Given: Traces of the performance systems behavior or advice about how to solve problems in the domain; Find: New or revised knowledge elements that improve system performance on novel problems. Find: New or revised knowledge elements that improve system performance on novel problems. We can state the general task of learning to improve reasoning as: Much of the early research on machine learning can be cast in just these terms.

5. Some Systems that Reason and Learn LEX (1981) SAGE (1982) UPL (1983) Soar (1984) MORRIS (1985) LEAP (1985) STRIPS (1972) ACT-F (1981) Anzai (1978) MacLearn (1985) Eureka (1989) Prodigy/E (1988) Bagger (1990) PRIAR (1990) Daedalus (1991) Prodigy/A (1993) Cascade (1993) SCOPE (1996)

6. Characteristics of Early Research 1.The performance system engaged in multi-step reasoning by dynamic composition of knowledge elements. 2.Learning methods were typically incremental and integrated with the performance system. 3.Learning was relatively rapid and took at least some domain knowledge into account. 4.Learning was embedded in a problem-solving architecture that made representational and performance assumptions. 5.Research emphasized support of cognitive abilities, such as planning and reasoning, rather than perception and execution. 6.Researchers looked to psychology and logic for ideas, rather than to statistics and operations research.

7. Some Historical Developments Some Historical Developments 1959Creation of the General Problem Solver 1972Development of STRIPS with MACROPs 1978First adaptive production systems developed 1980Carnegie symposium on learning and cognition 1981Growth of work on learning in problem solving 1983Active research on cognitive architectures 1986Growth of explanation-based learning movement 1988Recognition of the utility problem 1989Rise of experimental method, advent of UCI repository 1991ISLE/Stanford symposium on learning and planning 1992Influx of ideas from pattern recognition 1993Excitement about reinforcement learning 1995Influx of ideas from operations research 1998Reduced effort on learning and reasoning

8. Some Encouraging Signs academic courses and tutorials on learning and reasoning; academic courses and tutorials on learning and reasoning; AI Magazine survey of work on learning in planning domains; AI Magazine survey of work on learning in planning domains; interest in model-based and relational reinforcement learning; interest in model-based and relational reinforcement learning; broader interest in integrated cognitive architectures; broader interest in integrated cognitive architectures; DARPA workshop on rapid, embedded, and enduring learning; DARPA workshop on rapid, embedded, and enduring learning; prospects for DARPA program in learning for cognitive systems. prospects for DARPA program in learning for cognitive systems. In recent years, there have been some positive developments: Taken together, these suggested the time had arrived for another meeting on reasoning and learning.

9. Some Omitted Paradigms probabilistic learning and reasoning in Bayesian networks; probabilistic learning and reasoning in Bayesian networks; model-based approaches to learning from delayed reward; model-based approaches to learning from delayed reward; learning action models for use in planning and execution. learning action models for use in planning and execution. The meeting has some great speakers reporting on great topics, but some may wonder why there are no talks on: Each framework can learn knowledge that supports some form of multi-step reasoning or inference. However, research in these paradigms focuses on statistical issues rather than structural ones, which we emphasize here.

10. Some Open Research Problems focused on acquisition of relatively small knowledge bases; focused on acquisition of relatively small knowledge bases; dealt with learning over relatively short periods of time; dealt with learning over relatively short periods of time; emphasized mental processes over action and perception; emphasized mental processes over action and perception; preferred logical, all-or-none frameworks over alternatives; preferred logical, all-or-none frameworks over alternatives; downplayed the role of hierarchical knowledge structures; downplayed the role of hierarchical knowledge structures; relied primarily on initial, handcrafted representations. relied primarily on initial, handcrafted representations. Previous research in the area of learning and reasoning has: Each of these suggests open problems that should be addressed in future projects.

11. Challenge: Learning to Improve Reasoning Challenge: Learning to Improve Reasoning involve one-step decisions for classification or regression; involve one-step decisions for classification or regression; utilize simple reactive control for acting in the world. utilize simple reactive control for acting in the world. Current learning research focuses on performance tasks that: the acquisition of modular knowledge elements that the acquisition of modular knowledge elements that can be composed dynamically by multi-step reasoning. can be composed dynamically by multi-step reasoning. But many other varieties of learning instead involve: We should give more attention to learning such compositional knowledge. knowledge reasoning

12. Challenge: More Rapid Learning Challenge: More Rapid Learning methods for learning classifiers from thousands of cases; methods for learning classifiers from thousands of cases; methods that converge on optimal controllers in the limit. methods that converge on optimal controllers in the limit. Current learning research focuses on asymptotic behavior: learn reasonable behavior from relatively few cases; learn reasonable behavior from relatively few cases; take advantage of knowledge to speed the learning process. take advantage of knowledge to speed the learning process. In contrast, humans are typically able to: We need more work on knowledge-guided learning of this variety. experience performance

13. Challenge: Cumulative Learning Challenge: Cumulative Learning take no advantage of what has been learned before; take no advantage of what has been learned before; provide no benefits for what is learned afterwards. provide no benefits for what is learned afterwards. Current learning research focuses on isolated induction tasks that: incremental acquisition of knowledge over time that incremental acquisition of knowledge over time that builds on knowledge acquired during earlier episodes. builds on knowledge acquired during earlier episodes. In contrast, much human learning involves: We need much more research on such cumulative learning. initial knowledgeextended knowledge

14. Challenge: Evaluating Embedded Learning Challenge: Evaluating Embedded Learning favor work on minor refinements of existing component algorithms; favor work on minor refinements of existing component algorithms; encourage mindless bake offs that provide little understanding. encourage mindless bake offs that provide little understanding. Current evaluation emphasizes static data sets for isolated tasks that: a set of challenging environments that exercise learning and reasoning, a set of challenging environments that exercise learning and reasoning, that include performance tasks of graded complexity and difficulty, and that include performance tasks of graded complexity and difficulty, and that have real-world relevance but allow systematic experimental control. that have real-world relevance but allow systematic experimental control. To support the evaluation of embedded learning systems, we need: battle management in-city driving air reconnaissance

15. An Advertisement: Progress on I CARUS stores long-term knowledge as hierarchical skills and concepts; stores long-term knowledge as hierarchical skills and concepts; encodes short-term elements as instances of long-term structures; encodes short-term elements as instances of long-term structures; uses numeric value functions to select skill paths for execution; uses numeric value functions to select skill paths for execution; modulates reactive behavior with a bias toward persistence; modulates reactive behavior with a bias toward persistence; learns value functions for concepts and durations of skills; learns value functions for concepts and durations of skills; invokes means-ends analysis to handle unexecutable skills; invokes means-ends analysis to handle unexecutable skills; learns new hierarchical skills upon resolution of impasses; learns new hierarchical skills upon resolution of impasses; We are extending I CARUS, an integrated cognitive architecture that: Come to our poster this evening to hear more about the system.

16. End of Presentation