2. ARTIFICIAL INTELLIGENCE: THE MAIN IDEAS Nils J. Nilsson OLLI COURSE SCI 102 Tuesdays, 11:00 a.m. – 12:30 p.m. Winter Quarter, 2013 Higher Education Center, Medford Room 226 nilsson@cs.stanford.edu http://ai.stanford.edu/~nilsson/ Course Web Page: www.sci102.com/ For Information about parking near the HEC, go to: http://www.ci.medford.or.us/page.asp?navid=2117 There are links on that page to parking rules and maps

3. AI in the News ?

4. Perception Action Selection Memory PART ONE (Continued) REACTIVE AGENTS

5. Summary: Neural Networks Have Many Applications

6. But Some Are Not Very User- Friendly Fair Isaac Experience

7. Models of the Cortex Using Deep, Hierarchical Neural Networks All connections are bi-directional

9. The Neocortex

10. Two Pioneers in Using Networks to Model the Cortex Geoffrey Hinton Jeff Hawkins Hierarchical Temporal Memory

11. More About Jeff Hawkinss Ideas http://www.numenta.com/htm- overview/education/HTM_CorticalLearningAlgorithms.pdf

12. Dileep Georges Hierarchical Temporal Memory (HTM) Model A Convolutional Network George is a founder of startup, Vicarious http://vicarious.com/team.html

13. A Mini-Column of the Neo-Cortex From: HIERARCHICAL TEMPORAL MEMORY http://www.numenta.com/htm-overview/education/HTM_CorticalLearningAlgorithms.pdf

14. Figure 10. Columnar organization of the microcircuit. George, Dileep and Hawkins, Jeff: (2009) Towards a Mathematical Theory of Cortical Micro-circuits. PLoS Comput Biol 5(10): e1000532. doi:10.1371/journal.pcbi.1000532 http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1000532

15. Figure 9. A laminar biological instantiation of the Bayesian belief propagation equations used in the HTM nodes. George D, Hawkins J (2009) Towards a Mathematical Theory of Cortical Micro-circuits. PLoS Comput Biol 5(10): e1000532. doi:10.1371/journal.pcbi.1000532 http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1000532

16. Ray Kurzweils New Book

17. Unsupervised Learning

18. Letting Networks Adapt to Their Inputs All connections are bi-directional Massive number of inputs Weight Values Become Those For Extracting Features of Inputs Honglak Lee,et al., Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations, Proceedings of the 26th Annual International Conference on Machine Learning, 2009

19. Hubel & Wiesels Detector Neurons Short bar of light projected onto a cats retina Response of a single neuron in the cats visual cortex (as detected by a micro-electrode in the anaesthetized cat) David Hubel, Torsten Wiesel

20. Use of Deep Networks With Unsupervised Learning First Layer Learns Building-Block Features Common to Many Images All connections are bi-directional

21. Second Layer Learns Features Common Just to Cars, Faces, Motorbikes and Airplanes cars, faces, motorbikes, airplanes

22. Third Layer Learns How to Combine the Features of the Second Layer Into a Representation of the Input cars, faces, motorbikes, airplanes

23. Output Layer Can be Used to Make a Decision CAR

24. The Net Can Make Predictions About Unseen Parts of the Input

25. Building High-level Features Using Large Scale Unsupervised Learning Quoc V. Lee,et al. (Google and Stanford) 1,000 Google Computers, 1,000,000,000 Connections

26. 10 million 200x200 pixel images downloaded from the Internet (stills from YouTube) Unsupervised learning for three days Large Scale Unsupervised Learning (Continued) a face neuron Recognizes 22,000 object categories a cat neuron

27. 81.7% accuracy in detecting faces out of 13,026 faces in a test set One Result http://research.google.com/archive/unsupervised_icml2012.html For more information about these experiments at Google/Stanford, see:

28. Using Models (i.e., Memory) Can Make Agents Even More Intelligent Perception Action Selection Model of World (e.g., a map)

29. Types of Models Maps Memory of Previous States List of State-Action Pairs

30. Models can be pre-installed or learned

31. where am I? where is everything else? Learning and Using Maps Neato Robot Vacuum

32. Neato Robotics Mapping System

33. NEATO ROBOTICS XV11

34. S-R Rules Using State of the Agent Perception Action Selection determines thestateof the world Library of States and Actions (Memory) IF state 1, THEN action a IF state 2, THEN action b...

35. Ways to Represent States Lists of numbers, such as (1,7,3,4,6) Arrays, such as Statements, such as Color(Walls, LightBlue) Shape(Rectangular)...

36. Library of States & Actions (1,7,3,4,6) a (1,6,2,8,7) b (4,5,1,8,5) c... (7,4,8,9,2) k (1,5,2,8,6) Input (present state) Closest Match

37. Example: Face Recognition Using a large database containing many, many images of faces, a small set of building-block faces is computed: The average of all faces: http://cognitrn.psych.indiana.edu/nsfgrant/FaceMachine/faceMachine.html

38. Familiar Uses of Building Blocks A Musical Tone Consists of Harmonics

39. Library of Known Faces (Represented as composites of the building-block faces) (0,0,1,0,0,-2,-2,0,-1,-2,-2,-1,2,-1,0) (2,2,-2,0,0,1,2,2,-1,2,2,-1,,0,2,0) (-3,2,1,1,-2,1,-2,3,0,0,0,-4,-3,2,-2)(4,1,3,-1,4,0,4,4,1,4,4,-4,4,-4,-4) Plus Thousands More SamJoe Sue Mike

40. Library of Known Faces Query Face Represented as a composite of the building-block faces (present state) Sue is the Closest Match Face Recognition Sam Joe Sue Mike (-2,2,1,1,-2,1,-2,3,1,0,0,-4,-3,2,-2) (0,0,1,0,0,-2,-2,0,-1,-2,-2,-1,2,-1,0) (2,2,-2,0,0,1,2,2,-1,,2,2,-1,,0,2,0) (-3,2,1,1,-2,1,-2,3,0,0,0,-4,-3,2,-2) (4,1,3,-1,4,0,4,4,1,4,4,-4,4,-4,-4)

41. Another Kind of Model A table of states and actions and values State and ActionValue State 1, Action b State 1, Action c 13 7 State 2, Action g State 2, Action h State 2, Action j 394394 State 3, Action m State 3, Action n 2626

42. Why have values for multiple actions instead of just noting the best action? Because the values in the table can be changed (learned) depending on experience! REINFORCEMENT LEARNING (Another Point of Contact with Brains)

43. Pioneers in the Use of Reinforcement Learning in AI Andy BartoRich SuttonChris Watkins

44. An Example: Learning a Maze

45. But the Mouse Doesnt Have a Map of the Maze (Like We Do) Instead it remembers the states it visits and assigns their actions random initial values State and ActionValue State 1, up3 State 2, left State 2, down State 2, right 256256 <add more when encountered)

46. It Can Change the Values in the Table The First Step (state 1, up) gets initial random value 3

47. state 2, has 3 actions, each with initial random values There is only one action possible (up), and the mouse ends up in state 2

48. Now the mouse updates the value of (state 1, up) in its table 5 value propagates backward (possibly with some loss)

49. Sooner or later, the mouse stumbles into the goal and gets a reward

50. The reward value is propagated backward value propagates backward (with some loss) 99

51. And So On... With a Lot of Exploration, the Mouse Learns the Maze

52. Reward Centers Alter Dopamine Concentrations Reinforcement Learning in Animals

53. The Brains Reward Centers Associate Values With States A Neural Substrate of Prediction and Reward, Wolfram Schultz, Peter Dayan, P. Read Montague, Science, 275 :1593-1599, 14 March 1997. When The Actual Value of a State is Better Than The Predicted Value, Dopamine is Released; When Worse Than Expected Dopamine is Inhibited.

54. Learning to Flip Pancakes By Sylvain Calinon, Advanced Robotics Department Italian Institute of Technology

55. Learning to Fly a Model Helicopter

56. TD-Gammon Temporal Difference Learning and TD-Gammon, By Gerald Tesauro www.research.ibm.com/massive/tdl.html backgammon boards resulting from a move from current board predicted probabilities of winning (the values of the input boards) train neural network so that value of current board is closer to value of the best next board

57. Summary of TD-Gammon Results

58. Bottom Line: Reactive Agents Can Be Quite Effective! See: Daniel Kahneman, Thinking, Fast and Slow Perception Action Selection Model of World

59. An Interesting Novel About Neural Networks Galatea 2.2, Richard Powers