1. Hierarchical Temporal Memory (HTM)
A new computational paradigm based on cortical theory
Jeff Hawkins
May 10, 2006
IBM

3. Today’s PDA Market Indicator
Pipe Dream
Driven By
Greed
Mother
Of All
Markets

4. Today’s Cognitive Computing Indicator
Any
Moment
Now
Not in our
Lifetime

5. Not in our lifetime Decades of effort Not much successAI
neural networks
fuzzy logic
5th generation project
decade of the brain
Not much success
vision, language, robotics
Brain is very complex

6. Not in our lifetime Any moment now Decades of effort Not much successAI
neural networks
fuzzy logic
5th generation project
decade of the brain
Not much success
vision, language, robotics
Brain is very complex
Any moment now
Neocortex:
Fast
Flexible
Robust
100 years of data
Anatomical, physiological
Mathematics
Common cortical algorithm
Cortical Theory (HTM)

7. World Senses HTM/Cortex People Cars Buildings Words Songs Ideas
patterns
World
Senses
HTM/Cortex

8. “Causes” “Beliefs” World Senses HTM/Cortex People Cars Buildings Words
Songs
Ideas
cause
cause
cause
cause
cause
cause
patterns
World
Senses
HTM/Cortex

9. 1 Discover causes in the world 2 Infer causes of novel input
HTM
Causes
Representations
of Causes
What does an HTM do?
1 Discover causes in the world
2 Infer causes of novel input
3 Predict future
4 Direct motor behavior

10. HTMs use a hierarchy of memory nodes
Belief
Sensory data

11. HTMs use a hierarchy of memory nodes
Beliefs
Sensory data
Each node: Discovers causes (of its input)
Passes beliefs up
Passes predictions down

12. HTMs use a hierarchy of memory nodes
Beliefs
Sensory data
Each node: Discovers causes (of its input)
Passes beliefs up
Passes predictions down
Each node: Stores common sequences
Changing sensory data forms stable beliefs at top
Stable beliefs at top form changing sensory predictions

13. 1) Why does hierarchy make a difference?
2) How does each node discover and infer causes?

14. Why does hierarchy make a difference?
Shared representations lead to generalization and efficiency

15. Why does hierarchy make a difference?
Shared representations lead to generalization and efficiency
HTM hierarchy matches spatial and temporal hierarchy of causes in world

16. Why does hierarchy make a difference?
Shared representations lead to generalization and efficiency
HTM hierarchy matches spatial and temporal hierarchy of causes in world
Belief propagation techniques ensure all nodes quickly reach mutually compatible beliefs

17. Belief Propagation 90% cat 80% woof 70% pig image 20% meow
CPT
80% woof
20% meow
70% pig image
30% cat image

18. Why does hierarchy make a difference?
Shared representations lead to generalization and efficiency
HTM hierarchy matches spatial and temporal hierarchy of causes in world
Belief propagation techniques ensure all nodes quickly reach mutually compatible beliefs
Affords mechanism for attention

19. How does each node discover causes?

20. How does each node discover causes?
Learn common spatial patterns
Learn common sequences of spatial patterns

21. How does each node discover causes?
Learn common spatial patterns (things that happen at the same time are likely to have a common cause)

22. How does each node discover causes?
Learn common spatial patterns
Common patterns:
remember
Uncommon patterns:
ignore

23. How does each node discover causes?
Learn common spatial patterns
Learn common sequences of spatial patterns

24. How does each node discover causes?
Learn common spatial patterns
Learn common sequences of spatial patterns
Common sequence:
assign to cause
Common sequence:
assign to cause
Uncommon sequence:
ignore
time

25. How does each node discover causes?
Learn common spatial patterns
Learn common sequences
Use context from above in hierarchy

26. Do HTMs really work?

27. Simple HTM vision system (32x32 pixel)
Level 3
Level 2
Level 1
4 pixels

28. Training images

29. Training
images
Correct
Incorrect

31. Correctly recognized images

32. Numenta Plan
Develop a detailed computational theory of neocortical function (HTM)
On Intelligence (Times Books, 2004)
HTM white paper,
Biological mapping paper, August 2006

33. Numenta Plan
Develop a detailed computational theory of neocortical function (HTM)
Develop a software platform for HTM applications

34. Numenta Platform : Fileserver Run time environment Dev Tools
Node
Processor
Supervisor
API
Configurator
Supervisor
Trainer
Net list
Debugger
Node Processor 2
Gigabit switch :
Node Processor N
Fileserver

35. Numenta Plan
Develop a detailed computational theory of neocortical function (HTM)
Develop a software platform for HTM applications
Multiple processor/server architecture
Optimized C++ routines
Developer toolset with flexible scripting using Python
Supports Linux + MacOS. Windows to come.
Build a community of developers
Early access partners, 2nd meeting end of May 2006
Beta release early 2007

36. Numenta Plan
Develop a detailed computational theory of neocortical function (HTM)
Develop a software platform for HTM applications
Test HTM with a machine vision system

37. Numenta Machine Vision System
Robust Object Recognition From Natural Images
Recognition Task Defined
Data collection in process
Highly realistic 3D models and textures used to generate sequences
90,000 images and 102 sequences collected to date
Each image has accurate alpha channel for programmatic 2D modifications

38. HTM Applications What humans find easy and computers hard
vision, language, robotics
many apps from security to self-driving cars
extend with new senses, IR, sonar, radar…
Discovering causes in unusual worlds
geology, markets, weather, physics, genetics

39. HTM Capabilities
Discover causes
Inference
Prediction
Behavior
Beyond biology
Faster
Larger
Exotic senses

40.
(white paper posted this week)

41. Today’s Cognitive Computing Indicator
Any
Moment
Now
Not in our
Lifetime

42. Thank _ _ _

43. world world HTM models world, including hardwired motor behaviors HTM
Representations of motor behavior are auto-associatively paired with motor generators
world
motor

44. Hierarchical Temporal Memory
Powerful, flexible, robust
Can be applied to many problems - vision - language - robotics - manufacturing - business modeling - market modeling - network modeling - resource exploration - weather prediction - math, physics

45. ? Discovering and inferring causes has proven to be
Beliefs
(of causes) ?
Sensory data
Discovering and inferring causes has proven to be
very difficult, e.g.
- visual pattern recognition
- language understanding
- machine learning

46. “What is conspicuously lacking is a broad framework of ideas within which to interpret these different approaches.”
Francis Crick, 1979
Why would I want to do this, elaborate
- science
- commerce
Lots of data, no theory
I have solved/understand the problem conceptually.

47. Belief Propagation

48. Belief Propagation
“maybe diagonal line,
maybe vertical line”

49. Belief Propagation “maybe diagonal line, maybe vertical line”