1. Dynamics of Learning & Distributed Adaptation PI: James P. Crutchfield, Santa Fe Institute Second PI Meeting, 17-19 April 2001, SFe Dynamics of Learning: Single-agent learning theory Emergence of Distributed Adaptation: Agent-collective learning theory Strategies: Simulation: learning dynamics, collective behavior Theory: basic constraints, quantitative predictions Dynamics of Learning: Single-agent learning theory Emergence of Distributed Adaptation: Agent-collective learning theory Strategies: Simulation: learning dynamics, collective behavior Theory: basic constraints, quantitative predictions

2. REF: Control and Adaptation in Heterogeneous, Dynamics Environments Traffic dynamics Food delivery to major cities Electrical power grid Internet packet dynamics Market economies Dynamic task allocation by ant colonies Questions How do large-scale systems maintain coordination? How does one design such large-scale systems? Traffic dynamics Food delivery to major cities Electrical power grid Internet packet dynamics Market economies Dynamic task allocation by ant colonies Questions How do large-scale systems maintain coordination? How does one design such large-scale systems?

3. REF: Control and Adaptation in Heterogeneous, Dynamics Environments Common features Distributed systems with many subsystems Adaptive response to internal/external change No global control, but still perform function Local intelligence: – Controller Sensor Internal model Actuators Common vocabulary – Agents – Environment = Other agents + Exogenous Influences Common features Distributed systems with many subsystems Adaptive response to internal/external change No global control, but still perform function Local intelligence: – Controller Sensor Internal model Actuators Common vocabulary – Agents – Environment = Other agents + Exogenous Influences

4. What is an Intelligent Agent? The Learning Channel TLC: Adaptation of Communication Channel What are fundamental constraints on learning? – How to measure environmental structure? – How to measure “cognitive” capacity of learning agents? – How much data for a given complexity of inferred model? TLC: Adaptation of Communication Channel What are fundamental constraints on learning? – How to measure environmental structure? – How to measure “cognitive” capacity of learning agents? – How much data for a given complexity of inferred model?

5. Computational Mechanics: Preliminaries www.santafe.edu/projects/CompMech Observations: s  = s  s  Past  Future: … s -L s -L+1 …s -1 s 0 |s 1 …s L-1 s L … Probabilities: Pr(s  ), Pr(s  ), Pr(s  ) Uncertainty: Entropy H[P] = -  i p i log p i [bits] Prediction error: Entropy Rate h  = H[Pr(s i |s i-1 s i-2 s i-3 …)] Information transmitted to future: Excess Entropy E = H[Pr(s  )/ (Pr(s  )Pr(s  ))] Measure of independence: Is Pr(s  )=Pr(s  )Pr(s  )? Describes information in “raw” sequence blocks Observations: s  = s  s  Past  Future: … s -L s -L+1 …s -1 s 0 |s 1 …s L-1 s L … Probabilities: Pr(s  ), Pr(s  ), Pr(s  ) Uncertainty: Entropy H[P] = -  i p i log p i [bits] Prediction error: Entropy Rate h  = H[Pr(s i |s i-1 s i-2 s i-3 …)] Information transmitted to future: Excess Entropy E = H[Pr(s  )/ (Pr(s  )Pr(s  ))] Measure of independence: Is Pr(s  )=Pr(s  )Pr(s  )? Describes information in “raw” sequence blocks

6. Computational Mechanics: Mathematical Foundations Casual state = Condition of knowledge about future  -Machines = {Causal states, Transitions} Optimality Theorem:  -Machines are optimal predictors of environment. Minimality Theorem: Of the optimal predictors,  -Machines are smallest. Uniqueness Theorem: Up to isomorphism, an  -Machine is unique. The Point: Discovering an  -Machine is the goal for any learning process. Practicalities may preclude this, but this is the goal. (w/ DP Feldman/CR Shalizi) Casual state = Condition of knowledge about future  -Machines = {Causal states, Transitions} Optimality Theorem:  -Machines are optimal predictors of environment. Minimality Theorem: Of the optimal predictors,  -Machines are smallest. Uniqueness Theorem: Up to isomorphism, an  -Machine is unique. The Point: Discovering an  -Machine is the goal for any learning process. Practicalities may preclude this, but this is the goal. (w/ DP Feldman/CR Shalizi)

7. Computational Mechanics: Why Model? Structural Complexity of Information Source C  = H[Pr( S )], S = {Casual states} Uses: – Environ’l complexity: Amount/kind of relevant structure – Agent’s inferential capacity: Sophistication of models? Theorem: E  C  Conclusion: Build models vs. storing only E bits of history. – Raw sequence blocks do not allow optimal prediction, only E bits of mutual information in blocks. – Optimal prediction requires larger model: 2 C , not 2 E. – Explicit: 1D Range-R Ising spin system: C  = E +R h . Structural Complexity of Information Source C  = H[Pr( S )], S = {Casual states} Uses: – Environ’l complexity: Amount/kind of relevant structure – Agent’s inferential capacity: Sophistication of models? Theorem: E  C  Conclusion: Build models vs. storing only E bits of history. – Raw sequence blocks do not allow optimal prediction, only E bits of mutual information in blocks. – Optimal prediction requires larger model: 2 C , not 2 E. – Explicit: 1D Range-R Ising spin system: C  = E +R h .

8. Synchronizing to the Environment— Constraints on Agent Learning How does an agent come to know the environment? Agent synchronized to the environment when Agent Knows the (Hidden) State of the Environment Here an information theoretic answer Focus on Entropy Growth H(L) = H[Pr(s L )] Take derivatives and integrals of H(L) Recover in one framework all existing quantities h , E, and G Introduce a new quantity: Transient Information T How does an agent come to know the environment? Agent synchronized to the environment when Agent Knows the (Hidden) State of the Environment Here an information theoretic answer Focus on Entropy Growth H(L) = H[Pr(s L )] Take derivatives and integrals of H(L) Recover in one framework all existing quantities h , E, and G Introduce a new quantity: Transient Information T

9. Entropy Growth H(L)

10. Entropy Convergence h  (L) =  H(L)

11. Predictability Gain  2 H(L)

12. Example: All Period-5 Processes Three unique templates – 10000 – 10101 – 11000 Three unique templates – 10000 – 10101 – 11000

13. Example: Golden Mean Process “No consecutive 0s”

14. Example: Even Process “Even blocks of 1s”

15. Example: RRXOR Process...S 1 S 2 S 3 S 1 S 2 S 3... – S 1 random – S 2 random – S 3 = XOR(S 1,S 2 )...S 1 S 2 S 3 S 1 S 2 S 3... – S 1 random – S 2 random – S 3 = XOR(S 1,S 2 )

16. Example: NonDeterministic Process A Hidden Markov Model

17. Example: Morse-Thue Process Production rules: – 1  10 – 0  11 Infinite Memory Production rules: – 1  10 – 0  11 Infinite Memory

18. Regularities Unseen, Randomness Observed Consequence: Ignore Structure  More Unpredictable

19. Regularities Unseen, Randomness Observed Consequence: Assume Instant Synchronization  More Predictable (False) Consequence: Assume Instant Synchronization  More Predictable (False)

20. Regularities Unseen, Randomness Observed Consequence: Assume Synchronization  Less Memory

21. Regularities Unseen, Randomness Observed Conclusions Quantities key to synchronization, agent modeling h , E, T, and G Relationships between them via a single framework Derived consequences of ignoring them Can now distinguish kinds of synchronization Improved model building and control system design Conclusions Quantities key to synchronization, agent modeling h , E, T, and G Relationships between them via a single framework Derived consequences of ignoring them Can now distinguish kinds of synchronization Improved model building and control system design