1. Dynamic Bayesian Networks CSE 473

2. © Daniel S. Weld 2 473 Topics Agency Problem Spaces Search Knowledge Representation Reinforcement Learning InferencePlanningLearning LogicProbability

3. © Daniel S. Weld 3 Last Time Basic notions Bayesian networks Statistical learning Parameter learning (MAP, ML, Bayesian L) Naïve Bayes Structure Learning Expectation Maximization (EM) Dynamic Bayesian networks (DBNs) Markov decision processes (MDPs)

4. © Daniel S. Weld 4 Generative Planning Input Description of (initial state of) world (in some KR) Description of goal (in some KR) Description of available actions (in some KR) Output Controller E.g. Sequence of actions E.g. Plan with loops and conditionals E.g. Policy = f: states -> actions

5. © Daniel S. Weld 5 Simplifying Assumptions Environment Percepts Actions What action next? Static vs. Dynamic Fully Observable vs. Partially Observable Deterministic vs. Stochastic Instantaneous vs. Durative Full vs. Partial satisfaction Perfect vs. Noisy Deterministic vs. Stochastic

6. © Daniel S. Weld 6 StaticDeterministicObservable Instantaneous Propositional “Classical Planning” Dynamic Replanning/ Situated Plans Durative Temporal Reasoning Continuous Numeric Constraint reasoning (LP/ILP) Stochastic Contingent/Conformant Plans, Interleaved execution MDP Policies POMDP Policies Partially Observable Contingent/Conformant Plans, Interleaved execution Semi-MDP Policies

7. © Daniel S. Weld 7 Uncertainty Disjunctive Next state could be one of a set of states. Probabilistic Next state is a probability distribution over the set of states. Which is a more general model?

8. © Daniel S. Weld 8 STRIPS Action Schemata (:operator pick-up :parameters ((block ?ob1)) :precondition (and (clear ?ob1) (on-table ?ob1) (arm-empty)) :effect (and (not (clear ?ob1)) (not (on-table ?ob1)) (not (arm-empty)) (holding ?ob1))) Instead of defining ground actions: pickup-A and pickup-B and … Define a schema: Note: strips doesn’t allow derived effects; you must be complete! }

9. © Daniel S. Weld 9 Nondeterministic “STRIPS”? pick_up_A arm_empty clear_A on_table_A ? + arm_empty - on_table_A - holding_A

10. © Daniel S. Weld 10 Probabilistic STRIPS? pick_up_A arm_empty clear_A on_table_A P<0.9 + arm_empty - on_table_A - holding_A But what does this all mean anyway?

11. © Daniel S. Weld 11 Defn: Markov Model Q: set of states  init prob distribution A: transition probability distribution

12. © Daniel S. Weld 12 Probability Distribution, A Forward Causality The probability of s t does not depend directly on values of future states. Probability of new state could depend on The history of states visited. Pr(s t |s t-1,s t-2,…, s 0 ) Markovian Assumption Pr(s t |s t-1,s t-2,…s 0 ) = Pr(s t |s t-1 ) Stationary Model Assumption Pr(s t |s t-1 ) = Pr(s k |s k-1 ) for all k.

13. © Daniel S. Weld 13 What is the cost of representing a stationary probability distribution? Can we do something better?

14. © Daniel S. Weld 14 Representing A Q: set of states  init prob distribution A: transition probabilities how can we represent these? s0s0 s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 s 0 s 1 s 2 s 3 s 4 s 5 s 6 s0s1s2s3s4s5s6s0s1s2s3s4s5s6 p 12 Probability of transitioning from s 1 to s 2 ∑ ?

15. © Daniel S. Weld 15 Factoring Q Represent Q simply as a set of states? Is their internal structure? Consider a robot domain What is the state space? s0s0 s1s1 s2s2 s3s3 s4s4 s5s5 s6s6

16. © Daniel S. Weld 16 A Factored domain Variables : has_user_coffee (huc), has_robot_coffee (hrc), robot_is_wet (w), has_robot_umbrella (u), raining (r), robot_in_office (o) How many states? 2 6 = 64

17. © Daniel S. Weld 17 Representing  Compactly Q: set of states  init prob distribution How represent this efficiently? s0s0 s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 r u hrc w With a Bayes net (of course!)

18. © Daniel S. Weld 18 Representing A Compactly Q: set of states  init prob distribution A: transition probabilities s0s0 s1s1 s2s2 s3s3 s4s4 s5s5 s6s6 s 0 s 1 s 2 s 3 s 4 s 5 s 6 … s 35 s 0 s 1 s 2 s 3 s 4 s 5 s 6 … s 35 p 12 How big is matrix version of A?4096

19. © Daniel S. Weld 19 Dynamic Bayesian Network huc hrc w u r o o r u w huc 8 4 16 4 2 2 Total values required to represent transition probability table = 36 Vs. 4096 required for a complete state probablity table? T T+1

20. © Daniel S. Weld 20 Dynamic Bayesian Network huc hrc w u r o o r u w huc T T+1 Also known as a Factored Markov Model Defined formally as * Set of random variables * BN for initial state * 2-layer DBN for transitions

21. © Daniel S. Weld 21 This is Really a “Schema” huc hrc w u r o o r u w huc T T+1 “Unrolling”

22. © Daniel S. Weld 22 Semantics: A DBN  A Normal Bayesian Network o r u w hrc huc hrc w u r o 0 o r u w huc o r u w hrc huc 12 3

23. © Daniel S. Weld 23 Multiple Actions Variables : has_user_coffee (huc), has_robot_coffee (hrc), robot_is_wet (w), has_robot_umbrella (u), raining (r), robot_in_office (o) Actions : buy_coffee, deliver_coffee, get_umbrella, move We need an extra variable (Alternatively a separate 2DBN / action)

24. © Daniel S. Weld 24 Actions in DBN huc hrc w u r o o r u w huc T T+1 a

25. © Daniel S. Weld 25 What can we do with DBNs? State estimation Planning If you add rewards (Next class)

26. © Daniel S. Weld 26 State Estimation O R Huc Hrc U W ? ? MoveBuyC

27. © Daniel S. Weld 27 Noisy Observations huc hrc w u r o o r u w huc T T+1 So True state vars (unobserved) Noisy, sensor reading o So.9.2 a Action var Known with certainty

28. © Daniel S. Weld 28 State Estimation O R Huc Hrc U W ? ? MoveBuyC So=T …

29. © Daniel S. Weld 29 Maybe we don’t know action! huc hrc w u r o o r u w huc T T+1 So True state vars (unobserved) Noisy, sensor reading o So.9.2 a Action var (other agent’s action)

30. © Daniel S. Weld 30 State Estimation O R Huc Hrc U W ? ? ?? So=T …

31. © Daniel S. Weld 31 Inference Expand DBN (schema) into BN Use inference as shown last week Problem? Particle Filters

32. © Daniel S. Weld 32 Particle Filters Create 1000s of “Particles” Each one has a “copy” of each random var These have concrete values Distribution is approximated by whole set Approximate technique! Fast! Simulate Each time step, iterate per particle Use the 2 layer DBN + random # generator to Predict next value for each random var Resample!

33. © Daniel S. Weld 33 Resampling Simulate Each time step, iterate per particle Use the 2 layer DBN + random # generator to Predict next value for each random var Resample: Given observations (if any) at new time point Calculate the likelihood of each particle Create a new population of particles by –Sampling from old population –Proportionally to likelihood