What is a graphical model ? A graphical model is a way of representing probabilistic relationships between random variables. Variables are represented by nodes: Conditional (in)dependencies are represented by (missing) edges: Undirected edges simply give correlations between variables (Markov Random Field or Undirected Graphical model): Directed edges give causality relationships (Bayesian Network or Directed Graphical Model):
Graphical Models e.g. Bayesian networks, Bayes nets, Belief nets, Markov networks, HMMs, Dynamic Bayes nets, etc. Themes: – representation – reasoning – learning Original PGM reference book by Nir Friedman and Daphne Koller.
Motivation for PGMs Let X 1,…,X p be discrete random variables Let P be a joint distribution over X 1,…,X p If the variables are binary, then we need O(2 p ) parameters to describe P Can we do better? Key idea: use properties of independence
Motivation for PGMs (2) Two variables X and Y are independent if – P(X = x|Y = y) = P(X = x) for all values x, y – That is, learning the values of Y does not change prediction of X If X and Y are independent then – P(X,Y) = P(X|Y)P(Y) = P(X)P(Y) In general, if X 1,…,X p are independent, then – P(X 1,…,X p )= P(X 1 )...P(X p ) – Requires O(n) parameters
Motivation for PGMs (3) Unfortunately, most of random variables of interest are not independent of each other A more suitable notion is that of conditional independence Two variables X and Y are conditionally independent given Z if – P(X = x|Y = y,Z=z) = P(X = x|Z=z) for all values x,y,z – That is, learning the values of Y does not change prediction of X once we know the value of Z
Probability theory provides the glue whereby the parts are combined, ensuring that the system as a whole is consistent, and providing ways to interface models to data. The graph theoretic side of graphical models provides both an intuitively appealing interface by which humans can model highly- interacting sets of variables as well as a data structure that lends itself naturally to the design of efficient general-purpose algorithms. Many of the classical multivariate probabilistic systems studied in fields such as statistics, systems engineering, information theory, pattern recognition and statistical mechanics are special cases of the general graphical model formalism -- examples include mixture models, factor analysis, hidden Markov models, and Kalman filters. In summary….
Slide Credits Simon Prince – Computer vision, models and learning 29
Next class More on Graphical models Readings for next lecture: – Lecture notes to be posted Readings for today: – Lecture notes to be posted 30