1. The Sequence Memoizer Frank Wood Cedric Archambeau Jan Gasthaus Lancelot James Yee Whye Teh Gatsby UCL Gatsby HKUST Gatsby TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A AA A A A

2. Executive Summary Model – Smoothing Markov model of discrete sequences – Extension of hierarchical Pitman Yor process [Teh 2006] Unbounded depth (context length) Algorithms and estimation – Linear time suffix-tree graphical model identification and construction – Standard Chinese restaurant franchise sampler Results – Maximum contextual information used during inference – Competitive language modelling results Limit of n-gram language model as n !1 – Same computational cost as a Bayesian interpolating 5-gram language model

3. Executive Summary Uses – Any situation in which a low-order Markov model of discrete sequences is insufficient – Drop in replacement for smoothing Markov model Name? – ``A Stochastic Memoizer for Sequence Data ’’ ! Sequence Memoizer (SM) Describes posterior inference [Goodman et al ‘ 08]

4. Statistically Characterizing a Sequence Sequence Markov models are usually constructed by treating a sequence as a set of (exchangeable) observations in fixed-length contexts trigrambigramunigram4-gram Increasing context length / order of Markov model Decreasing number of observations Increasing number of conditional distributions to estimate (indexed by context) Increasing power of model

5. Finite Order Markov Model Example

6. Learning Discrete Conditional Distributions Discrete distribution $ vector of parameters Counting / Maximum likelihood estimation – Training sequence x 1: N – Predictive inference Example – Non-smoothed unigram model (u = ²)

7. Bayesian Smoothing Estimation Predictive inference Priors over distributions Net effect – Inference is “ smoothed ” w.r.t. uncertainty about unknown distribution Example – Smoothed unigram (u = ²)

8. A Way To Tie Together Distributions Tool for tying together related distributions in hierarchical models Measure over measures Base measure is the “ mean ” measure A distribution drawn from a Pitman Yor process is related to its base distribution – (equal when c = 1 or d = 1) concentrationdiscount base distribution TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A [Pitman and Yor ’ 97]

9. Pitman-Yor Process Continued Generalization of the Dirichlet process (d = 0) – Different (power-law) properties – Better for text [Teh, 2006] and images [Sudderth and Jordan, 2009] Posterior predictive distribution Forms the basis for straightforward, simple samplers Rule for stochastic memoization TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A Can’t actually do this integral this way

10. Hierarchical Bayesian Smoothing Estimation Predictive inference Naturally related distributions tied together Net effect – Observations in one context affect inference in other context. – Statistical strength is shared between similar contexts Example – Smoothing bi-gram (w = ², u, v 2 Σ )

11. SM/HPYP Sharing in Action Conditional Distributions Posterior Predictive Probabilities Observations TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A

12. CRF Particle Filter Posterior Update Conditional Distributions Posterior Predictive Probabilities Observations CPU

13. CRF Particle Filter Posterior Update Conditional Distributions Posterior Predictive Probabilities Observations CPU

14. HPYP LM Sharing Architecture Share statistical strength between sequentially related predictive conditional distributions – Estimates of highly specific conditional distributions – Are coupled with others that are related – Through a single common, more- general shared ancestor Corresponds intuitively to back-off Unigram 2-gram 3-gram 4-gram

15. Hierarchical Pitman Yor Process Bayesian generalization of smoothing n-gram Markov model Language model : outperforms interpolated Kneser-Ney (KN) smoothing Efficient inference algorithms exist – [Goldwater et al ’ 05; Teh, ’ 06; Teh, Kurihara, Welling, ’ 08] Sharing between contexts that differ in most distant symbol only Finite depth [Goldwater et al ’ 05, Teh ’ 06]

16. Alternative Sequence Characterization A sequence can be characterized by a set of single observations in unique contexts of growing length Increasing context length Always a single observation Foreshadowing: all suffixes of the string “ cacao ”

17. ``Non-Markov ’’ Model Example Smoothing essential – Only one observation in each context! Solution – Hierarchical sharing ala HPYP

18. Sequence Memoizer Eliminates Markov order selection Always uses full context when making predictions Linear time, linear space (in length of observation sequence) graphical model identification Performance is limit of n-gram as n !1 Same or less overall cost as 5-gram interpolating Kneser Ney

19. Graphical Model Trie Observations Latent conditional distributions with Pitman Yor priors / stochastic memoizers

20. Suffix Trie Datastructure All suffixes of the string “cacao”

21. Suffix Trie Datastructure Deterministic finite automata that recognizes all suffixes of an input string. Requires O(N 2 ) time and space to build and store [Ukkonen, 95] Too intensive for any practical sequence modelling application.

22. Suffix Tree Deterministic finite automata that recognizes all suffixes of an input string Uses path compression to reduce storage and construction computational complexity. Requires only O(N) time and space to build and store [Ukkonen, 95] Practical for large scale sequence modelling applications

23. Suffix Trie Datastructure

24. Suffix Tree Datastructure

25. Graphical Model Identification This is a graphical model transformation under the covers. These compressed paths require being able to analytically marginalize out nodes from the graphical model The result of this marginalization can be thought of as providing a different set of caching rules to memoizers on the path-compressed edges

26. Marginalization Theorem 1: Coagulation [Pitman ’99; Ho, James, Lau ’06; W., Archambeau, Gasthaus, James, Teh ‘09] G1G1 G2G2 G3G3 → G1G1 G3G3

27. Graphical Model Trie

28. Graphical Model Tree

29. Graphical Model Initialization Given a single input sequence – Ukkonen ’ s linear time suffix tree construction algorithm is run on its reverse to produce a prefix tree – This identifies the nodes in the graphical model we need to represent – The tree is traversed and path compressed parameters for the Pitman Yor processes are assigned to each remaining Pitman Yor process

30. Nodes In The Graphical Model

31. Never build more than a 5-gram

32. Sequence Memoizer Bounds N-Gram Performance HPYP exceeds SM computational complexity

33. Language Modelling Results

34. The Sequence Memoizer The Sequence Memoizer is a deep (unbounded) smoothing Markov model It can be used to learn a joint distribution over discrete sequences in time and space linear in the length of a single observation sequence It is equivalent to a smoothing ∞ -gram but costs no more to compute than a 5-gram