Presentation on theme: "Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)"— Presentation transcript:
Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)
Outline The EM Family Tree The Stochastic Cousins Some Odd Relations oNoNested EM and The Partially Blocked Gibbs Sampler A Newly Found Kinsman oAoA Stochastic ECME/AECM Sampler: The Partially Collapsed Gibbs Sampler
The EM Family Tree EM Algorithm Stochastic Simulation Variance Calculations Gauss- Seidel Efficient DA Monte Carlo Integration ECM ECME SEM DA sampler MCEM PXEM Efficient DA EM NEM AECM SECM Algorithms Methods 1977 (Dempster, Laird, & Rubin) 1987 (Tanner & Wong) 1990 (Wie & Tanner) 1998 (Liu, Rubin, & Wu) 1995 (van Dyk, Meng, & Rubin) 1991 (Meng & Rubin) 1993 (Meng & Rubin) 2000 (van Dyk) 1994 (Liu & Rubin) 1997 (Meng & van Dyk)
Stochastic Cousins EM Algorithm Gauss- Seidel Efficient DA Monte Carlo Integration ECM ECME MCEM PXEM Efficient DA EM NEM AECM DA Sampler Gibbs Sampler Marginal DA PX-DA Partially Blocked Gibbs Sampler ???
The EM and DA Algorithms p(M| ) p( |M) p(M| ) p( |M) EM Algorithm DA Sampler Expectation Step Maximization Step Random Draw BACK
An NEM Algorithmwith a Monte Carlo E-step NEM AlgorithmThe Stochastic Version E-Step M-Step Draw p(M 1 | ) p( |M 1 ) p(M| ) p(, M 2 |M 1 ) p(M 1 |M 2, )p(M 2 |M 1, )p(M 1 |M 2, )p(M 2 |M 1, ) p( |M)p(M 2 |M 1, )p( |M)
A Partially-Blocked Sampler NEM AlgorithmPartially-Blocked Sampler E-Step M-Step Draw p(M 1 | ) p( |M 1 ) p(M 1 |, M 2 ) p(, M 2 |M 1 ) p(M 1 |M 2, )p(M 2 |M 1, ) p( |M)p(M 2 |M 1, )p( |M) BACK
Ordering CM-steps in ECME ECME Algorithm E-Step M-Step Draw p(M| ) p( 2 | 1 )p( 1 |M, 2 ) Monotone Convergence ECME Algorithm p(M| ) p( 1 |M, 2 )p( 2 | 1 ) NO Monotone Convergence Reducing conditioning Speed up convergence! But BE CAREFUL! Step order Matters!
A Stochastic Version of ECME ECME Algorithm E-Step M-Step Draw p(M| ) p( 2 | 1 )p( 1 |M, 2 ) p(M| ) p( 2 | 1 )p( 1 |M, 2 ) Incompatible Conditional Distributions What is the stationary distribution of this chain??
Completely Collapsed Samplers E M D p(M| ) p(M, 2 | 1 )p(M, 1 | 2 ) p(M| ) p( 1 |M, 2 )p( 2 | 1 ) p( 1 | 2 ) ECME Collapsed Sampler Complete Collapse Blocking (ECME) is a special case of Partial Collapse (AECM). We expect Collapsed Samplers (CM) to perform better than Partially Collapsed Samplers (AECM). And we expect Collapsing (CM) to perform better than Blocking (ECME). Many of these relationships are known, I emphasize the connections between EM-type DA-type algorithms.
Reducing Conditioning in Gibbs: The Simplest Example Consider a two-step Gibbs Sampler: The Markov Chain has stationary dist’n With target margins but Without the correlation of the target distribution AND converges quickly! Iteration t Iteration t+1/2 We regain the target distribution with a one-step shifted chain.
Heads Up!! Reducing the conditioning within Gibbs involves new challenges: The order of the draws may effect the stationary distribution of the chain. The conditional distributions may no be compatible with any joint distribution. The steps sometimes can be blocked to form an ordinary Gibbs sampler with fewer steps.
An Example from Astronomy Parameterized Latent Poisson Process Underlying Poisson intensity is a mixture of a broad feature and several narrow features. The “line location” and mixture indicator are highly correlated. Photon energy Line location Emission Line Spectral Model for Photon Counts
An Example from Astronomy Standard sampler simulates Photon energy Line location Emission Line Spectral Model for Photon Counts which may converge very slowly or not at all.
An Incompatible Gibbs Sampler
Verifying the Stationary Distribution of Sampler 2