1. Audio-Visual Graphical Models Matthew Beal Gatsby Unit University College London Nebojsa Jojic Microsoft Research Redmond, Washington Hagai Attias Microsoft Research Redmond, Washington

2. Beal, Jojic and Attias, ICASSP’02 Overview Some background to the problem A simple video model A simple audio model Combining these in a principled manner Results of tracking experiments Further work and thoughts.

3. Beal, Jojic and Attias, ICASSP’02 Motivation – applications Teleconferencing –We need speaker’s identity, position, and individual speech. –The case of multiple speakers. Denoising –Speech enhancement using video cues (at different scales). –Video enhancement using audio cues. Multimedia editing –Isolating/removing/adding objects, visually and aurally. Multimedia retrieval –Efficient multimedia searching.

4. Beal, Jojic and Attias, ICASSP’02 Motivation – current state of art Video models and Audio models –Abundance of work on object tracking, image stabilization… –Large amount in speech recognition, ICA (blind source separation), microphone array processing… Very little work on combining these –We desire a principled combination. –Robust learning of environments using multiple modalities. –Various past approaches: Information theory: Hershey & Movellan (NIPS 12) SVD-esque: (FaceSync) Slaney & Covell (NIPS 13) Subspace stats.: Fisher et al. (NIPS 13). Periodicity analysis: Ross Cutler Particle filters: Vermaak and Blake et al (ICASSP 2001). System engineering: Yong Rui (CVPR 2001). Our approach: Graphical Models, Bayes nets.

5. Beal, Jojic and Attias, ICASSP’02 Generative density modeling Probability models that –reflect desired structure –randomly generate plausible images and sounds, –represent the data by parameters ML estimation p(image|class) used for recognition, detection,... Examples: Mixture of Gaussians, PCA/FA/ICA, Kalman filter, HMM All parameters can be learned from data!

6. Beal, Jojic and Attias, ICASSP’02 Speaker detection & tracking problem  mic.1mic.2 source at l x camera lxlx lyly Video scenarioAudio scenario

7. Beal, Jojic and Attias, ICASSP’02 Bayes Nets for Multimedia Video models –Models such as Jojic & Frey (NIPS’99, CVPR’99’00’01). Audio models –Work of: Attias (Neural Comp’98); Attias, Platt, Deng & Acero (NIPS’00,EuroSpeech’01).

8. Beal, Jojic and Attias, ICASSP’02 A generative video model for scenes (see Frey&Jojic, CVPR’99, NIPS’01) Mean  s Class s Latent image z Transformed image z Generated/observed image y Shift (l x,l y )

9. Beal, Jojic and Attias, ICASSP’02Example Hand-held camera Moving subject Cluttered background DATA Mean One class summary Variance 5 classes

10. Beal, Jojic and Attias, ICASSP’02 A generative video model for scenes (see Frey&Jojic, CVPR’99, NIPS’01) Mean  s Class s Latent image z Transformed image z Generated/observed image y Shift (l x,l y )

11. Beal, Jojic and Attias, ICASSP’02 A failure mode of this model

12. Beal, Jojic and Attias, ICASSP’02 Modeling scenes - the audio part  mic.1mic.2 source at l x camera mic.1mic.2

13. Beal, Jojic and Attias, ICASSP’02 Unaided audio model audio waveform  +15 -15 video frames +15 -15  Posterior probability over , the time delay. Periods of quiet cause uncertainty in  – (grey blurring). Occasionally reverberations / noise corrupt inference on  –and we become certain of a false time delay. time

14. Beal, Jojic and Attias, ICASSP’02 Limit of this simple audio model

15. Beal, Jojic and Attias, ICASSP’02 Multimodal localization Time delay  is approximately linear in horizontal position l x Define a stochastic mapping from spatial location to temporal shift:

16. Beal, Jojic and Attias, ICASSP’02 The combined model

17. Beal, Jojic and Attias, ICASSP’02 The combined model Two halves connected by  - l x link Maximize  n a log p(x t )+n v log p(y t )

18. Beal, Jojic and Attias, ICASSP’02 Learning using EM: E-Step Distribution Q over hidden variables is inferred given the current setting of all model parameters.

19. Beal, Jojic and Attias, ICASSP’02 Learning using EM: M-Step Audio: –Relative microphone attenuations 1, 2 and noise levels 1 2 AV Calibration between modalities – , ,  Video: –object templates  s and precisions  s –camera noise  Given the distribution over hidden variables, the parameters are set to maximize the data likelihood.

20. Beal, Jojic and Attias, ICASSP’02 Efficient inference and integration over all shifts (Frey and Jojic, NIPS’01) E Estimating posterior Q(l x,l y,  ) involves computing Mahalanobis distances for all possible shifts in the image M Estimating model parameters involves integrating over all possible shifts taking into account the probability map Q(l x,l y,  ) E reduces to correlation, M reduces to convolution Efficiently done using FFTs

21. Beal, Jojic and Attias, ICASSP’02 Demonstration of tracking A AV V n a /n v

22. Beal, Jojic and Attias, ICASSP’02 Learning using EM: M-Step Audio: –Relative microphone attenuations 1, 2 and noise levels 1 2 AV Calibration between modalities – , ,  Video: –object templates  s and precisions  s –camera noise  Given the distribution over hidden variables, the parameters are set to maximize the data likelihood.

23. Beal, Jojic and Attias, ICASSP’02 Inside EM iterations 1 2 4 10 Q(  |x 1,x 2,y) Q(l x |x 1,x 2,y)

24. Beal, Jojic and Attias, ICASSP’02 TrackingStabilization TrackingStabilization

25. Beal, Jojic and Attias, ICASSP’02 Work in progress: models Incorporating a more sophisticated speech model –Layers of sound Reverberation filters –Extension to y-localization is trivial. –Temporal models of speech. Incorporating a more sophisticated video model –Layered templates (sprites) each with their own audio (circumvents dimensionality issues). –Fine-scale correlations between pixel intensities and speech. –Hierarchical models? (Factor Analyser trees). Tractability issues: –Variational approximations in both audio and video.

26. Beal, Jojic and Attias, ICASSP’02 Basic flexible layer model (CVPR’01)

27. Beal, Jojic and Attias, ICASSP’02 Future work: applications Multimedia editing –Removing/adding objects’ appearances and associated sounds. –With layers in both audio and video (cocktail party / danceclub). Video-assisted speech enhancement –Improved denoising with knowledge of source location. –Exploit fine-scale correlations of video with audio. (e.g. lips) Multimedia retrieval –Given a short clip as a query, search for similar matches in a database.

28. Beal, Jojic and Attias, ICASSP’02 Summary A generative model of audio-visual data All parameters learned from the data, including camera/microphones calibration in a few iterations of EM Extensions to multi-object models Real issue: the other curse of dimensionality

29. Beal, Jojic and Attias, ICASSP’02 Pixel-audio correlations analysis SVD. Factor Analysis (probabilistic PCA). Original video sequence Inferred activation of latent variables (factors, subspace vectors)