1. Part I: Classifier Performance Mahesan Niranjan Department of Computer Science The University of Sheffield M.Niranjan@Sheffield.ac.uk & Cambridge Bioinformatics Limited Mahesan.Niranjan@ntlworld.com

2. BCS, Exeter, July 2004Mahesan Niranjan2 Relevant Reading Bishop, Neural Networks for Pattern Recognition http://www.ncrg.aston.ac.uk/netlab David Hand, Construction and Assessment of Classification Rules Lovell, et. Al. CUED/F-INFENG/TR.299 Scott et al CUED/F-INFENG/TR.323 reports linked from http://www.dcs.shef.ac.uk/~niranjan http://www.dcs.shef.ac.uk/~niranjan

3. BCS, Exeter, July 2004Mahesan Niranjan3 Pattern Recognition Framework

4. BCS, Exeter, July 2004Mahesan Niranjan4 Two Approaches to Pattern Recognition Probabilistic via explicit modelling of probabilities encountered in Bayes’ formula Parametric form for class boundary and optimise it In some specific cases (often not) both reduce to the same answer

5. BCS, Exeter, July 2004Mahesan Niranjan5 Pattern Recognition: Simple case O Gaussian Distributions Isotropic Equal Variances Optimal Classifier: Distance to mean Linear Class Boundary

6. BCS, Exeter, July 2004Mahesan Niranjan6 Distance can be misleading O Mahalanobis Distance Optimal Classifier for this case is Fisher Linear Discriminant

7. BCS, Exeter, July 2004Mahesan Niranjan7 Support Vector Machines Maximum Margin Perceptron X X X X X X O O O O OO O O O O X X X X X X

8. BCS, Exeter, July 2004Mahesan Niranjan8 Support Vector Machines Nonlinear Kernel Functions X X X X O O O O O O O X X X X X X O O OO O O O

9. BCS, Exeter, July 2004Mahesan Niranjan9 Support Vector Machines Computations Quadratic Programming Class boundary defined only by data that lie close to it - support vectors Kernels in data space equal scalar products in higher dimensional space

10. BCS, Exeter, July 2004Mahesan Niranjan10 Support Vector Machines The Hypes Strong theoretical basis - Computational Learning Theory; complexity controlled by the Vapnik-Chervonenkis dimension Not many parameters to tune High performance on many practical problems, high dimensional problems in particular

11. BCS, Exeter, July 2004Mahesan Niranjan11 Support Vector Machines The Truths Worst case bounds from Learning theory are not very practical Several parameters to tune –What kernel? –Internal workings of the optimiser –Noise in training data Performance? – depends on who you ask

12. BCS, Exeter, July 2004Mahesan Niranjan12 SVM: data driven kernel Fisher Kernel [Jaakola & Haussler] –Kernel based on a generative model of all the data

13. BCS, Exeter, July 2004Mahesan Niranjan13 Classifier Performance Error rates can be misleading –Imbalance in training/test data 98% of population healthy 2% population has disease –Cost of misclassification can change after design of classifier

14. BCS, Exeter, July 2004Mahesan Niranjan14 x xx x x x x x x x x x Adverse Outcome Benign Outcome Threshold Class Boundary

15. BCS, Exeter, July 2004Mahesan Niranjan15 True Positive False Positive Area under the ROC Curve: Neat Statistical Interpretation

16. BCS, Exeter, July 2004Mahesan Niranjan16 Convex Hull of ROC Curves False Positive True Positive

17. BCS, Exeter, July 2004Mahesan Niranjan17 Yeast Gene Example: MATLAB Demo here

18. Part II: Particle Filters for Tracking and Sequential Problems Mahesan Niranjan Department of Computer Science The University of Sheffield

19. BCS, Exeter, July 2004Mahesan Niranjan19 Overview Motivation State Space Model Kalman Filter and Extensions Sequential MCMC Methods –Particle Filter & Variants

20. BCS, Exeter, July 2004Mahesan Niranjan20 Motivation Neural Networks for Learning: –Function Approximation –Statistical Estimation –Dynamical Systems –Parallel Processing Guarantee Generalisation: –Regularise / control complexity –Cross validate to detect / avoid overfitting –Bootstrap to deal with model / data uncertainty Many of the above tricks won’t work in a sequential setting

21. BCS, Exeter, July 2004Mahesan Niranjan21 Interesting Applications Speech Signal Processing Medical Signals –Monitoring Liver Transplant Patients Tracking the prices of Options contracts in computational finance

22. BCS, Exeter, July 2004Mahesan Niranjan22 Good References Bar-Shalom and Fortman: Tracking and Data Association Jazwinski: Stochastic Processes and Filtering Theory Arulampalam et al: “Tutorial on Particle Filters…”; IEEE Transactions on Signal Processing Arnaud Doucet: Technical Report 310, Cambridge University Engineering Department Benveniste, A et al: Adaptive Algorithms and Stochastic Approximation Simon Haykin: Adaptive Filters

23. BCS, Exeter, July 2004Mahesan Niranjan23 Matrix Inversion Lemma

24. BCS, Exeter, July 2004Mahesan Niranjan24 Linear Regression

25. BCS, Exeter, July 2004Mahesan Niranjan25 Recursive Least Squares

26. BCS, Exeter, July 2004Mahesan Niranjan26 State Space Model State Process Noise ObservationMeasurement Noise

27. BCS, Exeter, July 2004Mahesan Niranjan27 Simple Linear Gaussian Model

28. BCS, Exeter, July 2004Mahesan Niranjan28 Kalman Filter Prediction Correction

29. BCS, Exeter, July 2004Mahesan Niranjan29 Kalman Filter Innovation Kalman Gain

30. BCS, Exeter, July 2004Mahesan Niranjan30 Bayesian Setting PriorLikelihood Innovation Probability Run Multiple Models and Switch - Bar-Shalom Set Noise Levels to Max Likelihood Values - Jazwinski

31. BCS, Exeter, July 2004Mahesan Niranjan31 Extended Kalman Filter Lee Feldkamp @ Ford Successful training of Recurrent Neural Networks Taylor Series Expansion around the operating point First Order Second Order Iterated Extended Kalman Filter

32. BCS, Exeter, July 2004Mahesan Niranjan32 Iterated Extended Kalman Filter Local Linearization of State and / or Observation Propagation and Update

33. BCS, Exeter, July 2004Mahesan Niranjan33 Unscented Kalman Filter Generate some points at time So they can represent the mean and covariance: Propagate these through the state equations Recompute predicted mean and covariance:

34. BCS, Exeter, July 2004Mahesan Niranjan34 Recipe to define: Recompute:

35. BCS, Exeter, July 2004Mahesan Niranjan35 Formant Tracking Example Linear Filter ExcitationSpeech

36. BCS, Exeter, July 2004Mahesan Niranjan36 Formant Tracking Example

37. BCS, Exeter, July 2004Mahesan Niranjan37 Formant Track Example

38. BCS, Exeter, July 2004Mahesan Niranjan38 Grid-based methods Discretize continuous state into “cells” Integrating probabilities over each partition Fixed partitioning of state space 

39. BCS, Exeter, July 2004Mahesan Niranjan39 Sampling Methods: Bayesian Inference Parameters Uncertainty over parameters Inference:

40. BCS, Exeter, July 2004Mahesan Niranjan40 Basic Tool: Composition [Tanner] To generate samples of

41. BCS, Exeter, July 2004Mahesan Niranjan41 Importance Sampling

42. BCS, Exeter, July 2004Mahesan Niranjan42 Particle Filters Prediction Weights of Sample Bootstrap Filters ( Gordon et al, Tracking ) CONDENSATION Algorithm ( Isard et al, Vision )

43. BCS, Exeter, July 2004Mahesan Niranjan43 Sequential Importance Sampling Recursive update of weights Only upto a constant of proportionality

44. BCS, Exeter, July 2004Mahesan Niranjan44 Degeneracy in SIS Variance of weights monotonically increases  All except one decay to zero very rapidly Effective number of particles Resample if

45. BCS, Exeter, July 2004Mahesan Niranjan45 Sampling, Importance Re-Sampling (SIR) Multiply samples of high weight; kill off samples in parts of space not relevant  “Particle Collapse”

46. BCS, Exeter, July 2004Mahesan Niranjan46 Marginalizing Part of the State Space Suppose Possible to analytically integrate with respect to part of the state space Sample with respect to Integrate with respect to Rao-Blackwell

47. BCS, Exeter, July 2004Mahesan Niranjan47 Variations to the Basic Algorithm Integrate out part of the state space –Rao-Blackwellized particle filters ( e.g. Multi-layer perceptron with linear output layer ) Variational Importance Sampling ( Lawrence et al ) Auxilliary Particle Filters ( Pitt et al ) Regularized Particle Filters Likelihood Particle Filters

48. BCS, Exeter, July 2004Mahesan Niranjan48 Regularised PF: basic idea Samples Kernel Density Resample Propagate in time

49. BCS, Exeter, July 2004Mahesan Niranjan49 Conclusion / Summary Collection of powerful algorithms New and interesting signal processing problems