1. NISS Metabolomics Workshop, 20051 Integrative Analysis of High Dimensional Gene Expression, Metabolite and Blood Chemistry Data Kwan R. Lee, Ph.D. and Lei A. Zhu, Amit Bhattacharyya, J. Alan Menius Biomedical Data Sciences GlaxoSmithKline kwan.lee@gsk.com

2. NISS Metabolomics Workshop, 20052 Overview Systems Biology Challenges for Statisticians Possible solutions Example of integrative data analysis Summary and discussion

3. NISS Metabolomics Workshop, 20053 Of mice and men ????

4. NISS Metabolomics Workshop, 20054 Integrate knowledge and technologies Reduce attrition by running coordinated studies in animal and man

5. NISS Metabolomics Workshop, 20055 Focusing on one platform may miss an obvious signal!!!

6. NISS Metabolomics Workshop, 20056 How can efficacy failures be attacked? Animal PhenotypeHuman Phenotype Classic Phenotypic Approach Animal PhenotypeHuman Phenotype Animal Biomarker Fingerprint Human Biomarker Fingerprint Integrative Biology Few data to support analogy Many data to support analogy

7. NISS Metabolomics Workshop, 20057 ‘Systems Biology’ approach to drug discovery

8. NISS Metabolomics Workshop, 20058 1 H NMR metabolites Affy Transcriptome LC-MS Lipid LC-MS metabolites “Non-omic” markers Veh A B C D Normal Disease A A Experimental Platforms Non-omics and Omics, what are they?

9. NISS Metabolomics Workshop, 20059 Experimental Platforms Non-omics and Omics, what are they? (cont.) Traditional Blood Chemistry (non-omics) Gene Expression (transcriptomics) Metabolite (metabonomics) Lipid (lipomics) Protein (proteomics)

10. NISS Metabolomics Workshop, 200510 Five Challenges 1.Data Pre-processing 2.High Dimensionality 3.Multiple Testing for Marker Selection 4.Data Integration 5.Validation of the Prediction Model

11. NISS Metabolomics Workshop, 200511 Peak Alignment (NMR, LC/MS) Normalization (Gene Chip, NMR, LC/MS data) –Why? Remove systematic bias in the data –Normalization within the platform makes data comparable across samples Challenge #1: Data Pre-processing

12. NISS Metabolomics Workshop, 200512 Challenge # 2: High Dimensionality # of subjects << # of variables Blood Chemistry: 9 markers Gene Expression: 22,000 probe sets Lipid LC/MS: 2, 000 peaks Metabolite LS/MS: 3,000 peaks NMR: 500 buckets Animal 1 Animal 2. Animal 100 probe set 1 …… 22,000Lipid 1...… 2,000Metabolite 1 … 3,000NMR 1 …… 500Choles, Trig,…...

13. NISS Metabolomics Workshop, 200513 NoiseSignalSignal+Noise No Adjustment for Multiple Testing FWER Adjustment FDR += Challenge #3: Multiple Testing in Variable Selection

14. NISS Metabolomics Workshop, 200514 1 H NMR metabolites Affy Transcriptome LC-MS Lipid LC-MS metabolites “Non-omic” markers Veh A B C D Normal Disease A Challenge #4: Data integration A

15. NISS Metabolomics Workshop, 200515 Platform A 20000s var. Platform B 1000s var. Combined Data Platform A 20000s var. Platform B 1000s var. Dimension Reduction (eg variable selection) Platform A 1000s var. Platform B 100s var. Combined Data Integration Approach 1: Integration Approach 2: Challenge #4: Data integration (cont.)

16. NISS Metabolomics Workshop, 200516 Integration approach 1: Simple data integration –Simply combining the platform data together, the platform with large amount of data and variability will dominate the other platforms Challenge #4: Data integration Example 1

17. NISS Metabolomics Workshop, 200517 PCA on Non-omics, Transcriptomics, and Combined. Non-omics (20) Transcriptomics (12,488) Combined (12,508) Mirror image!!! Transcriptomics data dominate Non-omics data!!!

18. NISS Metabolomics Workshop, 200518 PCA on Non-omics, Transcriptomics, and Combined. Non-omics (20) Transcriptomics (20 PCs) Combined (40) Like a mirror image!!!

19. NISS Metabolomics Workshop, 200519 Integration approach 2: Integrate on selected markers –9 blood chemistry + 2000 probe sets + 150 metabolites –There are still platforms with more selected markers –How to weight different platforms appropriately? Eg. 9 blood chemistry markers are known to relate to disease or drug –Identify relationship among the probe sets, metabolites, along with the blood chemistry markers in terms of biological pathways Challenge #4: Data integration Example 2

20. NISS Metabolomics Workshop, 200520 Normal Disease Principle Component Analysis (PCA ) Projection of 67 animals of 28 normal (black), 39 disease (red) (9 NO, 1991 TA, 115 MT) All markers used for projection

21. NISS Metabolomics Workshop, 200521 Loading Plot

22. NISS Metabolomics Workshop, 200522 Partial Least Square Discriminant Analysis (PLS-DA) Disease group only Vehicle Drug

23. NISS Metabolomics Workshop, 200523 PLS-DA: Corresponding projection of all markers (9 NO, 1991 TA, 115 MT), Which are important drug markers? Drug Veh

24. NISS Metabolomics Workshop, 200524 Ranked drug markers by importance or by coefficients. marker importance by variable importance on projection Up or down regulation by coefficients

25. NISS Metabolomics Workshop, 200525 Validation of the model: R2, Q2 and permutation tests 100 times (P < 0.01)

26. NISS Metabolomics Workshop, 200526 Variation explained by each platform PLS-DA for prediction of 2 experimental groups Two Groups HFD, vehicle HFD, Drug treated Q2(Y) = amount of variation among the 2 groups explained by the model (cross-validated) The above table is based on 2- component model. If the 4th model uses more components, 91% of the variation in the data can be explained by 4 components.

27. NISS Metabolomics Workshop, 200527 Challenge #5: Validation of the Prediction Model Correct way of doing cross-validation –Especially when the variables are selected Is your prediction accuracy significant?

28. NISS Metabolomics Workshop, 200528 Random Noise Data Simulate 20,000 marker columns of random noise for 100 patients and one additional column containing arbitrary labels of class indicators. Select 5 marker columns showing most correlation with class label. Make a prediction model for class indicators based on these 5 selected markers.

29. NISS Metabolomics Workshop, 200529 PCA of Full Markers

30. NISS Metabolomics Workshop, 200530 PLS-DA on Random Noise Data Running a full model on SIMCA does not yield a model – no significant Q2. –Multivariate approach is conservative. –Q2 computes prediction performance. But forced the software to fit a 6 - component model by PLS-DA (R2 = 1.0, Q2 = 0.225)

31. NISS Metabolomics Workshop, 200531 Full marker model PLS-DA

32. NISS Metabolomics Workshop, 200532 Was it real or by chance?

33. NISS Metabolomics Workshop, 200533 Select 5 Markers Selected top 5 markers using VIP from the over-fitted model and fit PLS-DA again on the same data. Now we have (R2 = 0.459, Q2 = 0.348)

34. NISS Metabolomics Workshop, 200534 Good prediction from PLS-DA? Q2 = 0.35

35. NISS Metabolomics Workshop, 200535 Validated by permutation test? Significance of Q2

36. NISS Metabolomics Workshop, 200536 Selection Bias When a prediction model is tested on the same data that were used in the first instance to select the markers, selection bias makes the test error overly optimistic. –Many publications claimed a small set of selected “genes” is highly predictive. –IBI practice is to use a data set to select markers and use the same data set to fit a prediction model based on selected markers.

37. NISS Metabolomics Workshop, 200537 How to correct for selection bias? External validation should be undertaken subsequent to feature selection process. 1.Independent test data set (hold-out data set) that never used for feature selection. 2.External cross-validation (ECV). Cross validation of the prediction model is external to the selection process. In other words, make a new selection for each cross validation round.

38. NISS Metabolomics Workshop, 200538 Externally Validated PLS. Model and variable selection Divide the data set randomly into d parts. Set ecv = 1; (this means hold-out one part and use d-1 parts for modeling) Set a =1 ; (the number of components, do until 10) Set k = total number of variables; Loop: Fit PLS model with given a and k, PLS (a,k); Predict hold-out set, compute PRESS (ecv, a, k) and save; Choose top half of the variables by appropriate statistics (coeff, vip, t-ratio etc); Set k = k/2; Go back to Loop until k = 2; Set a = a + 1; Go back to Loop until a =10; Set ecv = ecv + 1; Go back to Loop until ecv = d; Compute PRESS (a, k) = Sum over ecv {PRESS (ecv, a, k)}; Compute Q2(a, k) = 1 – PRESS (a, k)/TSS; Plot Q2(a,k) vs. log2(k);

39. NISS Metabolomics Workshop, 200539 Simulation of 2000 Random Data R. Simon 2003 20 x 6000 and 10/10 for class labels Repeat 2000 times Compute 3 different error rates –Re-substitution (wrong) –Cross validation after selection (wrong) –Cross validation before selection (correct)

40. NISS Metabolomics Workshop, 200540 Results of 2000 Random Data

41. NISS Metabolomics Workshop, 200541 Permutation testing Because of the high dimensionality of gene expression data, it may be possible to achieve relatively small error rates even for random data. To assess the significance of the classification results, permutation test may be suggested.

42. NISS Metabolomics Workshop, 200542 Challenge #5: Validation of the Prediction Model - summary Correct way of doing cross-validation –All the steps of the prediction modeling should be cross-validated. –Each cross validation step should start from scratch Is your prediction accuracy significant? –Random data can give you low prediction error –Permutation tests, bootstrap aggregation

43. NISS Metabolomics Workshop, 200543 Summary and Discussion Recent technological advances present challenging and interesting biological data at molecular level. Statistics and multivariate analysis play an important role in understanding and extracting knowledge from these type of data. Integrative analysis is even more challenging and we presented some solutions to these challenges. There is plenty of room for improvement.

44. NISS Metabolomics Workshop, 200544 Acknowledgement GlaxoSmithKline –High Throughput Biology –Biomedical Data Sciences –Genomics and Proteomics Science –Pathology, Cellular & Biochemical Toxicology –Discovery IT

45. NISS Metabolomics Workshop, 200545 Data exploration: Present Challenges Data is an extremely valuable asset, but like a cash crop, unless harvested, it is wasted. - Sid Adelman