1. Big Data and Large Scale Data Analysis Andrew Mead School of Life Sciences 23 rd October 2013

2. Big Data Modern technologies make it increasingly easy to collect large quantities of data – ‘Omics revolution – Remote sensing – Weather (and hence climate change applications) – Internet applications – Social networking – Shopping preferences – Health applications – … But how do we make the most of these data?

3. Gene expression microarrays Data on many thousands of genes (spots) on each array Comparisons of multiple samples (treatments, time, individual plants or animals, …) Processing of data for each gene separately or in combination

4. Landscape data Land-use/cover for each land-parcel Basis for simulation studies of changes in land-use Summary of spatial data into simple statistics

5. Challenges Storage of big data sets Management – Structured – Unstructured Analysis – Often similar questions as for smaller data sets – Computationally intractable as data volume increases

6. Multivariate Statistics and Data Mining Dimensionality reduction – Find the important combinations of variables – Use these in models Use computing power to search for “patterns” Challenge in connecting the analysis process to the data Distributed computing, massively parallel processing (MPP), machine learning, search- based applications (SBA), …

7. Statistics and Big Data Computing power is probably crucial! But statistical approaches are important – Designing the data collection Sub-sampling? – Defining the problem – Managing the data – Dimension reduction Finding the signal amidst the noise!