Institute for Personal Robots in Education (IPRE)‏ CSC 170 Computing: Science and Creativity

THE BIG DATA CHALLENGE How do we... process data sets that are too large for traditional algorithms and software tools? extract knowledge from large datasets in a wide variety of domains, from science to medicine to consumer data

DIKW PYRAMID Represents structural and/or functional relationships between d ata, i nformation, k nowledge, and w isdom Typically information is defined in terms of data, knowledge in terms of information, and wisdom in terms of knowledge wikipedia

DIKW PYRAMID Data : discrete raw facts, signals, observations of no use until...in a usable form Patient data, sensor data, scientific data

DIKW PYRAMID Data : discrete raw facts, signals, observations of no use until...in a usable form Patient data, sensor data, scientific data Information : processed, organized or structured data that is used for some purpose 14% of Americans are over 65

DIKW PYRAMID Data : discrete raw facts, signals, observations of no use until...in a usable form Patient data, sensor data, scientific data Information : processed, organized or structured data that is used for some purpose 14% of Americans are over 65 Knowledge and Wisdom: Thoughts in a persons mind believed to be true E = mc 2 God exists

DATA STORAGE HISTORY

SORTING AND SEARCHING Traditional Searching & Sorting Algorithms Data must fit into main memory (RAM) Data are processed sequentially (not in parallel)

ANALYZING DATA Spreadsheets Structured data: Columns and rows Small to large size data sets (50 MB - 2 GB) Analysis, visualization Widely used in business Code demo: Descriptive statistics in Google Sheets

VISUALIZING DATA Databases Structured Data: Tables Moderate to large data sets (2 GB - 2 TB) Storage and retrieval of relational data Logical searching and analysis of data Retrieve records for all accounts > 50000

STANDARD QUERY LANGUAGE SQL Data bases have tables: lists of records (1, 2,3) Record: a list of attributes (ID, Name, Address, City) Code Demo: Show a few data base queries from

DATASETS A data set (or dataset ) is a collection of data. Most commonly a data set corresponds to the contents of a single database table, or a single statistica data matrix, where every column of the table represents a particular variable, and eac row corresponds to a given member of the data set in question. The data set lists values for each of the variables, such as height and weight of an object, for each member of the data set. Each value is known as a datum. The data set may comprise data for one or more members, corresponding to the number of rows. wikipedia

GOOGLE PUBLIC DATA SETS Google Public Data This is the first part of Today’s lab Demo a bubble chart: Determine something interesting