Data Science Background and Course Software setup Week 1

Index Installation process Lecture 1: Introduction to big data and data science Lecture 2: Performing data science and preparing data

Installation process (I) The same development environment: Two free software packages: VirtualBox and Vagrant Virtual Machine Hardware and Software Prerequisites Minimum Hardware Requirements Free disk space: 3.5 GB RAM memory: 2.5 GB (4+ GB preferred) Processor: Any recent Intel or AMD multicore processor should be sufficient. Supported Operating Systems Windows, Linux, MAC OS X

Installation process (II) Installation of the Virtual box: virtualbox.org  Downloads  Choose the appropriate version of the Virtual box for your OS Installation of Vagrant: -> Downloads  Choose the appropriate version of the Vagrant for your OS Installation of the Virtual Machine: Create a custom directory (e.g., /home/marrval/myvagrant) Download the file: setup/archive/master.zip to the custom directory and unzip it. Copy Vagrantfile to the custom directory you created in step #1 Open a DOS prompt (Windows) or Terminal (Mac/Linux), change to the custom directory, and issue the command vagrant up (the Virtual box opened in the background) Sparkvm is running!

Installation process (III) Basic Instructions for Using the Virtual Machine To start the VM, from a DOS prompt (Windows) or Terminal (Mac/Linux), issue the command vagrant up. To stop the VM, use the command vagrant halt You should always stop the VM before you log off, turn off, or reboot your computer. To erase or delete the VM, vagrant destroy Once the VM is running, to access the notebook, open a web browser to " : start the iPython notebook on port 8001 (so we can have access to an IPython notebook with a Spark)

Installation process (IV) Running Your First Notebook Start the VM Open a web browser to " Upload the file "lab0_student.ipynb”, which is contained in the.zip Verify that you do not encounter any errors in the run of the cells

Introduction to big data and data science (I) Correlation doesn’t imply causation Use more data Explore more types of data/factors

Introduction to big data and data science (II) Big Data: Why all this excitement? From 2003 to 2008, they looked at weekly search queries  Identify 45 terms relevant to people searching about flu  Build a model Google rolled out flu stories in Google News during this period + reading stories  skewed the results

Introduction to big data and data science (III) Big Data: Why all this excitement? Bloggers used data science to analyze the elections The campaigns were using data science (database that modeled the behavior of the electorate) Pollsters try to predict the outcome by polling people  they have biases (+errors)  incorrect results Challenge: remove biases + errors

Introduction to big data and data science (IV) Cautionary tale How did they come to this conclusion? Look at Google trend searches for MySpace and use the same model to Facebook Correlation doesn’t imply causation Identify important factors

Introduction to big data and data science (V) Where Does Big Data Come From? Online (And can be recorded). Many data are collected and few analyzed Users (user-generated content) Individually is not very large

Introduction to big data and data science (VI) Where Does Big Data Come From? Health and scientific computing Graphs Log files (generated by servers around The Internet) The Internet of Things (e.g., sensors in a forest, toll collection transponder to traffic reporting)

Performing Data Science and preparing Data (I) What is Data Science? Data Science aims to derive knowledge from big data, efficiently and intelligently” Data Science encompasses the set of activities, tools, and methods that enable data-driven activities in science, business, medicine, and government Apply algorithms at scale to large amounts of data, and understand both the algorithms and the results Collect data, analyze them and understands the analytical process and results Collect knowledge, apply algorithms, but do not understand

Performing Data Science and preparing Data (I) What is Data Science? Data Science aims to derive knowledge from big data, efficiently and intelligently” Data Science encompasses the set of activities, tools, and methods that enable data-driven activities in science, business, medicine, and government Apply domain-specific knowledge at very large scale, and understand both the algorithms and the results

Performing Data Science and preparing Data (II) Contrasting Data Science: Database

Performing Data Science and preparing Data (III) Contrasting Data Science: Database Contrasting Data Science: Scientific computing

Performing Data Science and preparing Data (IV) Contrasting Data Science: Traditional Machine Learning

Performing Data Science and preparing Data (V) Doing data science Problem  Collect data  clean the data  build a model  communicate the results

Performing Data Science and preparing Data (V) Cloud computing: key enabler of data science Allows date science on a massive scale Data science practice

Performing Data Science and preparing Data (VI) What is hard about Data Science?

Performing Data Science and preparing Data (VII) Data acquisition and Preparation 1.Extract data from sources 2.Load data into the sink 3.Transform data (source, sink, staging area)

Performing Data Science and preparing Data (VIII) Data acquisition and Preparation We create pipelines or workflows, which can be scheduled Recording the execution of a workflow is known as capturing lineage or provenance (Spark does it automatically) Impediments to collaboration: diversity of tools/programming languages, finding a script is hard, most analysis work is ‘thrown away’

Performing Data Science and preparing Data (VIII) Data Science roles Individual Organizational