SUPPLY CHAIN OF BIG DATA

WHAT IS BIG DATA?  A lot of data  Too much data for traditional methods  The 3Vs  Volume  Velocity  Variety

UNIQUE PROBLEMS OF BIG DATA  Data is not always human-readable  Traditional storage and processing methods are too slow  Applications of data change

THE RISE OF BIG DATA  1 petabyte(PB) = 1,000 terabytes = 1,000,000 gigabytes  Average consumer hard drive is 500GB  Google processes 100PB of data a day  EBay processes 100PB of data day  Facebook has 300PB of data, growing by 600TB per day

THE SUPPLY CHAIN OF BIG DATA  Collecting  Storing  Processing  Applications  Analysis  Machine Learning

COLLECTING  Automated  Sensors in a machine  Analytics on a website  Public APIs  Publicly available  Limited  Can be free or paid  Buying Data  Generally unlimited  DataSift, FullContact

STORING  Relational databases  Slow after a certain point  “sharding” or distributed computing only does so much  NoSQL  Key: Value stores, Document stores  Distributing the database is effective  MongoDB, CouchDB, Cassandra  Cloud Solutions  Google Cloud Computing, AWS  Cloud solutions are cheap and managed  Designed to be fast and have high uptimes

PROCESSING  Cleaning  Missing Data  Outliers  Sampling  Is big data true population?  Do you need every bit of information

ANALYZING  Visualization  Tableau, Qlik  Easy to learn  Programming  R, Python  Tailored to the company  Distributed Computing  Hadoop, Spark, AWS

MACHINE LEARNING  Machine learning is making predictions based on data  Includes many algorithms  Can be supervised or unsupervised  Pitfalls  Poor data makes for poor predictions

ARTIFICIAL NEURAL NETWORKS (ANN)  Subset of machine learning  Contains one or more layers

GOOGLE BRAIN  Deep Neural Network (DNN)  Analyzed 10 million images from YouTube videos  Learned to identify a…

APPLICATIONS OF MACHINE LEARNING  Financial Trading  Advertising  Fraud Detection  Computer Vision  Natural Language Processing

ETHICS  Is collecting data ethical?  “Always listening”  Security  How safe is everyone’s data?  Who has access to the data?

EXAMPLES  Craiglist Missed Connections Craiglist Missed Connections  How Big is Snapchat? How Big is Snapchat?  18 th and 19 th century ship logs 18 th and 19 th century ship logs  Neural Network Painting Neural Network Painting