Big Data Analytics with Excel Peter Myers Bitwise Solutions

Cheap Storage $100 gets you 3 million times more storage in 30 years Inexpensive Computing 1980: 10 MIPS/$ 2005: 10M MIPS/$ > 5.5 billion (> 70% of global population) > 2 billion users Web traffic 2010: 130 exabyte (10 E18) 2015: 1.6 zettabyte (10 E21) > 10 billion

“Big data is a collection of data sets so large and complex that it becomes awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analysis, and visualization.” – Wikipedia “Big data is a collection of data sets so large and complex that it becomes awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analysis, and visualization.” – Wikipedia

VOLUME (Size) VARIETY (Structure) VELOCITY (Speed)

Data Complexity: Variety and Velocity Terabytes Gigabytes Megabytes Petabytes Big Data Log files Spatial and GPS coordinates Data market feeds eGov feeds Weather Text/image Click stream Wikis/blogs Sensors RFID Devices Social sentiment Audio/video

How do I optimize my services based on patterns of weather, traffic, etc.? What’s the social sentiment of my product? How do I better predict future outcomes?

TRADITIONAL RDBMS HADOOP Data Size Access Updates Structure Integrity Scaling DBA Ratio

+

Server Files Server

Demo Storing Files in Azure Blob Storage

Server RUNTIME Code

Distributed Storage (HDFS) Distributed Processing (MapReduce) Legend Red = Core Hadoop Blue = Data processing Green = Packages Purple = Microsoft integration points and value adds Orange = Data Movement

Demo Word Count with Pig (The “Hello World” for Hadoop)

Excess Data Logs ETL Some Data Data Warehouse

Logs Raw Data “Store it All” Cluster Raw Data “Store it All” Cluster

Hadoop Data Analytics

Demo Self-Service BI with Hive

Explore Everything PASS Has to Offer Free SQL Server and BI Web Events Free 1-day Training Events Regional Event Local User Groups Around the World Free Online Technical Training This is CommunityBusiness Analytics Training Session Recordings PASS Newsletter