Apache Hadoop on Windows Azure Avkash Chauhan

Agenda Presentation and Demos –Apache Hadoop Scaling in the Cloud –Apache Hadoop on Windows Azure Architecture Demo –Connecting Hadoop using HiveODBC Excel PowerPivot Demo Apache™ Hadoop™ – based services on Windows Azure ™

Project’s Current Status and availability: Limited CTP Release (refresh2) of Apache Hadoop on Windows Azure is available now. Visit: Hadooponazure.com There are no further details about any release date at this time. The details you will see here are part of limited CTP release available for limited users depend on available resources. You might have heard about Apache Hadoop on Windows Server however that is not part of this presentation

Apache Hadoop: Data Hadoop Flexibility A Single Repo for storing and analyzing any kind of data not bounded by schema Flexibility A Single Repo for storing and analyzing any kind of data not bounded by schema Scalability Scale-out architecture divides workload across multiple nodes using flexible distributed file system Scalability Scale-out architecture divides workload across multiple nodes using flexible distributed file system Low Cost Deployed on commodity hardware & open source platform Low Cost Deployed on commodity hardware & open source platform Fault Tolerant** Continue working event if node(s) go down Fault Tolerant** Continue working event if node(s) go down Hadoop is an Open Source (Java based), “Scalable”, “fault tolerant” platform for large amount of unstructured data storage & processing, distributed across machines. Intelligence

How Hadoop Works? Data Local FS Amazon S3 Azure Blob Data Local FS Amazon S3 Azure Blob NameNode DataNode JobTracker TaskTracker DataNode TaskTracker DataNode TaskTracker Hadoop Common HDFS Map/Reduce Intelligence

Scaling in Cloud Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis Resources Data Analysis

Usage Scenarios: Data Scientist Administration and Monitoring Visual Studio Windows Deployment Windows Azure Cloud Data Market Services Windows Deployment Windows Azure Cloud Data Market Services Analytics and Data Warehousing & PDW Private Cloud Clusters Active Directory SCOM EXCEL & PowerPivot Enterprise Integration Developer Business User EXCEL Web Shell for Hadoop Write Hadoop Jobs in Shell Visualization Interactive JS Visualization Interactive JS Consume Cloud/Storage Consume BI Platform Large Developer Toolset & Ecosystem Infrastructure Support Large Developer Toolset & Ecosystem Infrastructure Support

Components: Apache Hadoop on Windows Azure Azure Blob Store Amazon S3 HiveODBC Office Excel PowerPivot Hive Pig Mahout Zookeeper Flume Web Shell Monitoring by SCOM Integration with AD Interactive JS Sqoop Avro SQL Azure SQL Server

Running Apache Hadoop in Cloud Reduce the complexity Dramaticall y lower costs Enable flexible connectivity and delivery Business can focus on data and logic not infrastructure Instant Availability Leverage existing cloud services

Running Apache Hadoop in Windows Azure Apache Hadoop on Windows Azure Portal ( hadooponaz ure.com) hadooponaz ure.com Customer login and ask for a Hadoop Cluster with N Nodes: Customer can connect to existing Cloud services Customers can connect different data sources i.e. S3, Azure Storage or copy the data to HDFS HOT Instances are ready to use Hadoop service provision N nodes Hadoop Cluster in X amount of time Upload the Map/Reduce job to the cluster and start the Job

Apache Hadoop on Azure Portal:

Apache Hadoop on Windows Azure Demo

Connecting Excel to Hadoop Cluster on Windows Azure Have HadooponAz ure cluster ready Install HiveODBC Driver (64Bit) on a machine which will connect to Hadoop Cluster Configure Hadoop Cluster in System DSN Be sure to have HiveODBC port Open in Hadoop Cluster Verify that Microsoft Excel Bit shows Hive Panel in Data Tab Connect Hadoop Cluster from Excel 2010

Connecting Excel to Hadoop Cluster on Windows Azure

Connecting PoverPivot to Hadoop Cluster on Windows Azure Have HadooponAz ure cluster ready Install HiveODBC Driver (64Bit) on a machine which will connect to Hadoop Cluster Configure Hadoop Cluster Connection Using HiveODBC 64bit Driver Be sure to have HiveODBC port Open in Hadoop Cluster Launch PowerPivot from Microsoft Excel /64Bit Import Hive Tables to PowerPivot

Connecting PoverPivot to Hadoop Cluster on Windows Azure

Resources: Hadoop-based Services For Windows (en-US) on Technet: s/articles/6204.hadoop-based-services-for- windows-en-us.aspx My Hadoop on Azure Specific Blog on MSDN: an/archive/tags/hadoop/ Denny Lee Blog: twitter: