Working With Azure Batch AI 6/1/2018 2:41 AM Working With Azure Batch AI Alex Sutton Principal Program Manager Azure Big Compute Team © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Recipe for AI Innovation 6/1/2018 2:41 AM Recipe for AI Innovation Big Compute Powerful Algorithms Massive Data © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Iterative AI Workflows Ideate Train Operationalize Run

Infrastructure Can Get in Your Way Clusters Provision VMs Install drivers and software Interactive use Scheduling Queue work Prioritize jobs Start MPI Monitor Handle failures Data Scale access to training data Output logs & models Secure & compliant Cost Scale up and down Share reserved instances Low priority Workflow Choose efficient hardware Tooling integration Laptop to cloud

Announcing Azure Batch AI Managed service that enables data scientists to easily train and test deep learning and other AI models Work with clusters of GPUs to run experiments in parallel and at scale to reduce training time Do your AI with the scale and tools of Microsoft

6/1/2018 2:41 AM Demo Submit a Job © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Specialized for AI Scenarios 6/1/2018 2:41 AM Specialized for AI Scenarios Interactive, iterative, and batch jobs Framework-specific configuration Containers or virtual machines Shared storage options Standalone or part of workflows and tools © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6/1/2018 2:41 AM Built on Azure Batch Cloud-scale resource management and task execution Shipyard project for containers and CLI experience Specify what program to run with any parameters, where to run it, and how many instances in parallel Just pay for the compute you use Standard and low priority VMs https://azure.microsoft.com/services/batch/ © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Jupyter Notebook 6/1/2018 2:41 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Train From Configuration or Code 6/1/2018 2:41 AM Train From Configuration or Code Azure Resource Manager Azure Active Directory Role based access control (RBAC) Python, C#, Java, REST APIs Azure Command Line Interface (CLI) JSON parameter files © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Options Local disk Azure Files (stream with CLI) Azure Blob with FUSE (stream with CLI) Managed NFS (mount over SSH) Parallel File Server: Lustre, Gluster, BeeGFS… Mount remote volume to VM and into container Transfer to/from blob, FUSE, NFS, etc © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

BatchAI CLI cluster job file-server create list get (details, output files, remote login) terminate delete

6/1/2018 2:41 AM Demo Azure CLI © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Work in Parallel Parameter sweeps or experiments Network design Hyper-parameter tuning Multi-GPU Setup SSH, InfiniBand pool, launch MPI New Ubuntu 16.04 image for Azure RDMA InfiniBand support requires Intel MPI Not all frameworks support MPI (CNTK and Chainer do)

6/1/2018 2:41 AM Microsoft AI © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Train and Deploy Custom AI End-to-End CNTK, TensorFlow, Chainer… Python, Visual Studio, Spark… Azure Machine Learning Azure Data Lake SQL Server Azure Batch AI = Your Data With Any AI Tools + Training With Scale-Out GPU Clusters on Demand Intelligence In Your Apps and Data Services

Next-Gen GPU Compute VM: NC_v2 NC6s_v2 NC12s_v2 NC24s_v2 NC24rs_v2 Cores 6 12 24 GPU 1 x P100 GPU 2 x P100 GPU 4 x P100 GPU Memory 112 GB 224 GB 448 GB Disk ~700 GB SSD ~1.4 TB SSD ~3 TB SSD Network Azure Network InfiniBand

Next-Gen GPU Deep Learning VM: ND ND6s ND12s ND24s ND24rs Cores 6 12 24 GPU 1 x P40 GPU 2 x P40 GPU 4 x P40 GPU Memory 112 GB 224 GB 448 GB Disk ~700 GB SSD ~1.4 TB SSD ~3 TB SSD Network Azure Network InfiniBand

Training Workloads Performance Gains with P40 Speed-Up ranging to over 3x for training workloads Images Per Second CAFFE Microsoft Cognitive Toolkit

Batch AI Roadmap Private preview Public preview Oct 9 Future scenarios 6/1/2018 2:41 AM Batch AI Roadmap Private preview https://aka.ms/batchaitraining Public preview Oct 9 ARM, Azure CLI, Azure SDK and Portal Future scenarios Training target in Azure Machine Learning—data prep, experiment, operationalize Easy parameter sweep Serverless jobs (autopool) Testing and scoring © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Recipe for AI Innovation 6/1/2018 2:41 AM Recipe for AI Innovation Big Compute Powerful Algorithms Massive Data © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Please evaluate this session Your feedback is important to us! 6/1/2018 2:41 AM Please evaluate this session Your feedback is important to us! The slide will be replaced onsite through Silver Fox Productions with an updated QR code. This slide is required. Do NOT delete or alter the slide. From your PC or Tablet visit MyIgnite at http://myignite.microsoft.com From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting https://aka.ms/ignite.mobileapp © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6/1/2018 2:41 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.