Windows Compute Cluster Server Overview and Update Paris OpenFabrics Workshop 2006 Xavier Pillons – Principal Consultant Microsoft France

Top 500 Supercomputer Trends Industry usage rising GigE is gaining (50% of systems) Clusters over 70% x86 is leading (Pentium 41%, EM64T 16%, Opteron 11%)

Windows Compute Cluster Server 2003 Faster time-to-insight through simplified cluster deployment, job submission and status monitoring Faster time-to-insight through simplified cluster deployment, job submission and status monitoring Better integration with existing Windows infrastructure allowing customers to leverage existing technology and skill-sets Better integration with existing Windows infrastructure allowing customers to leverage existing technology and skill-sets Familiar development environment allows developers to write parallel applications from within the powerful Visual Studio IDE Familiar development environment allows developers to write parallel applications from within the powerful Visual Studio IDE

Compute Cluster Pack Support for Industry Standards MPI2Support for Industry Standards MPI2 Integrated Job SchedulerIntegrated Job Scheduler Cluster Resource Management ToolsCluster Resource Management Tools Compute Cluster Edition Support for high performance hardware (x64bit architecture)Support for high performance hardware (x64bit architecture) RDMA support for high performance interconnects (Gigabit Ethernet, Infiniband, Myrinet, and others)RDMA support for high performance interconnects (Gigabit Ethernet, Infiniband, Myrinet, and others) +

CCS Key Features Node Deployment and Administration Task-based configuration for head and compute nodes UI and command line-based n ode management Monitoring with Performance Monitor (Perfmon), Microsoft Operations Manager (MOM), Server Performance Advisor (SPA), & 3 rd -party tools Integration with existing Windows and management infrastructure Integrates with Active Directory, Windows security technologies, management, and deployment tools Extensible job scheduler 3 rd -party extensibility at job submission and/or job assignment Submit jobs from command line, UI, or directly from applications Simple job management, similar to print queue management Secure MPI User credentials secured in job scheduler and compute nodes Standardized MPI stack Microsoft provided stack reduces application/MPI incompatibility issues Integrated Development Environment OpenMP Support in Visual Studio, Standard Edition Parallel Debugger in Visual Studio, Professional Edition

User AppMPI Node Manager Job Execution Windows Compute Cluster Server 2003 DB/FS User Cmd line Desktop App Job Mgr UI Admin Admin Console Cmd line Head Node Job Mgmt Resource Mgmt Cluster Mgmt Scheduling High speed, low latency interconnect Tasks Management Jobs Policy, reports Active Directory Data Input Domain\UserA

Secure channel credential End-To-End Security Node Mgr Secure channel credential Client Scheduler MSDE Data Protection API credential Task Spawn Logon token Logon as user DB/FS Active Directory Data Kerberos Kerberos LSA Automatic Ticket renewal

MPI Network Private Network Public Network Corporate IT Infrastructure Compute Node Head Node ADDNSDHCP Windows Update Monitoring MOMMOM 3 rd party3 rd party Systems Management SMS SMS 3 rd party 3 rd party Compute Cluster Typical Cluster Topology Admin / User Cons RIS Job Scheduler MPI Management NAT Node Manager MPI Management Compute Node Node Manager MPI Management

Job/Task Conceptual Model Serial Job Task Proc Parallel MPI Job Task Proc Proc IPC Parameter Sweep Job Task Proc Task Proc Task Proc Task Flow Job Task Task Task Task

Compute Cluster Server’s Developer Environment Compute Cluster Server’s Scheduler software Programmatic job submission/control Compute Cluster Server’s MPI software Derived from Argonne National Lab’s MPI-2 implementation (MPICH2) MS MPI consists of 2 parts For ISVs: Full-featured API of160+ functions For Users: Command-line (mpiexec) or GUI tool to launch jobs Why did MS HPC team choose MPI? MPI has emerged as de-facto standard for parallel programming Visual Studios 2005 New parallel debugger! MPI & Open Message Passing (OpenMP) support

Proliferation of MPI libraries Application MPIImplementations Interconnects OSDistributions KernelVersions

Customers as integrators IT Manager’s Challenges What versions of Kernels? If application X has an MPI error, which vendor should I call for support? If application X drops Kernel version Y support, which nodes do I need to upgrade? If I remove a node for maintenance, what applications won’t be able to run anymore? MPICHLAM VendorNVendorNVendorNVendorN GigE IB Myrinet GigE Myrinet GigE Myrinet GigE IB RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE GigE IB Myrinet GigE IB RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE RH SuSE Application 1 Application 2 Application 3

CCS brings back the sanity Leverage Winsock Direct for best performance (latency & throughput) and CPU efficiency. Increased Flexibility: Users can upgrade their network interconnect with no changes to the application or MPI stack. Increased Utilization: All applications on the cluster benefit from the faster network…not just MPI applications. Targeting Industry Standard Interconnect Fabrics: Gigabit Ethernet to minimize costs for a fast network Infiniband for latency-sensitive and bandwidth-intensive applications MSMPI AnyInterconnect W2k3 WSD Application

MS-MPI Leverages Winsock Direct Networking Hardware Virtual Bus Driver IP TCP User Host Channel Adapter Driver IB WinSock Provider DLL Miniport (GigE) Miniport (IPoIB) Kernel API (verbs based) User API (verbs based) Host Channel Adapter Driver GigE RDMA WinSock Provider DLL IHV-provided component OS component NDIS WinSock DLL IB w/ RDMA GigE w/ RDMA Ethernet User Mode Kernel Mode Switch traffic based on sub-net Manage hardware resources in user space (eg., Send and receive queues) HPC Application MPI Winsock Switch

Networking Performance Continuum Application Characteristics Networking Offloads Legacy Sockets App Enhanced Sockets App Layer 2 traditional NIC only Legacy Sockets App Enhanced Sockets App RDMA-enabled NIC supporting WSD RDMA aware Sockets App RDMA-enabled NIC supporting RDMA Chimney Availability Now Future Windows Server release

WinIB Mellanox InfiniBand software stack for Windows Based on OpenFabrics development InfiniBand HCA verbs driver and Access Layer InfiniBand subnet management IPoverIB driver SDP driver WinSock Direct Driver (WSD) SCSI RDMA Protocol Driver (SRP) Windows Server 2003, Windows Compute Cluster Server 2003, Windows Server “Longhorn”* * WinIB on Windows XP SP2 is supported by Mellanox – It is unsupported by Microsoft

Win IB Software Stack HCA Hardware Windows Applications Win IB Hardware Access Layer Verbs Provider Driver WSD SAN Provider SRP Miniport StorPort IPoIB Miniport NDIS TCP/UDP/ICMP IP Verbs Provider Library Access Layer Library Management Tools SDP SPI Winsock Socket Switch Applications WinSock Provider MPI2 User Kernel MPI2* * Windows Compute Cluster Server 2003 Kernel Bypass

SDP vs. WSD Sockets Direct Protocol (SDP) Windows Sockets Direct (WSD) API Winsock 2.x, POSIX/BSD like API WHQLNone SAN / Winsock Direct Wire protocol specificationStandard Microsoft proprietary OS supported Windows Server code name “Longhorn” * Windows Server 2003 SP1, Windows Server code name “Longhorn” Wire protocol Interoperability Any OS that conforms to SDP specification Windows Server 2003, Windows Server code name “Longhorn” IHV module WinSock Service Provider Library SDP kernel module SAN Provider Library Implementation domain Mostly kernel mode Mostly user mode * SDP on Windows XP SP2 and Windows Server 2003 SP1 is supported by Mellanox. It is unsupported by Microsoft.

WHQL for InfiniBand Background Driven by Windows Networking Team Collaborated with OpenIB Available Since Mid May 06 Details A test suite for WSD providers and IP over IB Miniport drivers Include functional test only (no code coverage) Signature covers networking only (no storage)

Partners

Resources Main Windows HPC Page Windows HPC Community Scalable Networking mspx Download WinIB OpenFabrics InfiniBand Windows drivers development – sign up to contribute

Questions ?

© 2006 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.