Microsoft Keyboard

Cluster and Grid Computing Pittsburgh Supercomputing Center John Kochmar J. Ray Scott (Derek Simmel) (Jason Sommerfield)

Pittsburgh Supercomputing Center Who We Are Cooperative effort of –Carnegie Mellon University –University of Pittsburgh –Westinghouse Electric Research Department of Carnegie Mellon Offices in Mellon Institute, Oakland –On CMU campus –Adjacent to University of Pittsburgh campus.

Westinghouse Electric Company Energy Center, Monroeville, PA

Agenda HPC Clusters Large Scale Clusters Commodity Clusters Cluster Software Grid Computing

TOP500 Benchmark Completed October 1, 2001 MayAugustDecember FebruaryAprilMayAugust October 2000 March 2001 August - October 2001

Three Systems in the Top 500 HP AlphaServer SC ES40 TCSINI Ranked 246 with GFlops Linpack Performance Cray T3E900 Jaromir Ranked 182 with 341 GFlops Linpack Performance HP AlphaServer SC ES45 LeMieux Ranked 6 with TFlops Linpack Performance Top Academic System

Cluster Node Count RankInstallation SiteNodes 1Earth Simulator Center640 2Los Alamos National Laboratory1024 3Los Alamos National Laboratory1024 4Lawrence Livermore National Laboratory512 5Lawrence Livermore National Laboratory128 6Pittsburgh Supercomputing Center750 7Commissariat a l'Energie Atomique680 8Forecast Systems Laboratory - NOAA768 9HPCx40 10National Center for Atmospheric Research40

One Year of Production lemieux.psc.edu

Its Really all About Applications Single CPU with common data stream Large Shared Memory Jobs Multi-CPU Jobs …but, lets talk systems!

HPC Systems Architectures

HPC Systems Larger SMPs MPP- Massively Parallel Machines Non Uniform Memory Access (NUMA) machines Clusters of smaller machines

Larger SMPs Pros: –Use existing technology and management techniques –Maintain parallelization paradigm (threading) –Its what users really want! Cons: –Cache coherency gets difficult –Increased resource contention –Pin counts add up –Increased incremental cost

HPC Clusters Rationale –If one box, cant do it, maybe 10 can… –Commodity hardware is advancing rapidly –Potentially far less costly than a single larger system –Big systems are only so big

HPC Clusters Central Issues –Management of multiple systems –Performance Within each node Interconnections –Effects on parallel programming methodology Varying communication characteristics

The Next Contender? CPU 128 Bit CPU System Clock Frequency MHz 32MB Main Memory direct RDRAM Embedded Cache VRAM 4MB I/O Processor CD-ROM and DVD-ROM

Why not let everyone play?

Whats a Cluster? Base Hardware Commodity Nodes –Single, Dual, Quad, ??? –Intel, AMD –Switch port cost vs cpu Interconnect –Bandwidth –Latency Storage –Node local –Shared filesystem

Terascale Computing System Hardware Summary 750 ES45 Compute Nodes 3000 EV68 1 GHz 6 Tflop 3 TB memory 41 TB node disk, ~90GB/s Multi-rail fat-tree network Redundant Interactive nodes Redundant monitor/ctrl WAN/LAN accessible File servers: 30TB, ~32 GB/s Mass Store buffer disk, ~150 TB Parallel visualization ETF coupled Quadrics Control LAN Compute Nodes File Servers /tmp WAN/LAN Interactive /usr

Compute Nodes AlphaServer ES45 –5 nodes per cabinet –3 local disks /node

Row upon row…

PSC/HP Grid Alliance A strategic alliance to demonstrate the potential of the National Science Foundation's Extensible TeraGrid 16 Node HP Itanium2/Linux cluster Through this collaboration, PSC and HP expect to further the TeraGrid goals of enabling scalable, open source, commodity computing on IA64/Linux to address real-world problems

Whats a Cluster? Base Hardware Commodity Nodes –Single, Dual, Quad, ??? –Switch port cost vs cpu Interconnect –Bandwidth –Latency Storage –Node local –Shared filesystem

Cluster Interconnect Low End 10/100 Mbit Ethernet –Very cheap –Slow with High Latency Gigabit Ethernet –Sweet Spot –Especially with: Channel Bonding Jumbo Frames

Cluster Interconnect, cont. Mid-Range Myrinet – –High speed with Good (not great) latency –High port count switches –Well adopted and supported in the Cluster Community Infiniband –Emerging –Should be inexpensive and pervasive

Cluster Interconnect, cont. Outta Sight! Quadrics Elan – –Very High Performance Great Speed Spectacular Latency –Software RMS QSNET –Becoming more Commodity

way switch. (4096 & 8192-way same but bigger!) Switches.. 8: 8*(16-way) 8-16: 64U64D (13 for TCS) Federated switch

Overhead Cables

n Fully wired switch cabinet n 1 of 24. n Wires up & down Wiring: Quadrics

Whats a Cluster? Base Hardware Commodity Nodes –Single, Dual, Quad, ??? –Switch port cost vs cpu Interconnect –Bandwidth –Latency Storage –Node local –Shared filesystem

Commodity Cache Servers Linux Custom Software –libtcom/tcsiod –Coherency Manager (SLASH) Special Purpose DASP –Connection to Outside –Multi-Protocol *ftp SRB Globus 3Ware SCSI/ATA Disk Controllers

Whats a Cluster? System Software Installation Replication Consistency Parallel File System Resource Management Job Control

Installation Replication Consistency

Users Job Management Software queues submit Batch Job Management Simon scheduler TCS scheduling practices PBS/RMS Job invocation usage accounting database Monitoring Whats next? supply process distribution execution, control PSC NSF Visualization Nodes tcscomm Checkpoint / restart user file servers HSM tcscopy / hsm tcscopy requeue tcscomm node event management call tracking and field service db user notification demand Compute Nodes CPRCPR CPRCPR CPRCPR CPRCPR PSC Terascale Computing System

Monitoring Non-Contiguous Scheduling

Whats a Cluster? Application Support Parallel Execution MPI – Shared Memory Other… –Portals –Global Arrays

Building Your Cluster Pre-Built –PSSC – Chemistry –Tempest Roll-your-Own –Campus Resources –Web Use PSC –Rich Raymond –

Open Source Cluster Application Resources Cluster on a CD – automates cluster install process Wizard driven Nodes are built over network OSCAR <= 64 node clusters for initial target Works on PC commodity components RedHat based (for now) Components: Open source and BSD style license NCSA Cluster in a Box base Enable application scientists to build and manage their own resources –Hardware cost is not the problem –System Administrators cost money, and do not scale –Software can replace much of the day-to-day grind of system administration Train the next generation of users on loosely coupled parallel machines –Current price-performance leader for HPC –Users will be ready to step up to NPACI (or other) resources when needed Rocks scales to Top500 sized resources –Experiment on small clusters –Build your own supercomputer with the same software! scary technology

GriPhyN and European DataGrid Virtual Data Tools Request Planning and Scheduling Tools Request Execution Management Tools Transforms Distributed resources (code, storage, computers, and network) Resource Management Services Resource Management Services Security and Policy Services Security and Policy Services Other Grid Services Other Grid Services Interactive User Tools Production Team Individual InvestigatorOther Users Raw data source Illustration courtesy C. Catlett, ©2001 Global Grid Forum

Extensible Terascale Facility - ETF "TeraGrid"

Grid Building Blocks Middleware: Hardware and software infrastructure to enable access to computational resources Services: Security Information Services Resource Discovery / Location Resource Management Fault Tolerance / Detection

Thank You lemieux.psc.edu