1. High Performance Linux Clusters Guru Session, Usenix, Boston June 30, 2004 Greg Bruno, SDSC

2. Overview of San Diego Supercomputer Center Founded in 1985  Non-military access to supercomputers Over 400 employees Mission: Innovate, develop, and deploy technology to advance science Recognized as an international leader in:  Grid and Cluster Computing  Data Management  High Performance Computing  Networking  Visualization Primarily funded by NSF

3. My Background 1984 - 1998: NCR - Helped to build the world’s largest database computers  Saw the transistion from proprietary parallel systems to clusters 1999 - 2000: HPVM - Helped build Windows clusters 2000 - Now: Rocks - Helping to build Linux- based clusters

4. Why Clusters?

5. Moore’s Law

6. Cluster Pioneers In the mid-1990s, Network of Workstations project (UC Berkeley) and the Beowulf Project (NASA) asked the question: Can You Build a High Performance Machine From Commodity Components?

7. The Answer is: Yes Source: Dave Pierce, SIO

8. The Answer is: Yes

9. Types of Clusters High Availability  Generally small (less than 8 nodes) Visualization High Performance  Computational tools for scientific computing  Large database machines

10. High Availability Cluster Composed of redundant components and multiple communication paths

11. Visualization Cluster Each node in the cluster drives a display

12. High Performance Cluster Constructed with many compute nodes and often a high- performance interconnect

13. Cluster Hardware Components

14. Cluster Processors Pentium/Athlon Opteron Itanium

15. Processors: x86 Most prevalent processor used in commodity clustering Fastest integer processor on the planet:  3.4 GHz Pentium 4, SPEC2000int: 1705

16. Processors: x86 Capable floating point performance  #5 machine on Top500 list built with Pentium 4 processors

17. Processors: Opteron Newest 64-bit processor Excellent integer performance  SPEC2000int: 1655 Good floating point performance  SPEC2000fp: 1691  #10 machine on Top500

18. Processors: Itanium First systems released June 2001 Decent integer performance  SPEC2000int: 1404 Fastest floating-point performance on the planet  SPEC2000fp: 2161 Impressive Linpack efficiency: 86%

19. Processors Summary ProcessorGHzSPECintSPECfpPrice Pentium 4 EE 3.417051561791 Athlon FX-51 2.214471423728 Opteron 1502.416551644615 Itanium 21.5140421614798 Itanium 21.3116218911700 Power4+1.711581776????

20. But What You Really Build? Itanium: Dell PowerEdge 3250 Two 1.4 GHz CPUs (1.5 MB cache)  11.2 Gflops peak 2 GB memory 36 GB disk $7,700  Two 1.5 GHz (6 MB cache) makes the system cost ~$17,700 1.4 GHz vs. 1.5 GHz  ~7% slower  ~130% cheaper

21. Opteron IBM eServer 325 Two 2.0 GHz Opteron 246  8 Gflops peak 2 GB memory 36 GB disk $4,539  Two 2.4 GHz CPUs: $5,691 2.0 GHz vs. 2.4 GHz  ~17% slower  ~25% cheaper

22. Pentium 4 Xeon HP DL140 Two 3.06 GHz CPUs  12 Gflops peak 2 GB memory 80 GB disk $2,815  Two 3.2 GHz: $3,368 3.06 GHz vs. 3.2 GHz  ~4% slower  ~20% cheaper

23. If You Had $100,000 To Spend On A Compute Farm System # of Boxes Peak GFlops Aggregate SPEC2000fp Aggregate SPEC2000int Pentium 4 3 GHz 3542089810104370 Opteron 246 2.0 GHz 221765689257948 Itanium 1.4 GHz 121324660824528

24. What People Are Buying Gartner study Servers shipped in 1Q04  Itanium: 6,281  Opteron: 31,184 Opteron shipped 5x more servers than Itanium

25. What Are People Buying Gartner study Servers shipped in 1Q04  Itanium: 6,281  Opteron: 31,184  Pentium: 1,000,000 Pentium shipped 30x more than Opteron

26. Interconnects

27. Ethernet  Most prevalent on clusters Low-latency interconnects  Myrinet  Infiniband  Quadrics  Ammasso

28. Why Low-Latency Interconnects? Performance  Lower latency  Higher bandwidth Accomplished through OS-bypass

29. How Low Latency Interconnects Work Decrease latency for a packet by reducing the number memory copies per packet

30. Bisection Bandwidth Definition: If split system in half, what is the maximum amount of data that can pass between each half? Assuming 1 Gb/s links:  Bisection bandwidth = 1 Gb/s

31. Bisection Bandwidth Assuming 1 Gb/s links:  Bisection bandwidth = 2 Gb/s

32. Bisection Bandwidth Definition: Full bisection bandwidth is a network topology that can support N/2 simultaneous communication streams. That is, the nodes on one half of the network can communicate with the nodes on the other half at full speed.

33. Large Networks When run out of ports on a single switch, then you must add another network stage In example above: Assuming 1 Gb/s links, uplinks from stage 1 switches to stage 2 switches must carry at least 6 Gb/s

34. Large Networks With low-port count switches, need many switches on large systems in order to maintain full bisection bandwidth  128-node system with 32-port switches requires 12 switches and 256 total cables

35. Myrinet Long-time interconnect vendor  Delivering products since 1995 Deliver single 128-port full bisection bandwidth switch MPI Performance:  Latency: 6.7 us  Bandwidth: 245 MB/s  Cost/port (based on 64-port configuration): $1000 Switch + NIC + cable http://www.myri.com/myrinet/product_list.html

36. Myrinet Recently announced 256- port switch  Available August 2004

37. Myrinet #5 System on Top500 list System sustains 64% of peak performance  But smaller Myrinet-connected systems hit 70-75% of peak

38. Quadrics QsNetII E-series  Released at the end of May 2004 Deliver 128-port standalone switches MPI Performance:  Latency: 3 us  Bandwidth: 900 MB/s  Cost/port (based on 64-port configuration): $1800 Switch + NIC + cable http://doc.quadrics.com/Quadrics/QuadricsHome.nsf/DisplayPages/ A3EE4AED738B6E2480256DD30057B227

39. Quadrics #2 on Top500 list Sustains 86% of peak  Other Quadrics-connected systems on Top500 list sustain 70-75% of peak

40. Infiniband Newest cluster interconnect Currently shipping 32-port switches and 192-port switches MPI Performance:  Latency: 6.8 us  Bandwidth: 840 MB/s  Estimated cost/port (based on 64-port configuration): $1700 - 3000 Switch + NIC + cable http://www.techonline.com/community/related_content/24364

41. Ethernet Latency: 80 us Bandwidth: 100 MB/s Top500 list has ethernet-based systems sustaining between 35-59% of peak

42. Ethernet With Myrinet, would have sustained ~1 Tflop  At a cost of ~$130,000 Roughly 1/3 the cost of the system What we did with 128 nodes and a $13,000 ethernet network  $101 / port $28/port with our latest Gigabit Ethernet switch  Sustained 48% of peak

43. Rockstar Topology 24-port switches Not a symmetric network  Best case - 4:1 bisection bandwidth  Worst case - 8:1  Average - 5.3:1

44. Low-Latency Ethernet Bring os-bypass to ethernet Projected performance:  Latency: less than 20 us  Bandwidth: 100 MB/s Potentially could merge management and high-performance networks Vendor “Ammasso”

45. Application Benefits

46. Storage

47. Local Storage Exported to compute nodes via NFS

48. Network Attached Storage A NAS box is an embedded NFS appliance

49. Storage Area Network Provides a disk block interface over a network (Fibre Channel or Ethernet) Moves the shared disks out of the servers and onto the network Still requires a central service to coordinate file system operations

50. Parallel Virtual File System PVFS version 1 has no fault tolerance PVFS version 2 (in beta) has fault tolerance mechanisms

51. Lustre Open Source “Object-based” storage  Files become objects, not blocks

52. Cluster Software

53. Cluster Software Stack Linux Kernel/Environment  RedHat, SuSE, Debian, etc.

54. Cluster Software Stack HPC Device Drivers  Interconnect driver (e.g., Myrinet, Infiniband, Quadrics)  Storage drivers (e.g., PVFS)

55. Cluster Software Stack Job Scheduling and Launching  Sun Grid Engine (SGE)  Portable Batch System (PBS)  Load Sharing Facility (LSF)

56. Cluster Software Stack Cluster Software Management  E.g., Rocks, OSCAR, Scyld

57. Cluster Software Stack Cluster State Management and Monitoring  Monitoring: Ganglia, Clumon, Nagios, Tripwire, Big Brother  Management: Node naming and configuration (e.g., DHCP)

58. Cluster Software Stack Message Passing and Communication Layer  E.g., Sockets, MPICH, PVM

59. Cluster Software Stack Parallel Code / Web Farm / Grid / Computer Lab  Locally developed code

60. Cluster Software Stack Questions:  How to deploy this stack across every machine in the cluster?  How to keep this stack consistent across every machine?

61. Software Deployment Known methods:  Manual Approach  “Add-on” method Bring up a frontend, then add cluster packages  OpenMosix, OSCAR, Warewulf  Integrated Cluster packages are added at frontend installation time  Rocks, Scyld

62. Rocks

63. Primary Goal Make clusters easy Target audience: Scientists who want a capable computational resource in their own lab

64. Philosophy Not fun to “care and feed” for a system All compute nodes are 100% automatically installed  Critical for scaling Essential to track software updates  RHEL 3.0 has issued 232 source RPM updates since Oct 21 Roughly 1 updated SRPM per day Run on heterogeneous standard high volume components  Use the components that offer the best price/performance!

65. More Philosophy Use installation as common mechanism to manage a cluster  Everyone installs a system: On initial bring up When replacing a dead node Adding new nodes Rocks also uses installation to keep software consistent  If you catch yourself wondering if a node’s software is up- to-date, reinstall! In 10 minutes, all doubt is erased  Rocks doesn’t attempt to incrementally update software

66. Rocks Cluster Distribution Fully-automated cluster-aware distribution  Cluster on a CD set Software Packages  Full Red Hat Linux distribution Red Hat Linux Enterprise 3.0 rebuilt from source  De-facto standard cluster packages  Rocks packages  Rocks community packages System Configuration  Configure the services in packages

67. Rocks Hardware Architecture

68. Minimum Components X86, Opteron, IA64 server Local Hard Drive Power Ethernet OS on all nodes (not SSI)

69. Optional Components Myrinet high-performance network  Infiniband support in Nov 2004 Network-addressable power distribution unit keyboard/video/mouse network not required  Non-commodity  How do you manage your management network?  Crash carts have a lower TCO

70. Storage NFS  The frontend exports all home directories Parallel Virtual File System version 1  System nodes can be targeted as Compute + PVFS or strictly PVFS nodes

71. Minimum Hardware Requirements Frontend:  2 ethernet connections  18 GB disk drive  512 MB memory Compute:  1 ethernet connection  18 GB disk drive  512 MB memory Power Ethernet switches

72. Cluster Software Stack

73. Rocks ‘Rolls’ Rolls are containers for software packages and the configuration scripts for the packages Rolls dissect a monolithic distribution

74. Rolls: User-Customizable Frontends Rolls are added by the Red Hat installer  Software is added and configured at initial installation time Benefit: apply security patches during initial installation  This method is more secure than the add-on method

75. Red Hat Installer Modified to Accept Rolls

76. Approach Install a frontend 1. Insert Rocks Base CD 2. Insert Roll CDs (optional components) 3. Answer 7 screens of configuration data 4. Drink coffee (takes about 30 minutes to install) Install compute nodes: 1. Login to frontend 2. Execute insert-ethers 3. Boot compute node with Rocks Base CD (or PXE) 4. Insert-ethers discovers nodes 5. Goto step 3 Add user accounts Start computing Optional Rolls  Condor  Grid (based on NMI R4)  Intel (compilers)  Java  SCE (developed in Thailand)  Sun Grid Engine  PBS (developed in Norway)  Area51 (security monitoring tools)

77. Login to Frontend Create ssh public/private key  Ask for ‘passphrase’  These keys are used to securely login into compute nodes without having to enter a password each time you login to a compute node Execute ‘insert-ethers’  This utility listens for new compute nodes

78. Insert-ethers Used to integrate “appliances” into the cluster

79. Boot a Compute Node in Installation Mode Instruct the node to network boot  Network boot forces the compute node to run the PXE protocol (Pre-eXecution Environment) Also can use the Rocks Base CD  If no CD and no PXE-enabled NIC, can use a boot floppy built from ‘Etherboot’ (http://www.rom-o-matic.net)

80. Insert-ethers Discovers the Node

81. Insert-ethers Status

82. eKV Ethernet Keyboard and Video Monitor your compute node installation over the ethernet network  No KVM required! Execute: ‘ssh compute-0-0’

83. Node Info Stored In A MySQL Database If you know SQL, you can execute some powerful commands

84. Cluster Database

85. Kickstart Red Hat’s Kickstart  Monolithic flat ASCII file  No macro language  Requires forking based on site information and node type. Rocks XML Kickstart  Decompose a kickstart file into nodes and a graph Graph specifies OO framework Each node specifies a service and its configuration  Macros and SQL for site configuration  Driven from web cgi script

86. Sample Node File ]> Enable SSH &ssh; &ssh;-clients &ssh;-server &ssh;-askpass Host * CheckHostIP no ForwardX11 yes ForwardAgent yes StrictHostKeyChecking no UsePrivilegedPort no FallBackToRsh no Protocol 1,2 chmod o+rx /root mkdir /root/.ssh chmod o+rx /root/.ssh >

87. Sample Graph File Default Graph for NPACI Rocks. …

88. Kickstart framework

89. Appliances Laptop / Desktop  Appliances  Final classes  Node types Desktop IsA  standalone Laptop IsA  standalone  pcmcia Code re-use is good

90. Architecture Differences Conditional inheritance Annotate edges with target architectures if i386  Base IsA grub if ia64  Base IsA elilo One Graph, Many CPUs  Heterogeneity is easy  Not for SSI or Imaging

91. Installation Timeline

92. Status

93. But Are Rocks Clusters High Performance Systems? Rocks Clusters on June 2004 Top500 list:

95. What We Proposed To Sun Let’s build a Top500 machine … … from the ground up … … in 2 hours … … in the Sun booth at Supercomputing ‘03

96. Rockstar Cluster (SC’03) Demonstrate  We are now in the age of “personal supercomputing”  Highlight abilities of: Rocks SGE Top500 list  #201 - November 2003  #413 - June 2004 Hardware  129 Intel Xeon servers 1 Frontend Node 128 Compute Nodes  Gigabit Ethernet $13,000 (US) 9 24-port switches 8 4-gigabit trunk uplinks