1. Copyright Gordon Bell LANL 5/17/2002 Technical computing: Observations on an ever changing, occasionally repetitious, environment Los Alamos National Laboratory 17 May 2002

2. A brief, simplified history of HPC 1. Sequential & data parallelism using shared memory, Cray’s Fortran computers 60-02 (US:90) 2. 1978: VAXen threaten general purpose centers… 3. NSF response: form many centers 1988 - present 4. SCI: Search for parallelism to exploit micros 85-95 5. Scalability: “bet the farm” on clusters. Users “adapt” to clusters aka multi-computers with LCD program model, MPI. >95 6. Beowulf Clusters adopt standardized hardware and Linus’s software to create a standard! >1995 7. “Do-it-yourself” Beowulfs impede new structures and threaten g.p. centers >2000 8. 1997-2002: Let’s tell NEC they aren’t “in step ”. 9. High speed networking enables peer2peer computing and the Grid. Will this really work?

3. Copyright Gordon Bell LANL 5/17/2002 Outline Retracing scientific computing evolution: Cray, SCI & “killer micros”, ASCI, & Clusters kick in. Current taxonomy: clusters flavors deja’vu rise of commodity computng: Beowulfs are a replay of VAXen c1978 Centers: 2+1/2 at NSF; BRC on CyberInfrastructure urges 650M/year Role of Grid and Peer-to-peer Will commodities drive out or enable new ideas?

4. Copyright Gordon Bell LANL 5/17/2002 High performance architecture/program timeline 1950.1960.1970.1980.1990.2000 VtubesTrans.MSI(mini) Micro RISCnMicr Sequential programming---->------------------------------ (single execution stream) <SIMD Vector--//--------------- Parallelization--- Parallel programs aka Cluster Computing <--------------- multicomputers <--MPP era------ ultracomputers 10X in size & price!10x MPP “in situ” resources 100x in //sm NOW CC geographically dispersedGrid

5. Copyright Gordon Bell LANL 5/17/2002 DARPA SCI: c1985-1995; prelude to DOE’s ASCI Motivated by Japanese 5 th Generation … note the creation of MCC Realization that “killer micros” were Custom VLSI and its potential Lots of ideas to build various high performance computers Threat and potential sale to military

6. Copyright Gordon Bell LANL 5/17/2002 Steve Squires & G Bell at our “Cray” at the start of DARPA’s SCI c1984.

7. What Is the System Architecture? (GB c1990) X X X GRID SIMD X

8. Copyright Gordon Bell LANL 5/17/2002 Taxonomy:The Architectural Alternatives for scalability & high performance c1991 MIMD multicomputers (mC) (message passing) multiprocessors (mP) (shared memory) all are scalable multi, mainframe, super (limited-scalable) smP - scalable, with Dist'd. Shared Memory network mP?? (scalable, with DSM) multicomputers clusters workstations (ATM) workstations (LAN) Symm Asymm

9. Copyright Gordon Bell LANL 5/17/2002 Processor Architectures? VECTORS OR CS View MISC >> CISC >> Language directed RISC >> Super-scalar >> Extra-Long Instruction Word Caches: mostly alleviate need for memory B/W SC Designers View RISC >> VCISC (vectors)>> Massively parallel (SIMD) (multiple pipelines) Memory B/W = perf.

10. Copyright Gordon Bell LANL 5/17/2002 The Bell-Hillis Bet c1991 Massive (>1000) Parallelism in 1995 TMC World-wide Supers TMC World-wide Supers TMC World-wide Supers Applications Revenue Petaflops / mo.

11. Copyright Gordon Bell LANL 5/17/2002 Results from DARPA’s SCI c1983 Many research and construction efforts … virtually all new hardware efforts failed except Intel and Cray. DARPA directed purchases… screwed up the market, including the many VC funded efforts. No Software funding! Users responded to the massive power potential with LCD software. Clusters, clusters, clusters using MPI. It’s not scalar vs vector, its memory bandwidth! – 6-10 scalar processors = 1 vector unit – 16-64 scalars = a 2 – 6 processor SMP

12. Dead Supercomputer Society ACRI Alliant American Supercomputer Ametek Applied Dynamics Astronautics BBN CDC Convex Cray Computer Cray Research Culler-Harris Culler Scientific Cydrome Dana/Ardent/Stellar/Stardent Denelcor Elexsi ETA Systems Evans and Sutherland Computer Floating Point Systems Galaxy YH-1 Goodyear Aerospace MPP Gould NPL Guiltech Intel Scientific Computers International Parallel Machines Kendall Square Research Key Computer Laboratories MasPar Meiko Multiflow Myrias Numerix Prisma Tera Thinking Machines Saxpy Scientific Computer Systems (SCS) Soviet Supercomputers Supertek Supercomputer Systems Suprenum Vitesse Electronics

13. The evolution of Cray Inc. SELL SX5s

14. Copyright Gordon Bell LANL 5/17/2002 What a difference 25 years AND spending >10x makes! LLNL 150 Mflops machine room c1978 ESRDC: 40 Tflops. 640 nodes (8 - 8GFl P.vec/node)

15. Copyright Gordon Bell LANL 5/17/2002 Computer types Netwrked Supers… Legion Condor Beowulf NT clusters VPPuni T3E SP2 (mP) NOW NEC mP SGI DSM clusters & SGI DSM NEC super Cray X…T (all mPv) Mainframes Multis WSs PCs -------- Connectivity-------- WAN/LAN SAN DSM SM micros vector Clusters GRID & P2P Old World

16. Copyright Gordon Bell LANL 5/17/2002 Top500 taxonomy… everything is a cluster aka multicomputer Clusters are the ONLY scalable structure – Cluster: n, inter-connected computer nodes operating as one system. Nodes: uni- or SMP. Processor types: scalar or vector. MPP= miscellaneous, not massive (>1000), SIMD or something we couldn’t name Cluster types. Implied message passing. – Constellations = clusters of >=16 P, SMP – Commodity clusters of uni or <=4 Ps, SMP – DSM: NUMA (and COMA) SMPs and constellations – DMA clusters (direct memory access) vs msg. pass – Uni- and SMPvector clusters: Vector Clusters and Vector Constellations

17. Copyright Gordon Bell LANL 5/17/2002 Linux - a web phenomenon Linus Tovald - writes news reader for his PC Puts it on the internet for others to play Others add to it contributing to open source software Beowulf adopts early Linux Beowulf adds Ethernet drivers for essentially all NICs Beowulf adds channel bonding to kernel Red Hat distributes Linux with Beowulf software Low level Beowulf cluster management tools added

18. Copyright Gordon Bell LANL 5/17/2002 The Challenge leading to Beowulf NASA HPCC Program begun in 1992 Comprised Computational Aero-Science and Earth and Space Science (ESS) Driven by need for post processing data manipulation and visualization of large data sets Conventional techniques imposed long user response time and shared resource contention Cost low enough for dedicated single-user platform Requirement: – 1 Gflops peak, 10 Gbyte, < $50K Commercial systems: $1000/Mflops or 1M/Gflops

19. Copyright Gordon Bell LANL 5/17/2002 Innovation The Virtuous Economic Cycle drives the PC industry… & Beowulf Volume Competition Standards Utility/value DOJ Greater availability @ lower cost Creates apps, tools, training, Attracts users Attracts suppliers

20. Lessons from Beowulf An experiment in parallel computing systems Established vision- low cost high end computing Demonstrated effectiveness of PC clusters for some (not all) classes of applications Provided networking software Provided cluster management tools Conveyed findings to broad community Tutorials and the book Provided design standard to rally community! Standards beget: books, trained people, software … virtuous cycle that allowed apps to form Industry begins to form beyond a research project Courtesy, Thomas Sterling, Caltech.

21. Copyright Gordon Bell LANL 5/17/2002 Clusters: Next Steps Scalability… They can exist at all levels: personal, group, … centers Clusters challenge centers… given that smaller users get small clusters

22. Copyright Gordon Bell LANL 5/17/2002 Disk Evolution Capacity:100x in 10 years 1 TB 3.5” in 2005 20 TB? in 2012?! System on a chip High-speed SAN Disk replacing tape Disk is super computer ! Kilo Mega Giga Tera Peta Exa Zetta Yotta

23. Copyright Gordon Bell LANL 5/17/2002 Intermediate Step: Shared Logic Brick with 8-12 disk drives 200 mips/arm (or more) 2xGbpsEthernet General purpose OS 10k$/TB to 100k$/TB Shared – Sheet metal – Power – Support/Config – Security – Network ports These bricks could run applications e.g. SQL, Mail… Snap ~1TB 12x80GB NAS NetApp ~.5TB 8x70GB NAS Maxstor ~2TB 12x160GB NAS IBM TotalStorage ~360GB 10x36GB NAS

24. Copyright Gordon Bell LANL 5/17/2002 SNAP Architecture----------

25. RLX “cluster” in a cabinet 366 servers per 44U cabinet – Single processor – 2 - 30 GB/computer (24 TBytes) – 2 - 100 Mbps Ethernets ~10x perf*, power, disk, I/O per cabinet ~3x price/perf Network services… Linux based *42, 2 processors, 84 Ethernet, 3 TBytes

26. Computing in small spaces @ LANL (RLX cluster in building with NO A/C) 240 processors @2/3 GFlops Fill the 4 racks -- gives a Teraflops

28. Copyright Gordon Bell LANL 5/17/2002 Beowulf Clusters: space

29. Copyright Gordon Bell LANL 5/17/2002 Beowulf clusters: power

30. Copyright Gordon Bell LANL 5/17/2002 “The networks becomes the system.”- Bell 2/10/82 Ethernet announcement with Noyce (Intel), and Liddle (Xerox) “The network become the computer.” SUN Slogan >1982 “The network becomes the system.” GRID mantra c1999

31. Copyright Gordon Bell LANL 5/17/2002 Computing SNAP built entirely from PCs Wide & Local Area Networks for: terminal, PC, workstation, & servers Centralized & departmental uni- & mP servers (UNIX & NT) Legacy mainframes & minicomputers servers & terms Wide-area global network Legacy mainframe & minicomputer servers & terminals Centralized & departmental servers buit from PCs scalable computers built from PCs TC=TV+PC home... (CATV or ATM or satellite) ??? Portables A space, time (bandwidth), & generation scalable environment Person servers (PCs) Person servers (PCs) Mobile Nets

32. Telnet & FTP EMAIL WWW AudioVideo Voice! Standards Increase Capacity (circuits & bw) Lower response time Create new service Increased Demand The virtuous cycle of bandwidth supply and demand Incompence ?

33. Copyright Gordon Bell LANL 5/17/2002 Internet II concerns given $0.5B cost Very high cost – $(1 + 1) / GByte to send on the net; Fedex and 160 GByte shipments are cheaper – DSL at home is $0.15 - $0.30 Disks cost $1/GByte to purchase! Low availability of fast links (last mile problem) – Labs & universities have DS3 links at most, and they are very expensive – Traffic: Instant messaging, music stealing Performance at desktop is poor – 1- 10 Mbps; very poor communication links

34. Copyright Gordon Bell LANL 5/17/2002 Collaborative research sharing instrumentation, data, and programs We’ve talked about it for decades e.g. accelerators to telelescopes and zoology – Doer / “User, talker & meeter” = 4%. – http://www.all-species.org/ has the problem… http://www.all-species.org/ NSF focus has been and is on ops not bytes! – E.g. Pittsburgh center funded with no storage – Why have centers for computation at all? Don’t we need datacenter? – By having no storage, re-compute everything – Adding indexes i.e. databases, increases speed, lessens computation, and increases experimentation Computation centers become data centers since everyone/anyone builds a center Need for computational scientist database talent! Big question: Can distributed computing form to provide something better than a “center” can provide?

35. Scalable computing: the effects They come in all sizes; incremental growth 10 or 100 to 10,000 (100X for most users) debug vs run; problem growth Allows compatibility heretofore impossible 1978: VAX chose Cray Fortran 1987: The NSF centers went to UNIX Users chose sensible environment – Acquisition and operational costs & environments – Cost to use as measured by user’s time The role of gp centers e.g. NSF, state x is unclear. Necessity for support? – Scientific Data for a given community… – Community programs and data – Manage GRID discipline Are clusters ≈ Gresham’s Law? Drive out alts.

36. Copyright Gordon Bell LANL 5/17/2002 The end