1. Big Red, the Data Capacitor, and the future (clouds) Craig A. Stewart stewart@iu.edu 2 March 2008

2. License terms Please cite as: Stewart, C.A. 2009. Big Red, the Data Capacitor, and the future (clouds). Presentation. Presented 2 Mar 2009, University of Houston, Houston, TX. Available from: http://hdl.handle.net/2022/13940 Except where otherwise noted, by inclusion of a source url or some other note, the contents of this presentation are © by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work. 2

3. September 2, 2015 Big Red - Basics and history IBM e1350 BladeCenter Cluster, SLES 9, MPICH, Loadleveler, MOAB Spring 2006: 17 days assembly at IBM facility, disassembled, reassembled in 10 days at IU. 20.48 TFLOPS peak theoretical, 15.04 achieved on Linpack; 23rd on June 2006 Top500 List (IU’s highest listing to date). In production for local users on 22 August 2006, for TeraGrid users 1 October 2006 Upgraded to 30.72 TFLOPS Spring 2008; ??? on June 2007 Top500 List Named after nickname for IU sports teams

4. September 2, 2015 Motivations and goals Initial goals for 20.48 TFLOPS system: Local demand for cycles exceeded supply TeraGrid Resource Partner commitments to meet Support life science research Support applications at 100s to 1000s of processors 2nd phase upgrade to 30.72 TFLOPS Support economic development in State of Indiana

5. September 2, 2015

6. Why a PowerPC-based blade cluster? Processing power per node Density, good power efficiency relative to available processors Possibility of performance gains through use of Altivec unit & VMX instructions Blade architecture provides flexibility for future Results of Request for Proposals process ProcessorTFLOPS/ MWatt MWatts/ PetaFLOPS Intel Xeon 70411456.88 AMD2194.57 PowerPC 970 MP (dual core)2005.00

7. September 2, 2015 Feature20.4 TFLOPS30.7 TFLOPS Computational hardware, RAM JS21 componentsTwo 2.5 GHz PowerPC 970MP processors, 8 GB RAM, 73 GB SAS Drive, 40 GFLOPS Same No. of JS21 blades512768 No. of processors; cores1,024 processors; 2,048 processor cores 1,536 processors; 3,072 processor cores Total system memory4 TB6 TB Disk storage GPFS scratch space266 TBSame Lustre535 TBSame Home directory space25 TBSame Networks Total outbound network bandwidth40 Gbit/secSame Bisection bandwidth64 GB/sec - Myrinet 200096 GB/sec - Myrinet 2000

8. September 2, 2015

9. IBM e1350 vs Cray XT3 (data from http://icl.cs.utk.edu/hpcc/hpcc_results.cgi) Per process (core)Per processor

10. September 2, 2015 IBM e1350 vs HP XC4000 (data from http://icl.cs.utk.edu/hpcc/hpcc_results.cgi)

11. September 2, 2015 Difference: 4 KB vs 16 MB page size Linpack performance Benchmark set NodesPeak Theoretical TFLOPS Achieved TFLOPS % HPCC51020.4013.5366.3 Top50051220.4815.0473.4 Top50076830.7221.7970.9

12. September 2, 2015

13. Elapsed time per simulation timestep among best in TeraGrid

14. September 2, 2015 Simulation of TonB-dependent transporter (TBDT) Used systems at NCSA, IU, PSC Modeled mechanisms for allowing transport of molecules through cell membrane Work by Emad Tajkhorshid and James Gumbart, of University of Illinois Urbana-Champaign. Mechanics of Force Propagation in TonB- Dependent Outer Membrane Transport. Biophysical Journal 93:496-504 (2007) To view the results of the simulation, please go to: http://www.life.uiuc.edu/emad/ TonB-BtuB/btub-2.5Ans.mpg Image courtesy of Emad Tajkhorshid

15. September 2, 2015 WxChallenge (www.wxchallenge.com) Over 1,000 undergraduate students, 64 teams, 56 institutions Usage on Big Red: ~16,000 CPU hours on Big Red 63% of processing done on Big Red Most of the students who used Big Red couldn’t tell you what it is Integration of computation and data flows via Lustre (Data Capacitor)

16. September 2, 2015

17. Overall user reactions NAMD, WRF users very pleased Porting from Intel instruction set a perceived challenge in a cycle-rich environment MILC optimization with VMX not successful Keys to biggest successes: Performance characteristics of JS21 nodes Linkage of computation and storage (Lustre - Data Capacitor) Support for grid computing via TeraGrid

27. Cloud computing Cloud computing does not exist Infrastructure as a Service Platform as a Service Pitfalls of ‘Gartner hype curve’ and confusing IaaS and PaaS September 2, 2015

30. Support collaborative activities Application hosting (institutional apps) Content distribution Use cloud computing as a way to deliver reliable 7 x 24 services when the institutions IT organization does not run a 7 x 24 operation Internal collaboration within the college/university Question: What do you plan/want to do with cloud computing?

31. Is it safe (results of BOF at EDUCAUSE meeting)?

32. Not a clear choice now, might be compelling later Real worries about getting data and capability back after it has once been outsources Uses will be broad Cloud computing will cause major realignments in funding Cloud computing will push more computing to the individual Legal compliance issues may be solved as more universities and colleges push on clould vendors Utility model – is it here? Maybe Consensus – big changes 1-3 years

33. Look at history of local telcos, phone switches on campus Look at history of processors Cloud computing, long haul networks, and data management issues are deeply intertwined Our challenge *may be* to figure out how to make best use of IaaS in the future No one makes their own lab glassware anymore (usually)

34. Acknowledgements - Funding Sources IU’s involvement as a TeraGrid Resource Partner is supported in part by the National Science Foundation under Grants No. ACI-0338618l, OCI-0451237, OCI-0535258, and OCI-0504075 The IU Data Capacitor is supported in part by the National Science Foundation under Grant No. CNS-0521433. This research was supported in part by the Indiana METACyt Initiative. The Indiana METACyt Initiative of Indiana University is supported in part by Lilly Endowment, Inc. This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University. The LEAD portal is developed under the leadership of IU Professors Dr. Dennis Gannon and Dr. Beth Plale, and supported by NSF grant 331480. The ChemBioGrid Portal is developed under the leadership of IU Professor Dr. Geoffrey C. Fox and Dr. Marlon Pierce and funded via the Pervasive Technology Labs (supported by the Lilly Endowment, Inc.) and the National Institutes of Health grant P20 HG003894-01 Many of the ideas presented in this talk were developed under a Fulbright Senior Scholar’s award to Stewart, funded by the US Department of State and the Technische Universitaet Dresden. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF), National Institutes of Health (NIH), Lilly Endowment, Inc., or any other funding agency

35. Acknowledgements - People Maria Morris contributed to the graphics used in this talk Marcus Christie and Surresh Marru of the Extreme! Computing Lab contributed the LEAD graphics John Morris (www.editide.us) and Cairril Mills (Cairril.com Design & Marketing) contributed graphics This work would not have been possible without the dedicated and expert efforts of the staff of the Research Technologies Division of University Information Technology Services, the faculty and staff of the Pervasive Technology Labs, and the staff of UITS generally. Thanks to the faculty and staff with whom we collaborate locally at IU and globally (via the TeraGrid, and especially at Technische Universitaet Dresden)

36. Co-author affiliations Craig A. Stewart; stewart@iu.edu; Office of the Vice President and CIO, Indiana University, 601 E. Kirkwood, Bloomington, IN Matthew Link; mrlink@indiana.edu; University Information Technology Services (UITS), Indiana University, 2711 E. 10 th St., Bloomington, IN 47408 D. Scott McCaulay, smccaula@indiana.edu,UITS, Indiana University, 2711 E. 10 th St., Bloomington, IN 47408 Greg Rodgers; rodgersg@us.ibm.com; IBM Corporation, 2455 South Road, Poughkeepsie, New York 12601 George Turner; turnerg@indiana.edu; UITS, Indiana University, 2711 E. 10 th St., Bloomington, IN 47408 David Hancock; dyhancoc@iupui,edu; UITS, Indiana University — Purdue University Indianapolis, 535 W. Michigan Street, Indianapolis, IN 46202 Richard Repasky; rrepasky@indiana.edu,UITS, Indiana University, 2711 E. 10 th St., Bloomington, IN 47408 Peng Wang; pewang@ikndiana.edu; UITS, Indiana University — Purdue University Indianapolis, 535 W. Michigan Street, Indianapolis, IN 46202 Faisal Saied; faied@purdue.edu; Rosen Center for Advanced Computing, Purdue University, 302 W. Wood Street, West Lafayette, Indiana 47907 Marlon Pierce; Community Grids Lab, Pervasive Technology Labs at Indiana University, 501 N. Morton Street, Bloomington, IN 47404 Ross Aiken; rmaiken@us.ibm.com; IBM Corporation, 9229 Delegates Row, Precedent Office Park Bldg 81, Indianapolis, IN 46240; Matthias Mueller; matthias.mueller@tu-dresden.de; Center for Information Services and High Performance Computing (ZIH) Dresden University of Technology D-01062 Dresden, Germany Matthias Jurenz; Matthias.Jurenz@tu-dresden.de; Center for Information Services and High Performance Computing (ZIH) Dresden University of Technology D-01062 Dresden, Germany Matthias Lieber; Matthias.Lieber@tu-dresden.de;Center for Information Services and High Performance Computing (ZIH) Dresden University of Technology D-01062 Dresden, Germany

37. Thank you Any questions?