Die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)‏ Steinbuch Centre for Computing (SCC) A large Multi-Functional HPC Centre Rudolf Lohner, Frank Schmitz

2 | Steinbuch Centre for Computing | ZKI in Göttingen University 11faculties 120 institutes 4,000 employees 18,500 students 250 mio € budget University 11faculties 120 institutes 4,000 employees 18,500 students 250 mio € budget Forschungszentrum 10programs (research areas)‏ 21 large institutes 4,000 employees 310 mio € budget Forschungszentrum 10programs (research areas)‏ 21 large institutes 4,000 employees 310 mio € budget Karlsruhe Institute of Technology (KIT)‏ 10 km 15 min science education structures for the administration institutes services infrastructure KIT…

3 | Steinbuch Centre for Computing | ZKI in Göttingen KIT / SCC Karlsruhe Institute of Technology - two strong partners united  Universität Karlsruhe (TH)‏ One of the most research oriented universities in Germany Nominated to one of three top universities in October 2006 Capitalized with over 100 million € over 5 years  Forschungzentrum Karlsruhe Member of the Helmholtz Association Internationally acknowledged non-university institution Under the roof of KIT – Steinbuch Centre for Computing  Computing Center of the Universität Karlsruhe (TH)‏  Institute of Scientific Computing of the Forschungszentrum Karlsruhe

4 | Steinbuch Centre for Computing | ZKI in Göttingen IWR / FZK RZ Uni KA Virtual Computing Centre since 1996 IT-Services for UKA High Performance Computing Numerical Methods on HPC Integrated Information Mgmt. IT-Services for FZK Scientific Computing GridKa, CampusGrid, D-Grid‏ Nat. & Intl. Grid Projects Virtual Computing Centre Karlsruhe since 1996

5 | Steinbuch Centre for Computing | ZKI in Göttingen SCC-Board Four Professors and one Technical Director Networks and Telecommunication Data Management and Information- Services Systems and Server Desktop- Workstation- and Communi- cations Services SOA and Virtualisation Distributed Systems and Grid Print-, Plot- und Media Production I/O, Operational Services and Help Desk HPC -Systems, -Applications, -Grid IT-Security and Service- Management Project-Managemet and -Coordination Quality- und Product-Mgmt. Strategic Communiation & Relationship Mgmt. Organisation Teaching, Advanced Education Common Administration and Organisation 10 SCC Departments

6 | Steinbuch Centre for Computing | ZKI in Göttingen KIT / SCC Roadmap Karlsruhe Institute of Technology (KIT)‏  Foundation Contract to be signed in 2007  Official start of KIT: January 1st, 2008 Steinbuch Centre for Computing (SCC)‏  Founded simultaneously with KIT  Official start of SCC: January 1st, 2008

7 | Steinbuch Centre for Computing | ZKI in Göttingen SCC Portfolio Infrastructure Services Projects

8 | Steinbuch Centre for Computing | ZKI in Göttingen HPC platforms at SCC: HP XC6000 / XC4000 Q1 04 Q4 04 Q1 05 Q1 06 Q3 06 Phase 2: » Total of 3000 processor cores » Total of 15.6 TFlop/s peak performance » Total of 12 TB of main memory » Total 110 TB of local disk space (scratch)‏ Phase 0 (Q1 2004)‏ »12 2-way nodes (Itanium 2)‏ »4 file server nodes -2 TB shared storage »Single rail Quadrics interconnect Phase 1 (Q4 2004)‏ »108 2-way nodes -Intel Madison 6M, 1.5 Ghz, 12 GB »8 file server nodes -Approx. 11 TB storage system »Single rail Quadrics interconnect Phase 1 (Q1 2005)‏ »6 16-way nodes -Intel Madison 6M, 1.6 Ghz -2 partions, each 8 CPUs, 64 GB »Single rail Quadrics interconnect Phase 2 (Q3 2006)‏ »750 4-way nodes (two sockets)‏ -dual core AMD Opteron 2.6 Ghz -16 GB main memory »10 server nodes »10 file server nodes -Approx. 56 TB storage system »Infiniband DDR interconnect

9 | Steinbuch Centre for Computing | ZKI in Göttingen Globales Filesystem 56 TB Infiniband 4x (DDR) Fast Interconnect Ethernet Administration Login nodes Lustre Compute nodes Service nodes Local Disks à 72 GB... Configuration of the HP XC4000

10 | Steinbuch Centre for Computing | ZKI in Göttingen Parallel File System HP SFS on XC4000 Shared storage for all nodes of XC system 10 file server nodes  2 MDS / Admin  2 OSS for $HOME  6 OSS for $WORK 56 TB file space  8 TB $HOME  48 TB $WORK Bandwidth  Read / write from one node: 340 MB/s / 340 MB/s  Total read / write bandwidth of $HOME: 600 MB/s / 360 MB/s  Total read / write bandwidth of $WORK: 3600 MB/s / 2200 MB/s

11 | Steinbuch Centre for Computing | ZKI in Göttingen GridKa, the Tier-1 computer center for LHC LHC=large hadron collider (CERN) Grid based structure in a worldwide environment for high energy physics gLite storage resource manager (SRM) based on dCache Resource broker for world wide usage of LHC resources PBSpro for resource management Certification Authority (CA)‏

12 | Steinbuch Centre for Computing | ZKI in Göttingen GridKa, usage in Germany 22 Research Centers 44 Working Groups 350 Scientists Aachen (4) ● Bielefeld (2) ● Bochum (2) ● Bonn (3) ● Darmstadt (1) ▲ Dortmund (1) ● Dresden (2) ● Erlangen (1) ● Frankfurt (1) ● Freiburg (2) ● Hamburg (1) ▲ Heidelberg (1) ▲ (6) ● Karlsruhe (2) ● Mainz (3) ● Mannheim (1) ● München (1) ● (5) ▲ Münster (1) ● Rostock (1) ● Siegen (1) ● Wuppertal (2) ● ▲

13 | Steinbuch Centre for Computing | ZKI in Göttingen GridKa, future plans FZK has signed the WLCG MoU and is able to deliver resources for the LHC experiment up to 2022 as the Tier-1 center resources for WLCG (Worldwide LHC Computing Grid Projekt

14 | Steinbuch Centre for Computing | ZKI in Göttingen Motivation for the project CampusGrid heterogeneous IT-environment: vector-, SMP-, cluster-, blade- systems, SAN, NAS, Unix, Linux, Windows, Solaris, SuperUX, …. global view by the user only one user management (ADS from Microsoft)‏ one job management metacomputing (MPI,..), “real-time” applications access data for visualization at the local workstation global accounting seamless integration into different projects and middleware concepts like gLite, LCG, D-Grid, Unicore, GT4, ….

15 | Steinbuch Centre for Computing | ZKI in Göttingen Infrastructure of the project CampusGrid FSC Celsius (MetaDataServer)‏ 96 FSC RX 220 IBM eServer e325 IBM p630 Power4 IBM xSeries Sunfire V20z Brocade Silkworm qLogic/Silverstorm IB director IB-FC gateway FibreChannel InfiniBand 10/20 Gbit/s Brocade Switch and IBM Blade JS20/HS20/HS21 EMC Clariion CX700, S-ATA, FC disks Intel Whitebox 16 Gbit/s 4 Gbit/s FSC Celsius (MetaDataServer)‏ 4 Gbit/s 16 Gbit/s 32 Gbit/s 16 Gbit/s NEC SX-8R environment

16 | Steinbuch Centre for Computing | ZKI in Göttingen D-Grid (the German Grid initiative)‏ starting September 2005 to build a sustainable Grid infrastructure six Community Grid projects an the D-Grid Integration Project (DGI)‏ funded by BMBF, the Federal Ministry of Education and Research astronomy, climate research, high energy physics, engineering research, medical research, humanities gLite, GT4, UNICORE SRM/dCache for the data access heterogeneous CPU access (AIX, Linux, Solaris, SuperUX)‏

17 | Steinbuch Centre for Computing | ZKI in Göttingen Shosholoza Project: America’s Cup  Form optimization/Model reduction/Weather forecast  Head of the Scientific Advisory Board Metras Project: Weather Forecast  Cooperation with the University of Hamburg  Task force SCC (7 employees)‏ Simulation of Solid Oxid Fuel Cells  Detailed chemistry and 3D United Airways: Simulation of the respiratory tract  Cooperation with the clinical Center (2 employees)‏  Numerical HPC Numerics in HPC at the SCC - Projects

18 | Steinbuch Centre for Computing | ZKI in Göttingen United Airways Project

19 | Steinbuch Centre for Computing | ZKI in Göttingen Numerics in HPC at the SCC - Projects Multicore and coprocessor technology  Cooperation with HP and ClearSpeed Dedicated Research Group Junior Professor  Numerics on specific architectures Multicore Coprocessors Visualization Project: on HP SVA  Instationary 3D flow Software Development  HiFlow-Project: FEM, 3D, HPC (  HiVision-Project: Visualization  OpenLB: Lattice-Boltzmann/HPC (

20 | Steinbuch Centre for Computing | ZKI in Göttingen The influence of a Storage Lake in the Arid Elqui Valley in Chile on the Local Climate out: Δx = 800 m, Δy = 800m in(1): Δx = 400 m, Δy = 400 m in(2): Δx = 200 m, Δy = 200 m in(3): Δx = 100 m, Δy = 100 m (out)‏ (in)‏ 225 x 135 horizontal gridpoints 100 vertical levels

21 | Steinbuch Centre for Computing | ZKI in Göttingen Thank you! Questions?