SCARIe: using StarPlane and DAS-3 Paola Grosso Damien Marchel Cees de Laat SNE group - UvA

Outline The goal of my presentation: to give an overview of the DAS-3 and StarPlane architecture, their current status and outline which use SCARIe research can do on these infrastructure. Consequently the outline of this talk: -architecture of DAS-3 -architecture of StarPlane -current status of DAS-3 -current status of StarPlane -SCARIe use of DAS-3 and StarPlane

Some basics… DAS-3 is the ASCI distributed supercomputer, or better said the Distributed ASCI Supercomputer. DAS-3 is the third generation of DAS. StarPlane is an NWO-funded project. StarPlane goal is to develop the middleware for application controlled photonic networks. StarPlane performs its research on a portion of SURFnet6.

DAS-3: goals The goal of DAS is to provide a common computational infrastructure for researchers within ASCI, who work on various aspects of parallel and distributed systems, including communication substrates, programming environments, and applications. [Quoting Henri Bal - VU] Parallel processing on multiple clusters Study non-trivially parallel applications Exploit hierarchical structure for locality optimizations –latency hiding, message combining, etc.

DAS-3: cluster architecture Five clusters in four locations: UvA, VU, Leiden and Delft. Two clusters at the UvA. Key components: -1 head node -[] cluster nodes -Local/university interconnect -Fast interconnect -Photonic interconnect

DAS-3: clusters setup LUTUDUvAUvA-MNVU TOTALS Head * storage10TB5TB2TB 10TB 29TB * CPU2x2.4GHz DC 2x2.2GHz DC 2x2.4GHz DC 46GHz * memory16GB 8GB16GB8GB 64GB * Myri 10G1111 * 10GE11111 Compute (1) * storage400GB250GB 2x250GB250GB 89TB * CPU2x2.6GHz2x2.4GHz2x2.2GHz DC2x2.4GHz2x2.4GHz DC 1.9THz * memory4GB 1084GB * Myri 10G1111 Myrinet * 10G ports33 (7) (2) 203 * 10GE ports Nortel * 1GE ports32 (16)136 (8)40 (8)46 (2)85 (11) 542 * 10GE ports1 (1)9 (3)221 (1) 15

DAS-3: status As of today. All sites: - Operational with cluster and head nodes and bridge nodes installed. [Bridge nodes are used to connect to the photonic network, while the performance of the Myrinet cards is under test] -User accounts available. -Jobs can run on individual clusters not between clusters. Grid software: -SGE is the preferred scheduler -GLOBUS and KOALA are installed but not operational -MPICH-G2 is not installed Myrinet: -Switchcards installed but not operational -Tests of Myrinet-Ethernet bridging

DAS-3: photonic network The LAN connection to the University network, via Ethernet switches. The WAN/StarPlane connection to CPL, via Ethernet “bridging” card in a Myrinet switch. Connection from each site to: - OADM (fixed) equipment -Optical Add Drop Multiplexer - WSS - Wavelength Selectable Switches- in Amsterdam.

StarPlane: goals StarPlane is a NWO funded project with major contributions from SURFnet and NORTEL. StarPlane will use the physical infrastructure provided by SURFnet6 and the distributed supercomputer DAS-3. The vision is to allow part of the photonic network infrastructure of SURFnet6 to be manipulated by Grid applications to optimize the performance of specific e-Science applications. The novelty: to give flexibility directly to the applications by allowing them to choose the logical topology in real time, ultimately with subsecond lambda switching times.

StarPlane: WSS WSS will allow us to redirect a selected input color to the output fiber This allows us to flexibly reconfigure the network according to the application demands. Goal of StarPlane is sub-second switching, and topology reconfiguration.

Topology examples StarPlane: topology changes

StarPlane and NDL While on topologies. SNE group is working on NDL - Network Description Language. NDL is an RDF data model, based on idea of Semantic Web, for network topology descriptions. In StarPlane we are researching use of NDL for topology exchange and topology requests from clients.

StarPlane: status (1) First light between sites over the CPL infrastructure. Monitoring of connections being worked on. But no dynamic control over lightpaths yet.

StarPlane: status(2) Interaction between the management plane and DRAC over a testbed.

Conclusions References: DAS-3 website: StarPlane website: NDL website Questions? Now for the discussion…

For discussion Use of DAS-3 and StarPlane for SCARIe software correlation. 1.Does the DAS-3 supercomputer provide a suitable environment for software correlation? Computing power and number of nodes is adequate? Grid software suited to requirements? 2. Data needs to be on the DAS-3 cluster. How do we move the data in real-time to the cluster? Or we just focus on ust offline correlation? Do we have a lightpath connection to JIVE? From where to where? 3. Topology request. What are the optimal photonic topologies for software correlations?