Fast Data Transfer: Brazil Panel: Network Architecture for Globalization PRPv2 Workshop, UCSD Feb 21st, 2017 Michael Stanton RNP, Brazil

Objective: mention high-performance network support for e-science in Brazil Locally: High capacity circuits Science DMZ Internationally: High-capacity network support

Internal and international network infrastructure RNP national backbone: Connects capitals of each state and federal capital Most links currently 10G, except in the Amazon region Metro networks in 26 capitals and 13 other cities Built and owned by RNP and operated by local PoP International links: To US, shares with ANSP (state network in São Paulo) infrastructure of successive IRNC projects: currently Amlight ExP (240G currently available capacity) To Europe (5G) and Latin America (10G) via RedClara Terrestrial link between Porto Alegre and Buenos Aires (Argentina), used also by RedClara and InnovaRed (AR)

National RNP network 4Q2016 (100G by 2018) Southeast Region: Rio de Janeiro – São Paulo

Geography of high performance e-science: the states of Rio de Janeiro and São Paulo Cities/Towns: Campinas (M): Unicamp (U), CNPEM (Synchrotron Light Source) São Paulo (M): UNESP (U, HPC), USP (U) São José do Campos: INPE (Space Science including Earth Observation) Cachoeira Paulista: CPTEC (Weather Forecasting, Climate Studies, HPC-SC) Rio de Janeiro (M): CBPF (Physics, HPC), PUC-Rio (U), UERJ (U, HPC), UFRJ (U) Niterói (M): UFF (U) Petrópolis (M): LNCC (HPC-SC) Key: HPC: high-performance centre HPC-SC: supercomputer M: RNP-owned metro network U: university (all public, except for PUC-Rio)

International connections BR-US in 2016 (IRNC project AmLight-ExP) Partnership RNP & ANSP† with FIU, using paths on 2 submarine cable rings TIWS (SAm-1): 10G circuits on Atlantic and Pacific coasts (red and blue) LANautilus (SAC/LAN): New 100G circuits on the Atlantic and Pacific (yellow). + temporarily maintained 10G circuits (green/magenta) Total: 240G Operational since July 2016 † ANSP = network in São Paulo state

High-performance data transfer in Brazil Considerable experience has been gained since 2004 in providing such transfers, most of which have been carried out in the context of Supercomputing demos organised by high-energy physicists, led by Caltech These have involved two HEP centres at UERJ (Rio) and UNESP (São Paulo) which have demonstrated efficient use of the available bandwidth assigned to the demos. At SC16 in Salt Lake City, the UNESP group demonstrated > 95% use of the new (2016) 100G access to the US networks, via AMPATH (Miami)

100G Link Atlantic Ocean 100G Link Pacific Ocean (By courtesy of Rogério Iope,UNESP, São Paulo)

Data transfer results Atlantic Link 24 hour period (from Nov 16 to 17) (By courtesy of Rogério Iope,UNESP, São Paulo)

RNP and Science DMZ RNP has been exploring the support of high-performance data transfer to user sites, by use of the Science DMZ architecture from ESnet, with support from USP (U of São Paulo) This involves: Producing kits for DTNs + perfSonar for 1 and 10G sites Currently designing kit for 100G Identifying suitable use cases Tuning installed kits

Use Case: Inter-institutional e-Astronomy Lab (LIneA) at LNCC (Petrópolis, RJ) 2 use cases: Data transfers from NCSA and JHU to LIneA@LNCC in 2 steps. srvdatatransfer.linea.gov.br LIneA e-Astronomy lab RNP Office Rio de Janeiro 1Gbps RNP PoP São Paulo 10Gbps 20Gbps LNCC Petrópolis Science DMZ@RNP ps ps-bw.sdmz.rnp.br ps-lt.sdmz.rnp.br dtn.sdmz.rnp.br to Miami (Ampath) NCSA 141.142.160.4 desar2.cosmology.Illinois.edu JHU 128.220.233.83 dss001.pha.jhu.edu Internet2

Use Case: LIneA (cont’d) Transfer of data of the Sloan Digital Sky Survey – IV (SDSS) from Johns Hopkins University (Baltimore, MD, USA), via RNP (Rio de Janeiro, RJ) to LIneA/LNCC (Petrópolis, RJ): Disc to disc transfer of 5,7 Terabytes: w/o Science DMZ Using FTP with 8 parallel flows Reaching rates of 20 to 30 Mbps Complete transfer estimated in 2 weeks JHU (Maryland, EUA) LIneA/LNCC (Petrópolis, BR) with Science DMZ Using GridFTP with 8 parallel flows achieved rates of  1 Gbps Took  12 hours Using GridFTP with 8 parallel flows achieved rates of 2.2 Gbps 5 hours to complete JHU (Maryland, EUA) RNP (Rio de Janeiro, BR) LIneA/LNCC (Petrópolis, BR)

Using WGET to transfer  300 GB Took from 3 to 4 weeks to complete Use case: Federal U of Pernambuco (UFPE) in Recife, PE Data transfer scenario of Vegetable Genetics and Biotechnology Lab at UFPE and partner institutions NCBI (USA), Luxemburg (EU) e Unicamp (BR). Periodic transfers of 300 GB of data every 2 dias. w/o Science DMZ Using WGET to transfer  300 GB Took from 3 to 4 weeks to complete NCBI (EUA) LAB (UFPE, BR) with Science DMZ Using WGET takes  2 hours Using WGET now takes  4 days to complete NCBI (EUA) Science DMZ @ UFPE LAB (UFPE, BR)

PADEX: a project to enable efficient remote access to LNCC’s supercomputer In 2016, a new supercomputer, called Santos Dumont, was installed at the National Scientific Computing Centre (LNCC) in Petrópolis, RJ RNP is currently engaged in propitiating efficient remote use of the SC, by improving the access connection, potentially to 100G Use of DTNs and Science DMZ First potential clients are CNPEM (Campinas, SP) – Light Synchrotron sources CPTEC (Cachoeira Paulista, SP) – Weather and Climate Work in progress on the CNPEM case

Michael Stanton RNP, Brazil michael@rnp.br