Toward Production Level Operation of Authentication System for High Performance Computing Infrastructure in Japan Eisaku Sakane and Kento Aida National Institute of Informatics Eisaku Sakane and Kento Aida, National Institute of Informatics

Introduction High Performance Computing Infrastructure (HPCI)  national project promoted by Ministry of Education, Culture, Sports, Science and Technology (MEXT) in Japan  distributed computing infrastructure for high performance computing “K computer”, supercomputers and high performance storage  first production level infrastructure for high performance computing in Japan Roadmap  – Mar 2011basic design network, authentication, user management, shared storage, testbed for advanced software  Apr – Dec 2011 detailed design  Jan – Oct 2012test operation  Nov 2012 –production level operation Eisaku Sakane and Kento Aida, National Institute of Informatics This talk presents pilot operations of the authentication system for HPCI.

portal CA system shib. SP shared storage single sign-on apply certificate authentication network infrastructure computer resource shib. IdP shib. IdP shib. IdP shib. IdP shib. IdP shib. IdP HPCI acct. HPCI ID registration review proposals user management certificate repository HPCI Overview (at Nov. 2012) Eisaku Sakane and Kento Aida, National Institute of Informatics More resources will be connected after AICS, U. Tokyo NII HPCI Secretariat （ organized in 2011 ） acct. registration helpdesk computer resource computer resource AICS (K-computer) Supercomputer Centers in 9 Universities

SINET4 SINET4: Science Information NETwork 4 Eisaku Sakane and Kento Aida, National Institute of Informatics

user IX （ Tokyo) resource provider IX （ Osaka) AICS LAN user compt. resource storage university commercial network non-commercial network CA portal university user compt. resource storage resource provider university user compt. resource storage user compt. resource storage QoS VPN SINET4 (cont’d) connection to 700+ academic sites IX for commercial networks  134 （ 30Gbps ） in Tokyo  22 （ 11Gbps ） in Osaka Eisaku Sakane and Kento Aida, National Institute of Informatics 80Gbps backbone （ planned in 2011 ） L3VPN, L2VPN/VPLS, QoS

AICS and Supercomputer Centers in Japanese Universities Kyushu Univ. ： PC Cluster (55Tflops, 18.8TB) SR16000 L2 (25.3Tflops, 5.5TB) PC Cluster (18.4Tflops, 3TB) Kyushu Univ. ： PC Cluster (55Tflops, 18.8TB) SR16000 L2 (25.3Tflops, 5.5TB) PC Cluster (18.4Tflops, 3TB) Hokkaido Univ. ： SR11000/K1(5.4Tflops, 5TB) PC Cluster (0.5Tflops, 0.64TB) Hokkaido Univ. ： SR11000/K1(5.4Tflops, 5TB) PC Cluster (0.5Tflops, 0.64TB) Nagoya Univ. ： FX1(30.72Tflops, 24TB) HX600(25.6Tflops, 10TB) M9000(3.84Tflops, 3TB) Osaka Univ. ： SX-9 (16Tflops, 10TB) SX-8R (5.3Tflops, 3.3TB) PCCluster (23.3Tflops, 2.9TB) Kyoto Univ. T2K Open Supercomputer (61.2 Tflops, 13 TB) Tohoku Univ. ： NEC SX-9(29.4Tflops, 18TB) NEC Express5800 (1.74Tflops, 3TB) Univ. of Tsukuba ： T2K Open Supercomputer 95.4Tflops, 20TB Univ. of Tokyo ： T2K Open Supercomputer (140 Tflops, 31.25TB) AICS, RIKEN ： K computer K computer (10 Pfflops, 4PB) Available in 2012 AICS, RIKEN ： K computer K computer (10 Pfflops, 4PB) Available in 2012 A 1 Pflops machine without accelerator will be installed by the end of 2011 Tokyo Institute of Technology ： Tsubame 2 (2.4 Pflops, 100TB) source: Y. Ishikawa, Univ. of Tokyo Eisaku Sakane and Kento Aida, National Institute of Informatics

Hokkaido University Tohoku University University of Tokyo University of Tsukuba Tokyo Institute of Technology Nagoya University Kyushu University Osaka UniversityKyoto University AICS, RIKEN 12 PB+ storage 10 PB+ storage HPCI WEST HUBHPCI EAST HUB Gfarm2 is used as the global shared file system Storage source: Y. Ishikawa, Univ. of Tokyo Eisaku Sakane and Kento Aida, National Institute of Informatics

Authentication The goal is enabling single sign-on computer resources and shared storage in HPCI. survey of existing software technologies and operation of grid infrastructures account management  centralized or distributed? Eisaku Sakane and Kento Aida, National Institute of Informatics user portal HPCI acct/password login to computers access to shared storage single sign-on % gsi-ssh host.univ.ac.jp (1)sign-on the portal with HPCI acct. (2) ssh login to computers without password

Shibboleth + GSI Shibboleth for account management of HPCI  HPCI account = account to sign-on HPCI  federation of HPCI accounts managed in distributed way using Shibboleth A user has a HPCI account in one supercomputer center. Grid Security Infrastructure (GSI) for single sign-on  de facto in grid communities  enabling single sign-on using PKI  creating proxy certificate and delegation  mapping “Distinguished Name (DN)” in a client certificate and a local account in supercomputer centers grid-mapfile Eisaku Sakane and Kento Aida, National Institute of Informatics "/C=JP/O=NII/OU=CGRD/CN=Kento Aida” aida

Pilot Operations 1 st phase: Apr – Dec 2011  objective: for operation organizations to get used to operate GSI and Shibboleth systems  National Institute of Informatics operating CA system and Portal building an experimental CA system including a certificate repository –UMS provided by Shibbolized NAREGI Middleware v1.1 building an authentication portal with a proxy certificate repository –portal provided by Shibbolized NAREGI M/W  Supercomputer centers building Shibboleth IdP setting up a GSI-enabled ssh server and client as SP Eisaku Sakane and Kento Aida, National Institute of Informatics

Architecture Eisaku Sakane and Kento Aida, National Institute of Informatics Certificate Management System CA System (Shib. SP) Portal (Shib. SP) Proxy Cert. Repository Shib. DS Shib. IdP web browser GSI-SSH client National Institute of Informatics Supercomputer Centers SINET 4 apply certificate sign-on HPCI login to compt. resources Account DB GSI-SSH Server Supercomputer Centers, AICS storage Cert. Repository

Screenshots Eisaku Sakane and Kento Aida, National Institute of Informatics

Result of 1 st phase We confirmed the followings  Sign-on the authentication portal with Shibboleth federation mechanism  getting a end-user certificate via the authentication portal  generation a proxy certificate and downloading it to end-user’s terminal computer  logging in 9 supercomputer centers by using GSI-enabled SSH The system works as single sign-on system. Documents for HPCI users and administrators were revised according to feedback from participating organizations Problem  port number (22/tcp) collision between SSH and GSI-enabled SSH  Administrators are reluctant to stop sshd or replace with gsi-sshd because of security policy of supercomputer center.  We will unify the port number for gsi-sshd with another port number. Eisaku Sakane and Kento Aida, National Institute of Informatics

Pilot Operations (cont’d) 2 nd phase: Jan 2012 –  objective: evaluation of the authentication system and feedback  building a production level CA system preparing dedicated machines, HSM performing key ceremony examinations on normal or abnormal operations replacing certificates in 1 st phase with new certificates issued by new CA  building an authentication portal for HPCI  collaboration with the HPCI secretariat the role of the HPCI secretariat –proposal to use HPCI (including registration of HPCI ID) –notification of review –coordination among resource providers, … HPCI-ID is important because it connects subject DN with local account. combination examination between NII (CA), supercomputer centers (RPs) and HPCI secretariat Eisaku Sakane and Kento Aida, National Institute of Informatics

Connecting Subject DN with LN Flow until subject DN and local account name (LN) are connected  A HPCI-ID is assigned to an end-user.  The HPCI secretariat notifies CA and RPs of the HPCI-ID.  CA manage subject DN with HPCI-ID.  RP manages local account name with HPCI-ID.  RP inquires the information of CA, then generates grid-mapfile. Eisaku Sakane and Kento Aida, National Institute of Informatics CA HPCI secretariat RP HPCI-ID aida (LN) "/C=JP/O=NII/OU=CGRD/CN=Kento Aida” aida /C=JP/O=NII/OU=CGRD/CN=Kento Aida

Conclusions This talk presents an evaluation experiment of the authentication system for HPCI. current status and future work  network SINET4 has started production level operation in  authentication entering on 2 nd phase of evaluation experiment built a production level CA system in NII and evaluated its performance starting test operation of the production level system from Feb 2012 considering when we switch hash algorithm in digital signature to SHA-2  user management still preparing to start HPCI secretariat starting test operation as soon as possible Eisaku Sakane and Kento Aida, National Institute of Informatics