1. National Institute of Advanced Industrial Science and Technology Ninf-G - Core GridRPC Infrastructure Software Forum @ OGF19 Yoshio Tanaka (AIST) On behalf of the Ninf Project

2. Goal of this session Attendees will understand GridRPC programming model how to develop and run Grid applications using Ninf-G how it is easy to develop and run Ninf-G applications how Ninf-G is suitable for developing flexible, robust, and efficient Grid applications flexible: allow dynamic resource allocation/migration robust: detect various faults and recover from them efficient: utilize hundreds to thousands of CPUs efficiently. Interact with users Requirements? Bug reports? Questions?

3. Outline Introduction of GridRPC GridRPC: Grid-enabled Remote Procedure Call Introducition of Ninf-G What is Ninf-G? Ninf Project How to use Ninf-G Programming and execution of Grid-enabled applicationsExamples TDDFT long-run on PRAGMA Grid (by Yusuke) Large-scale hybrid QM/MD simulation on Japan-US Grid testbed Future direction

4. National Institute of Advanced Industrial Science and Technology Introduction of GridRPC

5. GridRPC: A programming model based on RPC What is GridRPC? Utilize remote procedure call (RPC) on the Grid Based on client-server model The GridRPC API is published as a proposed recommendation (GFD-R.P 52) at the GGF Usage scenarios Remote library call Executing compute-intensive tasks on a remote high performance computing resource Task parallel processing Executing large numbers of independent tasks on distributed computing resources Remote Library Call Task parallel processing

6. Server Client Compuer GridRPC Model Client Component Caller of GridRPC Manages remote executables via function handles Remote Executables Callee of GridRPC. Dynamically generated on remote servers. Information Manager Manages and provides interface information for remote executables. Func. Handle Client Component Info. Manager Remote Executables

7. Typical scenario 1: desktop supercomputing Utilize remote supercomputers from your desktop computer Reduce cost for maintenance of libraries ASP-like approach Numerical Libraries Applications client server arguments results

8. Typical scenario 2: parameter surevey Compute independent tasks on distributed resources eg. combinatorial optimization problem solvers Fault nodes can be discarded/retried Dynamic allocation / release of resources is possible Client Servers

9. Typical scenario 3: GridRPC + MPI Coarse-grained independent parallel (MPI) programs are executed on distributed clusters Combine coarse-grained parallelism (by GridRPC) and fine-grained parallelism (by MPI) Dynamic migration of MPI jobs is possible Practical approach for large-scale computation Client Servers

10. GridRPC v.s. MPI GridRPC GridRPC task parallel client/server GridRPC API dispensablegoodavailable can be dynamic easy to gridify existing apps. parallelismmodelAPIco-allocation fault tolerance private IP nodes resourcesothers MPI MPI data parallel SPMDMPIindispensable poor (fatal) unavailable static static well known seamlessly move to Grid * May be dynamic using process spawning

11. Sample Program Parameter Survey No. of surveys: n Survey function: survey(in1, in2, result) Input Parameters: double in1, int in2 Output Value: double result[] Main Program Survey Function Int main(int argc, char** argv) { int i, n, in2; double in1, result[100][100]; Pre_processing(); For(I = 0; I < n, i++){ survey(in1, in2, resul+100*n) } Post_processing(); survey(double in1, int in2, double* result) { Do Survey }

12. Build remote library (server-side operation) Survey Function survey (double in1, int in2, double* result) { Do Survey } Original Program Callee Function survey (double in1, int in2, int size, double* result) Do Survey } Specify size of argument IDL File Module Survey_prog; Define survey (IN double in1, IN int in2, IN int size, OUT double* result); Required survey.o Calls C survey(in1, in2, size, result); Client Program Callee Function IDL File IDL Compiler Stub Program Interface Info. LDIF File

13. Ninfy the original code (client-side) Int main(int argc, char** argv) int i, n, in2; double in1, result[100][100]; grpc_function_handle_t handle [100]; Pre_processing(); grpc_initialize(); for(I = 0; I < n; i++) { handle[i] = grpc_function_handle_init(); } For(I = 0; I < n, i++){ grpc_call_async (handles, in1,in2,100, result+100*n) } grpc_wait_all(); for(I = 0; i<n; i++){ grpc_function_handle_destruct(); } grpc_finalize(); Post_processing(); Int main(int argc, char** argv) { int i, n, in2; double in1, result[100][100]; Pre_processing(); For(I = 0; I < n, i++){ survey(in1, in2, resul+100*n) } Post_processing(); Declare func. handles Init func. handles Async. RPC Retrieve results Destruct handles

14. National Institute of Advanced Industrial Science and Technology Ninf-G Overview and Architecture

15. Ninf Project Started in 1994 Collaborators from various organizations AIST Satoshi Sekiguchi Hidemoto Nakada Yoshio Tanaka Atsuko Takefusa Yusuke Tanimura Hiroshi Takemiya University of Tsukuba Mitsuhisa Sato Taisuke Boku Osamu Tatebe Tokyo Institute of Technology Satoshi Matsuoka Kento Aida Tokyo Denki University Katsuki Fujisawa

16. Brief History of Ninf/Ninf-G 1994199720002003 Ninf project launched Release Ninf version 1 Start collaboration with NetSolve team Ninf-G development Release Ninf-G version 0.9 Release Ninf-G version 1.0 Standard GridRPC API proposed Release Ninf-G Version 1.1 GridRPC standared specification

17. Brief History of Ninf/Ninf-G (cont d) 2003200420052006 Draft GridRPC API 1 st submission to the GGF GridRPC API publishedas proposed recommendation GFD-R.P 52 Submitted experimental document 2 nd draft submission Ninf-G 2.0.0 Ninf-G 2.1.0 Ninf-G 2.2.0 Ninf-G 2.3.0 Ninf-G 2.4.0 Ninf-G in NMI Ninf-G 4.0.0 Ninf-G 4.1.0 Ninf-G 4.2.0

18. Ninf-G: A Reference Implementation of the GridRPC API A software package which implements the GridRPC API. Ninf-G includes C/C++, Java APIs, libraries for software development IDL compiler for stub generation Shell scripts to compile client program build and publish remote libraries sample programs and manual documents Latest version is 4.2.1

19. Features of Ninf-G Version 4 Enables RPC via WS GRAM Pre-WS GRAM Condor Unicore SSH NAREGI SS Has an interface for working with the other Grid middleware for RPC. E.g. ARC GridFTP Server (Nordu Grid)

20. Architecture of Ninf-G4 Client GRAM / UNICORE / Condor Invoke Executable Connect back IDL file Numerical Library IDL Compiler Ninf-G Executable Generate Interface Request Interface Reply Server side Client side MDS Interface Information LDIF File retrieve Globus-IO Invoke Server