11 Application of CSF4 in Avian Flu Grid: Meta-scheduler CSF4. Lab of Grid Computing and Network Security Jilin University, Changchun, China Hongliang Li (Simon) PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

22 CSF4-A cross-domain meta-scheduler Former application of OPAL-CSF model in Avian Flu Grid (AFG) Managing multiple public CloudsElastic virtual clusterConclusion and future plans PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

33 CSF4-A cross-domain meta-scheduler PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

44 CSF4 introduction Cross-domain meta-scheduler (grid-enable) Grid protocol (portable) –WS-GRAM, pre-WS-GRAM –Organizing resources from different domain under control of diverse local schedulers Scheduling plugin framework (extendable) –Default plugin –Arrayjob plugin –Workflow plugin –DataAware plugin –OPAL service plugin –Parallel job plugin PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

55 CSF4 modules Job Service, Queue Service, Resource Managers Supporting diverse local schedulers by grid protocols PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

66 Scheduling framework Support multiple scheduling plugins co- operate together PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

77 Default and Arrayjob plugins Arrayjob consists of multiple subjobs(SIMD) PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

88 Two plugins working together Workflow jobs are spitted to subjobs by Workflow plugin DataAware plugin allocate resources for these subjobs PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

99 Former application of OPAL-CSF model in Avian Flu Grid (AFG) PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

10 Integration of CSF4 and OPAL OPAL-CSF4 biomedical cloud –Enable large scientific applications (Virtual screening, Autodoc, 2000 Arrayjobs) –OPAL deals with service management and user interfaces –CSF4 deals with cross-domain job scheduling PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

11 OPAL-CSF4 cloud model CSF4 as a job manager of OPAL PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

12 System structure PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, Application management Cross-domain scheduling Input/Output file transfer

13 CSF4 stagein&stageout PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

14 CSF4 stagein&stageout PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

15 Improvements of CSF4 Cross-domain dynamic file transfer Recursively transmit files and folders for each job (subjob) Job re-submission Max walltime Default values in configuration file User defined with RSL files 2000 array jobs stable PRAGMA Grid testbed Latest CSF4 release(Version and 4.0.6). PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

16 Managing multiple public Clouds PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

OPAL as resource manager of CSF4 CSF4 allocate service instances of OPAL for jobs 17 New OPAL-CSF4 Cloud model PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

18 New OPAL-CSF4 Cloud model OPAL as virtual resource manager in CSF4 –Job submission, job monitoring CSF4 managing multiple OPAL sites –Site status (CPU, service) updates (modifying in OPAL) CSF4 allocate service resource of multiple sites –New interface of Job submission (URL to entire directory, URL to list file of directories) (modifying in OPAL) Scheduling OPAL service jobs and maintaining lifecycle of jobs –New scheduling plugin (OPAL Service plugin) –Monitoring job status using status files PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

19 New Resource manager Extend a new resource manager: –Resource Manger Opal Service PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

20 Scheduling plugin A: Select Opal sites according to service requirement; B: Sort opal resources according to CPU numbers; C: Spread arrayjobs to different sites PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

21 Communication mechanism Using SOAP protocol to cooperate with OPAL (URLs) Monitoring job status using status files PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

22 Configuration and Experiments PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, vm2-opal OPAL vm2.jlu.edu.cn /home vm4-opal OPAL vm4.jlu.edu.cn /home

23 Elastic virtual cluster PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

24 EVC model Customized, isolated and secure executing environment for parallel applications. PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, Resource manager Virtual Infrastructure VMM

25 Support EVC in CSF4 Objectives –Parallel job co-allocation –dynamic executing environment deployment (VJM) Extend VJM module to manage EVC (EVC manager) –Resource reservation using Vjobs, Vjobs manage virtual machines, EVC manages virtual clusters –Creating, reconstructing and rearranging virtual clusters New scheduling plugins: parallel job plugin –Parse VC requirements of jobs; prepare VCs dynamically in runtime; distribute parallel jobs to VC Others –Integrate VJM as a separate service in CSF4 –VC status monitoring using VJM –Real job monitoring PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

26 Parallel job scheduling in CSF4 Two phase resource allocation in parallel job plugin –Construct virtual clusters according to job requirements –Distribute real jobs to virtual clusters PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

27 Module design of EVC manager Interfaces and internal modules Organize VCs in a pool VM configuration (IP, image) VC configuration (subnet, cluster software, …) Support multiple VMMS (Xen, VMwareServer, etc.) PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

Two-phase scheduling are all based on GSI. –Resource co-allocation –Real job distribution 28 Process of parallel job scheduling PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

29 Image management Image configuration file (XML) Support image compression to save transmission time Support dedicated applications by dynamic installation (yum…) PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

30 Conclusion and future plans PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

31 Conclusion CSF4 have been evolved from traditional grid enabled to cloud support. –Powerful, usable, extendable New OPAL-CSF model –Sharing service resources by multiple OPAL sites. Elastic virtual cluster –Parallel job co-allocation –Dynamic executing environment pre-deployment PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

32 Ongoing works and plans Virtual cluster live migration strategy –Concurrent migration protocol Multi-domain service scheduling policies –Monitoring service utilization rate –Scheduling policies Elastic virtual cluster management strategies –Reconstruction –Virtual cluster pool –Multi-VO users PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.

33 PRAGMA 19 workshop, Changchun, Jilin, China, Sep.13-15, 2010.