Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow.

Similar presentations


Presentation on theme: "1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow."— Presentation transcript:

1 1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow

2 1 st December 2003 JIM for CDF 2 Contents What is JIM? What is SAMGrid? How does JIM relate to SAMGrid? Components of JIM Using JIM Job Types Station Setup Deployment Plan

3 1 st December 2003 JIM for CDF 3 What is JIM? Job and Information Management Job Management Infrastructure is the framework allowing job submissions for execution on a cluster that has enough resources to complete the job. Information Management is knowing what resources are available and the status of the jobs.

4 1 st December 2003 JIM for CDF 4 What is SAMGrid? SAMGrid is a grid infrastructure whose goal is to allow globally distributed computing for current experiments, DØ, CDF and Minos at Fermilab. Communicating with LHC experiments CMS (Fermilab) and Atlas (Brookhaven)

5 1 st December 2003 JIM for CDF 5 How does JIM relate to SAMGrid? Complements Sequencial Access via Metadata to provide complete grid services Job Management Information and Monitoring Data Handling

6 1 st December 2003 JIM for CDF 6 How does JIM relate to SAMGrid?

7 1 st December 2003 JIM for CDF 7 How does JIM relate to SAMGrid? JIM allows a user to submit jobs to SAMGrid and to access the output files on completion. JIM chooses which resources will be used to execute a job. This decision is based on the amount of data required by the job already cached at the sites. JIM submits the job to the local BS and SAM provides the I/O data management for the files.

8 1 st December 2003 JIM for CDF 8 How does JIM relate to SAMGrid? Condor MMS was expanded for SAMGrid –broker can query SAM station to see how much data is already present. –dynamic selection of a globus gatekeeper. –the match is determined by calling by calling external custom code, e.g. in SAMGrid the SAM station is invoked to determine the rank of a match. Globus toolkit is used for job transfer and monitoring

9 1 st December 2003 JIM for CDF 9 How does JIM relate to SAMGrid? Distinguishes grid level (global) scheduling (selection of a cluster to run) from local scheduling (distribution of the job within the cluster). Distinguishes structured jobs (where the details are known to Grid middleware) from unstructured jobs (where the whole job is mapped onto a single cluster).

10 1 st December 2003 JIM for CDF 10 Components of JIM All Sites –sam_gsi_config (includes sam_gridftp and globus_dh_client and server) for grid security –xmldb as a SAMGrid interface Client Site - used to submit jobs to SAMGrid –typically a remote server or workstation that is used to send jobs to a Submission site –jim-client

11 1 st December 2003 JIM for CDF 11 Components of JIM Submission Site - maintains a spool of jobs –acts as a client to the broker periodically sending jobs to available resources at an Execution site –jim_broker_client –www_jim_sandbox (optional)

12 1 st December 2003 JIM for CDF 12 Components of JIM Execution Site - runs the job –sam and sam_station –sam_batch_adapter –globus_rm_server –jim_jobmanager –jim_sandbox –jim_config and jim_advertise

13 1 st December 2003 JIM for CDF 13 Components of JIM Monitoring Site - provides information on the state of each submitted job and allows the output of completed jobs to be downloaded –globus_is_server and globus_is_client –jim_info_providers

14 1 st December 2003 JIM for CDF 14 Using JIM – Submitting a job User creates a jdl file such as the example shown: sam_dataset = jpmm08-1file executable = retrieve.sh input_sandbox = /home_scotgrid/m/mlyon/test/testjob cpu-per-event = 1s job_manager = sam job_type = sam_analysis sam_universe = prd sam_experiment = cdf log = testjob.log output = testjob.out error = testjob.err arguments = Download Output Yet? group = test instances = 1

15 1 st December 2003 JIM for CDF 15 Using JIM – Submitting a job The job is submitted by typing: samg submit testjobfile.jdf The progress of the job can be viewed by selecting the submission site from the list: http://samgrid.fnal.gov:8080 Select the job from the list. Details of the job state are displayed Once the job has been completed the output may be downloaded

16 1 st December 2003 JIM for CDF 16 Using JIM – Viewing Map

17 1 st December 2003 JIM for CDF 17 Using JIM – Viewing submission sites

18 1 st December 2003 JIM for CDF 18 Using JIM – Viewing submitted jobs

19 1 st December 2003 JIM for CDF 19 Using JIM – Downloading output

20 1 st December 2003 JIM for CDF 20 Types of Jobs Monte Carlo –events generated, passed through detector simulation and reconstructed –e.g. typically no input files, one output file per job –Each job part of a well defined (generator, parameters) set going to a given dataset Reconstruction –real data reconstruction –in general one input file from a dataset going to one output file in corresponding dataset Analysis –an entire dataset is input, many input files

21 1 st December 2003 JIM for CDF 21 Station Setup Initial installation uses current versions of all products. Care must be taken when upgrading –to avoid version incompatibility –to preserve existing configuration

22 1 st December 2003 JIM for CDF 22 What will JIM do once complete? Current functionality allows job submission and output retrieval Next steps: –Deployment of secure web-download –User support and defect fixing through testing phase –Roll out of SAMGrid to all CDF sites with available resources –Add more brokering criteria Directors review of Run II computing has recommended the expansion of SAM to be a lab- wide product

23 1 st December 2003 JIM for CDF 23 Deployment Plan Glasgow University has a complete installation of SAMGrid on both ScotGrid and the CDF cluster. This installation is being tested with Monte Carlo simulation. Oxford University has SAMGrid installed. Installation workshop organised for 20 th - 22 nd Jan 04

24 1 st December 2003 JIM for CDF 24 Credits Thanks to the JIM team for providing material for this presentation.


Download ppt "1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow."

Similar presentations


Ads by Google