1. 1 CMS user jobs submission with the usage of ASAP Natalia Ilina 16/04/2007, ITEP, Moscow

2. 2 What’s ASAP? ASAP: Arda Support for CMS Analysis Processing ASAP is a job creation and submission framework for CMS Analysis jobs:  command line tool to create and control jobs  optional server part which can assume responsibility for the users jobs resubmitting in case of failure

3. 3 What’s ASAP? 2 types of jobs can be submitted by ASAP:  standard user job - data are registered in CMS DBS/DLS system  private user job - any job (the limitation is that ASAP uses scramv1 environment)

4. 4 STEP I Starting to work with ASAP

5. 5  create the tracking directory which is opened for r/w for everybody:  setup enviroment:  create proxy: > mkdir user_path/ASAP > chmod a+w user_path/ASAP > chmod a+w user_path/ASAP > voms-proxy-init -voms cms -valid 192:0 > source /afs/cern.ch/sw/arda/install/CMS/asap3/setup.{sh,csh}

6. 6 Starting to work with ASAP  create CMSSW project area Usage of default library: Changed library:  > scramv1 project CMSSW CMSSW_1_2_0 > cd CMSSW_1_2_0/src > project CMSSW > cvs co -r CMSSW_1_2_0 JetMETCorrections/JetVertexAssociation > cd JetVertexAssociation Edit *.h, *.cc > scramv1 b > eval `scramv1 runtime -csh` > scramv1 project CMSSW CMSSW_1_2_0 > cd CMSSW_1_2_0/src > eval `scramv1 runtime -csh`

7. 7 STEP II Configure Job

8. 8 Configure Job: STANDARD JOB Requires 2 files: 1. CMSSW.cfg file 2. Configuration file for ASAP (task.conf)

9. 9 Example of task.conf # specify directory to store tasks jobdir = /afs/cern.ch/user/n/nilina/scratch0/ASAP_tracking # Store output at SE store_output = 1 # store output directly to castor (the dir should already exist and have # the correct permitions: rfchmod 777 /path/output) output_se = srm.cern.ch output_se_path = /castor/cern.ch/user/n/nilina/asap # bad_se=fnal.gov # specify grid to submit to (lcg, my) grid = lcg # or specify dataset path primary_dataset = mc-onsel-120_qqH135_2tau tier = FEVT processed_dataset = CMSSW_1_2_0-FEVT-1166726234

10. 10 Example of task.conf (continue) # pset file (CMSSW configuration file) pset_file = JetVertex.cfg # specify output sandbox, could be filename, # a comma sep list of filename or * output_files = jetvertex.root # total number of events you wish to process # (if unspecified assume all) events_required = 20 # events per job events_per_job = 5 # specify minimum time requirements for the job min_wall_clock_time = 100 min_cpu_time = 100

11. 11 Configure Job: PRIVATE JOB Requires 4 files:  CMSSW.cfg file  Configuration file for ASAP (task.conf)  dataset.conf - dataset file which tells ASAP which files to use for which jobs and with which parameters  template.sh – script that has to be run at working node

12. 12 template.sh #!/bin/sh lcg-cp --vo cms lfn:/my_lfn file:///`pwd`/my_lfn cmsRun --parameter-set `pwd`/test_analysis.cfg Instead of cmsRun one can use any other executable

13. 13 dataset.conf input_1.root 0 100 3.5 32 input_2.root 100 200 4.5 20 input_3.root 0 100 3.5 11 FORMAT: INPUT_FILE SKIP_EVENTS MAX_EVENTS ATTRIBUTE_1 … ATTRIBUTE_N Example N1: Example N2 (if the file is stored in a SE): Example N3 (no input data): srm://srm.cern.ch/castor/cern.ch/user/u/user/input.root 0 100 srm://srm.cern.ch/castor/cern.ch/user/u/user/input.root 100 100 - 0 100 345 9329 - 100 100 1345 93290 - 200 100 2345 13290 - 300 100 1111 345

14. 14 Changes in CMSSW.cfg file # Now it should be: source = PoolSource { untracked vstring fileNames = {${FILENAMES}} } service = RandomNumberGeneratorService { untracked uint32 sourceSeed = ATTRBUTE_1 PSet moduleSeeds = { untracked uint32 VtxSmeared = ATTRBUTE_2 }

15. 15 task.conf for private job # specify directory to store tasks jobdir = /afs/cern.ch/user/n/nilina/scratch0/ASAP_tracking # Store output at SE store_output = 1 # specify grid to submit to (lcg, my) grid = lcg # specify dataset file dataset_file = dataset.conf # pset file (CMSSW configuration file) pset_file = test_analysis.cfg

16. 16 task.conf for private job (continue) # template file (shell script to run at WN) application_wrapper = template.sh # specify output sandbox, could be filename, a comma sep list of filename or * output_files = *.root # specify minimum time requirements for the job min_wall_clock_time = 100 min_cpu_time = 100

17. 17 STEP III Job submition

18. 18 JOB SUBMITION  Create jobs: ASAP will package your executable for transfer to the Grid, check DBS/DLS if it is required, provide you with a TASK ID which you should use for all future operations You’ll see on the screen: > asap --create task.conf * starting asap3 (0_7_13), fasten your seatbelts * split 4 jobs with 20 events * using SCRAM environment * application: CMSSW_1_2_0 * use taskid 95312 for further tracking

19. 19 JOB SUBMITION  Match jobs: Checking if there are resources available to satisfy your jobs requirements  Submit manually: > asap --taskid TASKID --submit > asap --taskid TASKID --match

20. 20 JOB SUBMITION  Monitor status: You’ll see on the screen: > asap –taskid TASKID –update --list Job ASAP Status GRID Status GRID Reason ----------------------------------------------------------------------------------------------- 1 SUBMITTED Scheduled Job successfully submitted to Globus 2 SUBMITTED Scheduled Job successfully submitted to Globus 3 SUBMITTED Scheduled Job successfully submitted to Globus 4 SUBMITTED Scheduled Job successfully submitted to Globus -----------------------------------------------------------------------------------------------

21. 21 JOB SUBMITION  Fetch output from GRID: If you have not specified an output directory the output will be stored in jobdir/TASKID/output (jobdir was specified in your original config file) > asap --taskid TASKID --fetch

22. 22 Using monitor  Delegate your proxy:  Register Jobs with the Monitor: ASAP will submit and resubmit jobs to GRID upon completion the output. The address of a webpage will be displayed. Before looking at the task one need to insert the GRID certificate in the Browser. You can track the progress of your jobs by webpage. > > asap-user-register > > asap --taskid TASKID --register

23. 23 Using monitor  Monitor status with cmd line:  Unregister Jobs: Before retrieving the output one need to unregister job > > asap --taskid TASKID --unregister > > asap --taskid TASKID --list

24. 24 Other usefull commands  Kill jobs:  Fetch output:  Resubmit jobs: > asap --taskid TASKID --kill > > asap --taskid TASKID --fetch --output_dir OUTPUT_DIR > > asap --taskid TASKID –fetch-logs > > asap --taskid TASKID –fetch-data > asap --taskid TASKID --submit

25. 25 Usefull links  ASAP’s User Guide: http://lxarda13.cern.ch/docs/rn01.html  Contact: arda-support-cms@cern.charda-support-cms@cern.ch, arda-support-cms@cern.ch Craig.Munro@cern.chCraig.Munro@cern.ch, Craig.Munro@cern.ch Julia.Andreeva@cern.ch

26. 26 Practical part of tutorial  2 examples of standard job: ~nilina/scratch0/ASAP_TUTORIAL/ASAP/EXAMPLE1 ~nilina/scratch0/ASAP_TUTORIAL/ASAP/EXAMPLE2 ~nilina/scratch0/ASAP_TUTORIAL/ASAP/EXAMPLE2  1 example of private job: ~nilina/scratch0/ASAP_TUTORIAL/ASAP/EXAMPLE3