1. DIRAC Review (12 th December 2005)Stuart K. Paterson1 DIRAC Review Workload Management System

2. DIRAC Review (12 th December 2005)Stuart K. Paterson2 Contents Introduction & Overview The Life Cycle of a DIRAC Job Central Services & Interactions Distributed Workload Management WMS Strategies Outlook & Improvements

3. DIRAC Review (12 th December 2005)Stuart K. Paterson3 Introduction The WMS is a key component of DIRAC Realizes PULL Mechanism Community Overlaying Grid System COGS Paradigm WMS Services:- Rely on MySQL Job Database XML-RPC Protocol used for client service access Jabber is used for communication between services Condor Classad for Job JDL

4. DIRAC Review (12 th December 2005)Stuart K. Paterson4 The Life of a DIRAC Job Consider a typical DaVinci Analysis job since other use cases involve a subset of the steps for this Job is submitted to WMS via DIRAC API See tomorrow’s presentation User DIRAC submit() status() getOutput() DIRAC API GANGA

5. DIRAC Review (12 th December 2005)Stuart K. Paterson5 WMS Overview Job Receiver LFC Matcher Data Optimiser Job DB Task Queue Agent Director Pilot Agent LCG WMS Computing Resource Pilot Agent Monitor

6. DIRAC Review (12 th December 2005)Stuart K. Paterson6 Secure Job Receiver Currently the only secure service Assigns Job ID Saves job in JobDB Also uploads and saves proxy of user Notifies Optimiser Data or FIFO depending on requirements of job

7. DIRAC Review (12 th December 2005)Stuart K. Paterson7 Input/Output Sandbox Services At present use MySQL DB for storing I/O sandbox Very fast and efficient No problems observed for ‘small’ files (<10Mb) Limit on DB size of 4Gb Proposal is to move to Grid storage for this but: This will be slower Extra dependency on LFC Will be implemented and tested Final decision will be made based on performance

8. DIRAC Review (12 th December 2005)Stuart K. Paterson8 Input/Output Sandbox Mechanism

9. DIRAC Review (12 th December 2005)Stuart K. Paterson9 Job Database MySQL DB but this is not accessed directly Accessed through the JobDB class Contains full information about all the jobs Job description and status Primary job parameters Common for all jobs e.g. Owner, access optimized Extra Job Parameters Arbitrary key/value pairs

10. DIRAC Review (12 th December 2005)Stuart K. Paterson10 JobDB Interface Marks job status as ‘ready’ when added Not only a thin layer on top of SQL statements Performs high-level operations Adding jobs Removing jobs Provides bulk queries (e.g. for job monitoring) Scalability issues Test system up to 15000 (production and analysis) jobs without automatic cleaning, no problems so far…

11. DIRAC Review (12 th December 2005)Stuart K. Paterson11 WMS Overview Job Receiver LFC Matcher Job DB Task Queue Agent Director Pilot Agent LCG WMS Computing Resource Pilot Agent Monitor Data Optimiser Job DB Job Receiver

12. DIRAC Review (12 th December 2005)Stuart K. Paterson12 Data Optimizer Instantiates File Catalog Client (LFC) Retrieves requirements of job from JobDB Uses Condor Classad and MySQL Sets job status to ‘waitingdata’ Checks LFC for input data files and determines suitable SEs Job fails meaningfully if the data is not available Read-only LFC could speed up this process Sets job status to ‘waiting’ ‘PilotAgent Submission’ if data is available Inserts job into Task Queue

13. DIRAC Review (12 th December 2005)Stuart K. Paterson13 Data Optimizer - Improvements Currently uses proxy from the process owner certificate to access LFC Could use user proxy or Server certificate Read-only LFC would solve this issue Optimizer currently checks all input files for each job Can move to directories of datasets in the future Need to optimize LFC interaction for bulk queries In cooperation with LFC developers

14. DIRAC Review (12 th December 2005)Stuart K. Paterson14 Task Queue Deliberately have many task queues 1 Task Queue per set of requirements Drastically reduces the matching time Works ok for production jobs For analysis jobs with many varied requirements this remains to be seen ‘Double matching’ helps with this – see later Too many queues can cause problems Hierarchical organisation of queues with respect to requirements should improve matching

15. DIRAC Review (12 th December 2005)Stuart K. Paterson15 WMS Overview Job Receiver LFC Matcher Job DB Task Queue Agent Director Pilot Agent LCG WMS Computing Resource Pilot Agent Monitor Data Optimiser Job DB Job Receiver Data Optimiser Task Queue LFC

16. DIRAC Review (12 th December 2005)Stuart K. Paterson16 Agent Director (1) Agent Director is an API for Pilot Agent submission to LCG Sets up the proxy of the user for submission to LCG Monitors jobs in ‘waiting’ ‘PilotAgent Submission’ status Polls frequently After submission it enters ‘waiting’ ‘PilotAgent Response’ Since job remains in the ‘waiting’ state, can be picked up at any time by existing Agents from the same user

17. DIRAC Review (12 th December 2005)Stuart K. Paterson17 Agent Director (2) Currently only used for ‘user’ jobs Move to AD for production Can submit agents for each job in the Task Queue Currently Pilot Agents are submitted by cron job independently of the Task Queue state Can also potentially have agents working in filling mode (some infinite queues) Could be used for submission to other Grids…

18. DIRAC Review (12 th December 2005)Stuart K. Paterson18 Agent Monitoring Service Checks Pilot Agents of jobs in ‘waiting’ state Currently every 5 mins Monitoring of Pilot Agents on LCG allows: Catch jobs spending too much time in the ‘Waiting’ state Quickly spot the ‘Aborted’ status problem Aborted agents are tracked and can be accounted Can also spot the problem of Pilot Agents being submitted to batch queues Assigns ‘waiting’ ‘Proxy Expired’ state Can flag jobs for Agent Director to submit further Pilot Agents as necessary

19. DIRAC Review (12 th December 2005)Stuart K. Paterson19 Future Developments of AD and AM Services Ensure PilotAgents are submitted to different LCG sites if the job requirements permit it Not immediately obvious how to accomplish this Would need direct submission to LCG CE Make use of MyProxy Server Automatically renew user proxies before submission when necessary Pipe long life proxy with jobs? Run proxy monitor as a service which can deliver renewed user proxy?

20. DIRAC Review (12 th December 2005)Stuart K. Paterson20 WMS Overview Job Receiver LFC Matcher Job DB Task Queue Agent Director Pilot Agent LCG WMS Computing Resource Pilot Agent Monitor Data Optimiser Job DB Job Receiver Data Optimiser Task Queue LFC Agent Monitor Agent Director

21. DIRAC Review (12 th December 2005)Stuart K. Paterson21 Job State Machine up to this point Jobs may be picked up as soon as they enter the ‘Waiting’ state

22. DIRAC Review (12 th December 2005)Stuart K. Paterson22 Pilot Agent running on LCG WN Simple wrapper script sent as LCG job Installs DIRAC Runs a standard DIRAC Agent which polls for the particular Job from a particular User If not successful, requests any job from particular user which is satisfied by the requirements of the site it runs on ‘Filling’ Mode – see later DIRAC Agent starts JobAgent Module which performs the job requests

23. DIRAC Review (12 th December 2005)Stuart K. Paterson23 Matcher (1) Receives request from Pilot Agent Only responds to sites in ‘mask’ Contains list of allowed sites Checks available jobs in task queue Matches requirements of job (e.g. possible SEs) to requirements from Agent (e.g. owner, JobID at site with particular LocalSE) Double match, agent can put specific requirements on jobs (e.g. job of particular owner or certain priority level)

24. DIRAC Review (12 th December 2005)Stuart K. Paterson24 Matcher (2) Matcher has ‘semaphore’ mechanism to ensure job is only picked up once Assigns ‘matched’ state, sets the site for the job, logs this info and deletes job from task queue Sends job to WN Doesn’t need to be secure However, must ensure jobs are not picked up in error

25. DIRAC Review (12 th December 2005)Stuart K. Paterson25 WMS Workflow  Job State Machine

26. DIRAC Review (12 th December 2005)Stuart K. Paterson26 Job Agent Module Running on WN Job Agent Requests Job from WMS, gets JDL if successful Installs any software not available locally Links to any pre-installed software are created local to job during installation of DIRAC (dirac-install) Creates Job Wrapper using information local to the WN Template + job specific parameters, e.g. job JDL + + site specific parameters, e.g. software paths Job Agent executes Job Wrapper Local InProcess CE

27. DIRAC Review (12 th December 2005)Stuart K. Paterson27 Job Agent Job Preparation Installing Application Software Create Job Wrapper

28. DIRAC Review (12 th December 2005)Stuart K. Paterson28 WMS Overview Job Receiver LFC Matcher Job DB Task Queue Agent Director Pilot Agent LCG WMS Computing Resource Pilot Agent Monitor Data Optimiser Job DB Job Receiver Data Optimiser Task Queue LFC Agent Monitor Agent Director Matcher Pilot Agent LCG WMS

29. DIRAC Review (12 th December 2005)Stuart K. Paterson29 WMS Overview Pilot Agent Computing Resource

30. DIRAC Review (12 th December 2005)Stuart K. Paterson30 WMS Overview Computing Resource

31. DIRAC Review (12 th December 2005)Stuart K. Paterson31 Job Wrapper (1) Downloads input sandbox Currently InputSandbox is a DIRAC WMS specific service Can also use generic LFNs Provides access to the input data Resolves the input data LFN into a “best replica” PFN for the execution site Generates an appropriate Pool XML slice for protocol

32. DIRAC Review (12 th December 2005)Stuart K. Paterson32 Input Data Access Strategy Attempt to stage input data (multi-threaded, via lcg-gt) Try rfio and dcap Returns TURL for protocol (when it works) The returned TURLs currently don’t work inside the applications (also can’t ‘pin’ files yet…) Currently use globally constructed TURL from DIRAC Storage Element class Works fine with Gaudi applications If this isn’t available, bring datasets local

33. DIRAC Review (12 th December 2005)Stuart K. Paterson33 Job Wrapper (2) Invokes the job application in a child process Runs a watchdog process parallel to the application one: Provides heart-beats for the Job Monitoring Service Collects the application CPU and memory consumption to generate average numbers in the end May catch the application in a ‘stalled’ state if no CPU consumption detected

34. DIRAC Review (12 th December 2005)Stuart K. Paterson34 Job Wrapper (3) May receive messages through a messaging system Jabber messaging was demonstrated Possibility to kill the application gracefully or spy on the application output Not used currently as security issues should be sorted out first if at all possible Collecting job execution environment and consumption parameters, passing them to the Job Monitoring Service Local job ID’s, worker node characteristics and load, total CPU consumption, job timing, etc

35. DIRAC Review (12 th December 2005)Stuart K. Paterson35 Job Wrapper (4) Uploading output sandbox Currently OuputSandbox is a DIRAC WMS specific service Might be moved to a generic SE implementation soon Uploading output data Uploads output data to a predefined SE Default one or user defined Chooses PFN path according to the LHCb conventions May be overridden by user – not recommended Notifies the Job Monitoring Service of the changes in the job state

36. DIRAC Review (12 th December 2005)Stuart K. Paterson36 Job Wrapper Workflow

37. DIRAC Review (12 th December 2005)Stuart K. Paterson37 Overview of State Machine After Job Reaches WN (1)

38. DIRAC Review (12 th December 2005)Stuart K. Paterson38 Overview of State Machine After Job Reaches WN (2)

39. DIRAC Review (12 th December 2005)Stuart K. Paterson39 Overview of State Machine After Job Reaches WN (3)

40. DIRAC Review (12 th December 2005)Stuart K. Paterson40 Overview of Status Machine for Failed Jobs

41. DIRAC Review (12 th December 2005)Stuart K. Paterson41 DIRAC Job in Final State Once DIRAC Agent(s) executed, Pilot Agent terminates gracefully, freeing the resource If successful, job is in ‘outputready’ state which means it is retrievable User can request output at any time

42. DIRAC Review (12 th December 2005)Stuart K. Paterson42 Job Logging Logging is a mixture of primary (Job State) and secondary (App State) job states JOB STATE DATE TIME SITE submission 2005-12-11 13:40:21 LCG.CNAF.it ready 2005-12-11 13:40:23 LCG.CNAF.it waitingdata 2005-12-11 13:40:24 LCG.CNAF.it waiting 2005-12-11 13:40:26 LCG.CNAF.it matched 2005-12-11 13:53:09 LCG.CNAF.it Job received by Agent 2005-12-11 13:53:09 LCG.CNAF.it Installing Software 2005-12-11 13:53:09 LCG.CNAF.it Job prepared to submit 2005-12-11 13:55:01 LCG.CNAF.it scheduled 2005-12-11 13:55:01 LCG.CNAF.it queued 2005-12-11 13:55:01 LCG.CNAF.it running 2005-12-11 13:55:03 LCG.CNAF.it Starting DIRAC job 2005-12-11 13:55:03 LCG.CNAF.it DIRAC job initialization 2005-12-11 13:55:04 LCG.CNAF.it Getting Input Data 2005-12-11 13:55:04 LCG.CNAF.it Starting the application 2005-12-11 13:55:27 LCG.CNAF.it DaVinci step 1 started 2005-12-11 13:55:30 LCG.CNAF.it DaVinci execution, step 1 2005-12-11 13:55:38 LCG.CNAF.it DaVinci, step 1 done 2005-12-11 13:56:54 LCG.CNAF.it Job finalization 2005-12-11 13:56:54 LCG.CNAF.it Job finished successfully 2005-12-11 13:56:55 LCG.CNAF.it done 2005-12-11 13:56:55 LCG.CNAF.it outputready 2005-12-11 13:56:56 LCG.CNAF.it

43. DIRAC Review (12 th December 2005)Stuart K. Paterson43 Job Monitoring Service At all stages the Job Monitoring Service is used as an interface to update status information Changes status in Job DB directly Also updates the Job Logging information Two entry points, one for writing one for reading Move writing to secure service in the future Optimized for bulk queries One of the most solicited services Could maintain separate cache of state information to reduce future loads if necessary in the future

44. DIRAC Review (12 th December 2005)Stuart K. Paterson44 Cleaning Up Cleaning Agent is used for Production jobs Accounting Agent monitors jobs in final state Extracts accounting info and marks job as ‘deleted’ Cleaning Agent deletes job on next loop For analysis/user jobs, could mark as ‘Accounting Sent’ then mark as ‘Purgeable’ To be decided New Cleaning Agent can implement policy If output retrieved hold job for ~1 day more ?? If output not yet retrieved hold job for ~ 1 week ??

45. DIRAC Review (12 th December 2005)Stuart K. Paterson45 Overall Job State Machine

46. DIRAC Review (12 th December 2005)Stuart K. Paterson46 Current WMS Pilot Agent Strategy Now submit up to 4 Pilot Agents per job (unless all are being ‘Aborted’) These are submitted one at a time with a waiting period of 5 minutes between Agent ‘Filling’ mode When a Pilot Agent arrives at a WN, it first requests a particular job from the user. Next, the Pilot Agent will request any job from the same user If the requirements of the job match the site this is successful This should be optimized to make the most of the available resource – e.g. check time left on WN

47. DIRAC Review (12 th December 2005)Stuart K. Paterson47 Possible WMS Strategies (1) Simplest strategy is no strategy, 1Pilot Agent per job ‘Filling’ mode Multi-Threaded Agent infrastructure in place Can run jobs in parallel, can be a huge improvement Especially when mixing jobs of different priority and nature Reading data, downloading etc. can be complementary activities

48. DIRAC Review (12 th December 2005)Stuart K. Paterson48 Possible WMS Strategies (2) Picking up jobs with higher priority, can be achieved through ‘double matching’ mechanism Running jobs of different members of same VO by same Pilot Agent Very promising mechanism to optimize the workload for the LHCb VO as a whole

49. DIRAC Review (12 th December 2005)Stuart K. Paterson49 Outlook & Improvements (1) Input / Output Sandbox move to Grid storage Job ‘failures’ on LCG will be recovered as much as possible e.g. treatment of Stalled jobs Ensure PilotAgents are submitted to different LCG sites if the job requirements permit it To cope with troublesome sites

50. DIRAC Review (12 th December 2005)Stuart K. Paterson50 Outlook & Improvements (2) Extended life of user proxies using MyProxy Server or … Explore use of Multi-Threaded Agent on the Grid Provide at least minimal interactivity with running job Job killing/spying Spotting stalled Applications