CMS Grid Batch Analysis Framework HughTallini David Colling Barry Macevoy Stuart Wakefield

Contents Project objectives Requirements analysis Design outline Implementation Future plans

Objective Build an analysis framework to enable CMS physicists to perform end user analysis of DC04 data in batch mode on the Grid. i.e. run a private analysis algorithm over a set of reconstructed data i.e. not interactive – submit once and leave to run. Deliver by October ‘03

Approach Build tactical solution using (as far as possible) tools that already exist. Iterative development – deliver first prototype quickly covering major use cases, iterating design and implementation later to refine and extend the solution. Build scalable architecture that will inevitably have to evolve as requirements change (or get better determined). Use good software design practices (i.e. distinct analysis-design-implementation phases).

DC04 Calibration challenge Fake DAQ (CERN) Higgs DST Event streams Calibration sample Tier-0 challenge Data distribution Calibration challenge Analysis challenge Calibration Jobs MASTER Conditions DB TAG/AOD (replica) (10-100 kB/evt) Replica Conditions DB DC04 Analysis challenge 1st pass Recon- struction 25Hz 2MB/evt 50MByte/s 4 Tbyte/day DC04 T0 challenge HLT Filter ? CERN disk pool ~40 TByte (~10 days data) Archive storage CERN Tape archive Disk cache 25Hz 1MB/evt raw 0.5MB reco DST 50M events 75 Tbyte 1TByte/day 2 months PCP CERN Tape archive Higgs background Study (requests New events) Event server SUSY Background DST

What’s involved… Data handling Job splitting Job submission Job monitoring Job information archiving

Requirements Analysis (1) Typical Use Case 1 Private Single Step Analysis User wishes to run their analysis algorithm over 10Million MC events from a particular dataset. S/he compiles code and tests over a small sample of data on a local machine. Configures a JDL listing data sample to run over, ORCA libraries and ORCA executable to use, plus any steering files required. Submits to the framework. Uses AF to monitor jobs as they run. Uses AF to locate output data stored on Grid. Uses AF to keep record of analysis details, details of which are stored locally (privately).

Requirements Analysis (2) Typical Use Case 2 Private Multi Step Analysis As Use Case 1 but user wishes input data for analysis task to be the output data from a previous analysis. Typical Use Case 3 Group Multi Step Analysis Group of physicists wish to share input and output data Must also share details of how output data is created.

Requirements Analysis (3) Some other important requirements User’s analysis code should be identical whether running locally or on the Grid. No constraint on size of data sample to run over. Interface to Analysis Framework must be simple to use (users are physicists, not software developers) Single configuration file Single step submission

Batch Object Submission System Wrapper BOSS Local Scheduler farm node boss submit boss query boss kill BOSS DB farm node Accepts job submission from users Stores info about job in a DB Builds a wrapper around the job (jobExecutor) Sends the wrapper to the local scheduler The wrapper sends to the DB info about the job

COMMON BOSS/AF DATABASE System Design Schematic Architecture USER PHYSICS META-CATALOG UI MONITORING MODULE JOB SUBMISSION MODULE DATA INTERFACE RB BOSS GRID COMMON BOSS/AF DATABASE WN

Implementation 3 Development Areas Job preparation/submission module Data handling interface Monitoring module

Job preparation and submission Split analysis task into multiple jobs Prepare each job for submission to the Grid create JDL create job wrapper script Archive details of each task and job for future reference by user and to enable resubmission (input/output files, software versions, etc).

Prototype Tested on CMS-LCG-0 Simple shell scripts written to emulate framework ORCA 7.1.1 installed on CMS-LCG-0 WNs at Imperial Simple ORCA analysis code written and compiled on a local machine. Successfully submitted and ran analysis jobs to Grid. Important for development of job wrapper script

Job Preparation and Submission Object Model TASK TASKSPEC 1 -UserExecutable: string -InputFiles: vector<File> -OutputFiles: vector<string> -DataQuery: string -OrcaVersion : string -JDLRequirements: string 1 +createTask(UserSpecFile) +createJobs(..) +submitJobs(..) +killJobs(..) described by 1 1 +set(UserSpecFile) +getXYZ() creates 1 creates 1..* uses 1 uses JOB 1..* 1 1..* -DataSelection: int -UniqID: string -Executable: string -OutGUIDs: vector<string> -LocalInFiles: vector<File> -OutSandbox: vector<string> -InGUIDs: vector<string> PHYS_CAT WRAPPER 1 queries 1 1 Composed of +getRuns(DataQuery):vector<int> +getGUIDS(run):vector<string> +write(filename) Composed of 1 JDL BOSS +AddOutGUID(File) +AddOutSandbox(File) +AddInSandbox(File) +Submit() +Kill() +getXYZ() 1 uses 1 +write(filename) +submitJob(..) +getJobStatus(..)

Data Handling Input Data Output Data Physics catalogue (REFDB) contains metadata (user makes selection on) and data file GUIDs. Interface to REFDB exists (currently PHP) Job will be split to analyse one run at a time (ensures all data for a job is co-located). RB will send job to where data for the run is. Output Data Output file GUIDs and analysis details stored: Private analysis use case: local mySQL database Group analysis use case: centralised database (RefDB)

Monitoring Using Boss on the GRID BOSS DB Local BOSS gateway gatekeeper farm node BOSS DB farm node boss submit boss query boss kill GRID Scheduler gatekeeper boss registerScheduler Can be used now using native MySQL calls – tested and used for CMS productions on EDG and CMS/LCG-0 More scalable transport mechanism being investigated using RGMA (IC/Brunel team – see Peter Hobson’s talk).

Implementation Roadmap Requirements analysis Gain experience running ORCA locally Run ORCA on CMS-LCG-0 using simple prototype Design Implementation Build job submission module Build monitoring module Build data catalogue interface Commence testing on LCG-1 Iterate design/implementation as users give feedback new requirements. Completed 1-Oct