1. CMS Grid Batch Analysis Framework
HughTallini
David Colling
Barry Macevoy
Stuart Wakefield

2. Contents Project objectives Requirements analysis Design outline
Implementation
Future plans

3. Objective
Build an analysis framework to enable CMS physicists to perform end user analysis of DC04 data in batch mode on the Grid.
i.e. run a private analysis algorithm over a set of reconstructed data
i.e. not interactive – submit once and leave to run.
Deliver by October ‘03

4. Approach
Build tactical solution using (as far as possible) tools that already exist.
Iterative development – deliver first prototype quickly covering major use cases, iterating design and implementation later to refine and extend the solution.
Build scalable architecture that will inevitably have to evolve as requirements change (or get better determined).
Use good software design practices (i.e. distinct analysis-design-implementation phases).

5. DC04 Calibration challenge
Fake
DAQ
(CERN)
Higgs
DST
Event
streams
Calibration
sample
Tier-0 challenge
Data distribution
Calibration challenge
Analysis challenge
Calibration
Jobs
MASTER
Conditions DB
TAG/AOD
(replica)
(10-100
kB/evt)
Replica
Conditions DB
DC04 Analysis challenge
1st pass
Recon-
struction
25Hz
2MB/evt
50MByte/s
4 Tbyte/day
DC04 T0
challenge
HLT
Filter ?
CERN
disk pool
~40 TByte
(~10 days data)
Archive
storage
CERN
Tape
archive
Disk cache
25Hz
1MB/evt raw
0.5MB reco
DST
50M events
75 Tbyte
1TByte/day
2 months
PCP
CERN
Tape
archive
Higgs background
Study (requests
New events)
Event
server
SUSY
Background DST

6. What’s involved… Data handling Job splitting Job submission
Job monitoring
Job information archiving

7. Requirements Analysis (1)
Typical Use Case 1
Private Single Step Analysis
User wishes to run their analysis algorithm over 10Million MC events from a particular dataset.
S/he compiles code and tests over a small sample of data on a local machine.
Configures a JDL listing data sample to run over, ORCA libraries and ORCA executable to use, plus any steering files required.
Submits to the framework.
Uses AF to monitor jobs as they run.
Uses AF to locate output data stored on Grid.
Uses AF to keep record of analysis details, details of which are stored locally (privately).

8. Requirements Analysis (2)
Typical Use Case 2
Private Multi Step Analysis
As Use Case 1 but user wishes input data for analysis task to be the output data from a previous analysis.
Typical Use Case 3
Group Multi Step Analysis
Group of physicists wish to share input and output data
Must also share details of how output data is created.

9. Requirements Analysis (3)
Some other important requirements
User’s analysis code should be identical whether running locally or on the Grid.
No constraint on size of data sample to run over.
Interface to Analysis Framework must be simple to use (users are physicists, not software developers)
Single configuration file
Single step submission

10. Batch Object Submission System
Wrapper
BOSS
Local
Scheduler
farm node
boss submit
boss query
boss kill
BOSS DB
farm node
Accepts job submission from users
Stores info about job in a DB
Builds a wrapper around the job (jobExecutor)
Sends the wrapper to the local scheduler
The wrapper sends to the DB info about the job

11. COMMON BOSS/AF DATABASE
System Design
Schematic Architecture
USER
PHYSICS
META-CATALOG UI
MONITORING MODULE
JOB SUBMISSION MODULE
DATA INTERFACE RB
BOSS
GRID
COMMON BOSS/AF DATABASE WN

12. Implementation 3 Development Areas Job preparation/submission module
Data handling interface
Monitoring module

13. Job preparation and submission
Split analysis task into multiple jobs
Prepare each job for submission to the Grid
create JDL
create job wrapper script
Archive details of each task and job for future reference by user and to enable resubmission (input/output files, software versions, etc).

14. Prototype Tested on CMS-LCG-0
Simple shell scripts written to emulate framework
ORCA installed on CMS-LCG-0 WNs at Imperial
Simple ORCA analysis code written and compiled on a local machine.
Successfully submitted and ran analysis jobs to Grid.
Important for development of job wrapper script

15. Job Preparation and Submission Object Model
TASK
TASKSPEC 1
-UserExecutable: string
-InputFiles: vector<File>
-OutputFiles: vector<string>
-DataQuery: string
-OrcaVersion : string
-JDLRequirements: string 1
+createTask(UserSpecFile)
+createJobs(..)
+submitJobs(..)
+killJobs(..)
described by 1 1
+set(UserSpecFile)
+getXYZ()
creates 1
creates
1..*
uses 1
uses
JOB
1..* 1
1..*
-DataSelection: int
-UniqID: string
-Executable: string
-OutGUIDs: vector<string>
-LocalInFiles: vector<File>
-OutSandbox: vector<string>
-InGUIDs: vector<string>
PHYS_CAT
WRAPPER 1
queries 1 1
Composed of
+getRuns(DataQuery):vector<int>
+getGUIDS(run):vector<string>
+write(filename)
Composed of 1
JDL
BOSS
+AddOutGUID(File)
+AddOutSandbox(File)
+AddInSandbox(File)
+Submit()
+Kill()
+getXYZ() 1
uses 1
+write(filename)
+submitJob(..)
+getJobStatus(..)

16. Data Handling Input Data Output Data
Physics catalogue (REFDB) contains metadata (user makes selection on) and data file GUIDs.
Interface to REFDB exists (currently PHP)
Job will be split to analyse one run at a time (ensures all data for a job is co-located).
RB will send job to where data for the run is.
Output Data
Output file GUIDs and analysis details stored:
Private analysis use case: local mySQL database
Group analysis use case: centralised database (RefDB)

17. Monitoring Using Boss on the GRID BOSS DB
Local BOSS
gateway
gatekeeper
farm node
BOSS DB
farm node
boss submit
boss query
boss kill
GRID
Scheduler
gatekeeper
boss registerScheduler
Can be used now using native MySQL calls – tested and used for CMS productions on EDG and CMS/LCG-0
More scalable transport mechanism being investigated using RGMA (IC/Brunel team – see Peter Hobson’s talk).

18. Implementation Roadmap
Requirements analysis
Gain experience running ORCA locally
Run ORCA on CMS-LCG-0 using simple prototype
Design
Implementation
Build job submission module
Build monitoring module
Build data catalogue interface
Commence testing on LCG-1
Iterate design/implementation as users give feedback new requirements.
Completed
1-Oct