1. Michael P. McCumber Task Force Meeting April 3, 2006
Grid Computing Status
Michael P. McCumber
Task Force Meeting
April 3, 2006

2. Grid Operational Status
Submit from:
RCF
SBC
CCJ, UNM, VAN…
Process at:
RCF
SBC
CCJ, UNM, VAN…
Return to:
dCache
RCF
SBC
(SRM dcache_ferry)
(dotted returns - globus-url-copy needed cluster-wide
or additional SRM-like implementation on grid-host)
(green submit) – working pre-upgrade
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

3. Current Grid Job/SRM Job Logistics
Local Grid Host
Pacman
Repository
Remote Grid Host
Remote Node
scratch-dir
semaphore-dir
output-dir
To remove execution cluster dependence,
these directories have been standardized.
dCache
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

4. Grid Job XML Layout <xml tag defines XML version />
<job tag sets input file number >
<command tag>
call pacman
source environment
link output to $HOME/dcache_ferry/transfer
set seeds/initialize code
execute code
create semaphore
</command>
<input tag sets input file />
<stdout tag sets output log file/>
<stderr tag sets error log file />
</job>
PJS will generate this automatically for simulation users.
Scratch directory available in $SCRATCH
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

5. dCache_ferry (SRM job)
As before a crontab job checks for semaphore files, and runs an SRM transfer. The cycle is currently throttled to 5 mins to prevent local disk quota overruns. To the same end, I’ve designed the transfers to delete local copies once a dcache transfer has been achieved.
Also since running on the remote cluster without user login is required, I’ve re-designed SRM to open a grid-proxy when file transfers are needed.
(Put proxy password in ~/srm/bin/open_proxy.csh)
Could be run by a “ferry captain” with world writable directories…
Required directory structure
$HOME
dcache_ferry
semaphore
todo
doing
done
transfer
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

6. Raising a Semaphore
Once output has been sent to dcache_ferry/transfer directory, a semaphore is made in semaphore/todo such that:
1) The file has a unique name (usually based on JobID, run number, etc)
2) The file contains a line specifying the full path of the file to be transferred and the destination directory in dCache, seperated by a space.
Ex:
/phenix/u/mccumber/dcache_ferry/transfer/exampleOutput.root /pnfs/rcf.bnl.gov/user/mccumber/someDirectory
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

7. Current Available Policies
Usage: phenix-grid-submit –p <Policy Name> <jobXML>
Policies:
Grid_Policy (contains all queues)
Sim_Policy (general queues + simulation queues)
BNL_Only_Policy
SBC_Only_Policy
RCAS_Only_Policy
RCRS_Only_Policy
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

8. Current Development Issue
Old design makes pointless and cumbersome grid transfers
Old Structure:
Exodus
PISA
Reco
RCF
RCF
dCache
New Structure:
Exodus
PISA
Reco
Still solving internal conflicts…
dCache
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook

9. Other Current/Future Spearheads
Diagnose and solve upgrade issues
Updating changes to PJS in CVS and Grid documentation
Incorporate into grid, clusters at Vanderbilt (positive response), UNM & CCJ (died in Limbo, waiting for requested access)
Solve remote-transfer grid-proxy management problem (Done)
Add option of non-dCache return of output (Currently not feasible)
Need automated grid-monitoring software? Could be done via dynamic production of globalConfig.xml based on open path ways? Ferry Captain?
Simplify, simplify, simplify…
Task Force Meeting
April 3, 2006
Michael P. McCumber
SUNY Stony Brook