DØSAR a Regional Grid within DØ Jae Yu Univ. of Texas, Arlington THEGrid Workshop July 8 – 9, 2004 Univ. of Texas at Arlington
High Energy Physics –Total expected data size is over 5 PB (5,000 inche stack of 100GB hard drives) for CDF and DØ –Detectors are complicated Need many people to construct and make them work –Collaboration is large and scattered all over the world –Allow software development at remote institutions –Optimized resource management, job scheduling, and monitoring tools –Efficient and transparent data delivery and sharing Use the opportunity of having large data set in furthering grid computing technology –Improve computational capability for education –Improve quality of life The Problem
DØ and CDF at Fermilab Tevatron World’s Highest Energy proton-anti-proton collider –E cm =1.96 TeV (=6.3x10 -7 J/p 13M Joules on m 2 ) Equivalent to the kinetic energy of a 20t truck at a speed 80 mi/hr Chicago Tevatron pp p CDF DØ
650 Collaborators 78 Institutions 18 Countries DØ Collaboration
Centeralized Deployment Models Started with Lab-centric SAM infrastructure in place, … …transition to hierarchically distributed Model
Desktop Analysis Stations Institutional Analysis Centers Regional Analysis Centers Normal Interaction Communication Path Occasional Interaction Communication Path Central Analysis Center (CAC) DAS …. DAS …. IAC... IAC … RAC …. RAC DØ Remote Analysis Model (DØRAM) Fermilab
DØ Southern Analysis Region (DØSAR) One of the regional grids within the DØGrid Consortium coordinating activities to maximize computing and analysis resources in addition to the whole European efforts UTA, OU, LTU, LU, SPRACE, Tata, KSU, KU, Rice, UMiss, CSF, UAZ MC farm clusters – mixture of dedicated and multi-purpose, rack mounted and desktop, 10’s-100’s of CPU’s
UTA is the first US DØRAC Mexico/Brazil OU/ LU UAZ Rice LTU UTA KU KSU Ole Miss DØRAM Implementation Mainz Wuppertal Munich Aachen Bonn GridKa (Karlsruhe) DØSAR formed around UTA
UTA – RAC (DPCC) 100 P4 Xeon 2.6GHz CPU = 260 GHz 64TB of Disk space 84 P4 Xeon 2.4GHz CPU = 202 GHz 7.5TB of Disk space Total CPU: 462 GHz Total disk: 73TB Total Memory: 168Gbyte Network bandwidth: 68Gb/sec
The tools Sequential Access via Metadata (SAM) –Data replication and cataloging system Batch Systems –FBSNG: Fermilab’s own batch system –Condor Three of the DØSAR farms consists of desktop machines under condor –PBS Most the dedicated DØSAR farms use this manager Grid framework: JIM = Job Inventory Management –Provide framework for grid operation Job submission, match making and scheduling –Built upon Condor-G and globus
Operation of a SAM Station /Consumers Producers/ Station & Cache Manager File Storage Server File Stager(s) Project Managers eworkers File Storage Clients MSS or Other Station MSS or Other Station Data flow Control Cache Disk Temp Disk
Tevatron Grid Framework (JIM) UTA TTU
The tools cnt’d Local Task managements –DØSAR Monte Carlo Farm (McFarm) management Cloned to other institutions Various Monitoring Software –Ganglia resource –McFarmGraph: MC Job status monitoring –McPerM: Farm performance monitor DØSAR Grid: Submit requests onto a local machine and the requests gets transferred to a submission site and executed at an execution site –DØGrid Uses mcrun_job request script More adaptable to a generic cluster
Ganglia Grid Resource Monitoring Operating since Apr. 2003
Job Status Monitoring: McFarmGraph Operating since Sept. 2003
Farm Performance Monitor: McPerMMcPerM Designed, implemented and improved by UTA Students Operating since Sept. 2003
D0 Grid/Remote Computing April 2004 Joel Snow Langston University DØSAR MC Delivery Stat. (as of May 10, 2004) InstitutionInceptionN MC (TMB) x10 6 LTU6/ LU7/ OU4/ Tata, India6/ Sao Paulo, Brazil4/ UTA-HEP1/ UTA–RAC12/ D0SAR TotalAs of 5/10/0418.9
DØSAR Computing & Human Resources InstitutionsCPU(GHz) [future]Storage (TB)People Cinvestav131.11F+? Langston221.31F+1GA LTU25+[12]1.01F+1PD+2GA KU12??1F+1PD KSU401.21F+2GA OU (OSCER) (tape)4F+3PD+2GA Sao Paulo60+[120]4.52F+Many Tata Institute521.61F+1Sys UTA F+1sys+1.5PD+3G A Total943 [1075] (tape) 14.5F+2sys+6.5PD+10 GA
How does current Tevatron MC Grid work? Client Site Global Grid Sub. Sites Regional Grids Exe. Sites Desktop. Clst. Ded. Clst. SAM
Actual DØ Data Re-processing at UTA
Network Bandwidth Needs
Summary and Plans Significant progress has been made in implementing grid computing technologies for DØ experiment –DØSAR Grid has been operating since April, 2004 Large amount of documents and expertise accumulated Moving toward data re-processing and analysis –First set of 180million event partial reprocessing completed –Different level of complexity Improved infrastructure necessary, especially network bandwidths –LEARN will boost the stature of Texas in HEP grid computing world –Started working with AMPATH, Oklahoma, Louisiana, Brazilian Consortia (Tentatively named the BOLT Network) Need the Texan consortium UTA’s experience on DØSARGrid will be an important asset to expeditious implementation of THEGrid