Hans Wenzel PMG Meeting Friday Dec. 13th 2002 ”T1 status" Hans Wenzel Fermilab  near term vision for: data management and farm configuration management  Implementing the new WBS  work performed in the last few weeks.  Near term plans

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 What do we expect from dCache?  making a multi-terabyte server farm look like one coherent and homogeneous storage system.  Rate adaptation between the application and the tertiary storage resources.  Optimized usage of expensive tape robot systems and drives by coordinated read and write requests. Use dccp command instead of encp!  No explicit staging is necessary to access the data (but prestaging possible and in some cases desirable).  The data access method is unique independent of where the data resides.  High performance and fault tolerant transport protocol between applications and data servers  Fault tolerant, no specialized servers which can cause severe downtime when crashing.  Can be accessed directly from your application (e.g. root TDCacheFile class).  Can be used as scalable file store without HSM.

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 dCache GridFTP dCache GridFTP Catalog GridFTP CASTOR FNAL CERN Florida, … Catalog Tier 2 with dCache ENSTOREENSTORE

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Global data management w/SRB … GridFTP Local MCAT dCache GridFTP Local MCAT dCache GridFTP Local MCAT dCache Global Catalog Site A Site B Site N

Hans Wenzel PMG Meeting Friday Dec. 13th 2002

Node Front-end Node Network attached Disk dCache ENSTORE Web servers Db servers Dynamic partitioning and config. the farm

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Descriptive information to configure a node Compute Node Kickstart file IO ServerWeb Server Appliances Collection of all possible software packages (AKA Distribution) RPMs NPACI ROCKS

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Implementing the new WBS  Use new WBS to track our efforts.  Consequence of new WBS closer cooperation between T1 and T2 sites.  Established biweekly meetings to discuss R&D projects ( ).  Fermilab and SD collaborate on deploying and evaluating ROCKS. GOAL standard Rocks distribution for all US centers. ( , , )

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Implementing the new WBS (II)  SD and FNAL also collaborate on SRB and dCache ( ) dCache: combine disks on farm nodes, gridFTP as transport mechanism, SRB bookkeeping. dCache deployed at SD. Will package dCache.  Caltech evaluation and optimization of disk IO of the IDE based disk servers -> next step sustain 200MB/sec WAN speed ( )  Need even more collaboration next year

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 What was happening at FCC  Chris Brew (1/2) from SCS joint the CMS team in SCS (Joe Kaiser lead, Chris Brew, Merina Albert). ISD Michael Zalokar, Jon Bakken (dCache), Igor Mandrichenko (FBSNG). CMS: Hans Wenzel, Michael Ernst, Natalia Ratnikova  65 dual AMD athlon nodes are installed and commissioned. Acceptance period (30 days) finished right before Thanksgiving. (many delays, same vendor won bids for CDF, CMS, D0 purchases)  Monitored and evaluated daily, swapped broken parts. Achieved 98.3 % up time ( )  Upgraded monitoring software (temperature and fan speeds) ( )  We used the farm for hcal test beam production.  We used the farm to test the 3 TB aztera disk system ( benchmarking suite, ). Needed constant tuning and upgrading in the end we managed to double performance.

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 What was happening at FCC II  Used farm in tests of dCache system. Developed a root based benchmarking suite testing the entire data path from Application to mass storage. ( )  Used farm to install and configure ROCKS. ( )  7 dCache linux nodes: hardware has been upgraded (SCSI system disk). System upgraded to Kernel , XFS File system. System was usable during the upgrades ( ). dCache software and configuration upgraded.  Installed web servers electronic logbook. ( )  Work on transition from RH 6 to RH 7.  Missed milestone: Test of interactive farm based Analysis prototype ( ). Slipped by 2 -3 weeks.

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 The new farm and the dCache system

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Linux dCache node: Important Kernel to avoid memory management problems XFS file system: we found it’s the only one that scales still Delivers performance when File system is full Add SCSI system disk Need server specific Linux distribution!!! Need to tweak Many parameters to Achieve optimum Performance-> feed back to vendors Next generation: Xeon based, PCIX bus Large capacity disks Dual System disk (raid1)

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 First results with dCache system These tests were done before the hardware and configuration upgrade. The average file size is ~1 GByte the reads are equally distributed over all read pools. Reads with dccp from popcrn nodes into /dev/null # of concurrent reads (40 farm nodes) Aggregate input speed (sustained over hours) Mbyte/sec Mbyte/sec Mbyte/sec READS 2.7 MB/sec per process

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 First results with dCache system The following was done with 2 write pools. Using an Root Application utilizing TDCacheFile to write an Event tree into dCache. Only had three farm nodes available so we are network limited. # of concurrent writes (3 farm nodes) Aggregate output speed (sustained over hours) MByte/sec 5.2 MB/sec process WRITES

Hans Wenzel PMG Meeting Friday Dec. 13th pounds

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 AZTERA RESULTS

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Last weeks power outage (>1.5 days)  Made good use of the outage combining several maintenance projects. (knew it was coming sometime)  Completed remote power automation project, which required complete rewiring and racking of nodes.)  Addressed safety issues: Safer power distribution.  Better connectivity between cms computing and central switch.  Installation and configuration of private network for ROCKS.  Still causes delays…, things break ….  Many tasks were not listed in WBS (need to review more often)

Hans Wenzel PMG Meeting Friday Dec. 13th 2002 Plans for the near term future  Continue Evaluating disk systems (zambeel, dCache, dfarm, panasas, exanet…..). Procure a network attached system at the begin of next year? But lot happening on the market.  Upgrade of dCache system to follow the technology (serial ide, PCIX, xeon, memory bandwidth….)  Configure and deploy farm based interactive/batch user computing.  Continue evaluating ROCKS, SRB,…., Need more manpower for  Follow the marching orders given in the new WBS, refine WBS as we go along.