1 By: Solomon Mikael (UMBC) Advisors: Elena Vataga (UNM) & Pavel Murat (FNAL) Development of Farm Monitoring & Remote Concatenation for CDFII Production Project

08/08/2006Solomon Mikael 2 Outline CDF Experiment CDF Production Farm Goals & Structure Issues with Concatenation My Contributions 1 Control & Monitoring My Contributions 2 Summary Acknowledgments

08/08/2006Solomon Mikael 3 Goal of CDF Prodution Farm ● The main goal of the Production Farm is the reconstruction of available data for physics analysis as soon as possible, reprocess data when necessary, and generate Monte Carlo events

08/08/2006Solomon Mikael 4 CDF Experiment The CDF ( collider detector experiment at FermiLab) is an international collaboration involving many universities and national laboratories The CDF ( collider detector experiment at FermiLab) is an international collaboration involving many universities and national laboratories 2 intense beams of protons and anti-protons meet head on in the middle of the 100 ton solenoidal CDF detector 2 intense beams of protons and anti-protons meet head on in the middle of the 100 ton solenoidal CDF detector In order to observe the particles there are layers of subdetectors in the CDF detector each layer responsible for the detection of a different particles properties. In order to observe the particles there are layers of subdetectors in the CDF detector each layer responsible for the detection of a different particles properties. Information from electronic channels is recorded Information from electronic channels is recorded

08/08/2006Solomon Mikael 5 Production Farm - Hardware the Production Farm consists of 150 dual CPU PC's with a total computing power of 800 GHz The high throughput Linux clusters are used for event reconstruction and analysis The production farm PC have a total of 25 TB of disk space

08/08/2006Solomon Mikael 6 ● the CDF production farm performs computing and network intensive tasks in a cost effective manner Production Farm - Software ● SAM (sequential data access via metadata) a data handeling system organized as set of servers working together to store and retrieve files. SAM mitigates the problem of one person hogging the tape drives and/or flooding the tape system. SAM provides tools for the database bookkeeping. ● CAF (CDF Analysis Farm) is software and control systems for batch job submission on top of Condor batch system.

08/08/2006Solomon Mikael 7 Stucture of CDF Farm concatenation

08/08/2006Solomon Mikael 8 ● In the present scheme concatenator and tape uploader are running at the same time resulting in limited I/O from the stager. ● Disk access rate depends on the number of simultaneous I/O operations from the disk RAID 5. ● My project at CDF entailed removing the load of concatenation from the stagers to CAF to achieve higher data flow rates. Issues with Concatenation tape transfer rate (MB/s) Entries tape transfer rate (MB/s)

08/08/2006Solomon Mikael 9 Structure Stager mergeSubmit.py 1.Analyze input directory 2.Creates.tcl 3.Send CAF job Worker 1.Script copies input files 2.Run binary code for concatenator 3.Copy output file to stager Monitoring

08/08/2006Solomon Mikael 10 Required Skills Before I could make any changes to the CDF farm it was imperative I learned how the individual parts of the farm operated and how they are interrelated: Effective use of bash scripts & awk text editor Learning python to modifying the concatenating script MergeSubmit.py Modifying Tikiwiki pages using the online Tikiwiki editor & web pages

08/08/2006Solomon Mikael 11 What’s BASH & AWK Shell is a program which interprets commands, either typed in directly by the user or contained in a file called a shell script. Awk named after its developers ( Aho, Weinberger, and Kernighan ) is a programming language which permits manipulation of structured data and generation of formatted reports. A pattern scanning and processing language

08/08/2006Solomon Mikael 12 the_num=2; export JOB_NUM=3 echo $SEGMENT_NUMBER echo $JOB_NUM temp.awk -v seg_num=$SEGMENT_NUMBER #!/bin/awk BEGIN { flag = 0; } /SEGMENT_NUMBER/ { if ($5 == seg) { flag = 1 } } /include/ { if (flag == 1) { print $0 } } { if ($1 == "}") { flag = 0 } # if (($4 == "==") && (ENVIRON["SEGMENT_NUMBER"] == $5)) {print $0} } #!/bin/bash name=`basename $0`../cdfopr/scripts/common_procedures../cdfopr/scripts/parse_parameters $* > temp_parse_log echo $TCL_FILE echo $PARAM_USER echo $PARAM_HOST echo $PARAM_PATH cmd="fcp -c ${RCP} E}."; $cmd STATUS=$?; if[ STATUS -ne 0 ];then echo "$TCL_FILE was not able to be copied" exit 1; fi for file_loc in `grep "include file /" ${TCL_FILE} | awk '{print $3}'`;do cmd="fcp -c ${RCP} "; $cmd STATUS=$?; if [ STATUS -ne 0 ];then echo "$file_loc was not able to be copide fi done

08/08/2006Solomon Mikael 13 if { $env(SEGMENT_NUMBER) == 1 } { # # OutputDir = "/export/data1/cdfmc/concatTest/Monte_Carlo_Test1/mergeLogs/hphysr_0y_01/tmp" ; # total size: # set DATASET xbck0y include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbck0y/reco.xy0339c5.0284bck0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbck0y/reco.xy0339c5.028ebck0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbck0y/reco.xy0339c5.0298bck0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbck0y/reco.xy0339c5.02a2bck0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbck0y/reco.xy0339c5.02acbck0 } if { $env(SEGMENT_NUMBER) == 2 } { # # OutputDir = "/export/data1/cdfmc/concatTest/Monte_Carlo_Test1/mergeLogs/hphysr_0y_01/tmp" ; # total size: # set DATASET xbhd0y include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbhd0y/reco.xy0339c5.027abhd0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbhd0y/reco.xy0339c5.0284bhd0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbhd0y/reco.xy0339c5.028ebhd0 include file /export/data1/cdfmc/concatTest/Monte_Carlo_Test1/xbhd0y/reco.xy0339c5.0298bhd0 } Example.tcl File

08/08/2006Solomon Mikael 14 Control & Monitoring ● Tikiwiki software is used for web based documentation ● The tiki database keeps a history of all changes to the Farm Projects ● Tiki pages enable users to: ● keep track of all existing projects ● Start or stop a project ● Change resource sharing between projects ● Redirect output to another stager ● Forward execution to CAF without having to connect to the main server ● Python’s extensive support for XML, , RSS feeds and many other Internet protocols make it effective for developing custom web solutions

08/08/2006Solomon Mikael 15 Monitoring – Web Page Interface ● to ensure CDF production farm runs smoothly the hardware performance including status reporting must be monitered ● this is done using the production farm web interface (PFWI) ● PFWI parses, calculates, and displays all major characteristics of the farm with online results

08/08/2006Solomon Mikael 16 Tikiwiki Contributions In this page it shows the disk space on the 32 partitions on the different servers p = string.find(output, '/export/data4')stringfindoutput usage = string.strip(output[p-24:p])usagestringstripoutput fp.write(""" fncdfsrv5 %20s /export/data4 """ % usage)fpwriteusage fp.write("\n")fpwrite percentage = string.strip(output[p-5:p-2])percentagestringstripoutput if string.atoi(percentage) > 90 : IsFull=1stringatoipercentage1 Edited pythons script df_disk.py

08/08/2006Solomon Mikael 17 Tiki Editor Using the online tiki editor modifications were made to improve the functionality of the ProjectConfiguration page

08/08/2006Solomon Mikael 18 Summary In these weeks: Implemented improvements to Production Farm monitoring. Participated in development of remote concatenation. Acquired new skills Learned about the physics inside the FermiLab laboratory

08/08/2006Solomon Mikael 19 Acknowledgements SIST committee for giving me this opportunity Elena Vataga & Pavel Murat Ms. Engram & Dr. Elliott McCrory Dr. Davenport & Jamieson Olsen

08/08/2006Solomon Mikael 20 BACKUP

08/08/2006Solomon Mikael 21 History of Production Farm ● Fermilab has used clusters of processors to provide large computing power with dedicated processors like the Motoroloa ● CDF Run 2 data was processed using the first developed Farm Processing System (FPS) using FBSNG batch system (1) ● Farm Processing System was the software that managed, controlled, and monitered the CDF production farm from

08/08/2006Solomon Mikael 22 Monitoring ● PARSING - this layer access MySQL or CAF output files and after processing text and performing calculations the data is fed to cache layer ● CACHE – this layer does statistical preprocessing and has an interface to easily visualize the data. The data is then stored. (1) ● WEB – displays all the information collected by the parseres and gathers data not needing pre-processing. -- Uses PHP4 to generaet the web pages. ● Python tiki