ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on behalf of the TAG developers group ATLAS Software and Computing Week, 4~8 April, 2011

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 TAGS TAGS are produced as files. These files are uploaded to a database by the Tier0. – General (with links to TAG monitoring): – TAG Task Lister (upload and postTagUpload tasks): – TAG Dataset Lister: Details tracked in the TASK database – Data distribution, mapping to datasets, sites, etc. – Service installations and usages Other metadata is also needed for a meaningful event selection. – COMA: triggers, periods, streams, etc.

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 BACKBONES—Infrastructure development 3 Databases (Oracle) Routine uploads of TAGs and other metadata info (TASK and COMA [COditions Meta datA]) Topology CERN and several Tier-1 and Tier-2 sites (BNL, TRIUMF, DESY, PIC, RAL, and …) are hosting different sets (or subsets) of TAG data with or without overlaps and replicates of CATALOG and COMA data Data Volume and Complexity Grow/evolve and pose technical challenges on code flexibility and scalability, affect all the services Responsible: Florbela Viegas, Elizabeth Gallas and Elisabeth Vinek

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Event Metadata Event-File Map File Registry Sites Collection Registry Service Registry Runs COnditions MetadatA DDM POOLTAG TASK COMATASK

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 ELSSI Services Suite Web Tools – iELSSI – RunBrowser (used by iELSSI) Web Services – GUID Lookup (used by Pathena) – Trigger decoder (used by iELSSI, EventSelector in Athena) – Token Extraction (used by iELSSI, Ganga) – Skimming – Lumirange calculation (used by iELSSI, being factorized) – …

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 iELSSI--NOT just a browser 6 1.Does provide browsing functions Metadata info on full streams Triggers/Runs Physics metadata from AOD Event flags, e.g. Higgs candidate 2.Performs statistics on these metadata Are there enough dileptons with 2 forward jets to do an analysis requiring that? What is the overlap of the muon and egamma streams and what are the properties of those events? What is the data reduction if I want to build a DAOD stream of only the digamma events that pass triggers this and that?

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 iELSSI--NOT just a browser 7 Also provides an integrated, full-chain service by: 1.interconnecting with multiple DBs and sites (setting the right parameters for db handshakes) 2.Integrating GRL, JO files with event selection/skim 3.building efficient queries (valid attributes, data streams, triggers and runs/periods) 4.interfacing with ELSSISuite services 5.building inputs to Athena or DA tools such as Ganga or Pathena. 6.Submitting grid jobs on selected events to produce ntuples or skims of AOD, ESD or RAW.

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Trigger decoder service Map a trigger name to the appropriate bit in the TAG trigger words Provide the the SQL fragment (e.g.,operation="bitand(EFPassedTrigMask0, power(2,1))") for programs using the TAG databaseEFPassedTrigMask0 Clients: – iELSSI when querying triggers on the database – EventSelector in Athena--TAG file analyses to decode the trigger words in the TAG files. triggerDecoder Url:

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Trigger decoder service Usage: – From a browser: triggerDecoderUrl/getDecodedTriggerInfo.php?trignms=EF _e10_loose&runrange=162882&phys_raw=raw&filenam e_tag=data10_7TeV&ami_tag=f287_m588_p250_p160 – From a command line or within a program: curl -s -S --key server_hotkey --sslv3 --cert server_hotcert -k --url "triggerDecoderUrl/getDecodedTriggerInfo.php?trignms=E F_e10_loose&runrange=162882&phys_raw=raw&filename _tag=data10_7TeV&ami_tag=f287_m588_p250_p160"

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Token Extraction service Constructs a ROOT collection of selected events which can then be retrieved from the /afs space at CERN or directly from the web. Times can vary between seconds and minutes depending on the size of your selection (can be checked beforehand in iELSSI). A link to the results will be ed to you or it will also appear in iELSSI when the job completes. The will contain all the information on the inputs and outputs of the job which can be browsed or wget-ed. – Configuration (in xml) – Log files – Root Collection At the end of the Extract results, it gives instructions for running the command line or portable client version of Extract.

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Token Extraction service invoked by iELSSI

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Skimming service al_Skim al_Skim Skim the events selected in your ROOT collection Supply with status updates as it progresses through skim configuration, submission and completion (or failure). Create scripts and send them to the Ganga Service The skim does two jobs: – ‘Tag Prepare’ job finds (using a guid lookup) the datasets the selected events belong to. – ‘Tag Run’ job. The chosen AOD Athena JO is then run on the LCG on these datasets. The Tag run job is submitted to the Grid. notice with the name of the resulting dataset stored in dq2 (e.g. user09.thomasdoherty.ganga.users.thomasdoherty.ganga /) will be made available with a dq2-get.

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Skimming service invoked by iELSSI

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 TAGs for analysis? TAGs have proven extremely useful in commissioning and in early collision data TAGs discovered and quantified the scope of lumi blocks being split across multiple files in 2010 reprocessings – Problematic for cross-section calculation TAGs discovered the event duplication issues coming from TDAQ in early 2011 collision data If you have used event picking, you have used TAGs TAGs have identified anomalous and/or problematic events TAGS have been useful in understanding the overall cross-stream statistics of our data TAGs provide a quick means of checking whether your analysis needs to process the debug stream TAGs for analysis? Needs involvement of PAT group

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 SUMMARY 15 ELSSISuite services have been developed on top of an integrated worldwide database system to provide extended selection possibilities You are invited to more TAG related talks at the DB session this afternoon QIZHI ZHANG

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Thank you! Supporting slides or a live iELSSI demo

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Period-based selection

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 period-based selection-2 18

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Count per run/per trigger-1 19

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Count per trigger-2 20

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Stream mapped with runs Without run range

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Perform queries of different project-1

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Perform queries of different project-2

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 D E F Perform queries of different project-3

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 A B C Synchronized editing-1

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Synchronized editing-2 C

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Download GUIDs into a text file-1

Qizhi Zhang, Software and Computing Week, 4-8 April, 2011 Download GUIDs into a text file-2