Brookhaven Analysis Facility Michael Ernst Brookhaven National Laboratory U.S. ATLAS Facility Meeting University of Chicago, Chicago 19 – 20 August, – 20 August, 2009

20 August, 2009 M. Ernst Facility Workshop 2 Anticipated Analysis Workflow  Interactively develop and test code used in the athena frame work to access data via pAthena  Once the user has tested the code on a small number of events and she/he is satisfied w/ results the user will submit PanDA jobs to process the data located at the Tier-1 and/or Tier-2 centers. The output will be stored close to location where jobs ran  When user has created a DPD she/he wants to run over the data many times, she/he will use existing DQ2 tools to transfer the root files to the user’s site  Once the user data is at the analysis facility the user will analyze the DPD many times  User ntuples can be further reduced and shipped back to the home institution for further analysis

20 August, 2009 M. Ernst Facility Workshop 3 Aspects of Analysis Facility Implementation  Single machine access  Needed for “pre-PanDA” testing and Athena development work  Could be primary method for analyses with small data volume  Parallel processing w/ PROOF  Needed for interactive analyses with large data volume o One PROOF instance may last n hours o From interactive histo making to command line processing o Allows transparent use of multiple nodes (other than batch system) o Automatically distributes load and merges results  Distributed processing using Condor  Needed for long running analyses tasks (>1 day)  Needed for analyses that cannot use TSelector framework  Mature system that can be used for ntuple processing  Has queuing, prioritization and fair share mechanisms

20 August, 2009 M. Ernst Facility Workshop 4 Central Analysis Facility at BNL  Existing Computing Infrastructure of U.S. ATLAS meets requirements of the ATLAS CM  For Central Production and Distributed Analysis  Meet partially analysis needs of individual physicists and groups  Purpose: Serve the needs of data analysis at the “Post-AOD” stage  Allow members of the detector performance groups and individual users performing physics studies to analyze selected or reduced data with fast turnaround  Fraction of users will lack significant Tier-3 resources beyond a PC  Allow users to develop and debug code and run interactive jobs o Essential step prior to submitting pAthena and long analysis jobs o Need demonstrated by usage of interactive facilities at CERN and BNL  Selected information will be processed recursively many times at low latency before the data volume is further reduced o This may require several iterations  Primary intention is to help physicist to process large data volumes o Production of private datasets, generation of Toy MC will be discouraged  Not intended to replace the existing analysis model  Strengthen it by adding additional functionality (addressing needs of early analysis stages)  Users are strongly encouraged to make use of pAthena  AF useful as long as analysis code is in flux – once confidence in algor. reached produce DnPD

20 August, 2009 M. Ernst Facility Workshop 5 Central Analysis Facility at BNL  Will be kept separate from the (Tier-1) production facility  Functionality for parallel processing (interactive and batch)  Properly sized to satisfy the CPU and I/O intensive requirements  Will host selection of AOD & ESD, and primary DPD  Supports access of large data volumes multiple time w/o data movement, registration  Open to institutions opting to have BNL procure, install and maintain their equipment Analysis Facility will be an End-User facility Analysis Facility will be an End-User facility Combines functionalities currently in use but at larger scale A la ACAS with features available in PROOF (if applicable) and Condor Farms Analysis Facility can be thought of as a Tier-3 Facility Analysis Facility can be thought of as a Tier-3 Facility In addition to institutional Tier-3’s Providing support for users who do not have access to local Tier-3 resources Analysis Facility will allow users to work efficiently with “User Ntuples” Analysis Facility will allow users to work efficiently with “User Ntuples” Derived from a variety of data formats (produced using pAthena)

20 August, 2009 M. Ernst Facility Workshop 6 Data Management  AF will have finite amount of resources  Shared between ESD, AOD, DPD and User storage space  Data will be replicated from US ATLAS Tier-1 to AF  User tools to transfer data to home institutions  Data transfer into AF will be managed by group coordinator  Disk management of PROOF system handled by Xrootd  Tight binding of subjobs with associated data handled automatically

20 August, 2009 M. Ernst Facility Workshop 7 Central Analysis Facility GRID/PANDA AF Data transfer Tier1 Tier2s Users Multiple processing of data sets Interactive jobs Pathena Jobs Histos making Selected AOD ESD Tier 3 Selected D 1 PD D 2 PD D 3 PD Histos making DPD transfer DPD transfer User Ntuple Selected D 1 PD D 2 PD D 3 PD User Ntuple

20 August, 2009 M. Ernst Facility Workshop 8 Analysis Facility at BNL  Analysis Facility computer cluster with:  Excellent access to Atlas data  Easy access to Atlas databases (including conditions root files)  Available to any Atlas member (requires local account, currently ~750)  Augments Tier 3’s at Universities  Universities can contribute approved hardware o Fair share mechanism will allow Universities to use what they contributed  Open to U.S. ATLAS Institutions starting in September, 2009

20 August, 2009 M. Ernst Facility Workshop 9 Scientific Computing at BNL Joint BNL/Stony Brook Computational Science Center 100 TFlop BlueGene/L QCDOC RIKEN-BNL-Columbia: 10 TFlop DOE HEP: 10 TFlop RHIC/ATLAS Computing Facility (RACF) RHIC data and analysis center ATLAS Tier 1 Center Combined capacity: 20 MSI2k, 7 PB Disk, 9 PB Tape Platform for: ATLAS Tier 3 Computing LSST/astrophysics computing Computing for Daya Bay and LBNE experiments

20 August, 2009 M. Ernst Facility Workshop 10 RACF space and deployment planning Core structure: Brookhaven Computing Facility (13000 sq. ft.) Renovation (2000 sq. ft.) New addition (6400 sq. ft.) QCDOC NYBlue (7000 sq. ft. for RACF)

20 August, 2009 M. Ernst Facility Workshop 11 Analysis Facility at BNL  BNL has currently 1000 Batch slots available for user analysis  Usage of ~800 slots will be restricted to Group Analysis after start of LHC data taking (~mid November, 2009) according to ATLAS Computing Model  Acas nodes available for interactive analysis  Can be expanded if need – defined by user needs  Prototype Proof cluster – will expand as needed  150 TB (high quality, highly available) disk space for users  Non permanent – but not scratch space (managed)  University can contribute funds or “certified” hardware to place computing at BNL * Based on vendor price of $2,500 per compute server; incl. rack, cables, console, overhead, Acquisition ($) Operating ($ per yr) CPU (per 8-core node) 2,970*571 Disk (per TB) 80181

20 August, 2009 M. Ernst Facility Workshop 12 Hosting Services  Provide full integration of CPU and Disk into existing RACF infrastructure - Operating Cost covers  Space, Power and Cooling  “Racking and Stacking” including “out-of-band” console connection  Hardware service (typically next business day)  Line-rate 1 Gbps network connection for WNs and 10 Gbps for disk servers  Monitoring (system and services, nagios w/ notification as appr.)  OS installation, administration and maintenance (regular kernel patches (typically for cyber security), same as ATLAS farm)  Operating - typically 5days/week, 8 hours/day  Account Management  Shared file system access for ATLAS S/W releases  Batch queues (Condor)  Dedicated queues for institutional resources augmented by “General Queue” to share resources with the rest of the (US) ATLAS community in case cycles aren’t used  RACF provisions scalable gateway services for interactive access  Disk is procured in large quantities and distributed in TB quantities  Integration into Storage Management Infrastructure o