4/5/2007Data handling and transfer in the LHCb experiment1 Data handling and transfer in the LHCb experiment RT NPSS Real Time 2007 FNAL - 4 th May 2007.

Slides:

Advertisements

Similar presentations

Peter Berrisford RAL – Data Management Group SRB Services.

Advertisements

Overview of Database Administrator (DBA) Tools

Oracle9i Database Administrator: Implementation and Administration 1 Chapter 2 Overview of Database Administrator (DBA) Tools.

André Augustinus ALICE Detector Control System  ALICE DCS is responsible for safe, stable and efficient operation of the experiment  Central monitoring.

CHEP 2012 – New York City 1.  LHC Delivers bunch crossing at 40MHz  LHCb reduces the rate with a two level trigger system: ◦ First Level (L0) – Hardware.

1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.

Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.

6/4/20151 Introduction LHCb experiment. LHCb experiment. Common schema of the LHCb computing organisation. Common schema of the LHCb computing organisation.

MSc IT UFIE8K-10-M Data Management Prakash Chatterjee Room 3P16

Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage.

GLAST LAT ProjectOnline Peer Review – July 21, Integration and Test J. Panetta 1 Gamma-ray Large Area Space Telescope GLAST Large Area Telescope:

Harvard University Oracle Database Administration Session 2 System Level.

Database Management Systems (DBMS)

Exploiting the Grid to Simulate and Design the LHCb Experiment K Harrison 1, N Brook 2, G Patrick 3, E van Herwijnen 4, on behalf of the LHCb Grid Group.

Slide 1 of 9 Presenting 24x7 Scheduler The art of computer automation Press PageDown key or click to advance.

Overview SAP Basis Functions. SAP Technical Overview Learning Objectives What the Basis system is How does SAP handle a transaction request Differentiating.

Module 9: SQL Server 2005 Replication. Overview Overview of Replication Enhancements New Types of Replication Configuring Replication.

CERN IT Department CH-1211 Genève 23 Switzerland t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.

Irwin/McGraw-Hill Copyright © 2004 The McGraw-Hill Companies. All Rights reserved Whitten Bentley DittmanSYSTEMS ANALYSIS AND DESIGN METHODS6th Edition.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

LSC Segment Database Duncan Brown Caltech LIGO-G Z.

ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)

LHCb Applications and GRID Integration Domenico Galli Catania, April 9, st INFN-GRID Workshop.

CDF data production models 1 Data production models for the CDF experiment S. Hou for the CDF data production team.

Computing Infrastructure Status. LHCb Computing Status LHCb LHCC mini-review, February The LHCb Computing Model: a reminder m Simulation is using.

Database Design and Management CPTG /23/2015Chapter 12 of 38 Functions of a Database Store data Store data School: student records, class schedules,

Intro – Part 2 Introduction to Database Management: Ch 1 & 2.

Advanced Computer Networks Topic 2: Characterization of Distributed Systems.

Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.

08/30/05GDM Project Presentation Lower Storage Summary of activity on 8/30/2005.

The european ITM Task Force data structure F. Imbeaux.

Status of the LHCb MC production system Andrei Tsaregorodtsev, CPPM, Marseille DataGRID France workshop, Marseille, 24 September 2002.

ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR

 CASTORFS web page - CASTOR web site - FUSE web site -

Oracle's Distributed Database Bora Yasa. Definition A Distributed Database is a set of databases stored on multiple computers at different locations and.

Management of the LHCb DAQ Network Guoming Liu * †, Niko Neufeld * * CERN, Switzerland † University of Ferrara, Italy.

 Replication is the process of copying database information  Replication is used for:  Backing up your database  Migrating to a new server  Mirroring.

Introduction Database integral part of our day to day life Collection of related database Database Management System : software managing and controlling.

VMware vSphere Configuration and Management v6

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Implementation and performance analysis of.

The GridPP DIRAC project DIRAC for non-LHC communities.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF CF Monitoring: Lemon, LAS, SLS I.Fedorko(IT/CF) IT-Monitoring.

The DCS Databases Peter Chochula. 31/05/2005Peter Chochula 2 Outline PVSS basics (boring topic but useful if one wants to understand the DCS data flow)

1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.

AMGA-Bookkeeping Carmine Cioffi Department of Physics, Oxford University UK Metadata Workshop Oxford, 05 July 2006.

11/01/20081 Data simulator status CCRC’08 Preparatory Meeting Radu Stoica, CERN* 11 th January 2007 * On leave from IFIN-HH.

Markus Frank (CERN) & Albert Puig (UB).  An opportunity (Motivation)  Adopted approach  Implementation specifics  Status  Conclusions 2.

LHCC Referees Meeting – 28 June LCG-2 Data Management Planning Ian Bird LHCC Referees Meeting 28 th June 2004.

The GridPP DIRAC project DIRAC for non-LHC communities.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF CC Monitoring I.Fedorko on behalf of CF/ASI 18/02/2011 Overview.

Maria del Carmen Barandela Pazos CERN CHEP 2-7 Sep 2007 Victoria LHCb Online Interface to the Conditions Database.

Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.

uses of DB systems DB environment DB structure Codd’s rules current common RDBMs implementations.

M. Caprini IFIN-HH Bucharest DAQ Control and Monitoring - A Software Component Model.

Clara Gaspar, February 2010 DIM A Portable, Light Weight Package for Information Publishing, Data Transfer and Inter-process Communication.

Amazon Web Services. Amazon Web Services (AWS) - robust, scalable and affordable infrastructure for cloud computing. This session is about:

Cost9b 1 Living with Function Points Bernstein and Lubashevsky Text pp

Database 12.2 and Oracle Enterprise Manager 13c Liana LUPSA.

Controlling a large CPU farm using industrial tools

The COMPASS event store in 2002

LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.

Ákos Frohner EGEE'08 September 2008

CTA: CERN Tape Archive Overview and architecture

Monitoring of the infrastructure from the VO perspective

Database Management Systems

Status and plans for bookkeeping system and production tools

Production Manager Tools (New Architecture)

Offline framework for conditions data

Presentation transcript:

4/5/2007Data handling and transfer in the LHCb experiment1 Data handling and transfer in the LHCb experiment RT NPSS Real Time 2007 FNAL - 4 th May 2007 A.C. Smith, M. Frank, N. Neufeld, R. Stoica* (CERN) *On leave from IFIN-HH

4/5/2007Data handling and transfer in the LHCb experiment2 LHCb in a nutshell One of the 4 major LHC experiments at CERN 2 proton beams colliding at 7 TeV each Studies CP violation of b-mesons and other rare decays Single arm spectrometer channels to be read out at 40 MHz Foreseen to start at the end of 2007

4/5/2007Data handling and transfer in the LHCb experiment3 DAQ System overview ?

4/5/2007Data handling and transfer in the LHCb experiment4 File management requirements Functional requirements: Replicating files to the CERN Tape storage system (CASTOR) Creating appropriate entries in the GRID file catalogue (database that serves as a translation between the logical file name and its physical location) Managing metadata information and replicating it inside the LHCb Bookkeeping Database (database storing all information needed for all physics computing) Operational requirements: Integration with the Experiment Control System Error recovery, redundancy Interface to external Online applications Automated and flexible management

4/5/2007Data handling and transfer in the LHCb experiment5 File management implementation overview ECS = Experiment Control System DIRAC = Distributed Infrastructure with Remote Agent Control CASTOR = CERN Advanced STORage Manager

4/5/2007Data handling and transfer in the LHCb experiment6 Centralized point of file management. Simple but extensible SQL structure (not fixed yet!). Oracle database backend. Control and behaviour implemented using a server side Python application (using the cxOracle module) The management application is stateless, can be run in multiple instances for redundancy. Online Run Database

4/5/2007Data handling and transfer in the LHCb experiment7 File states inside DAQ System File in writing File closed File being transferred File is being deleted File in use by online process(es) Created Closed UsedNot Used Migrated Modified Deleting Deleted Modify BDB Migrating Modifying LHCb Bookkeeping Database Pseudo Petri net diagram models file behaviour Asynchronous transitions from one state to another File accesses are reference counted Timestamps are used for error recovery File migration is decoupled from file usage File deletion triggered by low space on the Online SAN

4/5/2007Data handling and transfer in the LHCb experiment8 Online Run Database interfaces Several possibilities for accessing/controlling the Online Run Database : XMLRPC interface for the server controlling the actual Oracle database. Required for communication with the DIRAC software. Also useful for creating various scripts/applications (C/C++, Python, etc) DIM interface to the Experiment Control System (PVSS). DIM is exposed to Python as an ordinary C library. Oracle specific tools for performing “big” changes to the database (e.g. table modification, backup restoring). HTTP based interface that can be opened using a common web browser. Should be the most used tool for viewing the content of the database and for small administrative tasks. Supports access control.

4/5/2007Data handling and transfer in the LHCb experiment9 File handler Functionality: file copy to CASTOR (triggering file migration) updates the LHCb Bookkeeping DB file deletion error recovery Can be run in multiple instances on different machines if needed. The Online Database is the central point of synchronization (e.g. assures that the same file is not migrated simultaneously by two independent processes). Can run independently of the other online processes (independent of the Data Writer). Integrated with the ECS using DIM

4/5/2007Data handling and transfer in the LHCb experiment10 WEB Management Tool

4/5/2007Data handling and transfer in the LHCb experiment11 Tests Functionality: files generated and transferred to CASTOR. Complete software chain is working. Stability and error recovery: Tested behaviour and error recovery in case of simulated errors. Performance: Load simulated for the data handler and Online Run Database on a single machine. Maximum data handling was 10 3 more than needed: files managed at a rate of 30/s CPU utilisation ~80% I/O ~4 MB/s files entries: 8 million

4/5/2007Data handling and transfer in the LHCb experiment12 (LHCb Cavern) Any questions? Thank you!