Parallel Reconstruction of CLEO III Data Gregory J. Sharp Christopher D. Jones Wilson Synchrotron Laboratory Cornell University.

Slides:



Advertisements
Similar presentations
IT Technical Support South Nottingham College. Aims Knowledge of the Registry Discuss the tools available to support a technician Gain an understanding.
Advertisements

B A B AR and the GRID Roger Barlow for Fergus Wilson GridPP 13 5 th July 2005, Durham.
Parasol Architecture A mild case of scary asynchronous system stuff.
Chapter 5: Server Hardware and Availability. Hardware Reliability and LAN The more reliable a component, the more expensive it is. Server hardware is.
Chap 5 Process Scheduling. Basic Concepts Maximum CPU utilization obtained with multiprogramming CPU–I/O Burst Cycle – Process execution consists of a.
Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
Dr Mohamed Menacer College of Computer Science and Engineering Taibah University CS-334: Computer.
Data Quality Assurance Linda R. Coney UCR CM26 Mar 25, 2010.
Reconstruction and Analysis on Demand: A Success Story Christopher D. Jones Cornell University, USA.
EventStore Managing Event Versioning and Data Partitioning using Legacy Data Formats Chris Jones Valentin Kuznetsov Dan Riley Greg Sharp CLEO Collaboration.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment Chapter 11: Monitoring Server Performance.
Chapter 11 - Monitoring Server Performance1 Ch. 11 – Monitoring Server Performance MIS 431 – created Spring 2006.
Computer Organization and Architecture
Utilizing Condor and HTC to address archiving online courses at Clemson on a weekly basis Sam Hoover 1 Project Blackbird Computing,
A web based Project Management and Tracking System Zheng Wang, Yuntian Zhao, Yanhong Li Biostatistics & Statistical programming.
CLEO’s User Centric Data Access System Christopher D. Jones Cornell University.
Introduction to Parallel Programming MapReduce Except where otherwise noted all portions of this work are Copyright (c) 2007 Google and are licensed under.
Yavor Todorov. Introduction How it works OS level checkpointing Application level checkpointing CPR for parallel programing CPR functionality References.
GRID job tracking and monitoring Dmitry Rogozin Laboratory of Particle Physics, JINR 07/08/ /09/2006.
Christopher Jeffers August 2012
GLAST LAT ProjectDOE/NASA Baseline-Preliminary Design Review, January 8, 2002 K.Young 1 LAT Data Processing Facility Automatically process Level 0 data.
March 3rd, 2006 Chen Peng, Lilly System Biology1 Cluster and SGE.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment, Enhanced Chapter 11: Monitoring Server Performance.
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
Central Reconstruction System on the RHIC Linux Farm in Brookhaven Laboratory HEPIX - BNL October 19, 2004 Tomasz Wlodek - BNL.
CDF data production models 1 Data production models for the CDF experiment S. Hou for the CDF data production team.
Proof Carrying Code Zhiwei Lin. Outline Proof-Carrying Code The Design and Implementation of a Certifying Compiler A Proof – Carrying Code Architecture.
1 DIRAC – LHCb MC production system A.Tsaregorodtsev, CPPM, Marseille For the LHCb Data Management team CHEP, La Jolla 25 March 2003.
INVITATION TO COMPUTER SCIENCE, JAVA VERSION, THIRD EDITION Chapter 6: An Introduction to System Software and Virtual Machines.
Computer Architecture and Operating Systems CS 3230: Operating System Section Lecture OS-1 Process Concepts Department of Computer Science and Software.
CSF4 Meta-Scheduler Name: Zhaohui Ding, Xiaohui Wei
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment, Enhanced Chapter 11: Monitoring Server Performance.
Lecture 8: 9/19/2002CS149D Fall CS149D Elements of Computer Science Ayman Abdel-Hamid Department of Computer Science Old Dominion University Lecture.
Chapter 1 (PART 1) Introduction to OS (concept, evolution, some keywords) Department of Computer Science Southern Illinois University Edwardsville Summer,
By Teacher Asma Aleisa Year 1433 H.   Goals of memory management  To provide a convenient abstraction for programming.  To allocate scarce memory.
Process Architecture Process Architecture - A portion of a program that can run independently of and concurrently with other portions of the program. Some.
1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana.
5/2/  Online  Offline 5/2/20072  Online  Raw data : within the DAQ monitoring framework  Reconstructed data : with the HLT monitoring framework.
DEPARTEMENT DE PHYSIQUE NUCLEAIRE ET CORPUSCULAIRE JRA1 Parallel - DAQ Status, Emlyn Corrin, 8 Oct 2007 EUDET Annual Meeting, Palaiseau, Paris DAQ Status.
A B A B AR InterGrid Testbed Proposal for discussion Robin Middleton/Roger Barlow Rome: October 2001.
Karsten Köneke October 22 nd 2007 Ganga User Experience 1/9 Outline: Introduction What are we trying to do? Problems What are the problems? Conclusions.
We will focus on operating system concepts What does it do? How is it implemented? Apply to Windows, Linux, Unix, Solaris, Mac OS X. Will discuss differences.
1 Computer Systems II Introduction to Processes. 2 First Two Major Computer System Evolution Steps Led to the idea of multiprogramming (multiple concurrent.
CERN – Alice Offline – Thu, 27 Mar 2008 – Marco MEONI - 1 Status of RAW data production (III) ALICE-LCG Task Force weekly.
CERN – Alice Offline – Thu, 20 Mar 2008 – Marco MEONI - 1 Status of Cosmic Reconstruction Offline weekly meeting.
Sep 13, 2006 Scientific Computing 1 Managing Scientific Computing Projects Erik Deumens QTP and HPC Center.
ATLAS Production System Monitoring John Kennedy LMU München CHEP 07 Victoria BC 06/09/2007.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition, Chapter 3: Process-Concept.
CPSC 171 Introduction to Computer Science System Software and Virtual Machines.
1  process  process creation/termination  context  process control block (PCB)  context switch  5-state process model  process scheduling short/medium/long.
HIGUCHI Takeo Department of Physics, Faulty of Science, University of Tokyo Representing dBASF Development Team BELLE/CHEP20001 Distributed BELLE Analysis.
PROOF and ALICE Analysis Facilities Arsen Hayrapetyan Yerevan Physics Institute, CERN.
Experiences Running Seismic Hazard Workflows Scott Callaghan Southern California Earthquake Center University of Southern California SC13 Workflow BoF.
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
General requirements for BES III offline & EF selection software Weidong Li.
© 2007 IBM Corporation Snehal S. Antani, WebSphere XD Technical Lead SOA Technology Practice IBM Software WebSphere.
AliRoot survey: Analysis P.Hristov 11/06/2013. Are you involved in analysis activities?(85.1% Yes, 14.9% No) 2 Involved since 4.5±2.4 years Dedicated.
15 December 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 15 th December 2000.
03/09/2007http://pcalimonitor.cern.ch/1 Monitoring in ALICE Costin Grigoras 03/09/2007 WLCG Meeting, CHEP.
Markus Frank (CERN) & Albert Puig (UB).  An opportunity (Motivation)  Adopted approach  Implementation specifics  Status  Conclusions 2.
Ch. 4 Memory Mangement Parkinson’s law: “Programs expand to fill the memory available to hold them.”
1 Build Your Own MySQL Time Machine Chuck Bell, PhD Mats Kindahl, PhD Replication and Backup Team Sun Microsystems 1.
Managing State Chapter 13.
Tree based validation tool for track reconstruction
LCGAA nightlies infrastructure
THINGS YOU SHOULD KNOW ABOUT JOB SCHEDULING. One such automation tool to run the essence of any organization or enterprise is Job scheduling. It is the.
湖南大学-信息科学与工程学院-计算机与科学系
Introduction to OS (concept, evolution, some keywords)
Introduction to OS (concept, evolution, some keywords)
Presentation transcript:

Parallel Reconstruction of CLEO III Data Gregory J. Sharp Christopher D. Jones Wilson Synchrotron Laboratory Cornell University

Outline Overview of CLEO reconstruction environment The problems with the old reconstruction system The solution - finer-grained parallelism The benefits

CLEO III Reconstruction Environment Uses a farm of more than 130 Sun Netras Sun Grid Engine ™ manages CPU allocation Events must be written to DB in event-number order Data read from & written to Objectivity/DB™ Reconstruction rate has to equal average DAQ rate

Former Reconstruction System Each run is processed in its entirety by a single CPU ~130 runs may be processed in parallel on the farm Up to 9 days to reconstruct a single run on a single CPU Output was written directly to the offline database All failures required operator and/or DBA intervention

Problems Need to maximize CPU utilization Load balancing between farms is difficult Takes a long time to stop the farm safely Output of the first few runs must be checked Debugging reconstruction code

More Problems Many locks held for long periods Low I/O rates to the database Large window for failures to occur Failure leaves database in an invalid state No automation of failure detection and recovery

The Solution Split each run into roughly equal-sized chunks Assign each chunk to a CPU Save sub-job output in intermediate binary files in event-number order Once all sub-jobs complete, collate binary files into database in event-number order

The Job Manager Once all reconstruction completes successfully the JM starts the collation sub-job The JM submits all the reconstruction sub-jobs and monitors their progress, retrying failures Once collation completes successfully the JM starts the merge histogram sub-job Can be restarted at any time if it dies

Structure Diagram

Automation A cron job generates status web pages Runs may be submitted automatically when SGE queue is (almost) empty JM restarts subjobs with transient failures

Implementation Details Written in Perl Uses Sun Grid Engine to submit and track jobs Uses CLEO III software infrastructure for reconstruction and population Uses PAW for merging histograms

Benefits The January ice storm Faster debugging Less operator intervention/management Increased CPU utilization, which offsets extra CPU use 20% Faster completion of reconstruction Just-in-time pre-staging of data from HSM file system

Future Steps Automate staging of data to cache disks Automate posting of staged runs info to Reconstruction

Conclusions Multiple file formats made this possible For more details: Substantial productivity gains Higher utilization of computing resources