1. Production Manager Tools (New Architecture)
LHCb Data Management Meeting 30 September 2003
Gennady Kuznetsov

2. Lessons Learnt © Andrei Tsaregorodtsev
DataGRID instability problems persisting :
MDS information system failures ;
Site miss configuration ;
Outbound IP connectivity is not available on all sites :
Needed for the LHCb software installation on per job basis ;
Needed for jobs exchanging messages with production services .
Data transfer :
bbftp file transport replaced by the replica manager tools ;
Next run of tests will hopefully be more successful :
better understanding of the system ;
Use other schema for job delivery to a worker node .
© Andrei Tsaregorodtsev
Presenter Name: Gennady Kuznetsov
Facility Name: PPD

3. Other Schema Other schema Flexible component architecture
Full forward and backward compatibility of jobs.
No needs for installation of the control or monitoring software on the node or site
Workflows, Steps, Modules stored on server or in Production Manager Library as XML file. Job is Python code generated from XML during submission.
Monitoring code got embedded into Jobs automatically
Agent (for development)
Agents can be run on any computer local or remote
Single agent can control one or many sites simultaneously
Agent implements Andrey’s pool job submission schema.
Presenter Name: Facility Name:

4. Components Step Workflow Production Module Job Step
We add Module as additional component of the architecture
Module
Step
Levels of usage:
Module – Programmer
Step – Production manager
Workflow – User
Step
Workflow
Job
Production
This module structure allow as to construct any algorithm as combination of modules.
Each step generates job as a Python program.
Presenter Name: Facility Name:
Presenter Name: Gennady Kuznetsov
Facility Name: PPD

5. Python code of single module. Can be many classes.
Module Editor
Python code of single module. Can be many classes.
Module Name
Stored as XML file
Description
Module variables.
Presenter Name: Facility Name:

6. Step Editor Stored as XML file, where all modules are embedded
Graphical representation of Module relationship (not ready)
Stored as XML file, where all modules are embedded
Step Name
Description
Variables of currently selected instance
Definitions of Modules
Instances of Modules
Selected instance
Presenter Name: Facility Name:

7. Workflow Editor Not ready yet.
It will links all jobs in a workflow instance.
The simplest workflow is a single step.
It will combine workflow and “Vector” and generate as many instances of workflow as many members in a vector. For example it can be list of numbers (job number or name) or it can be a list of files for reconstructions or analysis.
Presenter Name: Facility Name:

8. Architecture Central Services DIRAC Personal Desktop Node IDL+Python
scoreboard
Architecture
Production Manager
Agent on the Node replaced by
automatically generated Python program
No any software required except PYTHON
Central Services
DIRAC Personal Desktop
Node
Node
Grid Front End
IDL+Python
via SSH
Node
DIRAC Personal Agent
Node
Node
Batch Front End
shell+Python
via SSH
Node
Presenter Name: Gennady Kuznetsov
Facility Name: PPD

9. DIRAC Personal Desktop
Language C++ with QT
Provides tools to create Module Library for the experiment (for Programmer).
Provides tools to construct Steps and Workflows from Module Library. (for Production Manager)
Saves all objects as XML files.
Generates batch of Python jobs for the submission into GRID or Batch System.
DIRAC Personal Desktop
Module Editor
Step Editor
Workflow Editor
Production Manager
Monitoring Manager
Code Generator
Presenter Name: Gennady Kuznetsov
Facility Name: PPD

10. Personal Agent Language Java.
Represent User or Production Manager in the network on a permanent basis (jobs does not crash without agent).
Provides means to submit and monitor jobs to any number of Batch or GRID front ends via SSH of local calls.
Provides local disk space for data retrieval.
Allow job dependency.
DIRAC Personal Agent
Job Dependency Service
Job Submission Service
Internal Monitoring Srv.
Data Retrieval Service
Local Disk
Job pool
New
Not ready
Ready
Submitted
Running
Output ready
Done
Job states:
Presenter Name: Gennady Kuznetsov
Facility Name: PPD

11. Central Services (CERN)
Job Pool
Prepackaged Applications archive
Central Services
Production Service
Monitoring Service
Bookkeeping Service
Storage Service
Application distr. ser.
Data base with web front-end
Oracle at CERN
CASTOR or Replica Manager
Presenter Name: Gennady Kuznetsov
Facility Name: PPD