Ganga/Dirac Data Management meeting October 2003 Gennady Kuznetsov Production Manager Tools and Ganga (New Architecture)

Job Control & Monitoring Gennady Kuznetsov Requirements for the Job Control and Monitoring script 1.Interpreter. 2.Platform independent. 3.Well developed language with a rich API. 4.Object-oriented (we need separate variable scopes). Candidates: Python, Java, Rugby, shells We chose Python. (Java required compilation)

Current Design Gennady Kuznetsov Job Configuration, Data Production, Bookkeeping - Design Document, M. Frank (Editor), 04/12/2001

Components Module Step Workflow Job Production Levels of usage: 1.Module – Programmer 2.Step – Production manager 3.Workflow – User We add Module as additional component of the architecture Each step generates job as a Python program. This module structure allow us to construct any algorithm as a combination of modules. Gennady Kuznetsov

Module Gennady Kuznetsov Variable VariableCollection Module ModuleDefinitionModuleInstance n 1 StepDefinition n 1 n 1 Example of the generated code # Module Instance=iterration3 Type=PrintTest iterration3 = PrintTest() iterration3.value1 = iterration1.result # type=int iterration3.value2 = iterration2.result # type=double iterration3.execute() Each variable has Name Type (for information only) Value Stated as “input” & “output” Name of the instance of linked module Name of the variable of linked module Description ModuleInstances can be connected with each other and forms a Step. Each Step can be instantiated as a single Job

Module Editor Python code of single module. Can be many classes. Module variables. Description Module Name Stored as XML file Gennady Kuznetsov

Step Editor Step Name Description Definitions of Modules Instances of Modules Graphical representation of Module relationship (not ready) Variables of currently selected instance Selected instance Stored as XML file, where all modules are embedded Gennady Kuznetsov

Workflow Editor Not ready yet. It will link all jobs in a workflow instance. The simplest workflow is a single step. It will combine workflow and “Vector” and generate as many instances of workflow as many members in a vector. For example, it can be list of numbers (job number or name) or it can be a list of files for reconstruction or analysis. Gennady Kuznetsov

Layered Architecture Gennady Kuznetsov Module Module Library Step Workflow Module Editor Editors Step Editor Workflow Editor Programmer Programmer & Production Manager Production Manager & User Ganga Workflow Editor with limited functionality (configuration only) User Module library is a collection of XML files on shared file system (like AFS). Can be updated by anyone. It give us a lot of flexibility and freedom.

Combining all together Gennady Kuznetsov Flexible part of the code Permanent part of the code Integrated Environment Ganga User Interface to Workflow Job Execution Code Ganga Dirac Editors Module library Workflow Repacked as Modules Used as it is now Creates Use an interface specification from Workflow

Complete Structure Gennady Kuznetsov Ganga UI Permanent Module Library Flexible & Experiment dependent Dirac Editors Permanent Grid, Batch or Submission Service Submit Python job Use Create and use Submit Python job

What To Do? Gennady Kuznetsov If this proposal would be accepted: For the Data challenge (MC Simulation) 1.Extract execution code from Ganga and reorganize it as a modular structure and pack it as a Module Library using Dirac Editors. (Ganga, Dirac) 2.To make appropriate modification in the Job Handler class. (Ganga) For the User Analysis 1.Design formal specification for the User Interface (XML?) 2.Implement User Interface Rendering Engine (Ganga) 3.Reserve field in Workflow (XML) and design editing tool as part of the Workflow Editor (Dirac).

Collaboration with Ganga Gennady Kuznetsov Flexibility Save initial project development time. Good base for the code evolution. Wide and robust testing. Efficient code maintenance. Satisfy users with wide requirements. Combine development power of two groups Modular structure Reusable code Layered architecture. No functional limitations specified at the 1 st level. Modular architecture of the executables. The same modules could be tested in different experiments or by different groups. Service standardisation Code sharing 3 Levels of GUI (Programmer, Production manager, End user).