Parameter Study Principles & Practices. Outline Data Model of the PS Part I Simple PS –Generating simple PS Workflow by introducing PS Input port – using.

Slides:



Advertisements
Similar presentations
Network II.5 simulator ..
Advertisements

1 P-GRADE Portal and GEMLCA Legacy Code Architecture Peter Kacsuk MTA SZTAKI
Short Portlet Programming Guide in P-Grade (Gridsphere) environment G. Hermann.
LPDS MTA SZTAKI Equation Solver A Grid solution Important hint: A follow up example demonstrating the Parameter Study solution for the mass calculations.
1 CA202 Spreadsheet Application Combining Data from Multiple Sources Lecture # 6.
GUI Testing. High level System Testing Test only those scenarios and outputs that are observable by the user Event-driven Interactive Two parts to test.
CHAPTER 30 THE HTML 5 FORMS PROCESSING. LEARNING OBJECTIVES What the three form elements are How to use the HTML 5 tag to specify a list of words’ form.
WS-PGRADE: Supporting parameter sweep applications in workflows Péter Kacsuk, Krisztián Karóczkai, Gábor Hermann, Gergely Sipos, and József Kovács MTA.
Pseudo dynamic DAG control Version 1. Outline Goal Solution Restrictions Example Case Study.
Guide to Oracle10G1 Introduction To Forms Builder Chapter 5.
A Guide to Oracle9i1 Introduction To Forms Builder Chapter 5.
A Guide to Oracle9i1 Creating an Integrated Database Application Chapter 8.
Chapter 6. 2 Objectives You should be able to describe: Function and Parameter Declarations Returning a Single Value Pass by Reference Variable Scope.
Macromedia Dreamweaver 4 Advanced Level Course. Add Rollovers Rollovers or mouseovers are possibly the most popular effects used in designing Web pages.
A Visual Introduction to PC SAS. Start SAS by double-clicking on the SAS icon...
CHAPTER 6 FILE PROCESSING. 2 Introduction  The most convenient way to process involving large data sets is to store them into a file for later processing.
1 Chapter 20 — Creating Web Projects Microsoft Visual Basic.NET, Introduction to Programming.
Microsoft Visual Basic 2008 CHAPTER ELEVEN Multiple Classes and Inheritance.
FALL 2005CSI 4118 – UNIVERSITY OF OTTAWA1 Part 4 Web technologies: HTTP, CGI, PHP,Java applets)
WS-PGRADE Akos Balasko LPDS MTA SZTAKI. Content General Overview Parts of System Activities (of advanced and of common user) Simplified Graph of the hierarchy.
© Cheltenham Computer Training 2001 Macromedia Dreamweaver 4 - Slide No 1 Macromedia Dreamweaver 4 Advanced Level Course.
Simple Program Design Third Edition A Step-by-Step Approach
Chapter 8: Writing Graphical User Interfaces
CHAPTER 9 Introducing Microsoft Office Learning Objectives Start Office programs and explore common elements Use the Ribbon Work with files Use.
Copyright © 2007, Oracle. All rights reserved. Managing Concurrent Requests.
A First Book of C++: From Here To There, Third Edition2 Objectives You should be able to describe: Function and Parameter Declarations Returning a Single.
StAR web server tutorial for ROC Analysis. ROC Analysis ROC Analysis: This module allows the user to input data for several classifiers to be tested.
Eurotrace Hands-On The Eurotrace File System. 2 The Eurotrace file system Under MS ACCESS EUROTRACE generates several different files when you create.
CPSC 203 Introduction to Computers Lab 23 By Jie Gao.
Visual Basic.NET Comprehensive Concepts and Techniques Chapter 5 Decision Making.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI Creating the Autodock gateway from WS-PGRADE/gUSE and making it cloud-enabled.
Tools Menu and Other Concepts Alerts Event Log SLA Management Search Address Space Search Syslog Download NetIIS Standalone Application.
CSC 3210 Computer Organization and Programming Chapter 1 THE COMPUTER D.M. Rasanjalee Himali.
OCC Network Drives  H:\  P:\ 
Active Server Pages  In this chapter, you will learn:  How browsers and servers interacted on the Internet when the Internet first became popular 
LPDS MTA SZTAKI Equation Solver Light A Grid solution.
Parameter Study (PS) Principles & Practices Note This document uses as starting example the non PS case discussed in document EquationSolver.ppt If you.
1 Extend is a simulation tool to create models quickly, with all the blocks you need and without even having to type an equation. You can use a series.
ITCS373: Internet Technology Lecture 5: More HTML.
Working with the VB IDE. Running a Program u Clicking the”start” tool begins the program u The “break” tool pauses a program in mid-execution u The “end”
Introduction to Programming with RAPTOR
The EDGeS project receives Community research funding 1 Porting Applications to the EDGeS Infrastructure A comparison of the available methods, APIs, and.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Status report on Application porting at SZTAKI.
Parameter Study Principles & Practices. What is Parameter Study? Parameter study is the application of a single algorithm over a set of independent inputs:
Overview of the Automated Build & Deployment Process Johnita Beasley Tuesday, April 29, 2008.
Class Builder Tutorial Presented By- Amit Singh & Sylendra Prasad.
Lesson 3-Touring Utilities and System Features. Overview Employing fundamental utilities. Linux terminal sessions. Managing input and output. Using special.
1 P-GRADE Portal tutorial at EGEE’09 Introduction to hands-on Gergely Sipos MTA SZTAKI EGEE.
CS1Q Computer Systems Lecture 8
EGEE-II INFSO-RI Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows & parameter sweeps (Advanced features)
1 Support for Parameter Study applications in the P-GRADE Portal Cevat Şener Dept. Of Computer Engineering, METU.
JavaScript Introduction and Background. 2 Web languages Three formal languages HTML JavaScript CSS Three different tasks Document description Client-side.
1 P-GRADE Portal hands-on Gergely Sipos MTA SZTAKI Hungarian Academy of Sciences.
: Information Retrieval อาจารย์ ธีภากรณ์ นฤมาณนลิณี
07/02/2012 WS-PGRADE/gUSE in use Lightweight introduction Zoltán Farkas MTA SZTAKI LPDS.
WS-PGRADE/gUSE in use Advance use of WS- PGRADE/gUSE gateway framework Zoltán Farkas and Peter Kacsuk MTA SZTAKI LPDS.
1 Berger Jean-Baptiste
Remote Api Tutorial How to call WS-PGRADE workflows from remote clients through the http protocol?
1 Support for parameter study applications in the P-GRADE Portal Gergely Sipos MTA SZTAKI (Hungarian Academy of Sciences)
InSilicoLab – Grid Environment for Supporting Numerical Experiments in Chemistry Joanna Kocot, Daniel Harężlak, Klemens Noga, Mariusz Sterzel, Tomasz Szepieniec.
P-GRADE overview and introduction: workflows & parameter sweeps (Advanced features) Gergely Sipos MTA SZTAKI
Grid Application Support Group Case study Schrodinger equations on the Grid Status report 16. January, Created by Akos Balasko
Remote Api Tutorial How to call WS-PGRADE workflows from remote clients through the http protocol?
Lightweight introduction
Lightweight introduction
Multi-host Internet Access Portal (MIAP) Enhancement Guide
Using Templates and Library Items
Using SQL*Plus.
Capitolo 1 – Introduction C++ Programming
Workflow level parametric study support by the P-GRADE portal
Presentation transcript:

Parameter Study Principles & Practices

Outline Data Model of the PS Part I Simple PS –Generating simple PS Workflow by introducing PS Input port – using existing inputs Part II Complex PS –Making inputs for ourselves by Generator and Auto Generator Jobs using PS Output ports –Evaluate the results by Collector jobs using Special PS Input ports Appendix1: Auto Generator (separate example) Appendix2: Summary of overall PS execution Appendix3: Classification of possible PS outputs

PS_1_2 PS_1_1 PS_2_2 PS_2_1 PS_1_1PS_2_1PS_1_2PS_2_1PS_1_1PS_2_2PS_1_2PS_2_2 WF One or more Parameter Sets are defined. Each has different parameters. (Sets are represented by remote file directories, parameters by files) PS DATA FLOW Content of Parameter Sets are combined (Descartes product) Each combination will compose an input of an independent workflow Result_1_1 Result_2_1 Result_1_2 Result_2_2 Store the results of the calculations (Results are represented by remote files of one or more common directories)

Part I :Simple PS Start from a tested Workflow Activity 1: Change Input Ports to PS Input Port Activity2: Define placement of results

Part I: Example to start from: an simple WF It calculates (and checks the accuracy the calculation of) x of the matrix equation A*x=B, where the job Separator separates the value of A from B recognizing a separator # character in the common input file. AB A -1 A -1 *B=xA*xA*x - B

Input Port of the original workflow Changing the input of the workflow we want to define a container for a set of existing input files upon the workflow should be independently executed.

Creating a PS Input Port

Simple PS introducing the concept of PS Input Port PS Input port distinguished by dark green PS Input Port defines a container of input files in order to elaborate them separately by different turns of the workflow submissions. The container is represented by remote file directories. If there are more PS Input Ports the combination of all possibilities (Descartes product ) will be calculated.

PS Input Port of Simple PS Remote file Directory instead of FILE reference Do not use the prefix lfn: if the directory is EGEE Grid file catalogue

Simple PS Activity 2: placement of result If the Grid is LCG (EGEE) like, then the properties of the Grid File Catalog must be defined Menu item PS Properties can be called within the Workflow menu The Output directory will contain the set of individual compressed files. Each compressed file contains the outputs of an element Workflow have been elaborated over an item of the PS Input Set Do not use the prefix lfn: if the directory is EGEE Grid file catalogue

Workflow Manager List (base view) Only the different label on the button reveals that this is not a common but a PS Workflow

Workflow Manager List PS Details view showing eWF-s New, middle level list to render the details of a PS Workflow Statistics shows the progress of the elaboration of the whole PS The eWorkflow buffer list shows the state of the Workflows being processed.

Details view of the eWF Ax_EQU_B_voce_PS.6 Job level details of an eWorkflow See, that the button Attach is missing as there is not to much importance to access the WE until the eWorkflow list is exhausted

Part II:Data Model of advanced PS Gen A_Gen Input Coll Output Coll Output Coll Output The Autogenerator is a special job: May have no input Runs on the Portal

Creating a Generator (or Collector job) A Generator (or a Collector) job can be created if there is any PS property of the Workflow. A single PS input port indicates a PS property We want to create the inputs of the PS in a common frame, therefore we need a Generator

Creating a Generator (or Collector job) A Generator (or a Collector) job can be created if there is any PS property of the Workflow. A single PS input port indicates a PS property We want to create the inputs of the PS in a common frame, therefore we need a Generator

Creating an Auto Generator job from a common Generator job There are two subtypes of Generators: 1.“Normal” with user defined executable 2. “Automatic” a special macro processor running on the Portal Server

PS with Generator A generator job will be submitted before any PS jobs, and generates a set of parameter files gathered in a single remote file directory defined by a PS Output Port Generator jobs can be user defined (then it is the responsibility of the user defined executable of the Generator job to produce the parameter files in required way and number ) or may be “auto generated”. (See detailed on the next slides) PS Output Port must be connected to one or more PS Input Port(s)

Main Definition Window of Auto Generator Job Auto Generator is special macro processor. Keys define sets, whose elements will be substituted in the Input file text. The possible combinations are written in separate files and determine the Parameter Set. Note: a key occuring more than one times in the Input file text will be replaced by the same key set value in that element of the Parameter Set Hitting on a key opens the key definition window

A sub window for key definition of the Auto Generator Job There are four different ways to define the set of a distinguished key in our example of Reals Of B The values of the key set must have a common type, and they may be formatted Button Generate interprets the key set generation and enables a preview of the result in the list Generated Items

(Auto) Generator Attribute Editor for SE definition Attribute Editor defines the properties of remote files created by the Generator: 1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator

(Auto) Generator Attribute Editor for Grid File Catalog Attribute Editor defines the properties of remote files created by the Generator: 2. File Catalog Access must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator Note: The defined path (in the Generator PS output Port) will be automatically established whenever it is needed and possible. The user need not to bother with it!

(Auto) Generator PS Output Port Definition Remote file directory instead of FILE reference Do not use the prefix lfn: if the directory is EGEE Grid file catalogue

Workflow manager Details Window with terminated Auto Generator List of generators Buffer of the element workflows being observed and elaborated

PS with Auto Generator and Collector New job type Collector Runs after that all element workflows have terminated. It has a special PS Input port represented by a local directory. This local directory gets a copy of a remote file has been generated during each run of the preceding element workflows. It is the responsibility of user defined executable to read and elaborate the files of the local directory Collector PS Input Port is indicated by light green color

Remote Output Port connectable to a Collector File associated to a Collector PS Input port must be Remote. Its actual name will be extended by the name of the element Workflow You must use here the prefix lfn: if the directory is EGEE Grid file catalogue

Collector PS Input Port The inherited File Prefix name indicates that the names of the remote files to be generated will be extended by the names of the identifier of the corresponding element workflows Internal File Name is the prefix of copied local input files to be elaborated by the collector

A Collector Job Job Executable is user defined. It must be able to read and elaborate all the files (with the prefix name defined in the Collector PS Input Port)

Workflow Management Window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

Workflow Management Window showing terminated Generator and Collector The Output(s) of the collector(s) can be downloaded. Special case is if the WF results –or a fraction of them – may not be stored remotely. A log message (missing here) indicates the case and the user can download them together with the result of the Collector

Appendix 1 Auto Generator Summary Let be the template (represented by the Input file text ) “aXbYcX” and let be the substitution sets (represented by Keys) X={2,3}; Y={6,7} where a,b,c are final strings. Then supposing that the InternalFileName is “output” in the PS Output Port of the Generator then the generated grid files would look like: File NameContent output.1.1a2b6c2 output.1.2a2b7c2 output.2.1a3b6c3 output.2.1a3b7c3

Implementation of the example Auto Generator

Appendix2 Summary of overall PS execution First phase executed in Generator time Last phase executed in Collector time Second phase executed in eWorkflow submission time until the eWorkflow list is exhausted

Appendix3 Classification of possible PS outputs 1.Remote files with generated eWorkflow name extension – each for a remote output file of an eWorkflow (This sort of output is used as Collector input) 2.From the Portal downloadable compressed file of the eventual Collector results and remaining eventual eWorkflow results (See 3.) reluctant to be stored remotely. 3.Compressed remote files – each for the local output files of an eWorkflow (See menu item PS Properties of the Workflow Editor)