LEAD-VGrADS Day 1 Notes

Introduction & discussion Workflow discussion Interaction points Higher level resource requirements Where does bind happen? WCS is being developed right now, timeline not clear Does vgLaunch need to be retrofitted for a web service call? LEAD is BPEL service based – for FT, adaptation, etc GT4 toolkit is also web service based

LEAD is moving to GT4 WS-GRAM, notifications are in different format from LEAD workflow LEAD has focus on deployment and integration, very application focused LEAD might use more than one scheduler/resource solutions ISI (Ann/Ewa) are working with LEAD on the data side (DRS, RLS), domain side (THREDS) WCS handles application configuration such as namelist files

What do each of us need? Interfaces between LEAD and VGrADS? Performance Model? What in LEAD communicates with VGrADS? And what does it give to VGrADS? Interfaces between vgES and batch queue prediction What requirements does LEAD have and does it conflict with vgES philosophy? New features to implement in vgES inline with deadline for SC06

Short-term and long-term integration plan How do I use vgES? From LEAD workflow – concrete example of LEAD workflow? How to use batch queue from vgES? Batch queue waiting time aware Interface points are crucial Full ahead planning – resources might not be available at this time. In LEAD the first few tasks are light, last are intensive. So not much time lead time LEAD today will launch immediately. What if you start running before or after you expect to?

There is post-processing steps once we know what resources are needed Each workflow task – needs to be on a cluster, will need shared filesystem, good network connectivity

Practical resource constraints Where does and where can LEAD run today? (What are the real resource choices) WRF does not run on Itanium2 Have dependencies lot of libraries, binaries, compilers Today NCSA machines: copper, tungsten SDSC: tglogin IU teragrid PSC (luemix?) LEAD testbed (?) All GridFTP and GRAM enabled How about UC and UT machines? Possibilities in vgES UNC RTC supports GRAM but is Itanium2 Rice has another cluster. Can it be made Grid aware? Other sites? Credentials under which account – drlead account. vgES developers get access to drlead

We need to use LEAD workflow engine Scheduler makes assumption that queue prediction is accurate for first architecture interactions. What do we do when we are not accurate? vgES select and bind is tight Dan’s perl workflow doesn’t have notion of when to launch Having a deadline changes the way we look at this We need to submit job right at some time based on current wait queue times Prediction is used for which queue to submit, cannot use that to predict (selection happens apriori) Can’t guarantee when we can tell where to run resources What is the interface Scheduler is static right now. Is dynamic scalable?

vgLaunch might or not do WS-GRAM UNC/Lavanya will have to provide Suresh a standard interface to ask for resources and to submit jobs. Lavanya will turn around and do vgLaunch Data transfer is tightly coupled with computation. Can it be considered as a separate task? It is handled by myLEAD independently vgLaunch might or not do WS-GRAM How will we get the WS-GRAM notifications back through vgES? Application service is doing file staging (RFT didn’t seem to work right now) Notifications for files that are generated is based on job completion (this will need to change with file streaming) Data mining, visualization, etc Does vgMon help with some notification?

LEAD Data Management WE first registers the topic ID with myLEAD LEAD publishes events on the data status (task A has input data) myLEAD agent contacts DRS (which is managing the data) and tells it what to do with the data When a task gets a unique id (resolves to GridFTP) it resolves to the best possible input source

Everything has to happen through this. But this is changing Portal (user uses GUI to compose wf) App Service (AS) Everything has to happen through this. But this is changing Refer to Suresh’s diagram (in Lavanya’s Intro slides) LEAD Today The service (a wrapper) in LEAD need not be on the same cluster Right now LEAD submits sequential Is engine/application service smart enough to understand when resources are available How does LEAD deal with resources arrival times? The BPEL engine itself is just event based execution. Doesn’t have notion of time. The application service manages the state. How will an application service be called to launch into a queue before it runs (breaks the BPEL sequential call) LEAD does the PBS job submission What is the model for vgES launch?

After Lunch Demo Few slides from Suresh to understand data parts Service map of application service Workflow DAG What do you want to see the demo part? More about the interfaces