1. LEAD-VGrADS
Day 1 Notes

2. Introduction & discussion
Workflow discussion
Interaction points
Higher level resource requirements
Where does bind happen?
WCS is being developed right now, timeline not clear
Does vgLaunch need to be retrofitted for a web service call?
LEAD is BPEL service based – for FT, adaptation, etc
GT4 toolkit is also web service based

3. LEAD is moving to GT4 WS-GRAM, notifications are in different format from LEAD workflow
LEAD has focus on deployment and integration, very application focused
LEAD might use more than one scheduler/resource solutions
ISI (Ann/Ewa) are working with LEAD on the data side (DRS, RLS), domain side (THREDS)
WCS handles application configuration such as namelist files

4. What do each of us need? Interfaces between LEAD and VGrADS?
Performance Model?
What in LEAD communicates with VGrADS? And what does it give to VGrADS?
Interfaces between vgES and batch queue prediction
What requirements does LEAD have and does it conflict with vgES philosophy?
New features to implement in vgES inline with deadline for SC06

5. Short-term and long-term integration plan
How do I use vgES?
From LEAD workflow – concrete example of LEAD workflow?
How to use batch queue from vgES? Batch queue waiting time aware
Interface points are crucial
Full ahead planning – resources might not be available at this time.
In LEAD the first few tasks are light, last are intensive. So not much time lead time
LEAD today will launch immediately. What if you start running before or after you expect to?

6. There is post-processing steps once we know what resources are needed
Each workflow task – needs to be on a cluster, will need shared filesystem, good network connectivity

7. Practical resource constraints
Where does and where can LEAD run today? (What are the real resource choices)
WRF does not run on Itanium2
Have dependencies
lot of libraries, binaries, compilers
Today
NCSA machines: copper, tungsten
SDSC: tglogin
IU teragrid
PSC (luemix?)
LEAD testbed (?)
All GridFTP and GRAM enabled
How about UC and UT machines?
Possibilities in vgES
UNC
RTC supports GRAM but is Itanium2
Rice has another cluster. Can it be made Grid aware?
Other sites?
Credentials under which account – drlead account.
vgES developers get access to drlead

8. We need to use LEAD workflow engine
Scheduler makes assumption that queue prediction is accurate for first architecture interactions.
What do we do when we are not accurate?
vgES select and bind is tight
Dan’s perl workflow doesn’t have notion of when to launch
Having a deadline changes the way we look at this
We need to submit job right at some time based on current wait queue times
Prediction is used for which queue to submit, cannot use that to predict (selection happens apriori)
Can’t guarantee when we can tell where to run resources
What is the interface
Scheduler is static right now. Is dynamic scalable?

9. vgLaunch might or not do WS-GRAM
UNC/Lavanya will have to provide Suresh a standard interface to ask for resources and to submit jobs. Lavanya will turn around and do vgLaunch
Data transfer is tightly coupled with computation. Can it be considered as a separate task? It is handled by myLEAD independently
vgLaunch might or not do WS-GRAM
How will we get the WS-GRAM notifications back through vgES?
Application service is doing file staging (RFT didn’t seem to work right now)
Notifications for files that are generated is based on job completion (this will need to change with file streaming)
Data mining, visualization, etc
Does vgMon help with some notification?

10. LEAD Data Management WE first registers the topic ID with myLEAD
LEAD publishes events on the data status (task A has input data)
myLEAD agent contacts DRS (which is managing the data) and tells it what to do with the data
When a task gets a unique id (resolves to GridFTP) it resolves to the best possible input source

11. Everything has to happen through this. But this is changing
Portal
(user uses GUI
to compose wf)
App
Service
(AS)
Everything has to happen
through this. But this is changing
Refer to Suresh’s diagram (in Lavanya’s Intro slides)
LEAD Today
The service (a wrapper) in LEAD need not be on the same cluster
Right now LEAD submits sequential
Is engine/application service smart enough to understand when resources are available
How does LEAD deal with resources arrival times?
The BPEL engine itself is just event based execution. Doesn’t have notion of time.
The application service manages the state. How will an application service be called to launch into a queue before it runs (breaks the BPEL sequential call)
LEAD does the PBS job submission
What is the model for vgES launch?

12. After Lunch Demo Few slides from Suresh to understand data parts
Service map of application service
Workflow DAG
What do you want to see the demo part?
More about the interfaces