1. Future EU Grid Projects

2. EGEE (slides by P. Clarke) SUGAR
Outline
EGEE (slides by P. Clarke)
SUGAR
In this lecture you will have an overview of the incoming features that will be provided by future releases of the EDG middleware and
will get a quick overview of the Web Services based distributed computing architecture

3. EGEE (Enabling Grids for E-science in Europe) Goal Applications
Create a European wide production quality Grid
Build on
EU and EU member states major investments in Grid Technology
International connections (US and AP)
Several pioneering prototype results
Approach
Bind national and regional Grid infrastructures
Procure and deploy robust middleware
Applications
EGEE
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.
Geant network

4. The historical analogy
EU Geant  binds national NRENs and creates a high performance production network for Europe
EGEE  will bind national Grid infrastructures - focussing all activities towards establishing a production quality Grid for Europe
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

5. The EGEE Consortium
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.
Total of 70 full partners covering entire EU and beyond
Total requested budget: ~35 M€ !

6. EGEE: Network Activities
Management & coordination
User Training and Induction;
User Support and Consultancy;
Applications Interface.
Dissemination and Outreach;
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

7. EGEE: Specific Service Activities
The structure of the Grid services will comprise:
EGEE Operations Management
EGEE Core Infrastructure Centres
Regional Operations Centres, responsible for coordinating regional resources, regional deployment and support of services in all other countries
Network services through Geant & NRENs
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

8. EGEE: Joint Research Activities
Middleware Engineering Centres for key services:
Resource Allocation and Access ;
Data Management ;
Information Collection and Accounting
Resource Brokering
Quality Assurance team and a Grid Security team.
Middleware Integration team and Middleware Testing Centre.
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

9. SUGAR
(Support User Grid Application communities on the EU Research infrastructure)
Goal: disseminate the Grid paradigm within science, e-society services and industry/business by porting specific applications to the European Grid Infrastructure (EGEE)
36 partners from many EU and quasi-EU states led by INFN
Total requested budget: ~1.7 M€
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

10. SUGAR Project Organization
9 Work Packages:
WP1: Astrophysics
WP2: Computational Chemistry
WP3: Digital Libraries
WP4: Earth Observation Applications
WP5: e-Learning Applications
WP6: Industrial Applications
WP7: Technical coordination and relations w/ other projects
WP8: Dissemination and Outreach
WP9: Project Office (management)
Interactive Jobs:
The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs).
Not supported: remote signal sending & asynchronous interaction with the job.
Possible extensions will be evaluated after first deployment phase.
See Condor Bypass (Grid Console)
For check-pointed jobs, theintermediate state is that previously saved.
Job Partitioning takes place when a job has to process a large set of “independent elements”.
In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources.
At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output.
This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

11. Summary and conclusions
Computational grids could represent the “natural” environment for next generation scientific, social and industrial applications.
“Grid” could be the Internet “new age” where users can seamlessly and ubiquitously access not only information but also huge computing resources and mass storage systems distributed worldwide with their own applications.
However, in order to turn dreams into reality, grid access must be pervasive, ubiquitous, easy and intuitive especially for the vast majority of non-expert users.
Next EU Grid Projects are just to address all these issues. Stay tuned or, much better, come and join thi new wave of progress!