Future EU Grid Projects

EGEE (slides by P. Clarke) SUGAR Outline EGEE (slides by P. Clarke) SUGAR In this lecture you will have an overview of the incoming features that will be provided by future releases of the EDG middleware and will get a quick overview of the Web Services based distributed computing architecture

EGEE (Enabling Grids for E-science in Europe) Goal Applications Create a European wide production quality Grid Build on EU and EU member states major investments in Grid Technology International connections (US and AP) Several pioneering prototype results Approach Bind national and regional Grid infrastructures Procure and deploy robust middleware Applications EGEE Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism. Geant network

The historical analogy EU Geant  binds national NRENs and creates a high performance production network for Europe EGEE  will bind national Grid infrastructures - focussing all activities towards establishing a production quality Grid for Europe Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

The EGEE Consortium Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism. Total of 70 full partners covering entire EU and beyond Total requested budget: ~35 M€ !

EGEE: Network Activities Management & coordination User Training and Induction; User Support and Consultancy; Applications Interface. Dissemination and Outreach; Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

EGEE: Specific Service Activities The structure of the Grid services will comprise: EGEE Operations Management EGEE Core Infrastructure Centres Regional Operations Centres, responsible for coordinating regional resources, regional deployment and support of services in all other countries Network services through Geant & NRENs Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

EGEE: Joint Research Activities Middleware Engineering Centres for key services: Resource Allocation and Access ; Data Management ; Information Collection and Accounting Resource Brokering Quality Assurance team and a Grid Security team. Middleware Integration team and Middleware Testing Centre. Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

SUGAR (Support User Grid Application communities on the EU Research infrastructure) Goal: disseminate the Grid paradigm within science, e-society services and industry/business by porting specific applications to the European Grid Infrastructure (EGEE) 36 partners from many EU and quasi-EU states led by INFN Total requested budget: ~1.7 M€ Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

SUGAR Project Organization 9 Work Packages: WP1: Astrophysics WP2: Computational Chemistry WP3: Digital Libraries WP4: Earth Observation Applications WP5: e-Learning Applications WP6: Industrial Applications WP7: Technical coordination and relations w/ other projects WP8: Dissemination and Outreach WP9: Project Office (management) Interactive Jobs: The connection between WN and UI is always open from the job (we assume OutBoundIP connectivity available from WNs). Not supported: remote signal sending & asynchronous interaction with the job. Possible extensions will be evaluated after first deployment phase. See Condor Bypass (Grid Console) http://www.cs.wisc.edu/condor/bypass For check-pointed jobs, theintermediate state is that previously saved. Job Partitioning takes place when a job has to process a large set of “independent elements”. In these cases it may be worthwhile to decompose the job into smaller sub-jobs (which can be executed in parallel), in order to reduce the overall time needed to process all these elements, and to optimize the usage of all available Grid resources. At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output. This problem is being addressed in the context of job checkpointing and makes large use of the DAGMan mechanism.

Summary and conclusions Computational grids could represent the “natural” environment for next generation scientific, social and industrial applications. “Grid” could be the Internet “new age” where users can seamlessly and ubiquitously access not only information but also huge computing resources and mass storage systems distributed worldwide with their own applications. However, in order to turn dreams into reality, grid access must be pervasive, ubiquitous, easy and intuitive especially for the vast majority of non-expert users. Next EU Grid Projects are just to address all these issues. Stay tuned or, much better, come and join thi new wave of progress!