EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Steven Newhouse Technical Director Moving.

Slides:



Advertisements
Similar presentations
EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
Advertisements

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Francesco Giacomini JRA1 Activity Leader.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE-III Program of Work Erwin Laure EGEE-II / EGEE-III Transition Meeting CERN,
EGI: A European Distributed Computing Infrastructure Steven Newhouse Interim EGI.eu Director.
EGEE is a project funded by the European Union under contract IST JRA1 Testing Activity: Status and Plans Leanne Guy EGEE Middleware Testing.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks From ROCs to NGIs The pole1 and pole 2 people.
EMI INFSO-RI EMI Quality Assurance Processes (PS ) Alberto Aimar (CERN) CERN IT-GT-SL Section Leader EMI SA2 QA Activity Leader.
EMI INFSO-RI SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader EMI First EC Review 22 June 2011, Brussels.
INFSO-RI Enabling Grids for E-sciencE SA1: Cookbook (DSA1.7) Ian Bird CERN 18 January 2006.
Apr 30, 20081/11 VO Services Project – Stakeholders’ Meeting Gabriele Garzoglio VO Services Project Stakeholders’ Meeting Apr 30, 2008 Gabriele Garzoglio.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Steven Newhouse EGEE’s plans for transition.
Overview & Status of the Middleware & EGI Core Proposals Steven Newhouse Interim EGI Director EGEE Technical Director 26/05/2016 Status of EGI & Middleware.
European Middleware Initiative (EMI) – Release Process Doina Cristina Aiftimiei (INFN) EGI Technical Forum, Amsterdam 17. Sept.2010.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Oliver Keeble SA3 Activity Leader CERN EGEE-III.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks, An Overview of the GridWay Metascheduler.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks JRA1 summary Claudio Grandi EGEE-II JRA1.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Steven Newhouse Technical Director CERN.
INFSO-RI Enabling Grids for E-sciencE SA1 and gLite: Test, Certification and Pre-production Nick Thackray SA1, CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Oliver Keeble SA3 Activity Leader CERN EGEE-III.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks gLite Build Programme and Multi-Platform.
JRA Execution Plan 13 January JRA1 Execution Plan Frédéric Hemmer EGEE Middleware Manager EGEE is proposed as a project funded by the European.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE-EGI Grid Operations Transition Maite.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks CERN status report SA3 All Hands Meeting.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Bob Jones EGEE project director CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks SA1: Grid Operations Maite Barroso (CERN)
EGEE-III-INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE-III All Activity Meeting Brussels,
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Gergely Sipos Activity Deputy Manager MTA.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Antonio Retico CERN, Geneva 19 Jan 2009 PPS in EGEEIII: Some Points.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Steven Newhouse Technical Director CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGI Operations Tiziana Ferrari EGEE User.
EGI-InSPIRE Steven Newhouse Interim EGI.eu Director EGI-InSPIRE Project Director Technical Director EGEE-III 1GDB - December 2009.
European Middleware Initiative (EMI) The Software Engineering Model Alberto Di Meglio (CERN) Interim Project Director.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Robin McConnell NA3 Activity Manager 28.
EMI INFSO-RI European Middleware Initiative (EMI) Alberto Di Meglio (CERN)
INFSO-RI Enabling Grids for E-sciencE /10/20054th EGEE Conference - Pisa1 gLite Configuration and Deployment Models JRA1 Integration.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Update Authorization Service Christoph Witzig,
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks SA3 partner collaboration tasks & process.
EGEE-III INFSO-RI Enabling Grids for E-sciencE SA3 All Hands Meeting 'Cluster of Competence' Experience SA3 INFN Cyprus May 7th-8th.
EGEE is a project funded by the European Union under contract IST Roles & Responsibilities Ian Bird SA1 Manager Cork Meeting, April 2004.
EMI INFSO-RI European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE JRA1 All Hands Meeting July 10-12, 2006 Pilsen, CZ.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Middleware Update Maria Alandes Pradillo.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Operations Automation Team Kickoff Meeting.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Patch Preparation SA3 All Hands Meeting.
EMI Inter-component and Large Scale Testing Infrastructure Danilo Dongiovanni INFN-CNAF.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MSA3.4.1 “The process document” Oliver Keeble.
INFSO-RI Enabling Grids for E-sciencE gLite Test and Certification Effort Nick Thackray CERN.
EGEE-II INFSO-RI Enabling Grids for E-sciencE SA3 Report Markus Schulz EGEE-II SA3 Activity Leader IT Department,
Components Selection Validation Integration Deployment What it could mean inside EGI
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Steven Newhouse Technical Director CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Implementing product teams Oliver Keeble.
EGEE-III INFSO-RI Enabling Grids for E-sciencE JRA1 and SA3 All Hands Meeting December 2009, CERN, Geneva Product Teams –
Enabling Grids for E-sciencE EGEE-III-INFSO-RI EGEE and gLite are registered trademarks Francesco Giacomini JRA1 Activity Leader.
Collaboration ed. 11/11/2009gLite Open Collaboration1 The gLite Open Collaboration “WLCG Overview Board” M. Mazzucato gLite Collaboration Board Chair.
EMI INFSO-RI Testbed for project continuous Integration Danilo Dongiovanni (INFN-CNAF) -SA2.6 Task Leader Jozef Cernak(UPJŠ, Kosice, Slovakia)
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Job Management Claudio Grandi.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Towards an Information System Product Team.
INFSO-RI Enabling Grids for E-sciencE Padova site report Massimo Sgaravatto On behalf of the JRA1 IT-CZ Padova group.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks CREAM: current status and next steps EGEE-JRA1.
Security Area Christoph Witzig (SWITCH) on behalf of John White (HIP)
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Argus gLite Authorization Service Workplan.
Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE and gLite are registered trademarks Francesco Giacomini JRA1 Activity Leader.
GLite proto-Collaboration phone conference Mirco Mazzucato INFN Working Meeting 5 September 2009.
JRA1 Middleware re-engineering
Bob Jones EGEE Technical Director
JRA1 Middleware Re-engineering Status Report
Andreas Unterkircher CERN Grid Deployment
Claudio Grandi (INFN and CERN)
Presentation transcript:

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Steven Newhouse Technical Director Moving towards EGI - JRA1 & SA3 All Hands

Enabling Grids for E-sciencE EGEE-III INFSO-RI Where are we now? Approaching 10 years investment in ‘e-infrastructure’ –World leading development activities –Integration, testing & certification of production strength software –Deployment and operation on a worldwide 24/7/365 basis User Community –Number of users and their use of the infrastructure GROWS SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Where do we go next? EGI: Moving towards sustainability –What does that actually mean? It’s for the community not one community –Driven by the NGIs to contribute resources for their researchers –Provide an integrated distributed electronic infrastructure  High Throughput Computing  High Performance Computing  Data Resources & Data Sources (Instruments) Operations –Support the integration into a single European Infrastructure User Community –Provide the bridge to get communities onto the e-infrastructure  i.e. training, domain specific operations, domain specific software SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI And delivery of the middleware... An integrated e-infrastructure  integrated software –gLIte –ARC –UNICORE EGI will focus on maintenance & harmonisation –Incremental conservative improvements –Development can take place independently in parallel  Move developments across to production IF proven & required –Research takes place elsewhere Professional middleware for production use –Take note... for this software community to have a future! SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Moving towards UMD SA3 & JRA1 All Hands - May gLiteUNICOREARC Operating System Application Communities UMD Site Distribution UMD Community Distribution Domain SSCs Domain SSCs Domain SSCs Domain SSCs Domain SSCs UMD Client Distribution Alignment across key areas Compute Data Information Security Accounting Alignment across key areas Compute Data Information Security Accounting

Enabling Grids for E-sciencE EGEE-III INFSO-RI Middleware Components SA3 & JRA1 All Hands - May gLite UNICORE ARC UMD EGI Maintenance Direct EGI EU Funding Other EU Funding National Fund An NGI

Enabling Grids for E-sciencE EGEE-III INFSO-RI Accessing Resources SA3 & JRA1 All Hands - May Infrastructure Services Infrastructure Operations & Support Domain Support Domain Operations Domain Services Domain Service Development Domain Application Development Research & Researchers EGEEEGEE EGIEGI SSCSSC Funding from national or European research organisations Funding from national or European research organisations

Enabling Grids for E-sciencE EGEE-III INFSO-RI So what does this mean now? EGI provides the bridge between national activities –Small central organisation EGI (MU) owns the ‘UMD’ brand for the community... –Has to work with user, operations & middleware to define it –Work with middleware consortia to deliver it –Minimal resources (  0) to integrate and verify releases EGEE in Y2 has to transition to the EGI structures –Start now... find out problems now... not later SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Additional challenges EGI starting on 1 st May 2010 is unlikely –There may be scope to backdate start date Extend EGEE resources for an additional 3 months –Informed by the results from the CB questionnaire –Informed by the country reports & the activity view –Maintaining the critical tasks within the infrastructure What we are not doing... –Moving resources between partners  Look to move tasks not people –Stopping activities/tasks  May prioritise some tasks over others to provide extension –Formally moving work between activities  Different reimbursement rates between NA/SA/JRA Do the most with the resources we will have! SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Staged Rollout An Overview of JRA1/SA1/SA3 JRA1 SA1 SA3 Product Team Cluster of Competence Component Maintenance Testing PPS Pilot Services Deployment Testbed Product Team Cluster of Competence Component Maintenance Testing Product Team Cluster of Competence Component Maintenance Testing Certification EGI.org Middleware Unit NGI Rollout Exp. Services Pilot Services Certification This is what we have now! 10 SA3 & JRA1 All Hands - May 2009

Enabling Grids for E-sciencE EGEE-III INFSO-RI Detailed Changes – JRA1 & SA3 Identify and establish: –A gLite consortium bringing together SA3 & JRA1 partners –A EGI.org Middleware Unit (effectively SA3 CERN) Establish product teams –Responsible for delivering working certified deployable software  Obviously this includes testing, certification and deployment  But also packaging, integration into the node type & documentation –Some certification remains central – for the moment!  NB: Underfunded SA3 resources at CERN being raised at PMB Manage Workplan (upcoming MJRA1.3.2) –Integrate core cleanup tasks (documentation, error codes,...) –Group workitems into releases &monitor progress of releases Reduce certification workload –Group releases onto a node type SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Product Teams & Node Types - 1 SA3 & JRA1 All Hands - May Product Name (Lead Manager & Partner) Software Components (Partners) Allocated Node Type(s) Authorization (Christoph Witzig, SWITCH) Authz Service (SWITCH, HIP, INFN, NIKHEF) Shibboleth interoperability (SWITCH) PAP PDP VO Management (Vincenzo Ciaschini, INFN) VOMS (INFN) VOMSAdmin (INFN) VOMS Security Infrastructure Product Team (John White, HIP) Delegation Framework (CERN, HIP, STFC) Trustmanager (HIP), Util-Java (HIP), Hydra (HIP), DICOM (HIP) myProxy Integration (HIP), LCAS/LCMAPS (NIKHEF) glExec (NIKHEF), SCAS (NIKHEF), Gridsite (STFC) Hydra Information Systems (Laurence Field, CERN) BDII (CERN) GLUE Schema (CERN) BDII Compute Element (Massimo Sgaravatto, INFN) CREAM (INFN) CEMon (INFN) BLAH (INFN) CREAM Job Management (Marco Cecchi, INFN) WMS (INFN, ED)WMS Logging & Bookkeeping (Ales Krenek, CESNET) Proxy and attribute certificate renewal (CESNET) Logging &Bookkeeping (CESNET) Gsoap-plugin (CESNET) LB Data Management (Ákos Frohner, CERN) CGSI_gSOAP (CERN), DPM (CERN), FTS (CERN) GFAL /lcg_util (CERN), LFC (CERN) FTS (various), LFC DPM (various)

Enabling Grids for E-sciencE EGEE-III INFSO-RI Product Teams & Node Types - 2 SA3 & JRA1 All Hands - May Product Name (Lead Manager & Partner) Software Components (Partners) Allocated Node Type(s) Integrated Clients (SA3, CERN) Proxy Renewal ???) GSI-SSH (External - TBC) UI WN VO Box Torque (SA3, NIKHEF) Torque LSF (SAn, ???) LSF Condor (SAn, PIC) Condor SGE (SAn, CESGA) SGE MPI (SAn, CERN) Work with MPI-WG to find a new group to take this on.GLITE-MPI Other / Unassigned (SA3, CERN) dCache (External) AMGA ( Birger Koblitz, KISTI)

Enabling Grids for E-sciencE EGEE-III INFSO-RI You do not work in isolation! Responsible for delivering working software products –development vs. integration vs. documentation vs. testing vs. certification –Fixed resources: delay releases, lower quality, reduce scope  Lower quality: not an option with refunding coming up!  Delay release: what are the downstream consequences?  Reduce scope: what is really needed? Your software is a component in an infrastructure –Change the logging format: break operations –Change an interface (WSDL, API, CLI): break applications –Change a library: force unexpected upgrades on partners Really think about backwards compatibility –Follow an agreed policy major & minor releases for:  Software Component (i.e. RPM)  Software Element (i.e. Node type) SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI All work items in Savannah Define ‘tasks’ to reflect the structure & dependencies –Software Element (Node type) releases –Software Component (Library) releases –Work items that go into a software component Cross team dependencies –If your software element/component depends on another release then take a dependency on it –Allows teams to understand the impact of their delays/changes –All teams will use a base infrastructure to do their releases  ETICS configuration –Be transparent about dependencies and the impact of delays This plan will form a ‘contract’ with the customer –A snapshot of the plan will be in MJRA1.3.2 SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Work Items during Y2 Clean up error codes & messages in all components –Provide backwards compatibility Dependencies between different software components –Reduce dependencies between product teams Review and document all public APIs and CLIs. Implement project priorities: portability, IPv6,... Separate client and server binary deployment packages Correct copyright notices & licenses Generate source distributions from the ETICS builds –SA3 will verify that the source distributions build binaries What changes are NEEDED before the end of EGEE? SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Managing Work Items for Year II Issues found in production use, certification or testing –Triage aggressively... Is it really critical? Or just nice to have? Issues defined by the TMB –These are not done lightly... and are needed for project goals Issues in your own roadmaps –Is the item still valid? Is it needed? Is it critical? Can it wait? Items must have regular review –Daily within the product team? –Several times a week by the gLite consortium?  The role of the EMT? –Regular review by the ‘customer’ EGI MU (i.e. SA3) –Monitoring by the TMB (eventually the EGI MCB)  Metric: Predictable high quality releases SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Adoption of staged rollout Removing explicit SA1 PPS deployment testbed –Staged rollout on current deployment resources & NGI resources –Includes system testing on representative production resources  Exact mechanism TBC with SA1 Critical issues resolved by upgrading to ‘old’ release –V1.0.4 – deployed in production use –V1.0.5 – new release with critical undiscovered issue –V1.0.6 – new release containing V1.0.4 code base Must be able to make releases with JUST bug fixes –When you release you MUST branch –Not acceptable for a new bug fix release to have new functionality SA3 & JRA1 All Hands - May

Enabling Grids for E-sciencE EGEE-III INFSO-RI Summary EGI presents a disruptive opportunity to the EU middleware community –Will need to deliver working software on time –Will need to become more relevant to other e-infrastructures  Relationship with EU providers and elsewhere –Need to prove our worth –Focus on maintenance and development as distinct activities Changes planned for Y2 will smooth transition –Delivering certified software, recognising issues –Work together within and between the product teams –Ensure relevance to user and operations community SA3 & JRA1 All Hands - May Any questions?