1. EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org JRA1 and SA3 All Hands Meeting 15-16 December 2009, CERN, Geneva Product Teams – How to Manage Patches Maria Alandes Pradillo

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Summary 2 Introduction The new integration process Metapackage patch Internal patch Transition period Agenda to release old style patches Current status Conclusions

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 3

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 4 Patch repository cert metapackages Certification PPS/Staged roll-out release PPS/Staged roll-out metapackages Production release Production metapackages Integration Team Production Repository Building binaries Developers The Integration and Certification process up to now...

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 5 ETICS configuration maintenance Internal repository Staged roll-out release Production release Integration Team Building binaries Metapackages Patch repository Certification Developers The New Integration and Certification process... Production Repository

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 6 The main changes in the new process are: Product Teams will certify and integrate the middleware ETICS will replace the Integration Team to create: Metapackages Patch repositories More control and independence on the metapackage releases Metapackages are created once What it is certified is released to production One repository per metapackage Metapackages are not affected by bugs specific to other metapackages Versions across metapackage repositories should converge ETICS internal configurations Simplified process for Staged roll-out and Production releases

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 7 To take into account... The new process is not definitive It’s a starting point right now Open issues It will be improved with your feedback Thanks already to Fabio and Ales for their contributions We aim at automating things as much as possible Building binaries + Metapackages creation + Patch repository create-release -p project_configuration –c etics_configuration –t patch_number Certification Testsuites Regression tests

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 8 create-release -p project_configuration –c etics_configuration –t patch_number

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Introduction 9

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage patch New metapackage versions provided by the product team. Internal patch New package versions that are not part of any product team metapackage but that are needed by other product teams. i.e. GFAL or lcg_utils. New Integration Process 10 Against which metapackage an internal patch is certified? Some definitions... Internal ETICS configurations ETICS project configurations used to prepare the metapackages. They define the last certified version of the runtime dependencies in all metapackages. All external dependencies will be defined from the beginning even if they haven’t been certified within a metapackage

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Internal Repository Production Repository Savannah Patch Who and what is involved? Developer ETICS Certifier Beta Repository Product team roles Integration Team New Integration Process 11

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage Patch With provider 1.The product team creates a Savannah patch in the relevant platform and fills in: Summary -> SLX/ARCH metapackage name > summary Priority -> Normal, in most cases Assigned to -> product Team responsible gLite release -> 3.X Operating system -> SLX Architecture -> i386 or x86_64 Affected metapackages -> metapackage name Metapackages to be reconfigured: metapackage name if applicable Metapackages to be restarted: metapackage name if applicable Configuration changes: description if any Release notes: description 1. create patch Metapackage patch 12

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage Patch With provider 2. The product team builds new packages and prepares the metapackage in ETICS. The product team adds the following information to the Savannah patch: gLite subsystem tag(s)/ETICS configurations: ETICS configuration names RPM names: list of URLs of the released packages and metapackage 2. prepare metapackage + build new packages ETICS Metapackage patch 13

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 The packages and also the metapackage are created using ETICS Intructions on how to do this are here: https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt Metapackage patch 14 Metapackages in ETICS are a collection of runtime dependencies whose version is taken from a project configuration: glite_branch_3_1_0_internal for gLite 3.1 metapackages glite_branch_3_2_0_internal for gLite 3.2 metapackages Product teams manually specify the versions of new packages as static dependencies. Only the new packages + metapackage are specified in the patch.

15. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage patch 15 Example of Metapackage configuration

16. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage patch 16 glite_branch_3_1_0_internal

17. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage patch 17 glite_security_3_1_0_internal How does a product team know which packages are new? Metapackage versioning is now done by the Product Team, same convention as now?

18. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage Patch Ready for Certification 3. When the product team is happy, they change the state of the patch into Ready for Certification. 3. Ready for Certification Metapackage patch 18

19. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage Patch In certification 4. Certification starts. The product team changes the state of the patch into In certification. yum install metapackage-name 4. In certification ETICS Production Repository Metapackage patch 19 DAGOS Jpackage

20. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 What’s needed to do the certification: A machine with the platform of the patch that is going to be certified. The ETICS repo file: Repo file for non registered builds: http://etics-repository.cern.ch/repository/pm/volatile/repomd/name/metapackage-name/etics-volatile-build-by-name.repo Repo file for registered builds: http://etics-repository.cern.ch/repository/pm/registered/repomd/name/metapackage-name/etics-registered-build-by-name.repo The Production repo file: For 3.1: http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.1/ http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.1/ For 3.2: http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/ http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/ DAG, Jpackage and OS repositories, if needed Then, run yum install metapackage-name Metapackage patch 20

21. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Internal Repository Metapackage Patch Certified 5. When the product team finishes certification, the patch is moved to state certified. 6. The central team checks the patches in state certified and then: updates glite_branch_3_X_0_internal configuration in ETICS Copies and signs certified packages into the Internal Repository 5. Certified Update glite_branch_3_X_0_internal 6. certified? Copy (and Sign rpms for 3.2) ETICS Metapackage patch 21

22. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Beta Repository Metapackage Patch Rolling-out 7.a. The central team runs the script make-preview #patch_list 7.b. The central team updates the state of the patch in Rolling-out. 7. Rolling-out Make-preview Internal Repository 22 Metapackage patch

23. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metapackage Patch In Production 8.a. The central team runs the script make-release #patch_list 8.b. The central team changes the state of the patch in Production. 8. In production Make-release Production Repository Beta Repository Metapackage patch 23

24. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 It’s the same as a metapackage patch but it stops after certification Internal Patch ETICS With Provider Ready for Certification In Certification Certified Internal Repository UPDATE ETICS and Internal repository Packages in an internal patch are released as part of a metapackage patch Internal patch 24

25. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 25

26. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Product Teams should start creating Metapackage patches Internal patches Old style patches will no longer be supported Transition Period 26 For the existing old style patches, the following release schedule will be implemented: Patches in staged roll-out will go to Production: week 4-8 January 2010 See http://glite.web.cern.ch/glite/preview/default.asp for more detailshttp://glite.web.cern.ch/glite/preview/default.asp A set of Certified and PPS patches will go to staged roll-out: week 4-8 January 2010 When the staged roll-out is ready to go to production, these patches will go to Production (estimated date: end of January 2010) A second release of Certified and PPS patches will go again to staged roll- out, as long as Product Teams are not releasing anything before.

27. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Production 3.1 Update 59 3.2 Update 06 gLite 3.1 (SLC4 32bits) #3104 yaim-core 4.0.10 SL4/i386 #3110 yaim clients 4.0.9 SL4/i386 #3168 SL4/i386 New glite-swat rpms #3184 dCache security update dcache-server-1.9.1-11 Sl4 32 bit #3183 Proxy renewal update for SLC4/i386 #3189 yaim-core 4.0.11 SL4/i386 #3220 SLCS client update #3322 dcache-srmclient-1.9.5-2 bug fixes SL4/i386 #3361 dcap 1.9.3-5 dcache-srmclient 1.9.5-3 upgrades for SL4 i386 #3429 [yaim-wms] urgent patch for bug #58347 #3517 SGE InfoProvider support for "GlueCEPolicyAssignedJobSlots" in glite-SGE_utils release SL4/i386 gLite 3.1 (SLC4 64bits) #3105 yaim-core 4.0.10 SL4/x86_64 #3111 yaim clients 4.0.9 SL4/x86_64 #3185 dCache security update dcache-server-1.9.1-11 SL4 64 bit #3171 SL4/x86_64 New glite-swat rpms #3190 yaim-core 4.0.11 SL4/x86_64 #3315 dcache-srmclient-1.9.5-1 bug fixes SL4/x86_64 #3316 Proxy renewal update for SLC4/x86_64 #3327 dcache-srmclient-1.9.5-2 bug fixes SL4/x86_64 #3362 dcap 1.9.3-5 dcache-srmclient 1.9.5-3 upgrades for SL4 x86_64 gLite 3.2 (SLC5 64bits) #3191 yaim-core 4.0.11 SL5/x86_64 #3254 SL5 64 dcap new packages and setup #3281 new lcg-infosites version solving the bug #37572 SL5/x86_64 #3321 dcache-srmclient-1.9.5-2 bug fixes SL5/x86_64 #3364 dcap 1.9.3-5 dcache-srmclient 1.9.5-3 upgrades for SL5 x86_64 #3053 SL5/x86_64: glite-LFC_oracle #3225 SL5/x86_64: glite-MPI_utils #3344 SL5/x86_64 glite-SE_dpm_disk December 2009 Beta 3.1 Bundle 04 3.2 Bundle 03 Beta 3.1 Bundle 03 3.2 Bundle 02 Production 3.1 Update 60 3.2 Update 07 Production 3.1 Update 61 3.2 Update 08 Quarantine period Agenda to release old patches 27 January 2010 February 2010 Staged roll-out should be able to receive metapackage patches or some of the remaining old slyte patches in ~ February

28. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Current Status 28 ETICS glite_branch_3_1_0_internal glite_branch_3_2_0_internal Production 3.1 Update 59 3.2 Update 06 Beta 3.1 Bundle 03 3.2 Bundle 02 gLite 3.1 (SLC4 32bits) gLite 3.2 (SLC4 64bits) gLite 3.2 (SLC5 64bits) DONE On going PPS and certified patches To be done before the end of 2009 Current status of ETICS configurations: External dependencies DONE

29. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Conclusions 29 The first version of the new process is now defined and ready to be used Instructions are available under: https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt https://twiki.cern.ch/twiki/bin/view/EGEE/ProdInt Changes in the process are likely to happen. We foresee a period of at least one month (January 2010) to release pending old style patches Splitting up repositories will ease the combination of metapackage patches and old style patches The ETICS internal configurations need to be completed so Product Teams can actually start using them to prepare their metapackages (to be done this week)