1. Spring Cleaning the Software Repositories Matthias Schröder
Topics:
Introduction
The Issue
What can be done
Conclusion
27-March-2008
Spring Cleaning Repositories

2. Spring Cleaning Repositories
Introduction
Software Repositories hold all packages used on any quattor managed node
Organised by ARCH_OS
We keep all versions of packages ever put into the repositories
To be sure that we can reinstall a machine if needed
Number of packages grows with time…
ARCH_OS
i386_slc3
x86_64_slc3
i386_slc4
x86_64_slc4
Number of packages
22271
5389
15389
16369
27-March-2008
Spring Cleaning Repositories

3. Spring Cleaning Repositories
The Issue
The number of packages has reached a critical level
Going through the listings to find packages takes longer and longer
Panc compile times directly effected
Impossible to upload the repository listings to CDBs with little memory
Ok, 1GB is not a lot, but still…
Impossible to compile templates for enclosures with many nodes
Running out of memory even on CDBs with lots of installed memory…
We have to clean up the repositories now!
27-March-2008
Spring Cleaning Repositories

4. Possible Ways to Clean Up
Prepare a positive list of most frequent packages, keep all others
List of versions to keep is difficult to obtain
We can’t be sure that we will keep all versions required
…andwe can not count on panc to warn us
We might only discover that packages are missing when we have to reinstall a node
Prepare a list of packages to remove
Might be difficult to obtain for all packages
Same risk as above
Tedious
We probably keep many versions we don’t use any more => too little gain
Make list of versions still in use from xml profile of all nodes
Safest
But also have to keep newer versions not yet in use
Highest gain
Needs a new tool to extract list of packages and versions from xml profiles
27-March-2008
Spring Cleaning Repositories

5. Spring Cleaning Repositories
Tests Done
With a positive list for castor and CA’s for i386_slc3
But castor list now known to be too small…
Results very encouraging
Have to keep more castor versions
Can gain more on other packages
Full repo listing
Svelte repo listing
# packages
22271
14868
Compile time
54 s
35 s
Memory usage
3.8 GB
1.7 GB
27-March-2008
Spring Cleaning Repositories

6. Spring Cleaning Repositories
How to Continue
We have to be very careful with removal of versions
We have to reduce the repo listings now
Some profiles do not compile any more!
My proposal: two step approach
Step 1: Remove known obsolete castor and CA versions, keep all the rest
These are the packages for which the biggest gain can be achieved with little work
Step 2: Make a listing of all used versions from XML profiles
Maximum gain, safe method, but takes longer to prepare
Have to rethink the ways how we handle kernels (i386_slc3: 1122 “kernel” packages!)
Only keep these kernels that are configured on nodes?
27-March-2008
Spring Cleaning Repositories

7. What to do With Obsolete Versions
We should not delete them
Disk space is not an issue
Copy them to another place on SWREP
Then remove them via swrep-soap-client
27-March-2008
Spring Cleaning Repositories

8. Spring Cleaning Repositories
Conclusion
We have to reduce the size of the repository listings
This is a delicate operation, we have to ensure that all used versions are kept
Can we agree on two step approach?
27-March-2008
Spring Cleaning Repositories