1. Chuan-kai Lin Drake Diedrich Google Inc.
Debmarshal
Chuan-kai Lin
Drake Diedrich
Google Inc.
Theme created by
Sakari Koivunen and Henrik Omma
Released under the LGPL license.

2. Distributions for Service Owners
Service owners want changes to distributions while they are developing and deploying their services
Service owners want as few changes as possible once they've deployed.
Cannot meet both needs in a single distribution, so have two, one changing and one in near freeze.
New services are deployed continually, so new frozen releases are required continually.

3. Debmarshal Features
Maintain a Debian or Ubuntu software repository mirror for enterprise deployment
Process and publish an incoming queue of packaged software
Allow staging of software out to different classes of systems at different rates
Preserve all previous repository snapshots
Automate distribution-wide consistency checks

4. Debian Repository structure
A distribution is a list of packages stored in index files located in uri/dists/codename/...
A Packages.gz index file exists for each section – architecture - distribution tuple.
A package pool is a common directory that multiple distribution index files reference. uri/pool/...
Each index file contains the checksums of the packages, and each index is checksumed by a Release file, which is then gpg signed.

5. debian
|-- dists
| |-- etch
| | |-- Release
| | |-- Release.gpg
| | |-- main
| | | |-- binary-i386
| | | | |-- Packages.gz
| | | |-- source
| | | | |-- Sources.gz
| `-- testing -> etch
|-- pool
| |-- main
| | |-- a
| | | |-- aalib
| | | | |-- aalib-bin_1.4p5-13_i386.deb
| | | | |-- aalib-bin_1.4p5-22_i386.deb
| | | | |-- aalib1-dev_1.4p5-13_i386.deb
| | | | |-- aalib1-dev_1.4p5-22_i386.deb
| | | | |-- aalib_1.4p5-13.diff.gz
| | | | |-- aalib_1.4p5-13.dsc
| | | | |-- aalib_1.4p5-22.diff.gz
| | | | |-- aalib_1.4p5-22.dsc
| | | | |-- aalib_1.4p5.orig.tar.gz

6. Debmarshal repository structure
Same pool/ structure as upstream
Instead of just a distribution codename, use a distribution track and snapshot. eg. etch/23 or etch/latest
Put the same Release and Packages.gz files in the dapper/23 directory
deb etch/latest main contrib

7. debian
|-- dists
| |-- etch
| | |-- 23
| | | |-- Release
| | | |-- Release.gpg
| | | `-- main
| | | |-- binary-i386
| | | | |-- Packages.gz
| | |-- latest -> 23

8. Debmarshal programs Intended for use by other scripts and cron jobs
Persistent storage between invocations to reduce deb unpacking to single pass
Python dictionary access to Berkeley DB files
All logic in the make_release.py program
A few hours to fully index a new upstream distribution, and then minutes on subsequent changes and uploads.
Full consistency checks of new uploads in minutes

9. index_pool.py
Run in the top level directory of the archive to scan the pool directory for new packages
Populates dbs/ with DB files of package metadata
Run immediately after a debmirror or rsync of an upstream distribution, usually from cron.

10. make_release.py --dist codename
Takes an upstream Release file out of dists/codename as the list of packages to put in a new debmarshal snapshot.
The codename could be used as the debmarshal track name
make_release.py --dist dists/etch --track myetch
Points the latest symlink at the latest repository in the track
track names and architectures configured for tracking mode in config/repositories.

11. enter_incoming.py
Similar function as index_pool.py, but for a local package archive, typically a separate pool.
Checks gpg signatures on uploads in incoming/
Moves good files to the pool and generates the DB entries for the packages
Typically run every minute from cron
Allows use of standard Debian tools like debuild, debsign, and dupload.

12. make_release.py --track track --release snapshot/latest
Configure a “release specification file” for each track with the versions of packages allowed in
Run make_release.py, which will generate a new snapshot using the latest available packages and the release specification file.
Moves the latest label to point to the new snapshot number.
Typically run from same cron job immediately after enter_incoming.py for all currently mananged tracks.

13. make_release.py --track track --release track/89 diff track/88
Show the package differences between two snapshots in the same or different tracks
Useful for upload messages and other archive maintenance
Other syntaxes to specify a set of packages are possible
Works with symbolic snapshot names

14. handle_alias.py command
update track/label NNN – change label in track to point to snapshot number NNN
log track/label – print out the change history for label track
Typically called from interactive scripts used to manage the labels in each maintenance track

15. make_release.py --release track/label --track track verify
Check that each package in the release snapshot has all of it's dependencies met within the snapshot
Checks for file overlaps between packages without declared Conflicts meta data that would break apt-get.
Underlying: distribution may be specified in config/repositories to test derivative distributions.
Unit testing for release snapshots.

16. Workflow
Software developers produce packages and upload to archive. Packages go into the pool immediately.
Distribution maintainers decide which versions of which packages go into their tracks. Some very permissive for testing purposes.
Different labels in each track representing different levels of testing and early deployment.
Labels changed, and clients tracking each label update themselves using cron-apt.

17. Related software
snapshot.debian.net – full snapshots of all releases, several TB total. No local upload capability, verbatim indices.
reprepro – local repository maintenance software. No multi-track, very aggressive about cleaning pool.
dak – no local derivative repository. No integrity testing. Requires SQL DB. Undocumented.
dpkg-scanpackages – the original repository builder. Doesn't scale well to multiple tracks.

18. Future Work Publish good user-interface wrappers around scripts
Publish better docs on using and setting up
Publish manual of suggested workflow when more deployment experience is available
Publish integrity checks of upstream distributions for their use in bug stomping.
Verbatim Release files where possible, to ease use of upstream installers.
Integration with automated install testing.

19. Availability
Check internal debmarshal docs for debmarshal managed repositories inside Google.
Public software release under the GPLv2 at code.google.com/p/debmarshal