1. Building and Testing using Condor

2. Overview Goals The Condor build and test process
Transition to new NMI framework
Future plans

3. Goals
Automated nightly builds and tests of development and stable branches
Ability to submit on-demand build and test runs
View results of individual runs and overall status.

4. The build and test process
Builds and tests are submitted as Condor jobs to desired platforms.
For testing, the previously built Condor binaries are installed as a Personal Condor. Tests are compiled and submitted to the Personal Condor to run.
Condor returns the results when everything has finished.
We use Condor to build and test Condor.

5. Great results This system has worked great for us for some time.
Automated nightly build and test runs.
Condor works great to run our builds and tests.
We’ve learned from our experience and found some things we could improve on.
Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used.
Gcc - solaris vs. linux

6. Some lessons learned
Pool machines used for Condor and other project builds and tests.
Disk space and partitioning not well defined.
Not readily apparent which versions of software exist on a system or where to find it.
Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used.
Gcc - solaris vs. linux

7. Old framework problems
Builds and tests tightly coupled to the framework.
Difficult to make changes to scripts without undesirable side effects.
We created a new build and test system to solve these problems.
Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used.

8. The new NMI framework
Development funded by NMI - NSF Middleware Initiative - which includes groups such as the GRIDS Center
NMI project uses Condor and the pool machines for building and testing software releases.
NMI project benefits from new framework
New system came out of our collaboration with NMI project.

9. What does the new NMI framework provide?
The same generic framework is used for both builds and tests.
“Glue” scripts are used to connect the builds and tests with the framework.
Build and test scripts are simpler to write.
Inputs are well-defined

10. Example submit file inputs = input_build_runid.src, condor_test.src
pre_all = nmi_glue/test/pre_all
remote_declare = nmi_glue/test/remote_declare
remote_declare_args = all
remote_pre = nmi_glue/test/remote_pre
remote_task = nmi_glue/test/remote_task
remote_post = nmi_glue/test/remote_post
prereqs = perl-5.8.5, tar-1.14, patch-2.5.4, m , make-3.80, gzip-1.2.4
platforms = x86_rh_9,ia64_sles_8
Flexible inputs - source tarball, build id, ftp site

11. Workflow NMI Pool x86_rh_9 ia64_sles_8 disk archive database
submit file
NMI Pool
x86_rh_9
ia64_sles_8
Flexible inputs - source tarball, build id, ftp site
disk archive
database

12. Condor testing benefits
Condor test runs are now submitted per platform as the builds complete.
Condor tests for specific features or long-running tests may be run separately from the nightly runs.
Ability to pass arguments to tests at submit time.
Enhanced data storage of test results.
Testclasses example - define group or “class” of tests in Condor source code. Group name is passed to source via glue scripts to run the correct tests.
This allows flexibility in test options without directly modifying test scripts.
no need to wait for a build to finish on all platforms before starting tests.

13. New NMI pool management
Machines have minimum hardware and disk space requirements.
Any software installed or settings modified after the OS install are managed with cfengine.
Hawkeye used to advertise all of the software and versions installed in the machine ClassAds.
Condor jobs specify which software and version they require in the submit file, and are matched to the correct machine.
Hardware and disk requirements are to ensure enough disk space and memory is available for Condor builds and tests.
Cfengine - configuration agent - high level policy language to administrate and configure computer systems.
Cfengine, or the configuration engine is an agent/software robot and a high level policy language for building expert systems to administrate and configure large computer networks. Cfengine uses the idea of classes and a primitive intelligence to define and automate the configuration and maintenance of system state, for small to huge configurations. Cfengine is designed to be a part of a computer immune system, and can be thought of as a gaming agent.

14. The build and test pools
The old pool The new pool
Old pool=14 machines New pool=21 machines and growing

15. Future plans
Collaborators build and test software using our framework/pool
Multi-machine testing capabilities using the Parallel Universe
Automated Condor-G tests
External people already using pool for builds and tests. NMI, Globus, etc.

16. More information
Attend BOF NMI session on Wednesday for more details on the NMI pool setup, framework and glue scripts.