Building and Testing using Condor

Overview Goals The Condor build and test process Transition to new NMI framework Future plans

Goals Automated nightly builds and tests of development and stable branches Ability to submit on-demand build and test runs View results of individual runs and overall status.

The build and test process Builds and tests are submitted as Condor jobs to desired platforms. For testing, the previously built Condor binaries are installed as a Personal Condor. Tests are compiled and submitted to the Personal Condor to run. Condor returns the results when everything has finished. We use Condor to build and test Condor.

Great results This system has worked great for us for some time. Automated nightly build and test runs. Condor works great to run our builds and tests. We’ve learned from our experience and found some things we could improve on. Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used. Gcc - solaris vs. linux

Some lessons learned Pool machines used for Condor and other project builds and tests. Disk space and partitioning not well defined. Not readily apparent which versions of software exist on a system or where to find it. Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used. Gcc - solaris vs. linux

Old framework problems Builds and tests tightly coupled to the framework. Difficult to make changes to scripts without undesirable side effects. We created a new build and test system to solve these problems. Perl example /usr/local vs. /usr/local/bin - which version and modules where. Add other software that is PATH dependent and no way to tell which version is used.

The new NMI framework Development funded by NMI - NSF Middleware Initiative - which includes groups such as the GRIDS Center NMI project uses Condor and the pool machines for building and testing software releases. NMI project benefits from new framework New system came out of our collaboration with NMI project.

What does the new NMI framework provide? The same generic framework is used for both builds and tests. “Glue” scripts are used to connect the builds and tests with the framework. Build and test scripts are simpler to write. Inputs are well-defined

Example submit file inputs = input_build_runid.src, condor_test.src pre_all = nmi_glue/test/pre_all remote_declare = nmi_glue/test/remote_declare remote_declare_args = all remote_pre = nmi_glue/test/remote_pre remote_task = nmi_glue/test/remote_task remote_post = nmi_glue/test/remote_post prereqs = perl-5.8.5, tar-1.14, patch-2.5.4, m4-1.4.1, make-3.80, gzip-1.2.4 platforms = x86_rh_9,ia64_sles_8 Flexible inputs - source tarball, build id, ftp site

Workflow NMI Pool x86_rh_9 ia64_sles_8 disk archive database submit file NMI Pool x86_rh_9 ia64_sles_8 Flexible inputs - source tarball, build id, ftp site disk archive database

Condor testing benefits Condor test runs are now submitted per platform as the builds complete. Condor tests for specific features or long-running tests may be run separately from the nightly runs. Ability to pass arguments to tests at submit time. Enhanced data storage of test results. Testclasses example - define group or “class” of tests in Condor source code. Group name is passed to source via glue scripts to run the correct tests. This allows flexibility in test options without directly modifying test scripts. no need to wait for a build to finish on all platforms before starting tests.

New NMI pool management Machines have minimum hardware and disk space requirements. Any software installed or settings modified after the OS install are managed with cfengine. Hawkeye used to advertise all of the software and versions installed in the machine ClassAds. Condor jobs specify which software and version they require in the submit file, and are matched to the correct machine. Hardware and disk requirements are to ensure enough disk space and memory is available for Condor builds and tests. Cfengine - configuration agent - high level policy language to administrate and configure computer systems. Cfengine, or the configuration engine is an agent/software robot and a high level policy language for building expert systems to administrate and configure large computer networks. Cfengine uses the idea of classes and a primitive intelligence to define and automate the configuration and maintenance of system state, for small to huge configurations. Cfengine is designed to be a part of a computer immune system, and can be thought of as a gaming agent.

The build and test pools The old pool The new pool Old pool=14 machines New pool=21 machines and growing

Future plans Collaborators build and test software using our framework/pool Multi-machine testing capabilities using the Parallel Universe Automated Condor-G tests External people already using pool for builds and tests. NMI, Globus, etc.

More information Attend BOF NMI session on Wednesday for more details on the NMI pool setup, framework and glue scripts.