National Center for Supercomputing ApplicationsNational Computational Science Grid Packaging Technology Technical Talk University of Wisconsin Condor/GPT Meeting January 3, Michael Bletzinger Specialist, Software Developer Advanced Computational Environment and Security, NCSA

National Center for Supercomputing ApplicationsNational Computational Science GPT Purposes " Give developers an easy way to package software – All packaging data is specified from a source perspective. – Packaging data is strongly typed. " Provides a means of building and installing collections of packages " Provides standard version negotiation method for managing updates to packages.

National Center for Supercomputing ApplicationsNational Computational Science GPT and Globus " GPT was designed to re-implement the deployment requirements of Globus plus shared libraries. – Globus most of Globus is libraries. – Globus has deployment requirements for client tools, service daemons, and development/build environments. – Globus wants to release individual components separately – Globus wants to add outside software (ie. openssl, openldap) to these deployments. – Globus wants to work on multiple platforms.

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Splitting up the Source Code " Source Package Types – Source packages: contain source code and documentation. – Setup packages: contain configuration/localization scripts and files. – Test Packages: contain component test code.

National Center for Supercomputing ApplicationsNational Computational Science Source Packages " Contains source code for programs, scripts, and libraries. – All libraries are added to the link line. " Contains documentation and data files. " All of these share the same version number.

National Center for Supercomputing ApplicationsNational Computational Science Other Packages " Setup Packages – Contains scripts and templates. – These are assumed to have no build flavor. – Generate files outside of the control of the packaging system. " Test Packages – Contains test harnesses and code. – Installed in a special directory.

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Binary Package Types

National Center for Supercomputing ApplicationsNational Computational Science Package Transformations Test Package Setup Package Source Package pgm data docpgm data dev rtl docpgm data doc Source Code Installed Files localized files

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Dependencies " A Dependency is a need relationship to another package – RPM's "Requires:" does the same thing. " Different Dependency Types for Source an Binary Packages. " Types include both build and runtime dependencies.

National Center for Supercomputing ApplicationsNational Computational Science Runtime Dependencies " Source Runtime dependencies are transferred to the specific binary package. " Runtime dependencies require a binary package type in the specification.

National Center for Supercomputing ApplicationsNational Computational Science Build Dependencies " Are transferred to the binary dev packages " Are used in the configure stage of the source package build.

National Center for Supercomputing ApplicationsNational Computational Science Source Runtime Dependencies " pgm_runtime – Dependencies for the programs. " lib_runtime – Dependencies for the libraries. " doc_runtime – Dependencies for the documentation. " data_runtime – Dependencies for the data files.

National Center for Supercomputing ApplicationsNational Computational Science Source Build Dependencies " compile – Requires header file from another package " pgm_link – Programs require libraries from another package " lib_link – Libraries depend on libraries from another package

National Center for Supercomputing ApplicationsNational Computational Science Dependencies Header File Library Script Doxygen Page Library Source Code Documentation Program Source Code Source Package Package Foo Package Fee Package Foe Package Fum

National Center for Supercomputing ApplicationsNational Computational Science Binary Dependencies " Compile – Only in dev packages " Build_Link – Only in dev packages " Runtime_Link – rtl and pgm packages " Regeneration – Only in pgm packages " Runtime – in any binary package

National Center for Supercomputing ApplicationsNational Computational Science Source to Binary Dep Conversion compile lib_link pgm_link runtime compile runtime build_link runtime_link regeneration Static Linking Dynamic Linking Pass Thru Source Dependencies Binary Dependencies

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Build Flavors " A Build Flavor is a set of compilation and linking options which have to be defined to produce binaries that can be linked with each other. " A development environment issue for older versions of Globus. " Now a runtime issue with GT 2.0 because of shared libraries and plugins.

National Center for Supercomputing ApplicationsNational Computational Science Build Flavors " Build Flavor Choices – Compiler Choice: gcc, vendor cc – Size: 32 bit, 64 bit – Debugging: Debug Symbols included or stripped – Thread Package: pthreads, solaris, sproc, none – message passing library: mpi, none " Standard Flavors in green " Example – gcc32dbgpthr = gcc compiler, 32 bit, debugging symbols, pthreads

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Versioning Purposes " Source code change identifier. " Distribution identifier. " Marketing tool. " Interface compatibility identifier – for APIs. – for protocols. – for configuration files.

National Center for Supercomputing ApplicationsNational Computational Science Version Negotiation. " Agreement on compatibility between package developer and package user. – Package developer makes a compatibility recommendation. – Package user choose to accept or override the recommendation.

National Center for Supercomputing ApplicationsNational Computational Science Aging Version " Version scheme used by the package developer. " Based on libtool and other shared library schemes major version number minor version number - indicates a binary compatible change age - indicates backward compatibility to the major version number

National Center for Supercomputing ApplicationsNational Computational Science Version Requirements " Simple Version Requirement. – Package user specifies major version number. – Trusts the developer compatibility recommendation. " Version Range Requirement. – Package user specifies a range of major and minor version numbers. – Does not use the compatibility recommendation from the package developer. " Combinations of these can be used for more complex requirements.

National Center for Supercomputing ApplicationsNational Computational Science Versioning Example Version for the package is 5.1.3

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Installation Objectives " Standard structure for both runtime and build installations. " Multiple build flavors can exist in the same location. " Packaging data co-located with the installation.

National Center for Supercomputing ApplicationsNational Computational Science Installation Structure $GLOBUS_LOCATION programs libraries include etc setup test bin lib libexec gcc32 headers gcc32pthr headers packaging metadata sbin

National Center for Supercomputing ApplicationsNational Computational Science Multiple Installations $GLOBUS_LOCATION programs libraries include etc setup test bin lib libexec gcc32 headers gcc32pthr headers packaging metadata sbin Client Tools $GLOBUS_LOCATION programs libraries include etc setup test bin lib libexec gcc32 headers gcc32pthr headers packaging metadata sbin User Development $GLOBUS_LOCATION programs libraries include etc setup test bin lib libexec gcc32 headers gcc32pthr headers packaging metadata sbin Services

National Center for Supercomputing ApplicationsNational Computational Science How to move an installation. " Copy Directory. " Re-run setup scripts.

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Bundling Objective " Be able to install a collection of packages or even an entire distribution with one command. – Packages are not installed if compatible ones already exist. – All setups are consolidated into one post install script. " Be able to build a binary bundle with one command.

National Center for Supercomputing ApplicationsNational Computational Science Bundling Data " Name and version of the package. " Binary package types that are needed. " Build options to be avoided ie. no threads. " Build Flavors are chosen during builds for each platform.

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science Setup Objectives " Allow installed packages to be configured with a script. – Each script packaged in a setup package. " Script execution is recorded in the packaging data. " Setup packages can be replaced by distributing organizations to tailor a distribution.

National Center for Supercomputing ApplicationsNational Computational Science Setup dependencies " Source package indicate a setup need by a special setup dependency. " Setup packages have two names and versions. – One name/version for the specific package. " different setup packages have different name/version. – One name/version to fulfill the setup dependency. " All setup packages will have the same name/version for this.

National Center for Supercomputing ApplicationsNational Computational Science Setup Package Example Regular Package Foo Setup Package Fum Setup Package Fie Setup Package Foe

National Center for Supercomputing ApplicationsNational Computational Science Outline " Splitting up the source code " Binary Package Types " Dependencies " Build Flavors " Versioning " Installations " Bundling " Setup " The GPT Tools

National Center for Supercomputing ApplicationsNational Computational Science What does GPT offer? " An easy way for developers to package their software. " An easy way for people to build large sets of source packages. " Provides a package manager for those systems that don't have one. " Provides packaging metadata in an XML format. " Compatible with other package managers

National Center for Supercomputing ApplicationsNational Computational Science An easy way for developers to package their software " gpt_setup - Sets up Makefiles and autoconf to use packaging system. " gpt_edit - GUI to allow developers to edit the metadata. " patch-n-build - for those developers that want to keep their own build system. " globus-makefile-header - for those who want to build applications using Globus components.

National Center for Supercomputing ApplicationsNational Computational Science An easy way for people to build large sets of source packages. " globus-build -swiss army knife for building packages, bundles, and development directories. globus-build [ \ -verbose \ -help \ -installdir=path_to_installation \ -builddir=path_to_build directory \ -log=logfile \ -srcfile=source_metadata_file \ -srcdir=source_directory \ -static= \ -install-only \ -force \ -std-flavors \ -all-flavors \ ] \ [macro=value] [list of flavors to build] [list of source packages/bundles]

National Center for Supercomputing ApplicationsNational Computational Science Provides a package manager for those systems that don't have one. " globus-install - Installs/uninstalls binary packages " gpt_verify - Verifies that a Globus installation is complete.

National Center for Supercomputing ApplicationsNational Computational Science NMI Changes to GPT. " Changes for April NMI release. – New schema/dtd for packaging data. " conversion script to update existing packages – Add package signatures and other installation verification mechanisms. – Add tools for installation management. " Centralized file which records installations and installed bundles. – De-globus-ize the tools " replace "globus" with something else for script names variables etc. – probably keep old globus names for compatibility purposes.