OSCAR Cluster Configuration Challenges

Software on Nodes User can define “Package Sets” * User can define “Package Sets” * User applies a package set to a group of nodes User applies a package set to a group of nodes Basically works today with one to one associations between package sets and node groups Basically works today with one to one associations between package sets and node groups All collected package set data is stored in ODA All collected package set data is stored in ODA * Package Set: a saved set of OSCAR packages

Configurations on Nodes Works today in the one to one configuration to package set relationship Works today in the one to one configuration to package set relationship this is transitively one to one configuration to node group this is transitively one to one configuration to node group All collected configuration data can be stored in ODA All collected configuration data can be stored in ODA

Node Groups Due to package set exclusive package set and configuration associations with nodes, exclusive node groups are needed Due to package set exclusive package set and configuration associations with nodes, exclusive node groups are needed A non-exclusive grouping capability will certainly be necessary in the future A non-exclusive grouping capability will certainly be necessary in the future –C3, other packages –Storage nodes, compute nodes, user access points, etc –Capability will be very useful to admins, package authors

Arising Problems 1) Having two types of node grouping is confusing (exclusive and non-exclusive) 2) Configurations for some packages may want to be cluster wide, not limited to the package set associated nodes. However, some may need vice-versa. 3) Package Sets and Configurations are LOCKED to each other. This means that all nodes with a shared configuration must have the SAME Package Set applied. 4) $OSCAR_SERVER is configured in saved Configurations. What if two Configurations differ?

Arising Problems… cont. 5) Support is needed for client/server packages that need to use something other than $OSCAR_SERVER e.g. –server package is in rpm format a)Install server rpm in post_install b)Install server rpm everywhere, enable in post_install (could be overkill, especially for diskless nodes) c)Make special disk image just to reflect a single rpm difference per client/server package (results in many images with redundant data, and LOTS more work to manage for the sysadmin!) 6) Node names not available to package’s during their configuration steps. Package authors need the flexibility this would provide.

Solutions, and their problems… 1) Make many SIS images behind the scenes to reflect client/server rpm differences, and overlapping package sets  Really slick! PBS server node could be rebuilt with minimal post_install… it has it’s OWN unique image  Grossly inefficient in time for image creation and disk space  Having many images significantly reduces advantages gained by using images in the first place.  Solves no Configuration related problems  Extremely complicated to make a system which would adjust to changes in configuration along the way, change all necessary images, delete them, merge them, etc. No good!

Solutions, and their problems… 2) Go back to LUI type install (like Kickstart) and install nodes by rpm lists  Solves the “many image” problem for client/server packages – node definitions and differences are merely “lists”, not images  Inefficient, no hope for multicast, no multi-distro product (LUI is dead)  Solves no Configuration related problems  Doesn’t leverage SIS. We’re locked into SIS, and we know we want it. No Good!

Solutions, and their problems… 3) Continue on course and save a Configuration per Package Set. The package could have LOCAL (node group wide) or GLOBAL (cluster-wide) configuration options, saved per Package Set.  Keeps us on course, not as much backtracking  Still have to have exclusive and non-exclusive node groups  Adds much complexity to the configuration for package author  Configurations still bound to package sets. i.e. Solves problem #2, but not #3. (see Arising Problems slides)  Configuration routine must be gone through by the user N times (where N=number of occurrences in package sets) Mediocre.

But wait… let’s step back once. Take a simplistic view of what the user actually needs to do: Define nodes Define nodes Choose software for nodes (or node groups) Choose software for nodes (or node groups) Configure software Configure software Suppose we allow the user to bind a specific package’s saved “Configuration” to node directly, instead of via linking to a Package Set, which is bound to nodes… well, that would mean that a configuration could exist for a node without that package installed for it. That wouldn’t work- could have a conflict. What if… we didn’t define Package Sets at all and only configured? Application of the saved “Configuration” for the package to a node would IMPLY a selection itself. No Package Selection needed. No Package Sets needed. Different packages’ saved “Configurations” can now overlap different groups of nodes! Well what about the image then? We can’t have overlapping dynamic configurations and have different images. It’s like having our cake and eating it too, essentially. That conflict will cost a slight change in our API- if the operations currently done on the image were done during post_install instead… we could have a SINGLE BASE IMAGE. Currently, this means eliminating just 4 scripts, <300 lines. (ls packages/*/scripts/post_client_rpm_install, post_rpm_install)

[Insert more description here] Jeremy draws some windows on the whiteboard… Jeremy draws some windows on the whiteboard…

Solutions, and their problems… 4) Combine Package Selection and Configuration into the same step- just Configuration. Node definition would occur BEFORE configuration.  Simplifies wizard… what’s a Package Set? Who needs it? User just picks software and matches it to nodes; It’s the actual end-of-the-day, bottom-line process without all the junk in the middle.  Non-exclusive groups all around! No exclusive required with only one base image.  Node names are finally available during the Configuration step.  Image based maintenance is preserved as an option to users.  While still gaining the benefit of images, OSCAR is not so dependent on them, and is much closer to a state where it can install on pre-existing machines.  Treating the $OSCAR_SERVER the same as other nodes becomes incredibly easier, allowing us to treat it just like other nodes (all I need to do is ssh to it and sync it)  Add/Remove package is greatly simplified. Because packages would not be using image dependent scripts, the use of images is not crippling. A package would be essentially “Added” in the post_install anyway!  Requires an API change… post_rpm_install and post_clients can’t apply. It is the price of granularity, and using a single image.  Post_install is a little heavier. (note the original base install is also lighter) Good?