Recent Developments in the ICAT Job Portal a generic job submission system built on a scientific data catalog NOBUGS October 2016 Brian Ritchie, Rebecca Fair Steve Fisher, Kevin Phipps, Tom Griffin Dan Rolfe, Jianguo Rao STFC Rutherford Appleton Laboratory The ICAT Job Portal, or IJP, is a generic job submission system built on the ICAT scientific data catalog. In this presentation, I’m going to give an overview the IJP and cover some recent developments. Initial design and development on the IJP was carried out by Steve Fisher and Kevin Phipps, with more recent work by Rebecca Fair and myself. As our main (ahem, only) customers, Dan Rolfe and Jianguo Rao for Octopus in the Central Laser Facility have set the requirements for much of the design

Overview Search for datasets / datafiles using ICAT Configure and submit jobs to process selected datasets / datafiles on one or more batch servers Submit single job for all selected datasets/datafiles, or separate jobs for each Jobs use IDS to retrieve data (or ICAT for metadata) Monitor progress of jobs and inspect output The basic aims of the IJP are fairly straightforward: to allow users to search for datasets and datafiles using the ICAT catalogue; then configure and submit jobs to process that data on one or more batch servers. Depending on the job configuration, users can submit a single job for multiple datasets or datafiles, or submit separate jobs for each one. The jobs can use the ICAT Data Service to retrieve datasets or datafiles, or query ICAT for their metadata The IJP allows users to monitor the progress of their jobs and inspect their output (including while they are running)

Architecture … search retrieve ICAT & IDS IJP Server IJP web app User’s desktop GUI (browser) CLI REST Batch System 1 IJP batch connector 1 Batch server 1 Worker Node 1 Worker Node n … submit batch job batch server magic Batch System N IJP batch connector N Batch server N This diagram shows a rough outline of the logical architecture of the IJP. The central IJP server runs as a web application (currently within Glassfish). The primary interface is via the TopCAT GUI that runs in the user’s web browser. The IJP extends TopCAT via a plugin developed by Rebecca Fair. This communicates with the IJP server via a REST interface. There is also a command-line client that uses the REST interface. The interface supports job submission and monitoring. The IJP server can be connected to one or more “batch connectors” (via another RESTful interface). Each batch connector manages a particular batch processing system (e.g. Torque, Platform LSF, unix batch) and handles job submission and monitoring requests for that particular system. How the batch system handles jobs, e.g. farming them out to individual worker nodes, is up to it (for now, at least). Logical vs. physical : IJP server + batch connector usually on same machine. ICAT/IDS tend to be on separate machine. Demo system has it all in one box!

Finding data Use TopCAT to find data in ICAT Configure job for single dataset or datafile or build a cart with multiple datasets/datafiles Configure job for cart Here is an outline of how the IJP GUI can be used to find data. You won’t be able to see the fine details on these screenshots, but I just want to concentrate on the overall shape. The top screenshot shows the Browse tab; it has the usual TopCAT tools for filtering and for ordering results; but there’s an extra option to filter results by job type (more on that later). Also note that each row has a green Configure Job button to run a job on just that one dataset. To run jobs on multiple datasets or datafiles, the user can add them to a cart; and the cart view has an extra (green) button to configure a job on the entire cart.

Job Types Part of the IJP configuration Each Job Type specifies: Program (job script) to run Dataset types for which the job can be run If job is batch or interactive If job accepts datasets, datafiles or both If single job can take multiple datasets/datafiles Other job parameters / options GUI filters job types depending on selected data or filter data by selecting job type first A key part of the IJP configuration is the specification of the job types that it can run. Job Types are defined in XML. Each job type specifies… … whether a single job instance can … … other parameters and options that will be added to the command-line for the job … GUI filters jobtypes depending on the dataset type of the selected data

Job configuration Job options Submit options Again, you probably can’t see the details, but this just shows a dialog to configure a job; the contents of the dialog are determined by the options specified in the job type. When multiple datasets/datafiles are selected, submit options depend on whether the jobtype allows multiple datasets/datafiles per job instance. If it does, then the user can still choose to submit one job on the lot, or submit a separate job for each item in the cart. If the jobtype doesn’t allow multiple items per job, then Submit will always submit multiple jobs, one per item (mouseover makes this doubly clear!)

Job submission IJP server gets estimates from each batch connector Chooses one of the best Batch connector submits job to its batch system Jobscript executable defined in job type Job is passed dataset/file IDs, ICAT/IDS session tokens and job options Batch connector monitors submitted jobs Queue status, standard/error output IJP server monitors batch connectors IJP server holds job status and output Until user deletes the job Session tokens required so that jobscripts can resolve the ids. … and the GUI monitors the IJP server (see next slide) Deleting a job removes it from the IJP’s memory, but not any provenance records that the job creates in ICAT.

Monitoring batch jobs Job history, status, management Output of running job Here are a couple of screenshots showing how users can monitor jobs from the IJP GUI. The IJP plugin extends TopCAT with a My Jobs tab that lists all of the user’s jobs. The second-last column shows the job status (Queued, Executing, Completed etc.) The last column contains buttons to Cancel a job (if it is queued or running) or Delete it (after it has completed). Clicking on a job displays its standard and/or error output; if the job is still running this will be updated as it changes.

Interactive jobs Batch connector selects a worker node Node is removed from pool of available workers Sets up RDP session to run interactive executable RDP connection details passed back to IJP server GUI launches Remote Desktop (Windows) or gives pasteable command line (Linux) Batch connector releases worker once session is closed tries hard not to leave dangling interactive sessions As well as batch jobs, the IJP supports interactive jobs – jobs that launch an interactive GUI. When the user asks to run an interactive job, the chosen batch connector… Linux version is a little clunky – need to think about that… It’s possible to close Remote Desktop without exiting the RDP session, so batch connector will kill idle sessions.

Jobscripts Executable that runs on batch system workers Receives dataset/datafile IDs, options, session tokens on command line Uses IDS to retrieve datasets/datafiles (or ICAT for metadata) Should add provenance records to ICAT Does not communicate with the IJP Provenance: e.g. this job was run with these datasets and parameters, and created these new datasets

Developing jobs Create jobscript Deploy jobscript on batch system Python utility library for argument processing Python-icat or similar to work with ICAT / IDS Deploy jobscript on batch system Define jobtype XML Add to IJP server configuration (dynamic) Utility library mainly just extends the python args processor to declare the IJP arg options Torque batch processor uses puppet to manage the worker nodes; so add jobscripts to puppet config.

Recent developments New GUI TopCAT plugin (AngularJS) RESTful interface to IJP server Original GWT GUI still part of server, but won’t be developed further OK, that should be Recent Development (singular). See Future Development for what else we’d hoped to have done by now  The GUI work was done by Rebecca Fair, initially as a variant of TopCAT, then as a plugin.

Current status One active customer, Octopus (CLF) Test system in place (ingest, jobscript development) Not yet in production Batch connectors Torque Unix batch (for demos/tests) Platform LSF (incomplete) Perhaps should say more about Octopus, but may not have time! Central Laser Facility. Ingest process does not use the IJP Hopefully in production next year.

Future development Improve batch system brokering Add batch requirements to job types (e.g. requires GPU) Support versioning of datasets Specific requirement from Octopus – post-ingestion modifications IJP GUI should only show latest version of each dataset (custom results filtering) Version management separate from IJP, but may be developed as jobs Integration with / adoption of DaaS architecture? (Is DaaS specific to STFC?)