PanDA Integration with the SLAC Pipeline Torre Wenaus, BNL BigPanDA Workshop October 21, 2013

PanDA and LSST/DESC Gave a PanDA overview at the January 2013 LSST Dark Energy Sciences Collaboration (DESC) meeting Decision made to prototype PanDA as (another) submission engine for Tony Johnsons pipeline workload manager used by Fermi and now DESC (among others) – Gain experience in both directions – Gain DESC access to more resources via PanDA – Evaluate PanDAs usefulness In February Tony provided a Java interface to the pipeline for submission engines Torre Wenaus, BNL2 Oct

SLAC Workflow Pipeline and PanDA Prototype PanDA as distributed job manager and gateway to grid, cloud, HPC resources Torre Wenaus, BNL3 Oct

PanDA System Overview Torre Wenaus, BNL4 Oct

PanDA and LSST Pipeline => PanDA submission working, tested with HelloPanda and a standard DESC simulation workflow (phosim, DM stack) – Anyone who can submit a pipeline task can submit to PanDA Initial tests used ATLAS PanDA queue at BNL (epsilon level) and CERN based standalone pilot submission (no APF) – BNL PanDA instance being set up for non-ATLAS use in US – Discussions started with OSG (Miron so far) re: an OSG supported PanDA instance once PanDA is cleanly packaged and (re)ported to MySQL Currently borrowing BNL ATLAS resources during testing phase – Can submit jobs (pilots) as LSST VO as soon as desirable Plenty of US ATLAS resources to be had opportunistically right now; a quiet period (there will regularly be quiet periods like this until 2015) Torre Wenaus, BNL5 Oct

Pipeline/PanDA integration: Pipeline interface Java daemon JobControlClient accepts 3 control commands from the pipeline: submit, status, kill – Submit Receive command and arguments Submit the job Job s the pipeline server on job start and finish – Status Report job status to pipeline server – Kill Kill job PanDA implementation extends the JobControlClient class – Communicates with pipeline server via Java RMI Tested first with an emulator for the pipeline server (provided so a JobControlClient can be tested standalone) then PanDA daemon was connected to the real pipeline service at SLAC Torre Wenaus, BNL6 Oct

Pipeline/PanDA integration PandaJobControlService implemented supporting the submit, status, cancel functions Submit: – Service creates and submits a PanDA job configured with the specified command/arguments Job is configured to invoke a DESC-specific transformation (payload script) Following pipeline conventions, the script s the pipeline server on payload job start/finish. Depends on ing always working from WNs. – Returns PandaID as the ID identifying the job – LSST jobs are picked up from the PanDA job queue by standard PanDA production pilots – Amazon S3 used as staging area for job files – No special provision for data management at present; jobs are on their own in accessing data Torre Wenaus, BNL7 Oct

Pipeline/PanDA integration (2) Status: – Service queries PanDA for status of current pipeline jobs – Programmatic API to the PanDA monitor used to obtain job status (curl query returns json result) – Service maps PanDA job states, etc. onto pipeline conventions and returns pipeline JobStatus Cancel: – Service directs PanDA to cancel the job (request to PanDA server) – The PanDA server immediately sets the job state to cancelled – The job itself will initiate self-destruct when it gets its next heartbeat reply (every 30min) that will contain the cancel command Torre Wenaus, BNL8 Oct

LSST PanDA jobs in pipeline and PanDA monitors (June) Torre Wenaus, BNL9 Oct

Activities Since June Work program established with Open Science Grid (OSG) to move PanDA towards becoming an OSG service – US PanDA instance supporting OSG VOs Predicated on generalizing PanDA for non-ATLAS use: modularity, packaging, documentation, no Oracle requirement: the program of the BigPanDA project – Jarka in particular is working to take up LSST DESC as the first supported BigPanDA VO using the new BigPanDA infrastructure Jarkas work program since June: – Implement MySQL based PanDA server as a tandem (with Oracle) implementation for OSG and other non-ATLAS use (done) – Establish Amazon EC2 based MySQL PanDA service for OSG pre- production and development (done) to be cloned as stable production instance when need arises – Establish a VO-neutral PanDA monitor skeleton operating against the EC2 PanDA service for OSG VO PanDA monitoring (in progress) Oct Torre Wenaus, BNL10

Current Activities With the infrastructure now in place for OSG VO support, the next step and present work-in-progress is the LSST VO implementation PanDA-pipeline daemon (Tonys pipeline interface) moved from a dev EC2 instance to the PanDA OSG VO server Jarka and I working on implementing the pipeline-triggered PanDA job submission on the PanDA OSG VO server BNL now supports the LSST VO; a new PanDA site BNL-LSST created for LSST jobs – Submission to the BNL-LSST site being debugged Oct Torre Wenaus, BNL11

Next Steps Finish setting up the BNL-LSST PanDA site to receive real jobs from the pipeline – Software installed and DESC expert identified to help with simu workload Finish setting up LSST PanDA monitoring alpha version Enlist and work with users to get their workloads up and running – One or two DESCers at BNL are likely first candidates Get LSST user input to tailor LSST PanDA monitoring Extend to support opportunistic OSG (or non-OSG PanDA) sites – ie sites that dont necessarily have a local LSST person helping out – Key issues are software and data availability – For software, set up OSGs Oasis service for CVMFS based distribution – For data access, xrootd a good candidate Integrate the SLAC data catalog – The developer presented last week that it is a few months away from the new VO-neutral REST interface being ready Oct Torre Wenaus, BNL12

Utility of the pipeline/data catalog for BigPanDA? Pipeline provides a user-friendly web based interface to defining potentially very complex DAG workflows, saved and managed as tasks in projects Similar, at least superficially, to what we want from DEFT – Could it be useful? Data catalog similarly user-friendly, supports datasets and folders, dataset and file based metadata, VO- neutral… potentially interesting as a data management offering in association with BigPanDA? Torre Wenaus, BNL13 Oct