1.
R.Mashinistov (UTA)
July 20-21

2. Introduction
In 2017 the pilot project between BigPanDA and Blue Brain Project (BBP) (Swiss Federal Institute of Technology in Lausanne) teams began. The proof of concept project aimed to demonstrate efficient application of the PanDA software for the supercomputer-based reconstructions and simulations offering a radically new approach for understanding the multilevel structure and function of the brain.
In the first phase, the goal of this joint project is to support the execution of BBP software on a variety of distributed computing systems powered by PanDA. The targeted systems for demonstration include: Intel x86-NVIDIA GPU based BBP clusters located in Geneva (47 TFlops) and Lugano (81 TFlops), BBP IBM BlueGene/Q supercomputer ( 0.78 PFLops and 65 TB of DRAM memory) located in Lugano, the Titan Supercomputer with peak theoretical performance 27 PFlops operated by the Oak Ridge Leadership Computing Facility (OLCF), and Cloud based resources such as Amazon Cloud.

3. BBP computing infrastructure
ssh
slurm
slurm
slurm
pbs
api & sheduler
BlueGene/Q
viz cluster
lxviz cluster
Titan supercomputer
On-demand cluster
LUGANO
GENEVE

4. Initial PanDA test @ BBP
For initial test in March 2017 PanDA Server + Client were installed to the VM at the Campus Biotech (Geneva). Job submission has been done via the python based PanDA Client. Pilot were started manually on Geneve and Lugano clusters including BlueGene/Q. Titan & Amazon resources were integrated on later (on April).
SQL
https
PanDA Server
PanDA DB
PanDA Client
REST API
REST API
https
Pilot
Pilot
Pilot
Resource A
Resource B
Resource Z
July 20-21

5. PanDA Portal @ BBP (1/2) Authentication File Catalog
Data Transfer System
Data Management
System
Web Services
Pre/post processing
SQL
SQL
https
PanDA Server
PanDA DB
PanDA Monitor
PanDA Client
REST API
REST API
https
Pilot Scheduler
Pilot Scheduler
Pilot
Pilot
Pilot
Resource A
Resource B
Resource Z
July 20-21

6. PanDA Portal @ BBP (2/2) PanDA Portal Authentication File Catalog
Data Transfer System
Data Management
System
Web Services
Pre/post processing
SQL
SQL
https
PanDA Server
PanDA DB
PanDA Monitor
PanDA Client
REST API
REST API
https
Pilot Scheduler
Pilot Scheduler
Pilot
Pilot
Pilot
Resource A
Resource B
Resource Z
July 20-21

7. PanDA for BlueBrainProject
ssh
https
https
Various
interfaces
The test ‘Hello world’ jobs were successfully submited to the targeted resources via PanDA portal.
CLI
REST API
WEB UI
PanDA
Queues to execute jobs on different resources
PanDA
queues
BlueGene/Q
viz cluster
lxviz cluster
Titan
OLCF
GENEVE
LUGANO

8. User Interfaces Web forms to define, submit and monitor jobs
Authentication
LDAP as a first solution
Will be replaced by SSO
Possible to integrate with Jupyter notebooks (widely used by BlueBrain scientists)
Easy to change computing cluster (jobs are not atomic, typically scientist knows which cluster should be used with his computational job)
July 20-21

9. Impersonation Need to track job submitter on computing cluster
Pilot should have access only to the private files of job owner
It’s possible to propagate name and group of user inside job definition
Root access approach
Run pilot under root, use name/group as sbatch params
Root writes transformation wrapper with SUID, use name/group as sbatch params inside wrapper
No-root access approach
Use separate queue for every working group (cannot add cluster user for every project participant)
Run pilot under working group manager account
Working groups can be mapped with existing groups in LDAP or SSO
July 20-21

10. Conclusion Phase 1 of “Proof of the concept” was successfully finished
The project demonstrated that the software tools and methods for processing large volumes of experimental data, which have been developed initially for experiments at the LHC accelerator, can be successfully applied to BBP.
Phase 2 of “Pre-production” is under investigation currently
Next most significant tasks are:
Data management system
Jupyter integration