1. KIT visit to Cascina , 22 March 2019
Virgo Computing
Franco Carbognani
KIT visit to Cascina , 22 March 2019

2. KIT visit to Cascina , 22 March 2019
Virgo Computing Reorg
Virgo Offline and Online Computing management has been reorganized as one of the outcomes of the 2018 External Computing Committee (ECC ) recommendations
This has also lead to the definition of the “AdVirgo Computing and Data Processing Infrastructure Work Breakdown Structure (WBS)” document
AdVirgo Computing Model, Implementation Plan and Management Plan documents are being reviewed and updated as deliverables of WP10
KIT visit to Cascina , 22 March 2019

3. AdVirgo Data Flow schema
Advanced Virgo Computing Model

4. AdVirgo Data Flow: The GW170814 case
The signal arrives
Data composed into frames
Calibration of the data
Veto, DQ flags production
h(t) transfer
Low-latency matched-filter pipelines
Upload to GraceDB
Data written into on-line storage
Low-latency data quality
Low-latency sky localization
GCN Circular sent out
Data written into Cascina Mass Storage
Data transfer toward aLIGO and CCs
KIT visit to Cascina , 22 March 2019

5. AdVirgo Computing Layers
17/10/2018
Advanced Virgo Computing Model

6. Data Processing Infrastructure
Virgo Week Jan 2019
KIT visit to Cascina , 22 March 2019

7. Computing and Data Processing
CDP
WP8
Offline Computing Services
WP9
Data Analysis Tools and Pipelines
WP10
LVC Computing Strategy
WP12
New Developments
WP11
Public Data Open Science
WP8.1
CVMFS support at CCs
WP9.1
Continuous Waves (CW)
WP10.1
Updated Virgo Computing Model
WP11.1
O2 Data Release
WP12.1
ML/DL strategies for GW detections
WP8.2
Usage reporting in LVC accounting
WP9.2
Compact Binary Coalescence (CBC)
WP11.2
Open Data Science Workshop
KIT visit to Cascina , 22 March 2019

8. Virgo Platform & Services (WP1)
Expansion of Raw Data Circular Buffer
Expanded to 700TB => ~ 6.3 months at 45 MB/s
1PB final expansion (~12 months) ongoing
Online Storage buffer upgraded
Condor farm, used for Noemi, Detchar and cWB low latency pipeline expanded to reach 160 cpus
Cascina site link bandwidth increased from to 1Gb/s to 10Gb/s.
Bulk data transfer to Tier1 CC
Legacy solution, GFAL2/iRODS based running smoothly during ER14
LIGO/Virgo common web access via ligo.org authorization
Completed for VIM, other relevant web pages will follow
KIT visit to Cascina , 22 March 2019

9. Software Management (WP2)
Supporting release cycles associated to Engineering Runs and Science Run
A joint Ligo-Virgo Software Configuration Control Board (SCCB) is in place for the relevant Low Latency software
Software stored on a centralized SVN archive in Cascina.
Transition from SVN to Git ongoing. Currently mirroring relevant packages from Virgo SVN to git.ligo.org
Setting up CVMFS for Virgo software distribution
Joint Ligo-Virgo proposal “Sustainable software development and distribution with the conda package manager and the conda-forge” accepted and initial work has started
KIT visit to Cascina , 22 March 2019

10. Online System Processes (WP3)
KIT visit to Cascina , 22 March 2019

11. Data Quality & Detchar (WP4)
 All analysis run at Cascina on EGO machines
One dedicated machine for high CPU/memory analysis
5 virtual machines for online analysis
Condor farm for the data quality reports (DQRs)
Outputs uploaded to GraceDB to produce joint LIGO-Virgo DQRs
Analysis of the past day run on a nightly basis
Interactive machines: farmn & ctrl
 Main input: raw data
 Few TB of storage
 DBs: segments, noise lines, etc.
 Web server to display results and browse the associated reports
KIT visit to Cascina , 22 March 2019

12. Low Latency data distribution (WP5)
Activities and plans
Transfer LIGO h(t) frames from LLH and LHO to EGO directly (by-passing the CIT node)
Evaluating use of Kafka instead of Cm for intersite h(t) frame exchange (WP5.1)
KIT visit to Cascina , 22 March 2019

13. Low Latency data distribution (WP5)
Kafka evaluation
We are evaluating the use of Kafka for the Cascina – aLIGO link as a replacement for the current Cm (Virgo specific) based solution. Kafka is a modern message-query which embed a smart fail-over mechanism and will be used by aLIGo for all other Low Latency links (including toward Kagra).
Status:
Setup parallel live data Kafka streams (CIT <-> Cascina) being compared with Cm based ones
KIT visit to Cascina , 22 March 2019

14. Bulk Data Handling (WP7)
The main problem
We have been confronted with a strong diversity among Tier1 CC that has always prevented the development of a unified solution
For the post O3 timeframe we are aiming to a unified solution
Investigate and implement a unifying solution based on a EGI/OSG framework in coordination with Tier1 CCs
Must provide a full Data Transfer and Access integrated solution for a full mesh topology among all AdVirgo/LIGO data endpoints
KIT visit to Cascina , 22 March 2019

15. Offline-computing (WP8 – WP9)
The main problem
The majority of the LVC common pipelines:
Run on HTCondor clusters fully devoted to LIGO
Look for data and libraries in a local shared file system
Pipelines are intrinsically uncompliant with GRID and it’s difficult to have them running at CNAF, CCIN2P3, Nikhef, ….
For a better utilization of Virgo computing resources:
Pilot projects to enable main CBC detection pipelines to run on Virgo resources directly, via European grid submission, or via OSG
Exploration of DIRAC and Rucio for workflows
Support of O2 CW analyses with additional computing power provided at CNAF and Nikhef/SURFSara clusters
Setting up CVMFS areas at Virgo CCs for data
KIT visit to Cascina , 22 March 2019

16. Offline-computing (WP10 – WP12)
New draft of updated Virgo computing model (May 2019)
Longer-term goal: integrated LVC computing model
Open Data prep, formatting, publication, etc.
Translation to local languages; mirror server in Europe
Open Data Science Workshop in Paris (Apr )
Exploration and development of ML/DL pipelines
1st Conference on ML for GWs, Geophysics, Robotics and Control Systems (Jan 2019 at EGO)
KIT visit to Cascina , 22 March 2019