1. GridPP37, Ambleside 2016-08-31 Adrian Coveney (STFC)
APEL Update
GridPP37, Ambleside
Adrian Coveney (STFC)
TODO: REBUS, INDIGO, check portal link

2. Outline
The APEL team
Accounting system overview and accounting records
REBUS
Recent developments
Indigo DataCloud
APEL in Indigo
How you can help us
Contacts and links

3. The APEL Team Adrian Coveney John Gordon Greg Corbett
Team Leader
EGI and everything else
John Gordon
WLCG and EGI
Greg Corbett
Newest member
INDIGO DataCloud, EGI Federated Cloud
(Stuart Pullinger)
Career Break

4. Accounting System Overview
APEL collects accounting data from sites participating in the EGI and WLCG infrastructures, and sites in other Grid organisations collaborating with EGI, including OSG, NorduGrid and INFN.
Accounting data is collected from different sensors into a central repository, and then processed to generate statistical summaries that are available through the EGI/WLCG Accounting Portal.
Statistics are available to view by Users, VO Managers, Site Administrators and anonymous users according to defined access rights.

5. Accounting records Job Records Summary Records
Made from joining batch records with CE records.
Summary Records
Aggregations of job records.
Normalised Summary Records
As per summary records but with times normalised by benchmarks.
Others
Cloud accounting/summary records.
Storage accounting records.
(Data set usage records.)

6. Cloud System, e.g. OpenStack
Cream
Batch
APEL Parser
APEL
APEL S M S M S M
Client
Server
Batch
System S M
ARC or
QCG
EGI Message
Brokers
Cloud System, e.g. OpenStack
EGI S M S M
WEB
Caso/ONE
APEL clients can run an APEL parser to extract data from a batch system and place it in their client database, or they can use third-party tools to extract batch or cloud data. This data is then unloaded into a message format suitable for transmission.
APEL clients run a sending Secure Stomp Messenger (SSM) to send these messages containing records via the EGI Message Brokers to the central APEL server. The messages can contain either Job Records or Summary records. This is configurable in the APEL client.
The central APEL server runs an instance of the SSM, which receives these messages and a “loader” processes the records in the messages and loads them into a MySQL database.
A “summariser” process runs to create summaries of any Job Records received and load them into a “SuperSummaries” table along with any Summary records. This summariser runs as a cron job approximately once a day.
A database “unloader” process unloads the summary records into the message format to be sent on by the sending SSM via the EGI Message Brokers to the EGI Accounting Portal.
Portal

7. REBUS (I) WLCG REsource, Balance, and Usage
Not controlled by APEL but it both takes data from, and provides metadata to, the Accounting Portal.
A database at CERN which stores useful information for WLCG
Topology – the definitive metadata on T1 sites, T2 federations in countries and their constituent sites. Used by Accounting Portal
Pledges – WLCG resources pledged per T1 and T2 site. Input by WLCG country management (DB)
Capacities – harvests data from BDII on installed capacity at sites. Physical and logical cpus, average HS06, total capacity.

8. REBUS (II)
Reports – T1s can input their actual installed capacities per VO.
Trends – plots of a few metrics as a function of time. Data taken from Accounting Portal.
Accounting – Monthly WLCG reports showing usage vs pledges. Data loaded from Accounting Portal with possibility for T1s to correct/fudge their data. Reports then archived in WLCG wiki.

9. Recent Developments (I)
Storage Accounting
Thank you if you’ve helped us test storage accounting. This has let us track down a bug at the central repository.
We will have fixed the problem soon and hope to roll out to more sites afterwards.
The fix will also simplify the addition of storage summary records at a later date.
Data Set Accounting
Feedback received from EGI partners on first draft of metrics.
Looking to revise and test with a provider.

10. Recent Developments (II)
WLCG Accounting Task Force
Generally happy with the new Accounting Portal.
Trying to verify data in portal from experiment data. Some success with ATLAS but also discrepancies.
Tool created at CERN to help automate the process of comparing data (ATLAS, ALICE, and CMS).
They would like unscaled wallclock but some batch systems scale the wallclock. (On long-term wishlist.)
CERN revamping how they report to APEL, so their data should soon be much more correct than previously.
Task force set up to review the accounting tools and propose improvements.

11. Indigo DataCloud (I)
An H2020 project from April 2015 to September 2017
Who: 26 European partners in 11 European countries
What: develop an open source Cloud platform for computing and data tailored to science.
For: multi-disciplinary scientific communities
Where: deployable on hybrid (public or private) Cloud infrastructures
Why: answer to the technological needs of scientists seeking to easily exploit distributed Cloud/Grid compute and data resources.
Slides courtesy of Giacinto Donvito,
INFN - BARI, Italy

12. Indigo DataCloud (II) How the Service Works
Top level - users submit “jobs” via TOSCA files
Middle level determines what VMs, Docker Containers and Storage is needed
Lowest level is the underlying infrastructure to instantiate these resources
Indigo DataCloud Partners
Slides courtesy of Giacinto Donvito,
INFN - BARI, Italy

13. APEL in Indigo (I) APEL will account for the underlying resource usage
Developments undertaken
(Read + Write) REST interface to an underlying APEL Server
Summarises per day rather than by month
Packaged as a Docker container, deployable on Kubernetes
Future
Support for cloud v0.4 records
No changes to EGI Infrastructure
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
Support for cloud 0.4 will come after support in main APEL software.

14. Indigo DataCloud Micro-Services
APEL in Indigo (II)
Other
Indigo DataCloud Micro-Services
Cloud Records created via Caso and OneAcct (as in EGI).
Sent via POST requests.
Cloud Summaries are retrieved by other micro-services via GET requests.

15. How you can help us
Checking if batch systems change their output formats
Usually no notice or documentation.
When sites test a new release of a batch system, can they also check that the parsers still work and produce sensible results.
Sending summaries instead of individual job records
More efficient in terms of network traffic.
Less load on the central database.
Also protects against individual job records going missing in transit as summaries are sent for the current and previous month, every day.

16. Contacts and Links GGUS Support Unit – APEL: https://ggus.eu/
GitHub:
New Accounting Portal:
CERN comparison tool:

17. Thank you.