2. Plans for the renovation of the Post Mortem infrastructure
TE-MPE-TM 82
15/09/2016
Matthias Pöschl – TE-MPE-MS

3. Matthias Pöschl – TE-MPE-MS
Agenda
Current Post Mortem Architecture
Shortcomings
Improvements
New Post Mortem Storage
New File Format
Collaboration with CALS 2.0
Shared data and infrastructure
New Post Mortem Architecture
Conclusion
15/09/2016
Matthias Pöschl – TE-MPE-MS

4. Current Archtitecture
15/09/2016
Matthias Pöschl – TE-MPE-MS

5. Matthias Pöschl – TE-MPE-MS
Shortcomings
Direct user access to underlaying filesystem and file format
Outdated data collection stack
Manual load balancing
Very limited horizontal scaling
Unfit for future use-cases with strict time constraints
15/09/2016
Matthias Pöschl – TE-MPE-MS

6. Matthias Pöschl – TE-MPE-MS
Improvements
Update storage technology
Allow for dynamic load balancing
Update data collection stack and file format
User access only through REST API
15/09/2016
Matthias Pöschl – TE-MPE-MS

7. New Post Mortem Storage
Benchmark of Ceph, MongoDB and GlusterFS
Use of real Post Mortem Data (all of 2015)
Constraints:
Technology has to handle drive and node failures, avoid data inconsistencies and degrade gracefully
Three replicas of each object have to be stored
Acknowledge of write only after all three copies have been written
Adding more nodes should increase capacity and throughput by a known factor (“linear scaling”)
15/09/2016
Matthias Pöschl – TE-MPE-MS

8. New Post Mortem Storage
GlusterFS performed best in read-only and write-only benchmarks, very closely followed by Ceph
Ceph showed best mixed-workload performance:
Technology
Time to complete
Objects/s
Throughput
NFS
26.3h
340
8,018 KB/s
Ceph
20.4h
438
12,154 KB/s
MongoDB
23.5h
381
8,973 KB/s
GlusterFS
20.7h
432
10,187 KB/s
Explain Testbench, Servers, HDD rpms, …
15/09/2016
Matthias Pöschl – TE-MPE-MS

9. New Post Mortem Storage
Meeting with Dan van der Ster and Herve Rousseau from IT-ST-FDO to discuss Ceph
IT has had good experiences with Ceph
Biggest Ceph test so far with a ~30PB cluster
IT is running the VMs on top of Ceph
Not a single byte lost in +5 years of operation
IT sees no problem in using it for Post Mortem
IT is willing to provide support and assistance
15/09/2016
Matthias Pöschl – TE-MPE-MS

10. New Post Mortem Storage
Test with test Ceph cluser provided by IT
Explain variations in graph -> Filesize
Benchmark
Time to complete
Objects/s
Throughput
Import
12.9h
345
8,135 KB/s
Read
3.5h
1269
30,080 KB/s
Read/Write
16.8h
795
12,552 KB/s
15/09/2016
Matthias Pöschl – TE-MPE-MS

11. Matthias Pöschl – TE-MPE-MS
New file format
Aspirations on using Apache Avro as file format for storage and to serve the users
“Raw” RDA-data will still be stored for safety
Avro offers many useful features
Partial data retrieval (specific signals from a dump)
Efficient and fast compression
Self-describing schema
Libraries for almost every programming language
Allows direct conversion to JSON
15/09/2016
Matthias Pöschl – TE-MPE-MS

12. Collaboration with CALS 2.0
The Logging Service team is updating their whole infrastructure
Oracle cluster will be replaced with Hadoop
15/09/2016
Matthias Pöschl – TE-MPE-MS

13. Collaboration with CALS 2.0
New CALS design will allow BigData queries
“Show me the biggest deviations from the mean of a certain BLM in sector 67 in the year 2015 when dumping at 6.5 TeV”
Idea: Feed the Logging Service the high resolution Post Mortem data to make this kind of queries even more valuable for the users
15/09/2016
Matthias Pöschl – TE-MPE-MS

14. Collaboration with CALS 2.0
Meetings with Chris Roderick, Jakub Wozniak and Marcin Sobieszek from BE-CO-DS to evaluate common use of technologies and services as well as details of data ingestion
Result:
CALS might also use Avro for their data ingestion
Possible shared use of a Kafka cluster
Shared data storage not (yet) possible
15/09/2016
Matthias Pöschl – TE-MPE-MS

15. New Post Mortem Architecture
15/09/2016
Matthias Pöschl – TE-MPE-MS

16. Matthias Pöschl – TE-MPE-MS
Conclusion
Ceph cluster for Post Mortem data seems feasible and suitable to tackle future use cases
No common storage of CALS and PM (yet )
different timing constraints and data sizes
different preferred storage technologies
But: Data and infrastructure can be shared between both systems, providing a good trade-off for the users
15/09/2016
Matthias Pöschl – TE-MPE-MS

17. Matthias Pöschl – TE-MPE-MS
Conclusion
Avro allows efficient data storage, convenience for the users and easy integration with CALS 2.0
All data access through a REST API, serving
uncompressed JSON
compressed Avro
15/09/2016
Matthias Pöschl – TE-MPE-MS