1. A computational ecosystem for near real-time satellite data processing
Stefano Piani

2. L2VDP
Copyright 2016 EUMETSAT
Stefano Piani
2 of

3. Build a computational ecosystem
Topics
Re-engineering different software to build an end to end processing system
Improve the performance of the most time- consuming section of the code
Build a computational ecosystem
Main goal
Process the data coming from MetOp satellites in near real time
Stefano Piani
3 of

4. Python as a “glue” language A complete modular system
L2VDP structure
Python as a “glue” language
A complete modular system
All parameters can be set in one single file
Stefano Piani
4 of

5. Improve the performance of the processor as much as possible
Second Goal
Improve the performance of the processor as much as possible
(in two months!)
Stefano Piani
5 of

6. Methodology Profile the code Enhance the most time-consuming part
Test the improvement
Stefano Piani
6 of

7. Computational distribution
Stefano Piani
7 of

8. Step 1: Improving matrix multiplication
A product among three matrices used almost half of the time of the overall matrix multiplication
Changing the order decreases the number of floating point operations by a factor 4!
The code is 1.35 time faster
Stefano Piani
8 of

9. Step 3: Exploit matrix structure
We discover that an element of a matrix multiplication was a block matrix.
With this algorithm, the code is 1.37 times faster
Stefano Piani
9 of

10. Other analogous improvements were applied in others block and diagonal matrices
Stefano Piani
10 of

11. Step 6: Avoid wasting sparseness
The original code used the inverse of a sparse (banded) matrix.
Solving a linear system is usually better than compute the inverse of a matrix.
Removed the inverse and used a dedicated library to solve sparse linear systems.
Matrix multiplication improved by a factor 1.76
Stefano Piani
11 of

12. We delivered 6 version with one improvement for each
An overall view
We delivered 6 version with one improvement for each
Stefano Piani
12 of

13. Matrix multiplication improved of 7.5 SVD improved of 4.6
The final results
Improvement of 2.7 times (5.7 if we do not take into account the OSS Forward Model)
Matrix multiplication improved of 7.5
SVD improved of 4.6
Total time decreased from 178 seconds to 65 seconds
Stefano Piani
13 of

14. The computational ecosystem
Near real-time: we have to process 2700 FOVs every 3 minutes
Data should be elaborated as soon as possible
Stefano Piani
14 of

15. The computational ecosystem
A master/slave approach
Stefano Piani
15 of

16. The computational ecosystem
We need to embed the software inside a sort of “container”: a Docker
We need to embed the software inside a sort of “container”
Stefano Piani
16 of

17. The computational ecosystem
We need a way to distribute the workload
Stefano Piani
17 of

18. Deployment on TCE
Stefano Piani
18 of

19. Benchmark Run of 45 hours Observable Iasi A 6 X 8 core machines Iasi B
Granula arrived
877
873
Granula submitted
876
846
Granula discarded 1 37
Average process time
311 s
186 s
Average preprocess time
102 s
Min process time
31 s
37 s
Max process time
1702 s
976 s
Stefano Piani
19 of

20. A complete integration of all the processing phase has been performed
Conclusions
A complete integration of all the processing phase has been performed
The most time computing part of the code is now three time faster than before
The ecosystem has been proved to be stable and efficient, allowing to process in near real-time a large amount of data
Stefano Piani
20 of

21. Thank you!
Stefano Piani
21 of