1. Euclid Consortium OU-LE3 KOM Meeting, London 1March 22-23, 20121 OU /SDC Proposed Development Strategy Maurice Poncet (CNES, SDC-FR Lead) Andrea Zacchei (Inaf – SDC-IT Lead) Keith Noddle (IfA – SDC-GB Lead)

2. Euclid Consortium OU-LE3 KOM Meeting, London 2March 22-23, 2012 Euclid – Ground Segment Concepts The recurring acronyms of the ground segment:

3. Euclid Consortium OU-LE3 KOM Meeting, London 3March 22-23, 2012 Euclid – Ground Segment Concepts OUs (Organization Unit) : Responsible for the definition and prototyping of a given pipeline (i.e transform scientific req. from SWG into algorithms or mathematical methods) Responsible for the validation of a given pipeline (i.e providing baseline test data) SDCs (Science Data Centers) : Responsible for the S/W development of pipelines : SDC Dev Responsible for the H/W processing infrastructure : SDC Prod Responsible for the pipeline processing operations : SDC Prod OU SDC DEV SDC PROD

4. Euclid Consortium OU-LE3 KOM Meeting, London 4March 22-23, 2012 SDCs - Development Strategy SDC-xx DEV Team is a “delocalized” team – Distributed over several labs – Integration of other countries contributions This team is lead by only one responsible which is at SDC-xx Dev level SDC-xx DEV Team works through a collaborative web based development infrastructure A given SDC-xx DEV Team may contribute to SDC of other countries (under their responsibility) There is one dev team per pipeline SOC should be involved as any other SDC 4

5. Euclid Consortium OU-LE3 KOM Meeting, London 5March 22-23, 2012 SDCs – Implementation Involvement L : Lead C : Contribution (as corresp. OU Co lead) c : other contribution

6. Euclid Consortium OU-LE3 KOM Meeting, London 6March 22-23, 2012 SDCs – SDC Generic WPs WP-4-4-nn SDC- WP-4-4-nn-1000 SDC- Management WP-4-4-nn-2000 SDC- -DEV WP-4-4-nn-2100 SDC- -DEV Infra. WP-4-4-nn-2200 SDC- -DEV Implementation WP-4-4-nn-22sb pipeline Implementation WP-4-4-nn-22sb-st pipeline Implementation subtask WP-4-4-xx-2300 SDC- -DEV Integration WP-4-4-xx-23sb pipeline Integration WP-4-4-nn-23sb-st pipeline Integration subtask WP-4-4-nn-3000 SDC- -PROD WP-4-4-nn-3100 SDC- -PROD Infra. WP-4-4-nn-3200 SDC- -PROD Operation WP-4-4-nn-32sb pipeline operation WP-4-4-nn-32sb-st pipeline operation subtask 6 Feel free to propose any other set up.

7. Euclid Consortium OU-LE3 KOM Meeting, London 7March 22-23, 2012 SDC-DEV – Dev Environment Local Infra Dev Compilation Unit Test SDC-DEV Local Infra Dev Compilation Unit Test Local Infra Dev Compilation Unit Test Local Infra Dev Compilation Unit Tests S/W int tests e – Infra Conf Mangt Extraction Generation (Preint) tests Quality Check Dash Board SDC-PROD Integration Validation Deployment Generation Int/Val tests Perf. Test Quality Check Dash Board Production Data Ingestion Processing Data Distribution SDC Labs Located Computing Center(s) 7 EMA EC level S/W conf mangt )

8. Euclid Consortium OU-LE3 KOM Meeting, London 8March 22-23, 2012 SDCs - Development Environment & Context Tailored (i.e. simplified & adapted) ECSS process should be applied Linux platform Model based Approach : SysML Scripting language : Python Coding language : C/C++ Distributed Development (cf. open source best practices) Test driven approach Iterative & Incremental process (Agile Approach) Continuous S/W Integration Strong configuration management Automation (extraction, build, test, code quality check, …) Dashboards (motivating !) Online tools Collaborative documentation 8

9. Euclid Consortium OU-LE3 KOM Meeting, London 9March 22-23, 2012 OU / SDC-DEV – Agile Approach (1/2) Iterative & Incremental development – Sprints (e.g. monthly or less) : sub functions – Release (e.g. half year or less) => S/W available in early phases, ability to adapt the S/W

10. Euclid Consortium OU-LE3 KOM Meeting, London 10March 22-23, 2012 OU / SDC-DEV – Agile Approach (2/2) OU is “Product Owner”: – Backlog management : Prioritized list (e.g. priority, maturity, …) of pipeline “features”, – Definition of releases and sprints contents in coordination with SDC-DEV team (e.g. according to SDC-DEV capability), – Validation of pipeline with SDC-DEV team. SDC-DEV team responsible of sprint & release dev. OU SDC DEV

11. Euclid Consortium OU-LE3 KOM Meeting, London 11March 22-23, 2012 Continuous Integration Principles * Maintain a code repository * Automate the build * Make the build self-testing * Everyone commits to the baseline every day * Every commit (to baseline) should be built * Keep the build fast * Test in a clone of the production environment * Make it easy to get the latest deliverables * Everyone can see the results of the latest build * Automate deployment

12. Euclid Consortium OU-LE3 KOM Meeting, London 12March 22-23, 2012 SDCs - Development Phase Pipeline Development in respect with SGS and System rules – ECSS tailoring (documentation, dev cycle) – UML modeling (common look & feel) – Pipeline I/F – IAL principles – Products, Data modeling with modeling tool – Quality rules (check should be as much as possible integrated and automated inside the collaborative platform) Potential common S/W modules should be submitted at system level 12

13. Euclid Consortium OU-LE3 KOM Meeting, London 13March 22-23, 2012 SDCs – Integration Phase “target” pre-integration may be done at SDC-DEV side on virtual machines being representative of target configuration. Target Integration should be done on SDC-Prod infra. in a dedicated Integration « sand box » Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD and OU teams “Technical” tests designed at SDC level “Functional” tests designed at OU level Integration tests output data should be accessible to above teams for validation and further investigation 13

14. Euclid Consortium OU-LE3 KOM Meeting, London 14March 22-23, 2012 SDCs – Validation Phase Validation should be done on SDC-Prod infra. in a dedicated Validation « sand box » Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD, OU teams, and as needed SWG. “Scientific” tests designed at OU/SWG level Validation tests output data should be accessible to corresponding OU team and SWG for validation and in case of failure for further investigation. 14

15. Euclid Consortium OU-LE3 KOM Meeting, London 15March 22-23, 2012 SDCs – Collaborative Development Platform A Collaborative Development Platform should be set up: – Aims : verify tools readiness, simulate dev cycle, may be used for SGS mock-up. – O/S: Linux – Languages: Python, C++ – Configuration: SVN – Generation: Maven – Tests: xUnit – Integration: Jenkins – Quality: Sonar (with C++ plug-in) – IDE: Eclipse, Netbeans, vi, whatever… – Tracking: Jira, Redmine – Doc Gen.: Doxygen – Modeling: Sparx Enterprise Architect – Deployment: IzPack 15

16. Euclid Consortium OU-LE3 KOM Meeting, London 16March 22-23, 2012 SDCs - Collaborative Devlopment Platform Eclipse Jenkins Sonar Jira & co 16

17. Euclid Consortium OU-LE3 KOM Meeting, London 17March 22-23, 2012 Tools and …. Rules But, tools are not sufficient, needs for : Rules (e.g. naming conventions, formats, directory trees, …) Guidelines Code, Doc templates Good practices User support At SGS level 17

18. Euclid Consortium OU-LE3 KOM Meeting, London 18March 22-23, 2012 SDCs – Typical Product Tree 18 SDC-xx SDC-xx-DEV xxx pipeline yyy pipeline (contribution) Development Infrastructure TL/IAL Plug- ins SDC-xx- PROD Infrastructure Integration Infrastructure Validation Infrastructure Production Infrastructure TL/IAL Connectors

19. Euclid Consortium OU-LE3 KOM Meeting, London 19March 22-23, 2012 SDCs – Development Inputs OU-xx outputs : – Pipeline workflow, breakdown, inputs & outputs and algorithms specifications, Use cases,… – Pipeline V&V kit (test data, expected results, documentation) – Optionally, prototype code : should be reengineered in order to fit SGS rule Formal data modeling of all xx pipeline input (primary and secondary), output and intermediate data. Code and Data from OU-SIM SGS Level Documentation impacting the software development (RB, TS, DDF, DJF, MGT, PAF) to which the xx pipeline implementation should be compliant and/or should derive its own documentation.: – Architecture – Development principles – Common Rules & Tools – PA/QA – … Architecture constraints from production infrastructure

20. Euclid Consortium OU-LE3 KOM Meeting, London 20March 22-23, 2012 SDCs – Development Deliverables outputs Pipeline product including Software executables, documented source code, under strong configuration control pipeline documentation (RB, TS, DDF, DJF, MGT, PAF), in particular : – RB : Software system specifications (SSS), Interface requirements document (IRD) – TS : Software Requirements Specifications (SRS), Interface Control Document (ICD) – DDF : Software Design Document (SDD), Software Configuration File (SCF), Software Release Document (SRelD), Software User Manual (SUM) including in particular extraction, deployment, installation and generation procedures. – DJF : Software V&V Plan (SVVP), Software Test Specifications (STS), Software Test Report (STR), Software problem reports and non- conformance reports (SPR & NCR) – MGT : Software Development Plan (SDP) – PAF : Software Product Assurance Plan (SPAP), traceability & coverage matrixes (TCM). 20

21. Euclid Consortium OU-LE3 KOM Meeting, London 21March 22-23, 2012 SDCs - Integration Deliverables Outputs Operational VIS pipeline on SDC-FR PROD infra. Verification & Validation documentation : – Verification & Validation Plan (SValP) – Verification & Validation Tests Specification (SValS) – Verification & Validation Tests report (SValR) Contribution with OU & PO/System team to system 2by2 tests report Contribution to system E2E tests report 21

22. Euclid Consortium OU-LE3 KOM Meeting, London 22March 22-23, 2012 SDCs - Operation Deliverables Outputs Data products from pipeline Production reports : global metrics ( problems met during exploitation, performance : processing, storage,…) 22

23. Euclid Consortium OU-LE3 KOM Meeting, London 23March 22-23, 2012 SDCs – Versions & Overall Planning SGS Global schedule Phase F Disposal 23 Proposed Schedule