1. Long-Term Data Preservation Jamie.Shiers@cern.ch WLCG Overview Board, March 2013 Twitter: #DPHEP International Collaboration for Data Preservation and Long Term Analysis in High Energy Physics

2. Overview Summary of DPHEP Blueprint recommendations Opportunities: collaboration with other disciplines & funding A “2020 vision” and its implementation 2

3. DPHEP BLUEPRINT 3

4. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports 4

5. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports Implemented via multi- lateral Collaboration Agreement (draft circulated) 5

6. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports Chair of Study Group was Cristinel Diaconu / CPPM & DESY who continues in this role 6

7. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports CERN provides Project Manager 2013 – 2015 after which may rotate 7

8. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports Broadened to include “influential” names, e.g. from APA, SCIDIP-ES 8

9. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports Representatives of parties to Collaboration Agreement 9

10. DPHEP Entities Organisational BodyDescriptionInput and positioningDPHEP Output DPHEPOrganisation for Data Preservation in High- Energy Physics Projects in data preservation at experiment and laboratory level Working groups on common projects, status report documents DPHEP ChairOverall coordination of DPHEP Appointed by ICFA, represents DPHEP in relationship with other bodies Yearly reports to ICFA, representation to other related scientific bodies DPHEP Project ManagerProject management, administrative, technical, funding Main operational coordinator, maintain contacts, organises meetings, lead proposals for funding Reports to the steering committee Advisory committeeGroup of external personalities Synergy with the wider HEP community, input from other fields and initiatives Project proposals, documents for scrutiny Steering committeeInternal executive body, chaired by the DPHEP Chair Contributions from the participation members Strategic and operational decisions Funding bodiesFunding agencies are invited to take note on the progress reports and periodically analyse the relevance of the funding Direct funding to the DPHEP organisation, under the supervision of the Project Manager Quarterly progress reports e.g. EU, NSF, STFC, INFN, … 10

11. DPHEP Blueprint Deliverables ObjectiveDeliverable (Measurable) Positioning as forumCatalogue of technical knowledge and practical solutions Description of possible alternatives for governance. Co-ordination of projectsCommon R&D projects meet the expectations of the stakeholders. Harmonisation and liaison Synchronisation of preservation projects in the field. Identification of areas where external knowledge needs to be transferred to HEP. Design sustainable future Characterisation of discipline-wide toolkit for preservation Business plan for long-term preservation in HEP. Outreach and advocacyUnderstanding of needs/opportunities for medium- and small-sized collaborations. Concrete discussions with funding bodies/laboratories. Proposed activities of the DPHEP Organization – p85, Blueprint document. These deliverables are to be met within 2 years of becoming fully operational. 11

12. DPHEP Preservation Levels Preservation ModelUse case 1.Provide additional documentation Publication-related information search 2.Preserve the data in a simplified format Outreach, simple training analyses 3.Preserve the analysis level software and data format Full scientific analysis based on existing reconstruction 4.Preserve the reconstruction and simulation software and basic level data Full potential of the experimental data 12

13. DPHEP Levels Preservation ModelUse case 1.Provide additional documentation Publication-related information search 2.Preserve the data in a simplified format Outreach, simple training analyses 3.Preserve the analysis level software and data format Full scientific analysis based on existing reconstruction 4.Preserve the reconstruction and simulation software and basic level data Full potential of the experimental data 13

14. DPHEP Levels Preservation ModelUse case 1.Provide additional documentation Publication-related information search 2.Preserve the data in a simplified format Outreach, simple training analyses 3.Preserve the analysis level software and data format Full scientific analysis based on existing reconstruction 4.Preserve the reconstruction and simulation software and basic level data Full potential of the experimental data 14

15. DPHEP Levels Preservation ModelUse case 1.Provide additional documentation Publication-related information search 2.Preserve the data in a simplified format Outreach, simple training analyses 3.Preserve the analysis level software and data format Full scientific analysis based on existing reconstruction 4.Preserve the reconstruction and simulation software and basic level data Full potential of the experimental data HepMC / Rivet toolkit may play a useful – and sustainable – role here. See DPHEP7 15

16. DPHEP Summary There is a lot of knowledge and experience in the existing DPHEP community that can be leveraged for other efforts, e.g. LHC & LEP LHC is clearly of key interest to WLCG OB but we should not forget LEP before it is too late!LEP On-going (small) effort to document current situation and options for moving forward  CERNLIB felt to be (a) critical factor but there are many external distributions 16

17. OPPORTUNITIES & FUNDING 17

18. Collaboration with others Many other disciplines, ranging from science to arts & humanities, already (very) active Numerous conferences and workshops have been up and running for years We have been accepted – partly due to halo effect of the Higgs discovery – with open arms Concrete discussions on further collaboration are funding advancing well  Not limited to Data Preservation – e.g. SKA! 18

19. Funding DASPOS is up and running with NSF funding Research Data Alliance – with indirect EU, NSF, AUS and other funding – will play a role – Co-chair of RDA WG on DP Clear signs that EU Horizon 2020 will include Data Preservation – e-IRG meeting, EIROforum w/s, RDA, … Now is the time to firm up partnerships & prepare for up-coming projects  STFC and other UK bodies particularly active in above activities: how can we profit from this? 19

20. A 2020 VISION 20

21. 2020 Vision for LT DP in HEP Long-term: disruptive change(s), e.g. LC era – All archived data – e.g. that described in Blueprint, including LHC data – easily findable, fully usable by designated communities with clear (Open) access policies and possibilities to annotate further – Best practices, tools and services well run-in, fully documented and sustainable; built in common with other disciplines, based on standards  Vision achievable, but we are far from this today 21

22. Long-Term Commitment To achieve long-term data preservation, we need long-term commitment(s) By 2035, there will have been: – 3-4 updates to the ESPP; – 4-5 new DGs; – X re-organizations of CERN-IT.  We need commitments that outlive all of these! 22

23. 2020 Vision – The OAIS Model 23

24. OAIS Components In the OAIS model, there are the concepts of producer and consumer DASPOS aims to take data produced by e.g. CMS and show that e.g. ATLAS can reproduce a full analysis, using the software, meta-data, documentation etc. This exercise will be started at DPHEP7 (March 21-22) and hopefully repeated regularly – e.g. annually – so that by 2020 the entire process is well understood, documented and repeatable  It is proposed that the (Archive) Information Packages are simply XML documents stored in Invenio The exact tool-set and feature requirement is still TBD Some tools used on a daily basis – e.g. Twiki! – not suitable for long- term archives  Good opportunity for sharing experiences and best practices with other disciplines / projects, e.g. SCIDIP-ES, APA APA – “Too Big an Issue for any single organisation – we must work together” 24

25. Archival Storage Experience from WLCG and beyond tells us that data loss and corruption will (and does) occur! – See WLCG SIRs, Tim Bell’s presentation to DPHEP3WLCG SIRspresentationDPHEP3 But there are things that we can do to mitigate risks and recover (often), e.g. rule-based systems: apply checksum and other “tests” upon schedule and/or actions What is the current situation at WLCG sites? Can we coordinate / agree suitable actions? Coordinate via HEPiX, IEEE MSST, APA, EUDAT, RDA etc. Collaboration with industry, e.g. IBM-led FP7 project  Recovery often performed by experiments by re-replicating data: how will this be done in the long-term? 25

26. DPHEP Level 4 Retaining the full potential of the data is the only really interesting option – but it is by far the most difficult! Difficult does not mean impossible – and we can profit from a period of “meta-stability” while we concentrate on this Past experiments typically ported / re-wrote major parts of their offline environment several times over a period of decades This is inevitable for LHC too – we could make this easier, but it will require an initial investment!  Collaboration with others who face similar problems could help but much of this we have to solve ourselves 26

27. Where to Invest? Tools and Services, e.g. Invenio Archival Storage Functionality Support to the Experiments for DPHEP Level 4 27

28. Suggested Topics for DPHEP7 “Ingest Issues” (10’) – How did you (the experiment) decide what data to save, how to make it discoverable / available, how is it documented, where is the data / meta-data etc. What are the access policies and target communities? – What tools do you use? “Archive issues”: (10’) – How is the archive managed? How are errors detected and handled? What is the experience? – What storage system / services are used? “Offline environment issues”: (20’) – What have been the key challenges in keeping the offline environment alive? What are the key lessons learned / pitfalls to be avoided? What would you have done differently if long-term preservation had been a goal from the early days of the experiment?  DPHEP8: around or during CHEP? TBD in coming weeks… DoodleDoodle Outline for site / experiment talks at DPHEP7, March 21-22, CERN 28

29. S.W.O.T. StrengthsDPHEP is well established within the community and recent contacts to other disciplines are very encouraging WeaknessesEffort is very scarce within the project at a time when manpower is already stretched to the limit elsewhere OpportunitiesThrough a convergence of events there are clear possibilities for significant funding and collaboration in the EU’s Horizon 2020 programme and most likely corresponding programmes in other areas of the world, e.g. NSF-funded projects ThreatsFailure to invest now would jeopardise attempts to “rescue” LEP data as well as to take other preservation events (BaBar, Tevatron, Hera etc.) to a stable and sustainable state. It could also limit our ability to prepare for – and hence participate in – future projects 29

30. Summary We have outlined the current status of Long-Term Data Preservation in HEP and areas for fruitful collaboration with others Funding, e.g. through EU Horizon 2020, is looking good – we need to invest now to secure this!  Much work needs to be done to turn a dream into reality – particularly and critically in the area of future-proof offline environments However, this is expected to result in a cost- saving in the long-term by reducing effort in inevitable migrations 30

31. Where to Invest – Summary Tools and Services, e.g. Invenio: could be solved. (2-3 years?) Archival Storage Functionality: should be solved. (i.e. “now”) Support to the Experiments for DPHEP Level 4: must be solved – but how? 31

32. International Collaboration for Data Preservation and Long Term Analysis in High Energy Physics 32