1. The Database Project a starting work by Arnauld Albert, Cristiano Bozza

2. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20132 Server technology: Oracle Enterprise Server (RAC - Cluster) Well hosted and supported at CCIN2P3 Working licenses already available in several sites in Italy Discounts possible through CERN agreement Good experience of technical support by Oracle (also 24  7 for critical cases) Huge (unequalled?) variety of tools and libraries (by Oracle and independent software producers) Accessible through: C/C++, Java, C#, Python, PHP, VB, Perl, ODBC, ODP.NET, … OS-independent

3. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20133 System: Symmetric datacenters Continuous data synchronization

4. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20134 Schema for construction – detector description Locations Users Products PBS Descriptions Container Mapping Logical and hierarchical relationships among products are stored The structure of the detector (PBS) is also stored and documented in the DB

5. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20135 Schema for construction – activity management Locations Users Operations Status Decisions Each operation corresponds to a well defined task An operation may contain one or more sub-operations, like a tree Each operation is linked to the place where it is performed and the user that does it or is responsible for it. At a certain time, it is in a certain “status”; on completion, decisions may be taken concerning it.

6. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20136 Schema for construction – activity management Operations Decisions The Type of a certain operation contains links to programs and their operating parameters. This allows not only documentation but also management through the DB Each operation type has a set of possible decisions. Operation Types Possible Decisions Operation Types

7. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20137 Schema for construction – detailed bookkeeping information Operations Depending on the type of operation, additional information may be required Detail tables are linked to the main operation table For each kind of test, the list of parameters to be tested is defined For each component all the output of the testing is stored; it is easy to add new parameters as needed Product bookkeeping IntegrationTests Test types Parameter values Sets of parameters

8. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20138 Schema for construction – Content The DB is completely flexible about testing parameters The DB can contain explanation of parameters and the testing procedure can be described and documented We are collecting information from experts about testing procedures For the moment, we have some sample data (thanks to Tamas, Oleg and Emanuele)

9. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 20139 Access Three basic user roles have been defined: Administrator (km3net), Reader (km3read), Writer (km3write) with obvious meaning As for information, we distinguish between “information author/consumer” and DB user Information author/consumer: any person that produces or uses data from the DB, directly or indirectly  stored in Users table DB User: person or batch process that connects directly to the DB, using part of its memory, CPU, disk speed  Oracle DB Server authentication

10. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201310 Access A DB user can have multiple connections if needed A DB user can initiate transactions, lock rows/tables, run SQL queries A person should have a DB user account if he/she is aware of the related responsibility in resource usage User accounts can also be given on institution or group basis Most people will just need data, and also in friendly format Many people will want to stay focused on data and do not care about technicalities A special DB user, named km3web, is used by a dedicated Web site to provide a friendly user interface The Web site can provide information properly formatted and with detailed explanations The Web site can also be used to upload information about results of tests This can be also done by uploading data files

11. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201311 KM3Net Web DB access Login provided through Users table

12. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201312 KM3Net Web DB access For the moment, we are waiting for expert feedback to build useful pages The Web site can also provide “raw SQL access” (technical tests, maintenance, etc.) As the recent trends of Internet show, Web servers are increasingly becoming application servers for machine-to-machine data exchange

13. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201313 Outlook on Physics data storage It is possible to use a relational DB to store not only “data about the detector” but also “data from the detector” Moreover, intermediate output of reconstructions can also be stored, and specific datasets can be flagged and indexed – help and support analysis! There is recent fruitful experience concerning this technique (e.g. OPERA DB – designed to range between 50 and 100 TB, currently 34 TB) A first technical test has been successful for NEMO-phase-1 post-trigger data (thanks to Tommaso for data, discussion and help) Operational settings (“datacards”), trigger configuration, detector status, and sampled waveforms all easily stored A C++ library has also been developed to store data without any knowledge of SQL

14. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201314 Outlook on Physics data storage Test for NEMO-phase-1 post-trigger data

15. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201315 Outlook on Physics data storage Test for NEMO-phase-1 post-trigger data: code snippet to store a full run The only places where you know you’re storing to a DB are those in red: fill username/password to connect, and commit the transaction to tell the DB that all data were written and they can be stored

16. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201316 Outlook on Physics data storage Detector data storage is also related to the whole computing model Pro’sCon’s Reliable storage, with corruption checkAdditional checks load CPU and disk OS-independentData access requires linking a library (ODBC, OCI, ODP.NET, …) Languange and technology independent storage (C++,C#,Java,VB,Python,PHP,…) Easy to manage using SQL Consistency and integrity are automatically enforced Views help present data effectively with few lines of code In case of data model evolution, old programs still run without changes (no recompilation needed) Normally, also file-system storage uses an I/O library, so this is not a characteristic “con” of relational DB’s DB administrators of course take also care of developing and maintaining the I/O library

17. The Database Project A. Albert, C. Bozza – KM3Net Collaboration Meeting – Marseille, Jan 201317 Conclusions The work to set up the DB to document and support construction has already begun Startup schema defined User access defined Web site to make access user-friendly already set up, needs to be filled with useful pages (input from experts!) WE NEED DATA! Outlook It is possible to store not only construction data, but also raw data and physics output There is already know-how on that, and we can start a broad discussion