25th Nov 2005CERN AB Controls Post Mortem data conversion G.Kruk

CERN – AB DepartmentG. Kruk – Agenda  Conversion overview Why do we want to convert the data? Conversion process BinX library Final data format  Current state  Issues & further steps

CERN – AB DepartmentG. Kruk – Why do we want to convert the data?  Minimize processing on the client side --> send the data as it is  Data sent by different clients may have different structure  To browse and analyze PM events we want to access (read) data coming from different systems in the same way (using the same API)  Let’s convert the different data into one, standardized format

CERN – AB DepartmentG. Kruk – Conversion process PM Server PMD file (CMW data object) Binary file with data sent by client Final data format Conversion program extracts converts Notifies through “LastDump” property

CERN – AB DepartmentG. Kruk – Conversion program It consists of two components:  PMManager Is notified when new PM data arrives (monitors “LastDump” property) Prepares directories structures Calls appropriate PMConverter to convert the data  PMConverter (interface) Performs the actual conversion from binary file to the final format Uses BinX library At the moment two implementations: FGCConverter, DefaultConverter (now used only for tests)

CERN – AB DepartmentG. Kruk – BinX library  Library (C++) facilitating reading/writing and conversion of binary files  Uses XML descriptor to decode the binary  Takes care of the endian-ness  Using it we can potentially avoid writing a separate PMConverter for each system/class  Only the conversion program depends on it

CERN – AB DepartmentG. Kruk – Final data format Information that need to be stored  For each signal: name type (int, float, string, …) values (array) units/precision? description?  Additional meta-information?

CERN – AB DepartmentG. Kruk – Final data format  After considering several options we decided to use SDDS as the final format  SDDS Fulfils requirements Is simple API in Java, C/C++, Python, … Supports both: ASCII and binary formats We have positive experiences with it

CERN – AB DepartmentG. Kruk – Current state  Converter program installed on the PM server machine (cs-ccr-pm1)  Supports conversion of FGC PM data - tested this week  Ready to perform tests with other systems

CERN – AB DepartmentG. Kruk – Current state (directories structure) SYSTEMS FGC _11_24__14_44_45_ RPMBB.UA83.RQSX3.L8.bin binx.xml RPMBB.UA83.RQSX3.L8-50Hz.sdds RPMBB.UA83.RQSX3.L8-100Hz.sdds RPMBB.UA83.RQSX3.L8-1kHz.sdds RPMBB.UA83.RQSX3.L8-events.sdds PROCESSED PMD FILES FGC _11_24__14_44_45_ RPMBB.UA83.RQSX3.L8.pmd

CERN – AB DepartmentG. Kruk – Issues & further steps  Performance ~ ms to convert data from one PC There are places for improvements e.g. use binary SDDS rather than ASCII  No support for mapping from single bits into flag names e.g  “FLAG_A FLAG_C” For FGCs at the moment it is hardcoded In the future  move the mapping to BinX descriptor or to some other descriptor  Directory structure modification (refinement)  Before extending functionality of the converter program – tests should be performed with other systems (not only with FGCs) and with analysis team: To validate current functionality, formats used, directory structure To understand better required refinements