1 Archiving Workshop (Soleil, May 2010) Archiving System Status

2 Archiving Workshop (Soleil, May 2010) Plan  Archiving Systems Overview  HDB / TDB / SNAP / ADB  Soleil Deployment Status  Oracle DB infrastructure description  Latest modifications  Production  Missing features

3 Archiving Workshop (Soleil, May 2010) Archiving Systems Overview

4 Archiving Workshop (Soleil, May 2010) HDB/TDB Archiving: General Description  Operators Requirements  Storage of a large number of data produced by the Control System  Data exploitation at anytime Watch the vacuum temperature and the pressure evolution Look for an explanation of a beam loss  The tools  Historical DataBase (HDB) Data are kept for lifetime The minimal archiving period is 10 seconds On-line extraction available during x years (hardware capacity limitation)  Temporary DataBase (TDB) Data are kept x days (hardware capacity limitation) The minimal archiving period is 100 milliseconds The two are managed via the graphical tool named Mambo

5 Archiving Workshop (Soleil, May 2010) HDB/TDB Archiving: Mambo MAMBO : Configuration and Extraction application Web access also available via JBOSS

6 Archiving Workshop (Soleil, May 2010) HDB/TDB Archiving: Devices  Devices  Hdb/TdbArchiver : Collect of the attributes values and storage in the DB  ArchivingManager : Upper HDB/TDBArchiver, contains Start/Stop archiving commands  HDB/TDBExtractor : various extraction commands as getAttDataBetweenDate, getArchivingMode …  HDB/TDBArchivingWatcher : Supervision Device Diagnosis tool : detecting abnormal archiving interruption Recovery : reactivate archiving on the failed attributes

7 Archiving Workshop (Soleil, May 2010) SNAP Archiving: General Description  Operators Requirements  Capability to take a picture of a set of attributes at t time Motors positions before a planned electric halt  Compare quickly and easily the attributes values Before and after an experience to analyse the beamline parameters evolution  Send instructions easily to several equipments Set the beamline in a configuration reference  The tool  Snap DataBase (SNAP) Allows to take a snapshot of a set of attributes Allows to apply values on a set of attributes Allow to compare attributes values to each other It is managed via the graphical tool named Bensikin

8 Archiving Workshop (Soleil, May 2010) SNAP Archiving: Bensikin BENSIKIN : Configuration and Exploitation

9 Archiving Workshop (Soleil, May 2010) SNAP Archiving: Devices  Devices  SnapArchiver : takes the snapshot and stores it in the DB (Hibernate java technology)  SnapManager : triggers the launch snapshot to an available archiver  SnapExtractor : various extraction commands as getSnap

10 Archiving Workshop (Soleil, May 2010) ADB Archiving: General Description  Operators Requirements  Supervise attribute quality and device state changes  The tool  Alarms DataBase (ADB) Monitors the attributes alarm state It is managed via the graphical tool named Macarena

11 Archiving Workshop (Soleil, May 2010) ADB Archiving: Macarena Macarena : Current and Deleted Alarms

12 Archiving Workshop (Soleil, May 2010) ADB Archiving: Devices  Devices  AdbArchiver : monitors the attributes quality and stores the values in the DB  AdbManager : dispatches alarm supervision between the existing AdbArchiver devices Used by the GUI for each database transaction.  AlarmDiary : contains the available alarms list, access by the GUI only

13 Archiving Workshop (Soleil, May 2010) Common Points  SGAD: GUI tool generates a MySql DB creation script  for Unix or Windows  for HDB, TDB or/and SNAP DB  Common point :  Archiving systems are provided in Java 1.6

14 Archiving Workshop (Soleil, May 2010) Soleil Infrastructure

15 Archiving Workshop (Soleil, May 2010) Machine Deployment status  All the archiving systems use an Oracle Database  SNAP  Oracle engine updated  Following some Bensikin updates and a little ICA pressure the operators plan to work more with it during the next weeks Currently important data are stored in Matlab files Still now, the Vacuum group makes some manual comparison at every run startup  ADB  Before the data were stored into files  System recently migrated to be built as HDB/TDB system  GUI consolidation is ongoing and strongly awaiting by the operators to help them in the control system monitoring

16 Archiving Workshop (Soleil, May 2010) Machine Deployment status  HDB : 8900 attributes  8750 scalars spectrums 1800 at 10 s 4000 at [30 – 60] s 1800 at [200 – 900] s 1000 at [1000 – 3600] s The rest is upper 3600 s  25 DServers * 5 devices, so 125 devices hold by One Linux Server Each device supports about 70/75 attributes (initial estimation was 50 attributes / device)  5 years of online data  TDB : 6500 attributes  6470 scalars + 30 spectrums 10 at 100 ms 1900 at 500 ms 3100 at [1 – 5] s 650 at [10 – 60] The rest is upper 60 s  60 DServers * 5 devices, so 300 devices hold by Three Linux Servers Each device supports about 20/25 attributes (estimation is about 30 attributes / device)  Data are available 30 days

17 Archiving Workshop (Soleil, May 2010) Beamline Deployment status  All the archiving systems use a MySql Database  SNAP  Really often used to store motor positions, to set beamline in reference position...  TDB: at most 250 attributes  Mainly scalars  1 DServers * 5 devices  Data are available 20 days (easily configurable per beamline)  Minimal archiving period equals to 1 s due to a MySql limitation  HDB: Not available  Oracle installation is in progress  All the hardware infrastructure is available  Will be deployed progressively beamline per beamline

18 Archiving Workshop (Soleil, May 2010) Oracle DB Solution

19 Archiving Workshop (Soleil, May 2010) Oracle Solution  Refer to the Jérémy’s presentation

20 Archiving Workshop (Soleil, May 2010) Latest Modifications

21 Archiving Workshop (Soleil, May 2010) Latest Modifications: Mambo  Mambo  Better.lock file management At Mambo startup this file is stored in the directory linked to the connexion profile This file is still available when the application is brutally interrupted  The profile is no more usable Now.lock can be easily suppressed after confirmation

22 Archiving Workshop (Soleil, May 2010) Latest Modifications: Mambo  DB connection parameters location  Either in command line with … -DHDB_HOST=srv1 -DHDB_NAME=hdb -DTDB_HOST=srv1 -DTDB_NAME=tdb fr.soleil.mambo.Mambo … Useful to start 2 Mambo in front of 2 differents DB  Or in Hdb/Tdb Archiver class properties DbHost, DbName Properties also used by the archivers devices

23 Archiving Workshop (Soleil, May 2010) Latest Modifications: Mambo  Mambo  Display Multiple VC  Some parts has been transformed in java bean : Table Mode Connection Window VC (not completly done)  Start archiving error description in the log part

24 Archiving Workshop (Soleil, May 2010) Archivage HDB/TDB : Mambo  Archiving modes: something more in the Absolute and Relative modes  Definition Attribute value is archived when the variation between two successive values (in absolute value or in percentage) is upper configured bounds.  Default behavior the variation is computed with the attribute value previously read on the device and the current value  Slow drift behavior the variation is computed with the attribute value previously stored in the DB and the current value

25 Archiving Workshop (Soleil, May 2010) Latest Modifications: Mambo  Mambo  New Graphical tool usage named Comete  various impacts : Spectrum viewer X Axis configuration management Indice Time Time Stack

26 Archiving Workshop (Soleil, May 2010) Latest Modifications: Mambo  Mambo  Set All Colors in the New/Modify VC

27 Archiving Workshop (Soleil, May 2010) Latest Modifications: HDB/TDB Devices  HdbArchiver  Some attributes can be archived under 10 s by configuration (not fully tested)  Hdb/TdbArchivingWatcher Device state modification  Number of KO attributes ≥ MinNumberOfKoForAlarm (new property) then State = ALARM otherwise State = ON  Database Api refactoring  Done but Not completed

28 Archiving Workshop (Soleil, May 2010) Latest modifications: Bensikin  Context description  Number of attributes  SnapShot description  Contexte number  Number of attributes  SetEquipment With Command  Send a command with the Read or Write value as argin  At startup: last open snapshot is loaded (quickly opened)

29 Archiving Workshop (Soleil, May 2010) Latest modifications: Bensikin  SnapShots comparaison  String and boolean attributs types can be also compared  Table bean and Connection window bean usage

30 Archiving Workshop (Soleil, May 2010) Latest modifications: Bensikin  Application options (proper to the profile) :  Date Filter criteria Snapshot expanded Prefilled Time field

31 Archiving Workshop (Soleil, May 2010) Latest modifications: Bensikin  Columns order definition in the comparison window  Configuration of the Data included in the Snapshot export Application options (proper to the profile)

32 Archiving Workshop (Soleil, May 2010) Latest modifications: SNAP Devices  SnapArchiver / Manager  LaunchSnapshot method has been moved to its right place: it concerns the SnapManager and call the TriggerLaunchSnapshot command of the SnapArchiver

33 Archiving Workshop (Soleil, May 2010) Production

34 Archiving Workshop (Soleil, May 2010) Maven production  Now.classpath project is empty  pom.xml usage  Pom.xml  contains the main jars files  see Vincent Hardion presentation on Thursday  Archiving packaging ongoing  Either one zip file with the complete archiving systems distribution  Or one zip file per archiving system  Zip file will be stored in the Tango Pink site fr.soleil.lib hdbtdbArchivingApi fr.soleil.lib commonArchivingApi fr.esrf.tango TangORB

35 Archiving Workshop (Soleil, May 2010) To be improved

36 Archiving Workshop (Soleil, May 2010) Missing features  No archiving system manage the Tango::Image data_format  Archiving system monitoring must be improved for computer team and operators  Long term TDB: some data mustn’t be deleted  Logging information improvement with log 4 j usage  Database api code refactoring not fully finalized  MySql doesn’t support the milliseconds  Tdb/Mysql DB upgrade (versus Oracle) to ensure a heavy load (eq. SqlLoader, jobs, partitions, backup mechanism …)

37 Archiving Workshop (Soleil, May 2010) Summary

38 Archiving Workshop (Soleil, May 2010) To sum-up  Archiving Systems are functionally fully operational  many interesting features are missing  Deployment is a big part of the archiving system activity  we are lacking tools to ease this task  Oracle DB infrastructure management is a huge task  How to share the know-how for MySql management ?  Archiving Systems are always in evolution  Sharing modifications, with the community, is crucial to avoid forking Last but not least : the Archiving system philosophy Give Users the ability to be autonomous