ACS Configuration Database(s) ACS workshop presentation by Heiko Sommer, ESO Garching. Last updated 2010-11-12. Two alternative implementations in ACS: jDAL (“CDB”) hibernateDAL (“TMCDB”) Same purpose: To centrally maintain and provide structured system configuration data. This presentation supersedes “ACSCourse-CDB-Short.ppt” and “ACSCourse-CDB.ppt”.

What sort of data? Data used by ACS about system structure and behavior, e.g. To run components inside containers. Logging config for containers, components, manager. Notification channel configuration (deployment, QoS) Alarm definitions, reduction rules Configuration for baci properties (e.g. legal value range, monitoring settings). Custom data used by components. [Only with TMCDB extension by Alma]: Structured data about configuration of different hardware, HW startup, pointing model coefficients, SNMP configuration, … In the ACS workshop, we only configure data read by the containers and manager. Clients / components do not need to actively read from the CDB. Some other configuration data is still kept separately in files: Under $ACSDATA/config: orb.properties: Read directly by JacORB from every CORBA aware Java process. data used by Alma ARCHIVE: archiveConfig.properties to connect to databases (incl. TMCDB) tnsnames.ora read directly by the Oracle driver The Alma OMC tool still supports some proprietary XML config files NotifyService instances require configuration data (incl. runtime data about NamingService corbaloc and cache for subscriber reconnects) in files, which get copied from the template $ACSROOT/config/svc.conf/defaultNotify.svc.conf to $ACS_TMP/ACS_INSTANCE.0/conf/….Notify.svc.conf for the various instances.

Interface and implementations Corba interface DAL / DAO implements jDAL xercesj DOM hDAL Hibernate OO-RDB Service process (acsStart) other apps reads, writes reads, writes XML Software Config Hardware Monitoring ref DAL = Data Access Layer DAO = Data Access Object (for specific CDB node) The CDB service as an abstraction helps to - switch between different storage techniques - simplify maintenance because the files do not have to be copied to all machines (nor be accessed remotely in a shared file system). The CDB (DAL) is started by “acsStart” or similar action from the operator GUI. XML Data storage XML ref ref XML TMCDB Relational Database XML files with schemas

CDB implementation XML jDAL (often called just “CDB”) Initial implementation of the CDB’s interface in ACS. A future relational variant was always foreseen. Data stored as XML files and schemas (XSD) in a directory hierarchy, transparent to users. Human-readable and -writable. Offline XSD validation catches errors in single XMLs. Does not enforce relations between the XML files. Best for smaller projects and unit tests (although no performance limits have been reached in Alma yet). Configuration control is simple but must be done externally, e.g. by keeping XML files in CVS. The historical name “jDAL” refers to the Java language in which it is implemented, and the main corba interface called “DAL” . Especially now that the alternative implementation “HibernateDAL” is also implemented in Java, we better should have called the jDAL “xmlDAL” to refer to how it stores data, or simply “xmlCDB”. Probably it’s too late for a name change though. When ACS was conceived, we thought that an alternative RDBMS-based implementation of the same “DAL” interface would become necessary for performance reasons as Alma gets bigger. At least with the current emphasis on read-only access to the CDB, the XML implementation has not reached its limits in Alma testing. Instead, the limit was the lack of history built in, and the need to link hardware and monitoring data to the software configuration data in the CDB. Application code that uses the CDB can access data without ever seeing XML, based on the “node with values” DAO concept. The CDB public interface can also offer data as XML, which does not depend on the underlying storage as XML. For example, the TMCDB also returns XML data which it assembles from the relational tables. The tool “cdbChecker” adds some of the missing inter-XSD validation functionality. For the ACS workshop, we will use the jDAL CDB.

CDB implementation RDB HibernateDAL (“hDAL”, or wrongly just “TMCDB“) Implements same "DAL / DAO“ interfaces as jDAL. Data is stored in a relational database. Underlying DB engine: HSQLDB for small testing, Oracle for real-life deployment, or others (hibernate). Versioning support (using Oracle's "total recall“). Data relation consistency enforced through (cascading) constraints, beyond jDAL’s XSD abilities. Extensible for other data (accessed directly on DB). Needs special tool for data maintenance (the “TMCDB Explorer” in Alma). Offers initial XML import into the RDB. The term “TMCDB” generally refers to the entire database schema that allows storing software configuration data used by the HibernateDAL or other applications, along with other data related to hardware configuration, monitoring etc. The historical name “HibernateDAL” reflects the hibernate framework used for database access, and the main corba interface being the “DAL” . A more precise name would have been “rdbDAL” to refer to the relational DB storage, or simply “rdbCDB”. Probably it’s too late for a name change though. The TMCDB functionality is a super-set of that of the CDB. Both implement the same Corba interfaces, so that applications accessing the CDB do not need to be changed at all when switching from the jDAL to HibernateDAL implementation. After years of struggling with the hardware and monitoring part, the CDB functionality was fully retrofitted into the TMCDB tables and software in ACS 8.2, and gets used by Alma since ACS 9.0 (Alma R8.0). Before that, only the monitoring part of the TMCDB was used by Alma, in coexistence with the XML CDB and proprietary XML config files used by the control subsystem. Using a relational database currently does not give performance advantages over XML storage, because both CDB implementations keep all data into memory. The XML import is an additional feature of the HibernateDAL process, allowing us to import old jDAL data into the TMCDB tables. Of course the software configuration tables can also be filled from scratch using scripts or the TMCDB explorer. Will talk of the additional features of the HibernateDAL and the code generator tools that ACS offers for data definition later.

CDB runtime access Manager Component NC libs Container AlarmService Component deployment Container & component logging Baci properties config Manager logging NC config AlarmService Ping intervals Which alarm system? Alarm data, Reduction rules. DAL / DAO jDAL hDAL Default values from XSD. Default values from RDB. XML XML

DAL interface DAL server implements DAL IDL: string get_DAO( in string curl ) raises (RecordDoesNotExist,XMLerror); DAO get_DAO_Servant( in string curl ) raises (RecordDoesNotExist,XMLerror); oneway void shutdown(); //data change handling long add_change_listener( in DALChangeListener listener ); void listen_for_changes( in string curl, in long listenerID ); void remove_change_listener( in long listenerID ); // listing string list_nodes( in string name ); }; XML string for local processing Remote object for data field retrieval Get XML string for more efficient network usage (“object by value”). Get DAO reference for atomic data access without having to parse XML.

DAO interface interface DAO { long get_long( in string propertyName ) raises (WrongDataType, FieldDoesNotExist); double get_double( in string propertyName ) raises (WrongDataType, FieldDoesNotExist); string get_string( in string propertyName ) string get_field_data( in string propertyName ) stringSeq get_string_seq( in string propertyName ) longSeq get_long_seq( in string propertyName ) doubleSeq get_double_seq( in string propertyName) raises (WrongDataType, FieldDoesNotExist); };

CDB Node Structure CDB MACI Managers Manager alma Components myComp Containers myCont1 yourCont Manager myComp yourComp anotherComp CDB alma schemas

jDAL file - node structure FILES NODES Client (Container or Component) parent directory File1.xml CDB Data Access Layer (DAL) Server child directory1 File2.xml get “parent” get “parent/child1” DAO “File1”, “File2”, “File3”, “File4” child directory2 File3.xml File4.xml DAO “File2”

jDAL Component Config $ACS_CDB must point at CDB parent dir. CDB/MACI/Components/Components.xml <Components xmlns="urn:schemas-cosylab-com:Components:1.0" xmlns:cdb="urn:schemas-cosylab-com:CDB:1.0“ xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance> <_ Name="TELESCOPE " Code="TelescopeImpl" Type="IDL:alma/TELESCOPE_MODULE/Telescope:1.0" Container="telescopeContainer" ImplLang=“cpp” /> <_ Name="MOUNT_PY_1" [...] /> </Components> Examples in $ACSDATA/config/defaultCDB/ Every module can have a local test CDB, under the ./test/ directory. The strange element name “_” is an intented convention, to allow iterating over element lists.

jDAL XML schemas Schemas are available at Example: Container.xsd $ACSROOT/config/CDB/schemas/ $ACS_CDB/CDB/schemas/ http://www.eso.org/projects/alma/develop/acs/OnlineDocs/ACS_docs/schemas Example: Container.xsd <xs:complexType name=“Container"> <xs:sequence> <xs:element name="Autoload" type="cdb:Array"/> </xs:sequence> <xs:attribute name="CommandLine" type="xs:string" use="optional"/> <xs:attribute name="Timeout" type="xs:int" use="optional" default="20000"/> <xs:attribute name="UseIFR" type="xs:boolean" use="optional" default="1"/> <xs:attribute name="ManagerRetry" type="xs:int" use="optional" default="10"/> <xs:attribute name="ManagerReference" type="xs:string" use="optional"/> <xs:attribute name="CacheSize" type="xs:int" use="optional" default="10"/> <xs:attribute name="MinCachePriority" type="xs:int" use="optional" default="0"/> <xs:attribute name="MaxCachePriority" type="xs:int" use="optional" default="31"/> <xs:attribute name="CentralizedLogger" type="xs:string" use="optional" default="Log"/> <xs:attribute name="DALtype" type="xs:string" use="optional"/> </xs:complexType> http://www.eso.org/projects/alma/develop/acs/OnlineDocs/ACS_docs/schemas/urn_schemas-cosylab-com_Container_1.0/complexType/Container.html http://www.eso.org/projects/alma/develop/acs/OnlineDocs/ACS_docs/schemas/urn_schemas-cosylab-com_Manager_1.0/element/Manager.html http://www.eso.org/projects/alma/develop/acs/OnlineDocs/ACS_docs/schemas/urn_schemas-cosylab-com_EventChannel_1.0/complexType/EventChannel.html http://www.eso.org/projects/alma/develop/acs/OnlineDocs/ACS_docs/schemas/urn_schemas-cosylab-com_Channels_1.0/complexType/NotificationServiceMapping.html (The notify service where the NC objects run does not read from the CDB. The client-side supplier and subscriber libraries do this instead.)

Details Component Config Full specification for static components. None or partial config for dynamic components; missing data added at runtime. Characteristic components (those with baci properties) need extra configuration. -> Show CDB/schemas/TelescopeControl.xsd

CDB Browser

TMCDB Explorer

More CDB features Write access: WDAL interface, implemented by jDAL and hDAL. Content discovery using “DAL#list_nodes” jDAL component configuration: alternative scheme with one file per component. On-change notification, e.g. after cdbjDALClearCache cdbRead Example retrieving XML for all container config: cdbRead MACI/Containers –raw Get access to the CDB from within your code using the containerServices object.

TMCDB sw config core

TMCDB sw config extended