1. TANGO Harmonization Meeting (Edinburgh)
ANT Team SKA Common Alarms Handling Andrea DeMarco, Elisabetta Giani, Alessandro Marassi, Lize van den Heever
TANGO Harmonization Meeting (Edinburgh)
TANGO Core Alarms
ANT Team Objectives
TANGO Alarm Extensions Rationale
Elettra Alarm System
FormulaConf
Experimental Work/Prototyping

2. ANT Team Objectives Evaluate the TANGO core alarm system
Evaluate the TANGO alarm extensions by Elettra (C++) and ALBA (Python)
Discuss if it would be opportune to allow for LMC to use both these extensions, or only one of them (or none, meaning a custom SKA Alarm system should be implemented)
Evaluate the LIG requirements for alarms and whether these can be satisfied to the full by the chosen system
Look into whether we can define a common interface (or function) within a future SKA Generic Device that allows for any element to declare/undeclare alarms in a uniform manner
Discuss alarm hierarchies within element, and element to central coordination

3. TANGO Core Alarms: Setup
The standard mechanism to generate attribute alarms in the Tango core is the following:
Alarm Configuration: define attributes and set their alarm configuration levels
Alarm Condition: When alarm/warning thresholds are crossed, the attribute quality factor is set to ATTR_ALARM/ATTR_WARNING and the device State attribute is switched to ALARM.
Restore of Normal Conditions: When the attribute value returns to normal range, its quality factor is set to ATTR_VALID and the device State attribute set to ON.

4. TANGO Core Alarms: Clients
Clients wanting to be notified that an attribute alarm condition occurred in a device server shall perform these actions:
Subscribe to events (e.g. CHANGE_EVENT/PERIODIC_EVENT/...) defined for all server attributes for which an alarm configuration is set. When receiving attribute events, the attribute quality factor can be accessed and an alarm condition checked.
Subscribe to CHANGE_EVENT for server State and Status attributes and parse them.

5. TANGO Core Alarms: Drawbacks
Clients need to subscribe and monitor all server attributes and parse the quality factor
Detecting the State change alone would not provide sufficient information to know which of the attribute(s) was responsible for the alarm
Required: unique source of alarm information
Required: difference classes of alarms e.g. Communication, Quality-Of- Service, Software Processing/Exception, Equipment, Environmental, User
Required: different alarm priority levels
The idea of a new device event, e.g. ALARM_EVENT, a standardized device ALARM_EVENT was therefore advanced for this purpose

6. TANGO Alarm Extensions Rationale
SKA is not alone in requiring more robust alarm acknowledgement and more complex alarm definitions
Elettra/Alba developed extensions
Alarms can still be setup in the TANGO core
Extensions have a concept of raising events, in the same way as requested by most elements
Extensions also execute specified callback functions when alarms go ON/OFF
Extensions allow complex formulae, useful for logical/high-level alarms

7. ELETTRA: Alarm System

8. ELETTRA: Alarm Formulae
Formula can contain attribute names, numbers, Tango States, logical and mathematical operators. Expressions can be enclosed by round brackets.
Attribute names have to be of four fields separated by “/”, ex: b/dev/test/curr
Numbers can be real numbers (ex: 35, -23.5, ...) or hexadecimal (ex: 0xaf, 0x1A)
Tango States allowed are ON, OFF, CLOSE, OPEN, INSERT, EXTRACT, MOVING, STANDBY, FAULT, INIT, RUNNING, ALARM, DISABLE, UNKNOWN
Operators allowed are the following in order of precedence: *, /, +, -, <<, >>, <=, >=, >, <, ==, !=, &, |, ^, &&, ||. Unary operators abs, ! are allowed
An example:
(b/dev/test1/val * 2.5 >= abs(b/dev/test2/val) && (b/dev/test3/state == OFF) || b/dev/test4/stat & 0xA0)

9. ELETTRA: Callback Methods I
The Elettra formulae allow the definition of two callbacks for alarm configurations, one for alarm entry, another for alarm exit.
These two commands can be specified as the last field of the alarm string given to the “Load” command and have to be separated by the “;” character. This field is optional.
An example:
b/sec/test/curr (b/sec/test/current > 100) 0 log gr_all “Alarm Curr! b/sec/dev1/action;b/sec/dev2/action”
command b/sec/dev1/action is executed when changing state to ALARM
command b/sec/dev2/action is executed when changing state to NORMAL

10. ELETTRA: Callback Methods II
Callbacks either have no arguments (Tango::DEV_VOID), or take in a single string (Tango::DEV_STRING)
E.g. For the alarm loaded with the following:
b/sec/test/dev ((b/sec/test/current > 100) && (b/sec/test/voltage > 20)) 30 log gr_all “Alarm Device!” b/sec/dev/notif_alarm;b/sec/dev/notif_normal
The value passed to b/sec/dev/notif_alarm if it accepts a Tango::DEV_STRING as input would be:
b/sec/test/dev;b/sec/test/current=101.75;b/sec/test/volt age=22.6

11. Experimental Work
Start polling on the attributes needed for a specific rule.
Event firestarters need to be configured for the concerned attributes as well e.g. absolute/relative change criteria for change events to be fired by the TANGO kernel.
An alarm rule is then configured in the Elettra Alarm device server - which automatically sets up subscriptions to the change events of the involved attributes etc.
The TANGO core is only used as a source of events - the Elettra alarm server will itself make checks on specific values (and other logical conditions e.g. combined alarms) and decide for itself whether to raise an alarm event or not.
The TANGO core alarm levels can still be used separately if needed (e.g. to change Quality factor, which in turn could be utilized by the alarm system), but can otherwise be bypassed.

12. FormulaConf I
Another TANGO device, works in tandem with Elettra Alarm System
Essentially a parser generating dynamic attributes for each alarm formula. Types are (Bool, Long, Double, spectrum of the same types)
Handles complex alarm formulae
Properties need to be set up:
Attr_config_prop: an array of strings of the form: attr;type; dynamic attribute name; index; formula; description (optional)
Attr_config_file: the path of the file with an array of strings
Polling_period: if = -1, disabled, else period (in sec.) of the internal polling thread
Read_timeout: if internal polling is enabled, this is the time (sec) after which the attribute is read even if the corresponding event is not in error.

13. FormulaConf II
Same rules of Elettra Alarm System apply – polling and eventing
Perhaps a generic way of doing this can be formulated by the SKA
Devices have to be available prior to FormulaConf startup, otherwise a restart is needed
E.g.
admin_proxy = DeviceProxy(formula_conf_proxy.adm_name())
admin_proxy.restartserver()

14. AAVS Alarm Stream Device Prototype
LFAA/AAVS has prototyped an Alarm Stream device
Any number of stream devices can be deployed
E.g. streams for different levels
Centralized/uniform alarm handling
More on this in later sessions