A Universal Framework for Data Validation Giovanni Organtini INFN-Sez. di Roma

ACAT03-Giovanni Organtini2 Data Validation Data Validation (DV) – Fast analysis of data coming from a DAQ system to evaluate data usability – Each atomic analysis program is called task fast, light and single-objective Data Validation must be – Able to evolve on a short timescale – Flexible to manage multiple and different validation tasks – Robust to allow many different users to cooperate without interference

ACAT03-Giovanni Organtini3 DV Framework Concepts Sensors : elements that detect events such as an EOR or a Warning signal issued by analysis tasks Events : any Sensor outcome Mediator : a process that collects Events from Sensors and dispatches them to Solutions Solutions : objects that recognize Events and act according to their content

ACAT03-Giovanni Organtini4 Framework architecture

ACAT03-Giovanni Organtini5 Framework architecture Sensors

ACAT03-Giovanni Organtini6 Framework architecture Notify Event to Mediator

ACAT03-Giovanni Organtini7 The Sensors AlarmSensor: parses HTML-formatted files and detects comments of the form <!–- SensorHTML Creates events according to comment content EORSensor: parses a text file (runlist) and detects updates. A line is added to runlist at each EOR

ACAT03-Giovanni Organtini8 Framework architecture Dispatch Event to Solutions

ACAT03-Giovanni Organtini9 Framework architecture Solutions

ACAT03-Giovanni Organtini10 The Solutions Program: launch one or more programs in a given directory (repair known problems) send an with arbitrary text to a list of recipient Window: show a popup window on a given terminal with arbitrary HTML- formatted text

ACAT03-Giovanni Organtini11 A more formal view Implemented in Java – portable – distributed (RMI-server created to allow Sensors to run on remote processors)

ACAT03-Giovanni Organtini12 How it works Sensors detect changes in files – an EOR-Sensor can detect an EOR looking at the content of a given file containing the runlist – An AlarmSensor can parse output files produced by analysis tasks to detect potential problems The Mediator collects Events and notify them to all the existing Solutions The Solutions choose to activate themselves according to Event content

ACAT03-Giovanni Organtini13 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date= –ebeam=50 –beam=e

ACAT03-Giovanni Organtini14 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date= –ebeam=50 –beam=e directory containing analysis tasks for this application

ACAT03-Giovanni Organtini15 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date= –ebeam=50 –beam=e directory containing analysis tasks for this application parameters to be passed to analysis tasks (arbitrary)

ACAT03-Giovanni Organtini16 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date= –ebeam=50 –beam=e Program Solution activated directory containing analysis tasks for this application parameters to be passed to analysis tasks (arbitrary)

ACAT03-Giovanni Organtini17 Analysis tasks Analysis tasks can be written in ANY language, provided they can accept arguments on the command line – Examples: C, C++, Perl, shell scripts, etc. They must provide the output in HTML-formatted files to be located in a given directory – output can be either text and/or images – the result of any single task could be browsed from everywhere using any browser – a process will automatically generate the list of output files to be shown to the shifter

ACAT03-Giovanni Organtini18 Alarm Sensors They look for HTML-comments in output files and generate events according to the comment content – comments not shown by browsers – must be compliant with a set of rules – actions may be combined (i.e. can send s and notify a message to the shifter)

ACAT03-Giovanni Organtini19 Example <!– SensorHTML winFlag=false Flag=true commandFlag=false senderName=Giovanni Organtini - ped analyzer smtpServer=localhost subject=Problem with pedestals in test beam A problem has occurred during the analysis of the pedestal. May be you want to run to the detector to investigate…..-->

ACAT03-Giovanni Organtini20 Solution activation The Solution is activated – sends an to Coordinator and Task responsible – the body is arbitrary and may contain links and other HTML-tags Other informative message can be written on the output file containing the comment – output file may contain text images links

ACAT03-Giovanni Organtini21 Applications CMS Crystal Calorimeter Construction in Roma – Crystals measured by automatic machine – Results analyzed by validation tasks – Problems notified to construction supervisor via – Some known problems recovered via program Solution CMS Crystal Calorimeter Test-Beam at CERN – Calorimeter modules exposed to e,  beams – Data collected and analyzed by validation tasks – Problems notified to experts via and to shifters via popup windows

ACAT03-Giovanni Organtini22 Conclusion A Data Validation Framework has been developed – simple to manage and use – universal – flexible – robust – configurable – no requirements on analysis tasks – no library to link or file to include Package available soon (2004) for download on