1. A Universal Framework for Data Validation Giovanni Organtini INFN-Sez. di Roma

2. ACAT03-Giovanni Organtini2 Data Validation Data Validation (DV) – Fast analysis of data coming from a DAQ system to evaluate data usability – Each atomic analysis program is called task fast, light and single-objective Data Validation must be – Able to evolve on a short timescale – Flexible to manage multiple and different validation tasks – Robust to allow many different users to cooperate without interference

3. ACAT03-Giovanni Organtini3 DV Framework Concepts Sensors : elements that detect events such as an EOR or a Warning signal issued by analysis tasks Events : any Sensor outcome Mediator : a process that collects Events from Sensors and dispatches them to Solutions Solutions : objects that recognize Events and act according to their content

4. ACAT03-Giovanni Organtini4 Framework architecture

5. ACAT03-Giovanni Organtini5 Framework architecture Sensors

6. ACAT03-Giovanni Organtini6 Framework architecture Notify Event to Mediator

7. ACAT03-Giovanni Organtini7 The Sensors AlarmSensor: parses HTML-formatted files and detects comments of the form <!–- SensorHTML Creates events according to comment content EORSensor: parses a text file (runlist) and detects updates. A line is added to runlist at each EOR

8. ACAT03-Giovanni Organtini8 Framework architecture Dispatch Event to Solutions

9. ACAT03-Giovanni Organtini9 Framework architecture Solutions

10. ACAT03-Giovanni Organtini10 The Solutions Program: launch one or more programs in a given directory (repair known problems) E-mail: send an e-mail with arbitrary text to a list of recipient Window: show a popup window on a given terminal with arbitrary HTML- formatted text

11. ACAT03-Giovanni Organtini11 A more formal view Implemented in Java – portable – distributed (RMI-server created to allow Sensors to run on remote processors)

12. ACAT03-Giovanni Organtini12 How it works Sensors detect changes in files – an EOR-Sensor can detect an EOR looking at the content of a given file containing the runlist – An AlarmSensor can parse output files produced by analysis tasks to detect potential problems The Mediator collects Events and notify them to all the existing Solutions The Solutions choose to activate themselves according to Event content

13. ACAT03-Giovanni Organtini13 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date=20031111 –ebeam=50 –beam=e

14. ACAT03-Giovanni Organtini14 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date=20031111 –ebeam=50 –beam=e directory containing analysis tasks for this application

15. ACAT03-Giovanni Organtini15 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date=20031111 –ebeam=50 –beam=e directory containing analysis tasks for this application parameters to be passed to analysis tasks (arbitrary)

16. ACAT03-Giovanni Organtini16 EOR-Sensor The EOR-Sensor generate an Event as soon as a new line appear in the runlist runlist formatted as follows --date=20031111 –ebeam=50 –beam=e Program Solution activated directory containing analysis tasks for this application parameters to be passed to analysis tasks (arbitrary)

17. ACAT03-Giovanni Organtini17 Analysis tasks Analysis tasks can be written in ANY language, provided they can accept arguments on the command line – Examples: C, C++, Perl, shell scripts, etc. They must provide the output in HTML-formatted files to be located in a given directory – output can be either text and/or images – the result of any single task could be browsed from everywhere using any browser – a process will automatically generate the list of output files to be shown to the shifter

18. ACAT03-Giovanni Organtini18 Alarm Sensors They look for HTML-comments in output files and generate events according to the comment content – comments not shown by browsers – must be compliant with a set of rules – actions may be combined (i.e. can send e-mails and notify a message to the shifter)

19. ACAT03-Giovanni Organtini19 Example <!– SensorHTML winFlag=false emailFlag=true commandFlag=false senderEmail=Giovanni.Organtini@roma1.infn.it senderName=Giovanni Organtini - ped analyzer receiverEmail=tb.coord@cms.cern.ch,G.Organtini@cern.ch smtpServer=localhost subject=Problem with pedestals in test beam A problem has occurred during the analysis of the pedestal. May be you want to run to the detector to investigate…..-->

20. ACAT03-Giovanni Organtini20 Solution activation The e-mail Solution is activated – sends an e-mail to Coordinator and Task responsible – the e-mail body is arbitrary and may contain links and other HTML-tags Other informative message can be written on the output file containing the comment – output file may contain text images links

21. ACAT03-Giovanni Organtini21 Applications CMS Crystal Calorimeter Construction in Roma – Crystals measured by automatic machine – Results analyzed by validation tasks – Problems notified to construction supervisor via e-mail – Some known problems recovered via program Solution CMS Crystal Calorimeter Test-Beam at CERN – Calorimeter modules exposed to e,  beams – Data collected and analyzed by validation tasks – Problems notified to experts via e-mail and to shifters via popup windows

22. ACAT03-Giovanni Organtini22 Conclusion A Data Validation Framework has been developed – simple to manage and use – universal – flexible – robust – configurable – no requirements on analysis tasks – no library to link or file to include Package available soon (2004) for download on http://www.roma1.infn.it/people/organtini