14-Dec-2010IT/EE-Palaver, Dietrich Beck Status CS-Framework Introduction Status Lessons Learned

14-Dec-2010IT/EE-Palaver, Dietrich Beck Starting Point (  2001) Control System –Control devices (power supplies, function generators, timing, complex experimental procedure…) –No process control –DAQ and data processing is only a minor issue –Integrate things that were not developed to work together as a system Ever changing set-up –The same experiment is never done twice –Reconfiguration of complete hardware (seconds!) –Integration of new hardware (next beam-time) Small and medium size experiments –No dedicated staff for control systems –Different experiments but similar problems to solve (configuration, complex timing schemes, similar hardware,…) …  What about a “Framework”?

14-Dec-2010IT/EE-Palaver, Dietrich Beck "Framework" Wikipedia: “…a software framework is an abstraction in which common code providing generic functionality can be selectively overridden or specialized by user code, thus providing specific functionality…” Common Code – Standardization –re-usability of software components –maintainability of software components –same look and feel at different experiments –ease transfer of know-how User Code - Specialization –concentrates on experiment specific requirements –extension of framework, specializing code

14-Dec-2010IT/EE-Palaver, Dietrich Beck Different Points of View – A Framework is... Developer: User: (here: pocket knife)... a tool box... something ready to use (sth. to solve a problem) without additional work (= the solution!)

14-Dec-2010IT/EE-Palaver, Dietrich Beck Definition of the CS-Framework control system = framework + add-ons EE/KS and contributors experiment bug reports, new features requested add-ons may become part of framework bug fixes, new features, maintenance Cycle Control AFG Timing+DAQ HV GUI

14-Dec-2010IT/EE-Palaver, Dietrich Beck PHELIX (Petawatt High-Energy Laser for Heavy Ion Experiments) (slide by S. Götte, GSI) Up to 3 MJ electrical energy converted to 1kJ light energy or 0.5 Petewatt laser power documented “shots” (Dec. 2010). About 10k process variables distributed on 15+ nodes in 2 buildings

14-Dec-2010IT/EE-Palaver, Dietrich Beck Quad-Triple Trap Mass Spectrometer ISOLTRAP High-precision mass determination of unstable nuclei with a Penning trap mass spectrometer at ISOLDE/CERN Quad-Triple Trap Mass Spectrometer ISOLTRAP High-precision mass determination of unstable nuclei with a Penning trap mass spectrometer at ISOLDE/CERN –Nuclear-, Astro-, Fundamental Physics –  m/m  (typically!) –via cyclotron frequency of stored ions –“sub-µs ion-juggling” every second

14-Dec-2010IT/EE-Palaver, Dietrich Beck Cooking Recipe for the CS Framework One development tool  LabVIEW Standardization  object oriented approach, base classes Distribution to many nodes  DIM ( –Event driven communication for everything –Scaling to large systems by distribution –Remote access –… SCADA functionality (alarming, trending, …)  LabVIEW DSC module

14-Dec-2010IT/EE-Palaver, Dietrich Beck Event Driven Communication DIM ( –Peer-to-peer connection, no intrinsic bottle neck –Named service as concept (a name server aids in setting up connections) –“service”: publisher-subscriber-pattern, one-to-many –“command service”: command-pattern, many-to-one CS objects communicate via –DIM services: publishing/subscribing to state information –DIM commands: triggering actions synchronous commands (timeout, “expire date”,…) everybody may talk to everybody Events provide the interface between –device object and it’s GUI object –device object and application layer –hierarchical state machines –… Design Approach - First: communication layer, Second: framework Connectivity to other programming languages and OSs CallerCallee Back-end (SCADA, GUI,...) Middleware (communication,.) Front-end (devices, drivers,...) 3-layers

14-Dec-2010IT/EE-Palaver, Dietrich Beck Object Orientation (OO) with CS "BaseClass" provides basic functionality (communication layer, active threads,...) "DeviceClass" adds functionality according to specs of device type "DS345". Instantiation: one object per device "BaseClass" "Device Class" inheritance AFG1 AFG2 AFG3 of course: classes for GUIs, Sequencer, State machines,... OO implemented by CS using pure LabVIEW (no LVOOP)

14-Dec-2010IT/EE-Palaver, Dietrich Beck Standardization of Services and Cmds _birthday, time/date of creation _classID, class name _condition, object created successfully… _systemID, name of (sub)system _accessID, reservation mechanism... _evtCounter, number of cmds received status of threads…Ping, thread alive? GetDescriptors (Introspection!!!) Reset (override by child classes) … _deviceGUI, name of GUI classInitialize, init interface _deviceID, ID string of deviceClose, close interface _deviceState, OK, ERROR, …IDQuery, query device ID… _nomFrequency, frequency set-valueSetFrequency, sets nominal value _actFrequency, frequency get-valueGetFrequency, gets actual value… CSObj THE base class CAEObj send/rec. events BaseProcess threads for events and periodic action DeviceBase basic device properties AFGBase arbitrary function generator…

14-Dec-2010IT/EE-Palaver, Dietrich Beck Status CS-Framework Introduction Status Lessons Learned

14-Dec-2010IT/EE-Palaver, Dietrich Beck Status CS 3.20 based on LabVIEW 2009, WinXP 32bit, Suse 11.2 Scattered into packages (Main, device base (+GUIs), devices, GUIs (configuration, operation,…)) + >20 device classes maintained by db, hb GPL licensed … Test of operational stability (non-stop) Used by about +12 experiments/facilities at GSI and elsewhere CS works as a collaboration Additional tools –Configuration data base (“SQL-server”, editing tools, import, export…) …

14-Dec-2010IT/EE-Palaver, Dietrich Beck SourceForge: downloads, tracker (bugs, features,…)

14-Dec-2010IT/EE-Palaver, Dietrich Beck SCC via GSI

14-Dec-2010IT/EE-Palaver, Dietrich Beck Domain Management System: Process Management in a Distributed Environment

14-Dec-2010IT/EE-Palaver, Dietrich Beck Process Monitoring

14-Dec-2010IT/EE-Palaver, Dietrich Beck Packaging

14-Dec-2010IT/EE-Palaver, Dietrich Beck Documentation via Wiki

14-Dec-2010IT/EE-Palaver, Dietrich Beck Class Documentation via UML Tools

14-Dec-2010IT/EE-Palaver, Dietrich Beck Status CS-Framework Introduction Status Lessons Learned

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – What Users say about CS Lessons Learned – What Users say about CS (from a talk by Stefan Götte, at the NI BIG PHYSICS Round Table, Paris, 2009) Stress Field of the Responsible Person: The OS (XP) and/or the intranet are always unreliable, the programming language (LabVIEW) never really fulfills the need, the framework (CS) is only close to the requirement, the classes of other CS collaborators are typically not usable, there is no way to test things since there is no test system available while the real system is always in use, the users never define what the program has to do, but are not pleased with what the programmer delivers, and they misuse the system additionally. Anyhow: The goal is an easy system where the happy user does not realize what happens behind the scene, which works reliable and for ever (better: till the next LabVIEW version is installed).

14-Dec-2010IT/EE-Palaver, Dietrich Beck “The OS (XP) and/or the intranet are always unreliable” (assumption for design phase) Design principles of major importance A control system must be designed to crash and (auto-)recover from failures of all kinds –Fileserver down –Computer down/reboot Installation of updates Switched-Off/On by users … –Network down –Power cuts –… The system is stressed most, when –everything goes wrong (failures, alarms, sub-system crash…) and –all kinds of asynchronous signals fire at the same time (self-induced DoS-Attack) and –availability is most important.

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – What Users say about CS Lessons Learned – What Users say about CS (from a talk by Stefan Götte, at the NI BIG PHYSICS Round Table, Paris, 2009) Stress Field of the Responsible Person: The OS (XP) and/or the intranet are always unreliable, the programming language (LabVIEW) never really fulfills the need, the framework (CS) is only close to the requirement, the classes of other CS collaborators are typically not usable, there is no way to test things since there is no test system available while the real system is always in use, the users never define what the program has to do, but are not pleased with what the programmer delivers, and they misuse the system additionally. Anyhow: The goal is an easy system where the happy user does not realize what happens behind the scene, which works reliable and for ever (better: till the next LabVIEW version is installed).

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned - Documentation The fundamental problem about documentation Two lines of text are not enough, but Two pages of docs – (almost) nobody will read that * Docs out of date … Solutions HOW-TOs, FAQs Auto-generated documentation –Class documentation, CS2Java  UML –Requires well documented code (again: coding conventions!) –Release notes Demo-system (delayed…) * sometimes not even the first two lines…

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – Generic/Common Tools Generic tools (generic control system GUI, generic device GUIs) Sometimes difficult to understand (  documentation problem) Lot’s of work to maintain Don’t match the users’ needs exactly Either –least common denominator (missing features), or –include all requested features (too complex) “Nobody likes them, but everybody uses them.”

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – Missing Features Identifying missing features sometimes requires telepathic skills results in cool solutions that are finally not used (nice to have versus really useful) –CS Access System –Sequencer –ObjectNets, PetriNets –some base classes …

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – Compatibility Backward Compatibility (“does my old code still work?”) Is maybe THE most important feature of a framework Often results in a impressive mix of different versions (core system v1, base classes v2, GUI v4, device classes v3,…) Makes it hard to correct wrong design decisions Requires clear public library routines or class methods, but… using conventions on what is public/private does not work

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – Users Users are (of course!) the best motivation for the work of the framework- developer are mainly interested in running their experiment and not so much in software development sometimes keep on using old (but working) solutions (a GUI developed in 1991 is still used at about five experiments) – it’s sometimes hard to make a clear cut and this may influence design decisions sometimes develop nice code (reusability, coding conventions, etc...) … and coding conventions /  may use software in a completely different way than it was intended by the developer test our software (the first beam-time is the real test), we don’t have a department with 50 people who do nothing than testing and formal verification of our software

14-Dec-2010IT/EE-Palaver, Dietrich Beck Lessons Learned – The “Saving-Time-Myth” Wikipedia: “…software frameworks … reducing overall development time” (?) Amount of time for solving a problem decreases dramatically, if a problem may be solved with (generic) existing software. “configuration instead of coding” does not change – but the solution is much better may even increase (short-term), compared to a dedicated solution not (!) using the framework (required: training, courses, understanding and application of conventions) decreases (long-term): framework maintained by others, re- usability of code, replacing hardware, coding conventions enforced, maintainability, common language, know-how transfer, …

14-Dec-2010IT/EE-Palaver, Dietrich Beck Experiments using the CS framework... require high flexibility have a large variety of hardware types have up to 10,000 (1M possible) process variables require fast (  100 ns) timing control using dedicated hardware PHELIX PHELIX Motion CaveA SHIPTRAP ISOLTRAP REXTRAP REXTRAP LEBIT LEBIT GSI, Germany Mainz, Germany Greifswald, Germany CERN, Switzerland MSU, USA Lanzhou, China data taking development commissioning Motion CaveA FOPI RISING others... HITRAP LPT TrigaTRAP ClusterTRAP POLARIS, HIJ

14-Dec-2010IT/EE-Palaver, Dietrich Beck Conclusion and Outlook CS 3.20 released for LV2009. about 15 active applications 1,000,000 PVs demonstrated, an even larger number should be feasible 5,000 objects (  hardware devices) demonstrated, an even larger number should be feasible stability of a distributed CS system is better than a few hundred hours of continuous operation about 15 (60) hardware devices supported on SourceForge (Subversion) FAIR: MATS and parts of HITRAP have selected CS as control system framework.

14-Dec-2010IT/EE-Palaver, Dietrich Beck Acknowledgements... Holger Brand, Dietrich Beck, Mathias Richter, Alexander Schwinn, Falk Ziegler, Romain Savreux, Chabouh Yazidjian, Klaus Blaum, Dennis Neidherr, Melanie Wolf, Stefan Schwarz, Josh Savory, Stefan Götte, Maximilian Kugler, Tobias Habermann, Michael Block, Frank Herfurth, Christian Rauth, Stephen Koszudowski, Manas Mukherjee, Thomas Rechel, Martin Feldmann, ……………...