3-Feb-2011CS-Workshop 2011, Dietrich Beck Overview CS-Framework Introduction (Over)view Lessons Learned

3-Feb-2011CS-Workshop 2011, Dietrich Beck Starting Point (  2001) Control System –Control devices (power supplies, function generators, timing, complex experimental procedure…) –No process control –DAQ and data processing is only a minor issue –Integrate things that were not developed to work together as a system Ever changing set-up –The same experiment is never done twice –Reconfiguration of complete hardware (seconds!) –Integration of new hardware (next beam-time) Small and medium size experiments –No dedicated staff for control systems –Different experiments but similar problems to solve (configuration, complex timing schemes, similar hardware,…) …  What about a “Framework”?

3-Feb-2011CS-Workshop 2011, Dietrich Beck "Framework" Wikipedia: “…a software framework is an abstraction in which common code providing generic functionality can be selectively overridden or specialized by user code, thus providing specific functionality…” Common Code – Standardization –re-usability of software components –maintainability of software components –same look and feel at different experiments –ease transfer of know-how User Code - Specialization –concentrates on experiment specific requirements –extension of framework, specializing code

3-Feb-2011CS-Workshop 2011, Dietrich Beck Different Points of View – A Framework is... Developer: User: picture pocket knife... a tool box... something ready to use (sth. to solve a problem) without additional work (= the solution!)

3-Feb-2011CS-Workshop 2011, Dietrich Beck Definition of the CS-Framework control system = framework + add-ons EE/KS and contributors experiment bug reports, new features requested add-ons may become part of framework bug fixes, new features, maintenance Cycle Control AFG Timing+DAQ HV GUI

3-Feb-2011CS-Workshop 2011, Dietrich Beck PHELIX (Petawatt High-Energy Laser for Heavy Ion Experiments) (slide by S. Götte, GSI) Up to 3 MJ electrical energy converted to 1kJ light energy or 0.5 Petewatt laser power documented “shots” (Dec. 2010). About 10k process variables distributed on 15+ nodes in 2 buildings

3-Feb-2011CS-Workshop 2011, Dietrich Beck Quad-Triple Trap Mass Spectrometer ISOLTRAP High-precision mass determination of unstable nuclei with a Penning trap mass spectrometer at ISOLDE/CERN Quad-Triple Trap Mass Spectrometer ISOLTRAP High-precision mass determination of unstable nuclei with a Penning trap mass spectrometer at ISOLDE/CERN –Nuclear-, Astro-, Fundamental Physics –  m/m  (typically!) –via cyclotron frequency of stored ions –“sub-µs ion-juggling” every second

3-Feb-2011CS-Workshop 2011, Dietrich Beck Cooking Recipe for the CS Framework One development tool  LabVIEW Standardization  object oriented approach, base classes Distribution to many nodes  DIM ( –Event driven communication for everything –Scaling to large systems by distribution –Remote access –… SCADA functionality (alarming, trending, …)  LabVIEW DSC module

3-Feb-2011CS-Workshop 2011, Dietrich Beck Event Driven Communication DIM ( –Peer-to-peer connection, no intrinsic bottle neck –Named service as concept (a name server aids in setting up connections) –“service”: publisher-subscriber-pattern, one-to-many –“command service”: command-pattern, many-to-one CS objects communicate via –DIM services: publishing/subscribing to state information –DIM commands: triggering actions synchronous commands (timeout, “expire date”,…) everybody may talk to everybody Events provide the interface between –device object and it’s GUI object –device object and application layer –hierarchical state machines –… Design Approach - First: communication layer, Second: framework Connectivity to other programming languages and OSs CallerCallee Back-end (SCADA, GUI,...) Middleware (communication,.) Front-end (devices, drivers,...) 3-layers

3-Feb-2011CS-Workshop 2011, Dietrich Beck Object Orientation (OO) with CS "BaseClass" provides basic functionality (communication layer, active threads,...) "DeviceClass" adds functionality according to specs of device type "DS345". Instantiation: one object per device "BaseClass" "Device Class" inheritance AFG1 AFG2 AFG3 of course: classes for GUIs, Sequencer, State machines,... OO implemented by CS using pure LabVIEW (no LVOOP)

3-Feb-2011CS-Workshop 2011, Dietrich Beck Standardization of Services and Cmds _birthday, time/date of creation _classID, class name _condition, object created successfully… _systemID, name of (sub)system _accessID, reservation mechanism... _evtCounter, number of cmds received status of threads…Ping, thread alive? GetDescriptors (Introspection!!!) Reset (override by child classes) … _deviceGUI, name of GUI classInitialize, init interface _deviceID, ID string of deviceClose, close interface _deviceState, OK, ERROR, …IDQuery, query device ID… _nomFrequency, frequency set-valueSetFrequency, sets nominal value _actFrequency, frequency get-valueGetFrequency, gets actual value… CSObj THE base class CAEObj send/rec. events BaseProcess threads for events and periodic action DeviceBase basic device properties AFGBase arbitrary function generator…

3-Feb-2011CS-Workshop 2011, Dietrich Beck (Over)view CS-Framework Introduction (Over)view Lessons Learned

3-Feb-2011CS-Workshop 2011, Dietrich Beck SourceForge: downloads, tracker (bugs, features,…)

3-Feb-2011CS-Workshop 2011, Dietrich Beck SCC via GSI

3-Feb-2011CS-Workshop 2011, Dietrich Beck Domain Management System: Process Management in a Distributed Environment

3-Feb-2011CS-Workshop 2011, Dietrich Beck Process Monitoring

3-Feb-2011CS-Workshop 2011, Dietrich Beck Packaging

3-Feb-2011CS-Workshop 2011, Dietrich Beck Documentation via Wiki

3-Feb-2011CS-Workshop 2011, Dietrich Beck Class Documentation via UML Tools

3-Feb-2011CS-Workshop 2011, Dietrich Beck Status CS-Framework Introduction Over(view) Lessons Learned

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – What Users say about CS Lessons Learned – What Users say about CS (from a talk by Stefan Götte, at the NI BIG PHYSICS Round Table, Paris, 2009) Stress Field of the Responsible Person: The OS (XP) and/or the intranet are always unreliable, the programming language (LabVIEW) never really fulfills the need, the framework (CS) is only close to the requirement, the classes of other CS collaborators are typically not usable, there is no way to test things since there is no test system available while the real system is always in use, the users never define what the program has to do, but are not pleased with what the programmer delivers, and they misuse the system additionally. Anyhow: The goal is an easy system where the happy user does not realize what happens behind the scene, which works reliable and for ever (better: till the next LabVIEW version is installed).

3-Feb-2011CS-Workshop 2011, Dietrich Beck “The OS (XP) and/or the intranet are always unreliable” (assumption for design phase) Design principles of major importance A control system must be designed to crash and (auto-)recover from failures of all kinds –Fileserver down –Computer down/reboot Installation of updates Switched-Off/On by users … –Network down –Power cuts –… The system is stressed most, when –everything goes wrong (failures, alarms, sub-system crash…) and –all kinds of asynchronous signals fire at the same time (self-induced DoS-Attack) and –availability is most important.

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – What Users say about CS Lessons Learned – What Users say about CS (from a talk by Stefan Götte, at the NI BIG PHYSICS Round Table, Paris, 2009) Stress Field of the Responsible Person: The OS (XP) and/or the intranet are always unreliable, the programming language (LabVIEW) never really fulfills the need, the framework (CS) is only close to the requirement, the classes of other CS collaborators are typically not usable, there is no way to test things since there is no test system available while the real system is always in use, the users never define what the program has to do, but are not pleased with what the programmer delivers, and they misuse the system additionally. Anyhow: The goal is an easy system where the happy user does not realize what happens behind the scene, which works reliable and for ever (better: till the next LabVIEW version is installed).

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned - Documentation The fundamental problem about documentation Two lines of text are not enough, but Two pages of docs – (almost) nobody will read that * Docs out of date … Solutions HOW-TOs, FAQs Auto-generated documentation –Class documentation, CS2Java  UML –Requires well documented code (again: coding conventions!) –Release notes Demo-system (delayed…) * sometimes not even the first two lines…

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – Generic/Common Tools Generic tools (generic control system GUI, generic device GUIs) Sometimes difficult to understand (  documentation problem) Lot’s of work to maintain Don’t match the users’ needs exactly Either –least common denominator (missing features), or –include all requested features (too complex) “Nobody likes them, but everybody uses them.”

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – Missing Features Identifying missing features sometimes requires telepathic skills results in cool solutions that are finally not used (nice to have versus really useful) –CS Access System –Sequencer –ObjectNets, PetriNets –some base classes …

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – Compatibility Backward Compatibility (“does my old code still work?”) Is maybe THE most important feature of a framework Often results in a impressive mix of different versions (core system v1, base classes v2, GUI v4, device classes v3,…) Makes it hard to correct wrong design decisions Requires clear public library routines or class methods, but… using conventions on what is public/private does not work

3-Feb-2011CS-Workshop 2011, Dietrich Beck Lessons Learned – The “Saving-Time-Myth” Wikipedia: “…software frameworks … reducing overall development time” (?) Amount of time for solving a problem decreases dramatically, if a problem may be solved with (generic) existing software. “configuration instead of coding” does not change – but the solution is much better may even increase (short-term), compared to a dedicated solution not (!) using the framework (required: training, courses, understanding and application of conventions) decreases (long-term): framework maintained by others, re- usability of code, replacing hardware, coding conventions enforced, maintainability, common language, know-how transfer, …

3-Feb-2011CS-Workshop 2011, Dietrich Beck Experiments using the CS framework... require high flexibility have a large variety of hardware types have up to 10,000 (1M possible) process variables require fast (  100 ns) timing control using dedicated hardware PHELIX PHELIX Motion CaveA SHIPTRAP ISOLTRAP REXTRAP REXTRAP LEBIT LEBIT GSI, Germany Mainz, Germany Greifswald, Germany CERN, Switzerland MSU, USA Lanzhou, China data taking development commissioning Motion CaveA FOPI RISING others... HITRAP LPT TrigaTRAP ClusterTRAP POLARIS, HIJ

3-Feb-2011CS-Workshop 2011, Dietrich Beck Conclusion and Outlook CS 3.21 released for LV2009. about 15 active applications 1,000,000 PVs demonstrated, an even larger number should be feasible 5,000 objects (  hardware devices) demonstrated, an even larger number should be feasible stability of a distributed CS system is better than a few hundred hours of continuous operation about 15 (60) hardware devices supported on SourceForge (Subversion) FAIR: MATS and parts of HITRAP have selected CS as control system framework.

3-Feb-2011CS-Workshop 2011, Dietrich Beck Acknowledgements... Holger Brand, Dietrich Beck, Mathias Richter, Alexander Schwinn, Falk Ziegler, Romain Savreux, Chabouh Yazidjian, Klaus Blaum, Dennis Neidherr, Melanie Wolf, Stefan Schwarz, Josh Savory, Stefan Götte, Maximilian Kugler, Tobias Habermann, Michael Block, Frank Herfurth, Christian Rauth, Stephen Koszudowski, Manas Mukherjee, Thomas Rechel, Martin Feldmann, ……………...