1. The GRIDCC Instrument Element: from prototypes to production environments Roberto Pugliese On Behalf of the GRIDCC Collaboration EU FP6 Project

2. www.gridcc.org (EU FP6 Contract 511382) 1. The GRIDCC - Grid Enabled Remote Instrumentation with Distributed Control and Computation – project has the main aim to bring Instruments to the GRID 2. It is a 3-years EU FP6 project started in September 2004 3. Web site: www.gridcc.orgwww.gridcc.org 4. More in particular the project goals are: 1. Definition of a “Instrument Element” allowing a standard remote access to any type of instrumentation 2. Tight integration between instrument grid and classical computational grid 3. Human interaction with Grids via Virtual Control Room (collaborative environment) 4. Enactment of complex workflows The GRIDCC project: Goals & Objectives

3. www.gridcc.org (EU FP6 Contract 511382) The GRIDCC partners Participant nameCountry Istituto Nazionale di Fisica NucleareItaly Institute Of Accelerating Systems and ApplicationsGreece Brunel UniversityUK Consorzio Interuniversitario per TelecomunicazioniItaly Sincrotrone Trieste S.C.P.AItaly IBM (Haifa Research Lab)Israel Imperial College of Science, Technology & MedicineUK Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche Italy Universita degli Studi di UdineItaly Greek Research and Technology Network S.A.Greece

4. www.gridcc.org (EU FP6 Contract 511382) Execution ServicesCollaborative Environment GRIDCC: Architecture (I) Instruments GridComputational Grid IE CE SE Instrument Element Problem Solver Instrument Manager Information & Monitoring System DATA ES VCR Workflow Service Oriented Architecture – SOA The GRIDCC services are Web Services compliant, according to the WS-I convention. Any type of client WS-I compliant, can access the GRIDCC services without any specific software library. X509 based security is used When performances are required, a kerberos based mechanism is used to access IEs.

5. www.gridcc.org (EU FP6 Contract 511382) GRIDCC Architecture Web Service Interface Execution Service WfMS WMS AS Instrument elements (IE) Storage Element (SE) Instrument elements (IE) Storage Element (SE) Instrument Element (IE) Storage Element (SE) Compute element (CE) Compute element (CE) Computing Element (CE) Storage Element (SE) Storage Element (SE) Storage Element (SE) Global Problem Solver Information and Monitoring Services (IMS) Information System (BDII) Security Services Virtual Control Room Virtual Control Room Collaborative Services (CS) WMS Work Management System WfMS Work Flow Mng System AS Agreement Service Broker

6. www.gridcc.org (EU FP6 Contract 511382) GridCC PermanentTest Bed VOMS AS, KrbServer VCR, SE, IE, BDII IE, CE/CREAM,SE, LFC WMProxy WfMS VCR, IE

7. www.gridcc.org (EU FP6 Contract 511382) GRIDCC main target areas (Large-scale) scientific experiments High energy particle physics (Radio-) Telescopes Remote process control Accelerator control (Tele-) Biomedicine Robotics Automotive Electronic microscopes Widely Sparse Instrumentation Power Grids Monitoring of the territory Monitoring of the sea Geo-hazard prediction Distributed laboratories Transportation monitoring Sensor network GRIDCC Middleware

8. www.gridcc.org (EU FP6 Contract 511382) GRIDCC pilot applications Power Grid High Energy Physics CMS High Energy Physics CMS Particle Accelerator Device Farm Geohazard Monitoring Meteorology

9. www.gridcc.org (EU FP6 Contract 511382) Accessing GRID Enabled Instrumentation: Requirements 1. Interactive access to allow the control and the monitor of the instrumentation. Uniform access to the physical devices. 2. Fast Data Publishing to allow the publishing acquired by the instrumentation 3. Information (logs, errors, etc.) Publishing to track the behaviour of the instrumentation and possibly to correct the malfunctioning 4. GRID integration. The acquired data should be visible by the GRID protocols both to : 1. Move them to large GRID data storage (Storage Element) 2. Process them from the GRID computing power (Computing Element)

10. www.gridcc.org (EU FP6 Contract 511382) A new 4 channels element to fit the requirements: The Instrument Element (IE) Instrumentation Contorl GRID Access Instruments Access Data/Info Pubblishing Instrument Element Web Service SRM/GRIDFTP Custom (JMS,..... ) Custom I’m here Discovery QoS Advance Reservation

11. www.gridcc.org (EU FP6 Contract 511382) The IE permits  SOA accessible operations to control and monitor the instruments (via VIGS), such as: execute a command get / set parameters  different data outputs: data mover to/from a grid Storage Element (via SRM) high bandwidth channel for data publishing (via IMS). low bandwidth channel for logs, states etc. (via IMS). Instrument Element: Versatile I/O multichannel VIGS Commands Status Data Subscribers Grid Interaction SRM VCR Storage Element (SE) Storage Element (SE) Storage Element (SE) Logs, Errors, States, Monitors Instruments IMS Parameters ES VIGSVirtual Instrument Grid Service IMSInformation and Monitor Service SRMStorage Resource Management IE

12. www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Proxy Control Manager Data Collector Real Instruments Data Flow Control Flow State Flow Error Flow Monitor Flow The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Input Manager Event Processor FSM Engine Resource Proxy Control Manager IMS Instrument Element Architecture Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Instrument Protocols

13. www.gridcc.org (EU FP6 Contract 511382) Configuration Examples Top IM Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Top IM Top IM Resource Service Instruments Pool A BC Configuration A, B, C can run concurrently sharing services like RS, PS, IMS

14. www.gridcc.org (EU FP6 Contract 511382) Instrument Element Framework Characteristics The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled. Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments. the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM). relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP). Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS). IE Resource Service IMS Local Problem Solver Instrument Manager Access Control Manager Data Mover Instrument Manager VIGS Instrument IMS Proxy Control Manager Data Collector Input Manager Event Processor FSM Engine Resource Proxy Control Manager

15. www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Instrument Manager Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Instrument Manager IMS Proxy Control Manager Data Collector

16. www.gridcc.org (EU FP6 Contract 511382) 16 Instrument Manager IM is composed by 3 main components: - Control Manager: - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability - FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources. - Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS IMS Proxy Data Collector Instrument Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager Instruments Data Flow State Flow Error Flow Monitor Flow Control Flow Customizable Plug-in modules to interface to the instruments

17. www.gridcc.org (EU FP6 Contract 511382) Status data collection data publishing on RMM-JMS data on file in progress data on DB to be started sync commands (enable/disable done trough the “set param” of the IE façade (VIGS) Data Collector Resource Service IMS Local Problem Solver Instrument Manager Instrument Element IMS Proxy Control Manager Data Collector Access Control Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager Data Mover VIGS IE File System RMM-JMS DB persistency enable/disable data stream Instrument data stream Data Subscribers (Monitor, data analysis, etc.)

18. www.gridcc.org (EU FP6 Contract 511382) VIGS IMS Proxy Control Manager Data Collector IMS Resource Service Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Resource Service

19. www.gridcc.org (EU FP6 Contract 511382) Resource Service Architecture The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date. Discovery Manager Subscribe Manager Partition&Lock Manager Configuration Manager Available Resources Partition Definitions Configuration Definitions RS Data Bases Partition/Configuration retrieve methods Partition and Lock setting methods Configuration setting methods Discovery methods

20. www.gridcc.org (EU FP6 Contract 511382) VIGS Control Manager Data Collector IMS Information and Monitor Service Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Publisher Access Control Manager SRM/SE GridFTP SE IMS Proxy

21. www.gridcc.org (EU FP6 Contract 511382) 21 Information and Monitor System (IMS) The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs. The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. IMS Proxy IMS Proxy IMS Proxy PERSISTENCY Publish / Subscrib SUBSCRIBERS

22. www.gridcc.org (EU FP6 Contract 511382) IMS Fast Data Publishing: RMM-JMS IE IE Data ProducerData Subscribers Same data are sent to several subscribers. Multicast protocols can have a benefic impact on performances JMS provides a standard set of APIs for the communication Many commercial and academic JMS implementations both in C/C++ and Java (NaradaBrokering, Sun, IBM) GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication

23. www.gridcc.org (EU FP6 Contract 511382) Status: installed and running SRM 2.2 STORM implemetation (INFN) Bestman implementation (Berkeley)) XFS IE file system Embedded SRM compliant Grid Storage Element (SE) Resource Service IMS Local Problem Solver Instrument Manager Instrument Element Access Control Manager SE Backend VIGS IE File System SRM GRID accessibile data

24. www.gridcc.org (EU FP6 Contract 511382) Querring Client Querring Client Discovery Instruments (I) The discovery of the instruments or of the IEs is an issue when the number of elements is high We can have two cases:  Quasi static cases The number of IEs are well defined and the single IE is quite complex with a good hardware support In this case a register based discovery mechanism can be used. GRIDCC tesbed is using the LCG BDII ( Berkely Database Information Index ) based on LDAP The information collected in the BDII follow a GLUE schema and can be used for match making querries IE BDII Local BDII Querring Client

25. www.gridcc.org (EU FP6 Contract 511382) Discovering Instruments (II)  Dynamic cases The number of IEs can change very quickly, they are very simple devices, often with poor hardware support The discovery is just use to know which are the online IEs A new approach has been developed based on Peer to Peer (P2P) protocols See demo on this workshop IE M1 M2 Querring Client Querring Client Querring Client

26. www.gridcc.org (EU FP6 Contract 511382) Instrument Elements at Work

27. www.gridcc.org (EU FP6 Contract 511382) The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory. CMS Magnet Test and Cosmic Challenge (MTCC) I and II, a milestone in the CMS construction, positively carried out. Pilot applications: CMS - I Top IE CSC IE Tracker IE HCAL IE DAQ IE RPC IE Trigger IE DT IE ECAL IE DAQ IM FBRBFF xdaq DAQ RS DAQ IMS User Interface CMS Control Structure CMS Detector

28. www.gridcc.org (EU FP6 Contract 511382)

29. CMS MTCC phase I and phase II carried out in 2006 Scale MTCC versus CMS FEDs: 20 out of 600 3% EVB RUs: 6 out of 6001% Filter Farm: 14 out of 20000.3% Trigger rate: 100 Hz out of 100 kHz 0.1% Event size: 200 kB out of 1 MB20% IEs15 out of 150 10 % Scale MTCC versus CMS FEDs: 20 out of 600 3% EVB RUs: 6 out of 6001% Filter Farm: 14 out of 20000.3% Trigger rate: 100 Hz out of 100 kHz 0.1% Event size: 200 kB out of 1 MB20% IEs15 out of 150 10 %

30. www.gridcc.org (EU FP6 Contract 511382) Pilot applications: Power grid Instrument Manager Instrument Element... Virtual Control Room Virtual Control Room Gas Solar Panel Power Grid V.O. GRIDCC deployed to monitor: A 50kW generator A 1 kw Photo-Voltaic array Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array See DEMO on this workshop

31. www.gridcc.org (EU FP6 Contract 511382) Pilot applications: Remote Operation of an Accelerator Elettra Synchrotron See DEMO on this workshop

32. www.gridcc.org (EU FP6 Contract 511382) Meteorology (Ensemble Limited Area Forecasting)  Weather forecasting system to detect hazardous weather Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories  The Device Farm consists of a pool of Measurement Instruments for Telecommunication Experiments Geo-hazards: Remote Operation of Geophysical Monitoring Network  The monitoring net will be characterized by different levels of activity: stand-by, pre-alert, alert, plus a control modality  An event worth to be monitored is for example the evolution of a “landslip” GRIDCC: other applications

33. www.gridcc.org (EU FP6 Contract 511382) Instrument Element: Scalable on embedded systems FPGA PPC 405 Xilink Virtex IV Custom Board Custom Electronics 1 Gbps Ethernet Web Service GridCC IM JavaVM Linux USER INTERFACE Custom Logic Montavista www.montavista.com JamVM http://jamvm.sourceforge.net Standalone Axis IE Instrument Manager

34. www.gridcc.org (EU FP6 Contract 511382) IE technologies Web Service compliant (WS-I) Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project  Sun  Narada Brokering  RMM - JMS (GridCC IBM) MySQL and Oracle are used as Data Base for the RS SRM interface version 2.2 used

35. www.gridcc.org (EU FP6 Contract 511382) Conclusion - I The GridCC project is integrating instruments into the “classic” computational/storage Grids. One of the novel concepts introduced by GridCC are:  The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid  hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times  Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. The IE is highly customizable and can be adapted in different environments.

36. www.gridcc.org (EU FP6 Contract 511382) Conclusion - II 3 pilot applications deployed, one of them in a real production environment The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation Several heterogeneous pilot applications are deploying and running the IE middleware We support and encourage the adoption of our middleware in other projects/experiments.