G. Maron, EGEE06 Conference,CERN, September, 2006 Instruments and Sensors on the Grid The GridCC Project Gaetano Maron Istituto Nazionale di Fisica Nucleare Laboratori Nazionali di Legnaro, Legnaro Italy

G. Maron, EGEE06 Conference,CERN, September, 2006 Overview Bringing Instrument into the Grid The GridCC Project The Instrument Element The GridCC Test-bed: Pilot applications Conclusions

G. Maron, EGEE06 Conference,CERN, September, 2006 Bringing Instruments into the Grid

G. Maron, EGEE06 Conference,CERN, September, 2006 The Grid Technologies to extend the limit of a single computer (center) User Interface Computing Element Computing Element Computing Element Storage Element

G. Maron, EGEE06 Conference,CERN, September, 2006 Extending the Grid Concepts Satellite views to monitor the volcano Virtual Control and Monitor Room To model calculations and disaster predictions Terrestrial probes to monitor The volcano activities Instrument Element

G. Maron, EGEE06 Conference,CERN, September, 2006 The GridCC Project Instruments Grid Computational Grid Data for Model Calculations Predictions Virtual Control and Monitor Room

G. Maron, EGEE06 Conference,CERN, September, 2006 The Gridcc Project

G. Maron, EGEE06 Conference,CERN, September, 2006 General on the GridCC Project Participant name Country Istituto Nazionale di Fisica NucleareItaly Institute Of Accelerating Systems and Applications Greece Brunel UniversityUK Consorzio Interuniversitario per Telecomunicazioni Italy Sincrotrone Trieste S.C.P.AItaly IBM (Haifa Research Lab)Israel Imperial College of Science, Technology & Medicine UK Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche Italy Universita degli Studi di UdineItaly Greek Research and Technology Network S.A. Greece It is a 3 years project. Started the 1st September 04 Funded by EU in the Frame Program 6 (contract ) 10 Partners from 3 EU Countries + (Israel) About 40 people engagged

G. Maron, EGEE06 Conference,CERN, September, 2006 GridCC Application Fields Experimental Sciences –Take control of a experiment from a distance (remote operation and control, data taking and data analysis): High Energy, Nuclear and Solid State Physics Electronic Microscopes Telescopes Monitoring and analysis of the territory (e.g. disaster analysis) –Meteorology –Geophysics Bio-medics –Integration of remote operation, data taking, data analysis and data storage of sophisticated instruments like: Mammography Pet, TAC, NMR etc. Industrial Applications –widely distributed controls Electrical power grid Public transportation ……

G. Maron, EGEE06 Conference,CERN, September, 2006 The GridCC Architecture Web Service Interface Execution Service WfMS WMS AS Instrument elements (IE) Storage Element (SE) Instrument elements (IE) Storage Element (SE) Instrument Element (IE) Storage Element (SE) Compute element (CE) Compute element (CE) Compute element (CE) Storage Element (SE) Storage Element (SE) Storage Element (SE) Global Problem Solver Information and Monitoring Services (IMS) Information System (IS) Security Services Virtual Control Room Virtual Control Room Collaborative Services (CS) WMS Work Management System WfMS Work Flow Mng System AS Agreement Service AS allows to negotiate advance Reservation on the IE, SE and CE

G. Maron, EGEE06 Conference,CERN, September, 2006 The Instrument Element

G. Maron, EGEE06 Conference,CERN, September, 2006 IE Requirements Web Services Instrument Element Any Protocol or physical connection Sensor Network Instrument Grid Computing Element Storage Element Computing Element Instrument Element 1: Provide a uniform access to the physical device 2: Allow a standard grid access to the instruments. (Low bandwidth) 3: Allows the data acquired from the Instruments to be published to subscribers (high bandwidth) 4: Allows standard access to the other Grid Element 5: Provide hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times

G. Maron, EGEE06 Conference,CERN, September, 2006 Device Virtualization Model Instrument Parameters Attributes Control Model XML Based Language 1.Parameters hold configuration information 2.Attributes hold instrument variables 3.Control Model hold actions 4.XML Based Language to allow the device to describe itself Parameters: Maximum Voltage, Minimum voltage Attributes: measured Voltage Commands: Perform a measure Voltmeter

G. Maron, EGEE06 Conference,CERN, September, 2006 Instrument Element interaction paths VIGS Commands Status Data Subscribers Grid Interaction SE 1)A Virtual Instrument Grid Service (VIGS) interface has been defined. The VIGS provides a Web Service acccess to the instrumentation 2)The Data Publishing (DP) channel disseminates the acquired information to the data subscribers 3) A SRM/SE interface is provided to allow file movement with other Grid Element, like SE, CE and other IEs. 4)Information messages (logs, errors, etc.) are published from the IE (via IMS) Virtual Control Room Execution Service WfMS WMS AS Compute element (CE) Compute element (CE) Computing Element (CE) Storage Element (SE) Storage Element (SE) Storage Element (SE) IE Logs, Errors, States, Monitors IMS DP VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SE Storage Element DP Data Publishing

G. Maron, EGEE06 Conference,CERN, September, 2006 Instrument Element Interconnections IE Control Panel Computing Element (CE) Fast Data Display Fast Data Publishing Information and Monitor System Log Display Control and Status Existing Grid Elements Log Persistency Data Consumer VCR

G. Maron, EGEE06 Conference,CERN, September, 2006 Instrument Element Architecture VIGS Resource Service IMS Problem Solver Instrument Manager Instrument Element Data Mover IMS Proxy Control Manager Data Collector Real Instruments Data Flow Control Flow State Flow Error Flow Monitor Flow The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Access Control Manager Input Manager Event Processor FSM Engine Resource Proxy Control Manager SRM/SE GridFTP SE DP IMS

G. Maron, EGEE06 Conference,CERN, September, 2006 IE middleware technologies Tomcat + Axis (and Java) are the main technologies of the IE All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project –Sun –Narada Brokering –RMM - JMS (IBM), see below for more details MySQL and Oracle are used as Data Base for the RS WfMS engine based on BPEL EGEE gLIte is the reference framework for –WMS –CE (gLite CREAM is used where possible) –SE + SRM A modified version of STORM is used as SRM for the IE

G. Maron, EGEE06 Conference,CERN, September, 2006 Instrument Element Implementations Resource Service Inf & Mon Service Problem Solver Instrument Manager Instrument Element Data Mover Access Control Manager The IE components are typically implemented into a fully equipped Machines (e.g. dual core cpus, large memory, large disks, etc). This is true for RS, IMS and PS. For IM (and DM) there are 2 possibilities, according to the application type: IM implemented in a fully equipped machine IM embedded into the instrument that should be controlled IM RS IMS IM

G. Maron, EGEE06 Conference,CERN, September, 2006 Instrument Element on a Chip Grid on a Chip FPGA PPC 405 Xilink Virtex IV Custom Board Custom Electronics 1 Gbps Ethernet Web Service GridCC IM JavaVM Linux USER INTERFACE VIGS DP SE IMS

G. Maron, EGEE06 Conference,CERN, September, 2006 Fast Data Publishing Typical Use Case: 1 to N IE Data Subscribers (Monitor, local storage, etc.) IE Data Producer Same data are sent to several subscribers. Multicast protocols can have a benefic impact In term of performance J MS Provide a standard set of API that standardize this communication system Many Commercial and academic implementation of this API exist in both C/C++ and Java (NaradaBrokering, Sun, IBM, SonicMQ etc etc ) No one has a good Multicast implementation Gridcc (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant RMM-JMS works within a LAN, but an efficient bridge technology has been developed to allow inter-LAN multicast communication

G. Maron, EGEE06 Conference,CERN, September, 2006 RMM-JMS Performance one to many case 60 MByte/s 500 kmsg/s IE

G. Maron, EGEE06 Conference,CERN, September, 2006 RMM-JMS Performance round trip time (RTT, Latency) Two machines with a single publisher and a single subscriber on each one Average round trip time computed over 1000 samples

G. Maron, EGEE06 Conference,CERN, September, 2006 GridCC Pilot Applications

G. Maron, EGEE06 Conference,CERN, September, 2006 Main IE Pilot Applications: Power Grid Instrument Manager Instrument Element... Virtual Control Room Virtual Control Room Gas Solar Power Grid V.O

G. Maron, EGEE06 Conference,CERN, September, 2006 Main GridCC Pilot Applications: Control and Monitor of high energy experiments In collaboration with the CERN CMS/TriDAS group

G. Maron, EGEE06 Conference,CERN, September, 2006 The CMS Data Acquisition O(10 4 ) distributed Objects to – control – configure – monitor On-line diagnostics and problem solving capability Highly interactive system (human reaction time - fraction of second) World Wide distributed monitor and control electronics channels 40 MHz 100 Hz

G. Maron, EGEE06 Conference,CERN, September, 2006 CMS Prototype

G. Maron, EGEE06 Conference,CERN, September, 2006 CMS Prototype: IEs at work Det 1 DAQ TTS FedBuilderRuBuilder FilterFarm Trigger TOP GTPe DAQ Detector GridCC middleware used for CMS MTCC (Magnet Test and Cosmic Challenge) - 11 Instrument Elements with a hierarchical topology - Instruments are in these case Linux hosts where the cms on-line software is running - More than 100 controlled hosts - First phase of data taking successfully ended - Second phase starting in few weeks CMS Instrument Elements DAQ IE Instrument Managers

G. Maron, EGEE06 Conference,CERN, September, 2006 Main GridCC Pilot Applications: Remote Operation of an Accelerator Elettra Synchrotron Demo available in the Demo session

G. Maron, EGEE06 Conference,CERN, September, 2006 The other GridCC pilot applications Meteorology (Ensemble Limited Area Forecasting) Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories Geo-hazards: Remote Operation of Geophysical Monitoring Network

G. Maron, EGEE06 Conference,CERN, September, 2006 Conclusion (I) The GridCC project is integrating instruments into the “classic” computational/storage Grids. Novel concepts introduced by GridCC are: –The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid –The Virtual Control Room, providing an highly interactive environment with IEs/CEs/SEs. VCR even provides cooperative tools to allow the cooperation (logbook, chat, videoconf., etc.) between remote users –hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times –Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. The IE is highly customizable and can be adapted in different environments.

G. Maron, EGEE06 Conference,CERN, September, 2006 Conclusions (II) The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation Several heterogeneous pilot applications are deploying and running the IE middleware We support and encourage the adoption of our middleware in other projects/experiments. Direct help can be provided since now, tutorial will be available by the end of 2006

G. Maron, EGEE06 Conference,CERN, September, 2006 Question? Thx for your time Acknowledgement: The GridCC project is supported under EU FP6 contract More information: On-line Demo at: