Wojciech Sliwinski Beams Department, Controls Group CERN

Outline  Defining Middleware  Types of Middleware  CERN Acc. Controls Messaging Infrastructure  Middleware Review for CERN Acc. Controls  Conclusions 2Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Outline  Defining Middleware  Types of Middleware  CERN Acc. Controls Messaging Infrastructure  Middleware Review for CERN Acc. Controls  Conclusions 3Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Why Middleware?  How do we build/connect applications over a network?  How do we facilitate Distributed Computing (2-tier, …, n-tier)?  How to support the heterogeneous environments? 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...4

What is Middleware?  Software, which allows an application to interoperate with other software, without requiring the user to understand and to code the low-level operations required to achieve interoperability Software layer between OS and the applications Hides complexity & heterogenity of distributed system Handles issues related to OS, Network Protocols & hardware platforms 5 Application Operating System1 Communication system MW Application Operating System2 MW Application Operating SystemN MW Standard API Specific API Standard API Specific API 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Introduction to Middleware  Middleware provides support for: Naming, Location, Service discovery, Replication Protocol handling, Communication faults, QoS Synchronisation, Concurrency, Failover, Scalability Access control, Authentication  Middleware dimensions: Request-Reply vs.Asynchronous Messaging Language-specific vs. Language-independent Proprietary vs. Standards-based Small-scale vs. Large-scale Tightly-coupled vs. Loosely-coupled components Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...6 Middleware is everywhere. You might think you can leave it but it just finds you again :-) 26th October 2011

Outline  Defining Middleware  Types of Middleware  CERN Acc. Controls Messaging Infrastructure  Middleware Review for CERN Acc. Controls  Conclusions 7Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Middleware – large domain … which to choose? 8Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 direct p2p broker req/reppub/sub rpc/orbmessage LAN WAN/ Internet NetworkEmbedded

Types of Middleware  Remote Procedure Call (RPC)  Object-Oriented Middleware (OOM)  Message-Oriented Middleware (MOM)  Event-Based Middleware a.k.a Publish-Subscribe 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...9

Remote Procedure Call - RPC  Masks remote function calls as being local  Client-Server model with Request-Reply paradigm  Synchronous communication  Marshalling of function parameters and return value  Provides Location & Platform transparency  Technologies: DCE-RPC XML-RPC 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...10 Client App. IDL Stub Network Server App. IDL Skeleton Network Req/Reply

RPC - Disadvantages  Synchronous Request-Reply interaction Tight coupling between client and server Client may block for a long time if server loaded ○ Multi-threaded programming at client Slow/failed clients may delay servers when replying ○ Multi-threaded programming at servers  RPC paradigm is not object-oriented Invoke functions on servers as opposed to methods on objects 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...11

Object-Oriented Middleware - OOM  Client-Server model with Request-Reply paradigm  Synchronous communication  Marshalling of function parameters and return value  Provides Location & Platform transparency Objects can be local or remote Remote objects accesible via proxy objects  Technologies: OMG CORBA MS DCOM Java RMI SOAP 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...12 Client App. IDL Stub IIOP Client ORB Server App. IDL Skeleton IIOP Server ORB POA Adapter Req/Reply

Object-Oriented MW - Disadvantages  Synchronous Request-Reply interaction Tight coupling between client and server Client may block for a long time if server loaded ○ Multi-threaded programming at client Slow/failed clients may delay servers when replying ○ Multi-threaded programming at servers  Distributed garbage collection Releasing memory for unused remote objects Error-prone and difficult to debug  Static mappings and heavy-weight development Difficult to extend an operational system 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...13

Message-Oriented Middleware - MOM  Data exchange via Messages & Message Queues  Asynchronous communication (Point-to-point  One-to-One)  Client and Server are loosely coupled  Message Servers may do filtering, transforming, logging…  Technologies: IBM WebSphereMQ, TIBCO Rendez-vous, JMS, SOAP 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...14 Client App. local message queues Server App. local message queues message queues Network Message Servers

Message-Oriented MW - Disadvantages  Poor programming abstraction (but has evolved) Rather low-level Request-Reply more difficult to achieve, but can be done  Message formats originally unknown to middleware No type checking (solved in JMS)  Queue abstraction only gives One-to-One communication Limits scalability (solved in JMS with Publish-Subscribe) 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...15

What we lack, so far  General interaction patterns We have Request-Reply and One-to-One Publish-Subscribe? One-to-Many? Many-to-Many? Dynamic joining and leaving?  Location transparency Anonymity of communicating entities 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...16

Event-Based Middleware a.k.a Publish-Subscribe  Publishers (advertise and) publish Events (Messages)  Subscribers express interest in Events with Subscriptions  Event Service notifies interested Subscribers of published Events  Events can have arbitrary content (typed)  Technologies (Brokers): JMS (ActiveMQ, RabbitMQ), AMQP (…) 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...17 Publisher1 Event Service (Broker) PublisherN … Publisher2 topics publish Subscriber1 SubscriberN … Subscriber2 subscribe notify subscribe notify subscribe notify

Properties of Publish-Subscribe  Asynchronous communication Publishers and Subscribers are loosely coupled  Many-to-Many interaction between Pubs and Subs Scalable scheme for large-scale systems Publishers do not need to know Subscribers, and vice-versa Dynamic join and leave of Pubs & Subs (thanks to Brokers)  Topic based Publish-Subscribe very expressive Filtered information delivered only to interested parties 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...18

Outline  Defining Middleware  Types of Middleware  CERN Acc. Controls Messaging Infrastructure  CERN Acc. Controls Middleware Review  Conclusions 19Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

CERN Controls Middleware – in all SW tiers… 20 CMW client (C++/Java) GUIs, LabView, RADE CMW client (Java servers) Logging, LSA, InCA, SIS CMW server (C++/Java) Proxy, DIP, AlarmMon, AQ CMW server (C++) FESA, FGC, GM CMW server (C++) PVSS (Cryo, Vacuum) JMS client (Java) GUIs JMS client (Java) Servers: Logging, InCA, SIS Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

JMS for Accelerator Controls  Java Messaging Service  Purpose Reliable transport of data between Java Processes Controls ○ Alarm System ○ Data Rendering Services ○ (Control) GUIs ○ Logging Services ○ Beam Security System(s) No JMS - No Beam ! 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS for Accelerator Controls Devices / Hardware JMS C/C++ Process JAVA Process JAPC CMW JMS Front End GUI Process JAPC CMW JMS CMW ~50 Processes~100 Processes ~46K Subscriptions 280 Connections machines 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Examples of JMS ”aware” systems 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

History of JMS in Accelerator Controls  Usage of SonicMQ JMS for LHC Alarm System since 2001  ActiveMQ introduced in 2005  Other project started to use JMS service  Today vital part of Beam Operations 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Technical Aspects  Broker Configuration Non-persistent messages on topics No subscription recovery policy No durable subscribers No exclusive consumers No replyTo mechanism Broker is persistent  Clients Pure JAVA - Wireformat Protocol 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

General JMS Deployment Situation  SonicMQ : 12 Brokers TIM, LASER, DIAMON Always as two node cluster  ActiveMQ : 16 Brokers One main default Broker Service Otherwise project dedicated Brokers  All Brokers on Technical Network 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Overall Broker Service Jan th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS Service Data Handling  Central JMS Service (276 days in 2010) 58 Million msg / day 120 TB IN 650 TB OUT  Additional handling of Heavy Connection and Subscription handling Bursts loads  Connected subscribers grew from 2K to 12K in 6 months => 440 GB /day => 2.5 TB /day YES, ActiveMQ can! 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Data Handling during LHC startup 2010 Total Data Out Increase in data demand 2.57 TB/day 1TB/day LHC startup 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS Service Availability 2010* * November th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Monitoring and Troubleshooting Tools  DIAMON & Lemon  JConsole / JVisualVM  Dump of broker information to SQLite DB  Traffic Monitoring Tool  Various Message Listeners 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Easy Identification of Bad Producers Fixed Problematic Producer 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS Service – Lessons Learnt  ActiveMQ was a good choice Running reliably Performs well Provides many metrics for diagnostic and monitoring Good integration in BE-CO monitoring tools via JMX  Projects are not always aware of sending so much data Service Level Agreements with projects required More analyzing tools for end users required 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS Service – Lessons Learnt  Monitoring is a vital part of operations Message reception speed test Memory Percent usage JMX attributes (#topics, #consumers, #subscriptions)  Broker Configuration is not always obvious 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

JMS Service – Lessons Learnt  Clustering Good for failover and load balancing Additional complexity Updating clients for new cluster members ○ Automatic rebalancing of clients  Memory is important Tuning of Java GC options 26th October Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...

Outline  Defining Middleware  Types of Middleware  CERN Acc. Controls Messaging Infrastructure  CERN Acc. Controls Middleware Review  Conclusions 36Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Hardware Devices GUI operator consoles Front ends “virtual devices” CERN Controls Middleware – CMW 37Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Controls Middleware Embedded Middleware

CMW Mandate & Scope  Standard set of MW solutions & libraries  Centrally managed services  Track & optimize runtime parameters  Scope: CERN Accelerator Complex Operational 24*7*365 Must be Reliable & High Quality ~80’000 HW devices, ~4’000 servers ~1700 machines, ~2’000’000 IO points  Huge infrastructure! 38Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

CMW Data Flow 39Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Linux/Windows CERN Controls Middleware 40Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 C++/Java process Controls Middleware API TCP/IP CORBA Controls Middleware RT Linux/LynxOS C++ process Controls Middleware API CORBA Problems: - C++ and Java implementations differ  - Heavy in memory usage  - Complex error prone API  - No direct support for pub/sub  - Blocking issues (JacORB)  - Shrinking community  - Lack of new releases and bug fixes  Long LHC shutdown at the end of 2012 A unique possibility! Instead of patching let’s provide a new one!

CERN Middleware Requirements 41Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Desirable Mandatory Fundamental Lightweight Active community Friendly API, documentation Request/reply & pub/sub patterns Open source, redistributable license QoS Stability, Maturity & Longevity Performance & Scalability Asynchronous C++/Java Linux/Windows Over TCP/IP LAN

How did we evaluate –> our criteria 42Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 CRITERIA QoS resources, binary size, memory performance Communications patterns API, look & feel, documentation Community, maturity Appearance Creators specification documentation Users forums bug reports Internet Simple usage Download licensing Compile Linux LynxOS Run examples Testing Communication patterns Performance QoS Exceptional situations

Surprising how many great new products! 43Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 BSD sockets RPC CORBA ??? ARPANET TODAY 1974 TCP 1998 IP v UDP 1981 IP v

Evaluated middleware products 44Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Ice Thrift omniORB YAMI OpenSpliceDDS OpenAMQ CoreDX RTI DDS ZeroMQ QPid MQtt RSMB JacORB Mosquito All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution. RabbitMQ

CORBA (omniORB, JacORB) 45Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Object-oriented communication platform standardized by OMG. It is over 20 years old, and the standard is well established but...  Complex, hard to learn and use   Differences between implementations   Big memory footprint   Shrinking community   Seldom updates and bug fixes  Complex Heavy Shrinking community Dying out

Object-oriented middleware by ZeroC, people who left CORBA to fix its problems. Conceptually similar to CORBA but...  Better design and implementation  Modern easier to use API and IDL mapping  Concise C++ and Java implementation  Support for versioning  Active community Ice 46Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Well designed Active community

Thrift 47Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Middleware by Facebook (moved to Apache)  Lightweight, free of unnecessary dependencies  Modern, clean API  Active community  Ongoing development, still incomplete   No documentation, empty tutorial  Poor documentation Work still in progress

Message-oriented middleware developed by one of our former colleagues. Used by some CERN Controls services.  Lightweight, free of unnecessary dependencies  Well designed, modern API  Asynchronous req/rep and pub/sub patterns  Dynamic type specification (paid in performance)  Small community  YAMI4 48Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Lightweight Well designed Small community

Middleware where messages are distributed by a broker. A few independent implementations of the broker and clients.  Big community  Designed to support pub/sub  Protocol standard issues   Broker – single point of failure   Broker – additional hop, slower communication   Broker – additional, non compliant monitoring tools  AMQP – OpenAMQ, Qpid, RabbitMQ 49Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Slower for req/rep Broker issues Active community

Data-oriented middleware with p2p communication. Standard defined by OMG, with a few compatible implementations.  Big industrial and military community  Possible compatibility with CORBA  Designed to support pub/sub  Req/rep possible but needs 2 channels   Steep learning curve   Complex API  DDS – RTI, CoreDX, OpenSplice 50Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Heavy Complex Big industry presence

Message-oriented middleware by iMatix (they dropped OpenAMQ.) API resembles BSD sockets but it is so much more.  Simple yet powerful API  Support for in-proc, inter-proc, TCP, PGM  Lightweight, free of unnecessary dependencies  Many communication patterns  May be used as a concurrency framework  Active community ZeroMQ 51Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Lightweight Well designed Active community

Performance requirements reviewed 52Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Instead of estimating  We asked our users what they will need  Gathered statistics from the current system  Most demanding users identified  Definition of a few performance tests

C++ server with client in: Performance tests, reliable Request-Reply 53Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October msg/sec Payload = 4B 5 msg/sec Payload = 10MB Many small messages A few big messages

Performance/Scalability tests, reliable Pub-Sub 54Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October msg x 8 B 10 clients, <50ms 30 msg x 8 B 10 clients, <20ms ThroughputLatency

Products comparison (according to the criteria) 55Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011 Sync, async & msg patterns QoS Dependencies & memory f-p Performance Look & feel, API, docs Community & maturity Score ZeroMQ 6 Ice  5 YAMI4   4 RTI  3 Qpid  3 CORBA  2 Thrift  2

Middleware Review Summary  Several good middleware solutions available.  The choice is dictated by the most critical requirements for any given application. Not easy  QoS policies and performance matter, but also ease of use, community, … 56  Concerning CERN Controls Middleware…  Prototype with the most promising candidates:  Deploy the new middleware before the long accelerator shutdown at the end of 2012 ZeroMQ Ice YAMI Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...26th October 2011

Conclusions  Middleware is an important abstraction for building the distributed systems  Synchronous vs. asynchronous communication  One-to-one vs. many-to-many communication  Better performance with direct communication  Better scalability and less coupling with Brokers  System requirements  Messaging patterns to apply 26th October 2011Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems...57