A. Dworak BE-CO-IN, CERN

Agenda 228th June 2012  Sum up of the previous report  Middleware prototyping  Transport  Serialization  Design concepts  Keep proven solutions  Modularity  Data Model  Transport  Client  Server  Work progress Andrzej Dworak, The new CMW-RDA

Previous reports 328th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC Andrzej Dworak, The new CMW-RDA

Previous reports 428th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC  Requirements gathering, user interviews, internal reviews  Shortcomings of the system, missing functionality Andrzej Dworak, The new CMW-RDA

Previous reports 528th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC CORBA:  C++ and Java implementations differ  Shrinking community  Lack of new releases and bug fixes  Expensive on resources  Complex error prone API  Problems with asynchronous and non-blocking communication  No support for pub/sub  Missing support for versioning or flexible interfaces (hard to upgrade) Current design:  Monolithic (hard to maintain, extend and test)  Not scalable (poor parallelization)  Not optimal design makes it unnecessarily heavy and slow  Lack of QoS, message priorities  Missing server-side management of client connections  After all the years and fixes it is BIG Major problems with CORBA and current code base using it: From users’ perspective  Service instability  Client may influence Server  Fails when overloaded  Does not scale  Impossible to optimize the usage  Unnecessarily limited (e.g. connection #), slow and heavy.  Needs workarounds (e.g. Proxy)  Unfriendly API  Lack of full diagnostics ===========================================================

Andrzej Dworak, The new CMW-RDA Previous reports 628th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC

Previous reports 728th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC Andrzej Dworak, The new CMW-RDA

Previous reports 828th June 2012  “Middleware Trends and Market Leaders 2011”  14/10/2011, ICALEPCS 2011  20/10/2011, BE-CO-IN TC Ice YAMI4 Andrzej Dworak, The new CMW-RDA

Middleware prototyping 928th June 2012  More tests  Scalability  Correctness  Error handling  Exceptional situations  Prototyping of the system parts Ice YAMI4 Andrzej Dworak, The new CMW-RDA USE

Serialization Prototyping 1028th June 2012  Necessary to transport data between different architectures  Currently self-describing serialization  Based on CORBA (on top of IDL)   Stay with self-describing serialization, chose better library  Used so far in RDA  No need to manage IDLs  Easier to upgrade or change the protocol  Need to encode type description   Slightly slower  Slightly bigger output  Review of +10 serialization libraries  Testing and prototyping with:  Google Protocol Buffers  MessagePack  Apache Avro  Compared with CORBA MessagePack USE Andrzej Dworak, The new CMW-RDA

More details at: 1128th June 2012  Middleware  General review and tests  Presentation of the review and test results talk.pdf?version=1&modificationDate= talk.pdf?version=1&modificationDate=  More details on the 1 st step also in A. Dworak “Middleware trends and market leaders 2011”  More details on the 2 nd step also in A. Dworak “The new CERN Controls Middleware”  Serialization  Presentation with review details and test results describing+Serialization.pdf?version=1&modificationDate= describing+Serialization.pdf?version=1&modificationDate= Andrzej Dworak, The new CMW-RDA

Design concepts – proven concepts 1228th June 2012 Keep the proven concepts:  Device/property model  Communication through get, set and subscribe/publish calls  Properties represented as dynamic Data objects  Self-describing serialization  Authorize and authenticate user access Andrzej Dworak, The new CMW-RDA

Design concepts – modularity 1328th June 2012  Less coupling between system parts  Easier to test and maintain  dependency injection  mocking  Will allow to exchange parts of the system (e.g. transport = ZeroMQ or serialization)  Will help during the migration process (old RDA to new RDA) Andrzej Dworak, The new CMW-RDA

Design concepts – transport 1428th June 2012  Transport executed asynchronously and independently of the user code execution  Separate channels for different types of communication  Administrative  Get and Set  Subscriptions  Optimization  Internal and through user API  Intelligent buffering on the transport level  Immutable data allows better and safer parallelization  0-copy (when sending, for receiving not 0 but also optimized)  Extended diagnostics Andrzej Dworak, The new CMW-RDA

Design concepts – data model 1528th June 2012  Property as a set of named fields with dynamic types  Current type set extended to support:  Fixed length integer types (C++ int64_t, int32_t, int16_t, int8_t)  Multidimensional arrays  Nested Data objects  Management of data ownership  Allows 0-copy for arrays and nested data objects  Context class with direct support for  Filters  RBAC tokens Andrzej Dworak, The new CMW-RDA

Design concepts – client 1628th June 2012  Can get, set and monitor values of properties  Uses predefined policies which will handle communication events and errors  Connect/reconnect policy  Subscribe policy  Underlying communication is always non-blocking and asynchronous  For convenience API allows to perform the calls as  Asynchronous  Synchronous  Buffered asynchronous  Buffered synchronous  RBAC role may guarantee a client a higher priority  Client may be managed on-line (similarly to servers and cmw-admin) Andrzej Dworak, The new CMW-RDA

Design concepts – server 1728th June 2012  Asynchronous execution of implemented get, set and (un)subscribe handlers  Non blocking data publishing  May set delivery policy (normal/background)  Message priority and QoS  Extended management (also on-line via CMW Admin)  Turn on/off get, set, publishing  (Un)block client / low-priority clients  Disconnect client / low-priority clients  Send commands to clients  Ping/hello  Disconnect request  “Failover to” request Andrzej Dworak, The new CMW-RDA