Diirt status data integration in real time Gabriele Carcassi

diirt: data integration in real-time Connection/data processing layer for general purpose Java clients – Includes pvmanager/graphene/… Main two clients: – CS-Studio (Java based Eclipse RCP) – WebPODS (Web-socket + JSON “gateway”)

Request/response Publish/subscribe pvmanager Ch Access Processing Visualization vTypes Data Definition Aggregation formula graphene Data integration in real time (Diirt) architecture BOYWeb-pods (aka web gateway) DB (jdbc) Data serialization/conversion csv datasources supportservices support list CS-Studio (general purpose clients)Pods (Protocol Oriented Distribution Service) PV AccessWeb PodsFileLocal…PV Access Cmd line (exec) Channel Finder … JSON Probe…

Funding situation Funding at BNL was drastically cut – Closed what I could – Cleanly dropped the rest I’ll report on each area – Current state – What we wanted to do but won’t get done

Overall project General improvements – All projects in a single repository Easier to move code where it belongs Easier to make releases (Maven lesson learnt: change version in feature branches) – All elements configurable through a single configuration directory structure DIIRT_HOME – Documentation review (Wiki for user documentation, javadocs for developer documentation, documented source code examples in the repository)

VTypes Review types with Java 8 in mind – Use javax.time – Support for unsigned integers – Reorganize type hierarchy to overcome some implementation and usage issues – Reorganize utility methods within the type hierarchy (instead of utility classes) Work was started, currently unfunded Frozen in org.diirt.vtype.next package

VTypes Extract VType conversions from JCA/PVA support – Allow to use same exact type conversion for those who do not need/want the whole stack Currently unfunded

Datasources (pvmanager) Final review to the datasource spec – Generalize type conversion for writes (so that all datasources behave the same) – Use Java 8 standard classes where possible – Generalize caches from sources of external data (i.e. better support for archive data, better link between services, …) – Finalized documentation and examples for writing datasources – Performance improvements – Streamline error handling (i.e. do more work in the framework, clients are simpler, error handling more homogeneous) Currently unfunded

Archiver integration Client side temporal cache that can be filled by different archiver sources and real-time updates Under development by ITER.

Services Final review to the services spec – Uses Java 8 standard classes for callback – Defines order for arguments and results – Allows both synch and asynch execution – Allows both synch and asynch implementation – Automatically matches/wraps execution with implementation (e.g. asynch execution on synch implementation) – Finalized documentation and examples No obvious functionality missing: this part can be considered done

public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); Map result = method.executeSync(arguments); System.out.println("Result: " + result); } Sync execution example Output Locating service method... Service method found: add(VNumber arg1, VNumber arg2): VNumber result Preparing arguments... Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]} Executing service... Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}

public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); method.executeAsync(arguments, result -> { System.out.println("Result: " + result); }, ex -> { ex.printStackTrace(); }); } Locating service method... Service method found: add(VNumber arg1, VNumber arg2): VNumber result Preparing arguments... Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]} Executing service... Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]} Async execution example Output

Services Supported services: – v4 PVAccess services – ChannelFinder – Database queries – Command-line execution Plenty of copy and paste examples

Formula Final review to the function spec and parser – Reorganized implementation, more consistent and reusable – Parser generates Abstract Syntax Tree (AST) AST can be manipulated and then converted to the actual function – Finalized documentation and examples Current functionality is done

Formula

Other enhancements – Writeable formulas – Passive scanning for formula (performance improvement) – User defined functions (e.g. site specific function to handle custom v4 PVStructure, naming conventions, …) Currently unfunded

Graphene Contributions from students – Improvements to intensity graph color map Import color map from Matlab Support for both relative and absolute value color maps (e.g. RED is at 10% of the range, or at value 25) – Time axis and plot Supports automatic time scale (e.g. sec -> min -> hours -> day -> week -> month -> year) Support for daylight savings time, different time zones – Preliminary JavaFX components based on the graphs Student work finished Other work needed to integrate new features

HSVRadian.XML: <!-- Varies the hue component of the HSV color model. The map is useful for displaying periodic functions, and expects the value to be in radians. --> JET.XML:

JavaFX line graph integration:

JavaFX There is a general desire to develop widgets and tools in JavaFX – Can be used within SWT/Eclipse/CS-Studio – More full featured than SWT – Can be developed/tested/used outside Eclipse – Allows testing contribution to diirt outside of CS- Studio Work was started, currently unfunded

JavaFX What is done – Almost all basic tools re-implemented and expanded in JavaFX – Determined best approach for JavaFX integration general purpose binding between channel updates and JavaFX properties

JavaFX Probe Browseable event history

JavaFX Service probe Autogenerates field based on service data Currently handles only scalar strings and numbers

JavaFX Formula Function Browser

JavaFX Service Browser

JavaFX What needs to be done – Set of general purpose JavaFX widgets that understand VTypes and channels/formula/services Can be used in standalone Java UI and in Eclipse/CS-Studio Can be used to create BOY widgets Ideally, use the same definitions for JavaFX/BOY and Web – Finalize Graphene JavaFX widgets – Finalize toolbox Work was started, currently unfunded

CS-Studio CS-Studio integration (what’s done) – Renaming packages – Jars are OSGi bundles (no extra build step) – Bridge from Java standard dependency injection (ServiceLoader) to OSGi implemented – Proof of concept for updating CS-Studio codebase to new version These items are done

CS-Studio CS-Studio integration (what’s not done) – Finish the integration – Testing – Replace old SWT application with JavaFX substitutes Items that were on my horizon – Integrate example BOY screens – BOY JavaFX widgets – Simplify error reporting for connection problems (e.g. collect pvmanager/caj/pva logs to report to developers) Work currently unfunded

WebPODS Web gateway based on WebSocket + JSON and pure HTML/CSS widgets Main benefits: – Uses web standards (any web client in any language) – Widgets can be configured through standard CSS – Data access outside of control network (possibly WAN) – Web pages do not need to live on the same server where the gateway is

WebPODS Specs available on the WIKI: – Protocol-Specification-v1 Protocol-Specification-v1 – serialization-for-vTypes-v1 serialization-for-vTypes-v1

Chrome extension: Simple Web Socket Client

WebPODS when installed

HTML Probe (Danielle Connolly, UMich)

Text monitor examples

LED examples

WebPODS Specify server location Adding text-monitor Adding led Specify color for “LOW” enum value

Using CS-Studio off site through Web Pods

Web Pods server configuration ~]# more.diirt/pods/web/mappings.xml Access security planned, not yet implemented Use wss (like https) for authentication Use username/role/unix group/host for authorization Be careful not to expose too much

Web Pods What’s done: – Version 1 specification done and implemented – Preliminary Javascript API – Finalized HTML/CSS compliant widgets (text-monitor, led, table, bubble-graph) – Being used in production at UMich for 6 months under light load. No downtime. – Client-side Java API – Client-side and server-side formulas What needs to be done: – More widgets – Client-side security (server side is implemented) – Load tests – Review Javascript API – Client-side reconnected and ping/pong What could be done: – Investigate binary protocols (for large data or pre-calculated graphs) – Make single widget spec for HTML and JavaFX widgets