1. IRMIS3 Status Gabriele Carcassi 15 Oct 2010

2. IRMIS3 Status  This talk is about: what is currently downloadable from the website what is ready for production what is actively supported

3. Project infrastructure Production server at BNL since January 2009 “alpha” - AuthZ and AuthN only designed Likely implemented by the end of the year Build and test server Automated tests IE or firefox on Windows/Linux/Mac supported SourceForge project (mailing list, bug tracking, wiki). Monthly release cycle.

4. IRMIS3 SERVICE ARCHITECTURE

5. IRMIS3 architecture IRMIS DB (MySQL) IRMIS DB (MySQL) Data service Java applets AJAX components AJAX components Scripts and CLI (perl, Python, …) Browser (Firefox, IE, …) XML protocol Web server (Glassfish) Data service enforces business rules (clients can be dumb) Data service allows db schema or business rule changes while in production Data service provides a good abstraction layer on which to integrate

6. Software stack XML protocol (REST style WS) Few database utilities: backup, consistency check, etc… Client Server Integration with external tools (i.e. physcs) 3 rd party Perl/Pyton scripts 3 rd party Java apps Jython scripts Web applications JavaScript bridge Applets and Widgets Java Client API Data Service layer Database layer

7. Getting data... XML returned REST style (data service) Id references De-normalized data

8. Writing data <transaction xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://xml.bnl.gov/schema/irmis' xsi:schemaLocation='http://xml.bnl.gov/schema/irmis irmis.xsd'> XML describing a transaction sent through a POST allows remote/disconnected work no lock “FOR UPDATE” idempotent actions (error or same result) different independent “commands”

9. Implementation MySQL JPA JAXB SAX Business logic Business logic Servlet Read Write Stream out XML as soon as possible to reduce delay JAXB to convert transaction request to Java objects JPA vs JDBC: simpler queries to maintain 6% performance loss

10. Performance  Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz - APS dataset Entire component data set streamed in about 5 seconds (31338 components, 63911 relationships) Most other queries under 100 ms (manufacturers 88ms, interfaces 95ms)  Roughly half of the time is spent in DB access and half in XML generation Streaming starts right after DB access  Only tuning done is on the database query (was “good enough”) Other areas: hardware/os tuning, xml generation improvements, profiling, caching, …

11. IRMIS3 CLIENT API

12. Client API  To access the service, XML and HTTP is all you need Available in all languages  Two libraries provided that allow to work with objects PHP API from MSU Java API from BNL  Only going to talk about the Java API

13. Java API: design principles  The Java API implement the XML protocol and provides objects that represent the data  All reads are cached: no need to keep caches in your code.  All reads are synchronized you can have reads on multiple threads on the same data and will not trigger multiple reads on the server (extremely useful for GUIs)

14. Java API design principles  The read access is object oriented Data is exposed through interfaces - Component, ComponentType, Manufacturer Some classes function as “finders” for the classes containing the data - Components.getById(123), FormFactors.getByName(“Rack”) You can follow references from one object to the other - component.getComponentType().getManufacturer()

15. Java API design principles  The write access is protocol oriented You create a transaction You call methods to add elements in the transaction - Manufacturers.createManufacturer(transaction, “ACME”) You save the transaction, which may fail The read access is refreshed only after the data is saved

16. Importing components from Excel (using Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful"

17. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Load excel file

18. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Connect to the service

19. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Loop over the rows

20. Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data")) transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Get data out Create components

21. Fluent API  Navigating through hierarchies can be tedious  On top of the “beans” API there is a fluent API (internal domain specific language) Search API Builders Change Set API

22. IRMIS3 USER INTERFACE

23. UI design  Web interface: uses applets and Web Start (both greatly improved with Java 6u10)  Uses Swing standard components plus open source (SwingX, Flamingo, JIDE, …)  UI elements can optionally be used in custom pages and integrated with javascript  UI available only for Components and Cabling

24. UI elements Component hierarchy tree widget (filtering, sorting, search, …) Component path widget

25. Component applications

26. Cabling applications

27. STATUS BY AREA

28. Components  Support for component and component types is production level Data model has been tested extensively through previous use at other sites Performance tests were done using APS datasets  Most of the work now is in UI development Most applications at their 3 rd iteration

29. Cabling  Support for cabling is still beta No considerable amount of data has been put into it No performance tests done (see lack of data)  UI at their second iteration  Need users!

30. Lattice  There is preliminary support for lattice Only through the API, no UI  A separate tree structure of “elements”, each of an “element type”. Each relationship has alignment information.  Lattice cannot be modified in place: copy on write creates a new version  Right now, provided as-is 1 1 1 1 1 2 2

31. PV  PV service and API is considered alpha. No UI yet. No crawler scripts provided yet.