ALMA Integrated Computing Team Coordination & Planning Meeting #1 Santiago, April 2013 Relational APDM & Relational ASDM models effort done in online software Jorge Avarias & Rafael Hiriart

ICT-CPM April 2013 Used by scheduling subsystem to:  Improve performance of the scheduling panel in OMC to show the list of observable Projects/SchedBlocks.  Run the selection step of the DSA based on criteria defined by users (Scheduling DSA Policy)  In Planning Mode Simulator as data source. The original motivation was to reduce the start-up time of the Scheduling Panel in the OMC. Relational APDM

ICT-CPM April 2013 The model is a subset of the APDM, plus some extra fields required by Scheduling. Currently in use as the schema of the SWDB. The SWDB is populated by the Scheduling updater:  In a incremental way if there is already data in the SWDB, otherwise Scheduling updater performs a full import of the APDM projects.  Scheduling Updater validates the APDM projects rejecting every inconsistent entity. Current Status (1/2)

ICT-CPM April 2013 Recently scheduling updater performance has been improved, creating a custom made read-only DAO for the XML store:  The DAO reads the APDM ObsProjects, ObsPrograms and SchedBlocks directly from Oracle using hibernate.  These retrieved XML Documents are converted to a DOM tree in memory, then passed to castor to convert them into Castor's mappings at the moment of the conversion to the scheduling data-model. Current Status (2/2)

ICT-CPM April 2013 To be used by Data Capturer to save observation metadata into a database (most likely Oracle) The main motivation is to solve the memory usage constraint issue found in the past in Data Capturer during long observations with a large number of antennas.  DC uses ~1.5 GB of RAM to store ~40 min. of an observation using 32 antennas.  Main contributor is the Pointing table (saved as binary and pushed into bulk store at the end of the observation) Relational ASDM

ICT-CPM April 2013 Workaround implemented in ALMA-9_1_4-B.  It considers only the Pointing table, which is written to a file in binary format (/tmp directory).  Then the file is pushed to BulkStore incrementally (implemented new method storeDataIncrementally in BulkStore) Long term implementation considers every single ASDM table defined as XML Schema. Current Status (1/4)

ICT-CPM April 2013 Prototype has been implemented. It is based on hibernate  The hibernate mapping is generated using Java reflection to know what are the columns and the keys of every table.  ASDM java row classes cannot be used as mapping classes for hibernate.  New wrappers classes are created for mapping: Defining the no-param constructor. Doing public some setters. Fixing some getters returning Exceptions. DB is auto-generated by Hibernate. Relationships (FK) have not yet defined among tables. Current Status (2/4)

ICT-CPM April 2013 New hibernate simple and complex types classes (to convert table value to POJO and vice versa) were implemented to handle custom ASDM types:  Simple types for angle, temperature, speed, etc.  Complex types for timeInterval and others.  Complex type for arrays (including IDL enumerations): Arrays are saved serialized (current XML format) as CLOB into DB.  Complex types for IDL enumerations. In the table two columns are used: IDL_Type and actual valor Newly defined hibernate types classes are easy because the ASDM data is immutable. Current Status (3/4)

ICT-CPM April 2013 Defined a DAO to read-write data.  Supports concurrent transactions a sessions using the ThreadedSessionFactory coming by default in Hibernate.  Save methods receive Row ASDM class as parameter, then the DAO encapsulates the parameter into the respective RowWrapper class, the the data is saved into DB.  Get methods support Criteria queries and incremental iterators to avoid to put the entire ASDM table in memory. Current Status (4/4)

ICT-CPM April 2013 Unit test implemented as part of the prototype:  Test zero: tests hibernate simple and complex types classes.  Test one: Take a XML ASDM on disk, load it on memory using ASDM Java classes, then every row in every table is saved into DB (HSQLDB file mode). Finally the ASDM is reloaded from DB and every row is compared as XML string with the original XML rows loaded from disk. Integration test used is CONTROL/IntTest  Interferometry with 4 arrays and flagging is tested.  So far no performance problems are detected. Current Status: Testing

ICT-CPM April 2013 Scalability tests, focused in known what is the maximum throughput.  They can be done after the DC scalability test are completed.  There are place for optimizations ;) Well defined DB model.  Model should not be generated automatically by Hibernate.  To define relationships (FK and constraints) among tables. This API is just for Java. Should Python and C++ be supported as well? Pending