Netlobs Manipulating Gridded Data in a Relational World Neil Stamps Technical Architect

Agenda Introduction to Lost Wax The problem framed Oracle 10g capabilities NetLobs – NetCDF meets database

Introduction to Lost Wax Advanced Software Engineering Multi-agent Systems Continual R&D Roles based business analysis Agent framework Products (SPL) Innovation projects Server-side Systems J2EE / MS.NET SOA and Web Services Legacy integration / wrapping GIS Mapping Solutions Mobile Distributed computing Java PDAs / phones

The problem framed DTI DEWS research project partners: Met Office, Reading University, BADC, BMT, IBM Provides web services to multiple domains GADS provides marine services Oracle target platform Interne t Web Service Interne t Web Service Forecast Data SQL GADS

Oracle – Blob support Oracle 10g deployment platform Large object support Blob – max size (4GB –1)*block e.g. 32k block = 128TB max Clob, nClob – max size as per Blob Extension support Java, C extensions Java stored procedures Custom data types (cartridges) Remote symbolic debugging (JDeveloper)

Oracle - Custom data types Provide encapsulation of attributes and methods Introduce OO capabilities into relational world Allow unstructured data to be queried Extensions to indexes allow efficient queries Nested tables provide collection capability

NetLobs – NetCDF SmartLobs Provides NetCDF file capability to Oracle Encapsulates data and meta-data in single type Physical implementation agnostic Automatic extraction and storage of meta-data Interrogate meta-data without blob enquiry Extraction over single or multiple Netlobs

NetLobs – NetCDF SmartLobs NetCDF 2.2 open source Java libraries NetLob wraps Oracle for NetCDF Files Extraction interfaces based upon current GADS requirements: Subset Reduction Concatenation Higher level interfaces to be layered over basic functionality

Cartridge invocation PL/SQL interface maps to Java call (or C) Oracle instantiates NetLob object Object implements SQLData interface Blob pointer passed to Java, Random access provided via internal JDBC

NetLobs – Data ingestion Upload using Oracle SQL*Loader Upload in two–phase method Validation at Netlob creation System optimisation based upon once-only performance hit at extraction of meta-data Meta-data chunks will facilitate query by value

NetLobs – Performance Reference GADS system provides predictable, linear extraction performance NetLob cartridge aims to achieve similar performance characteristic over large data extractions Optimisation tailored to larger extractions

Moving forward Storage and retrieval of rotated data Pluggable interpolation framework Offloading processing to GRID Enhanced meta-data to meet community needs Query by-value enhancements

Any Questions?