1. MBARI Data Management Initiatives John Graybeal Information Applications Group Lead

2. 2003.07.151 Established in 1987 Monterey Bay Aquarium Research Institute David and Lucile Packard Foundation

3. 2003.07.152 Santa Cruz Monterey Monterey Canyon MBARI MBARI Location

4. 2003.07.153 Monterey Ocean Observing System Suitable for deep ocean or coastal studies Suitable for deep ocean or coastal studies low power, long term moorings and benthic nodes low power, long term moorings and benthic nodes low bandwidth communication links to shore low bandwidth communication links to shore Configurable, re-deployable instruments and platforms (using ships and ROVs) Configurable, re-deployable instruments and platforms (using ships and ROVs) Smart nodes on deployed platforms Smart nodes on deployed platforms some on-board data processing some on-board data processing facilitate autonomous event detection facilitate autonomous event detection perform on-board calculations/detections perform on-board calculations/detections handle responses from shore handle responses from shore

5. 2003.07.154 MOOS Concept of Operations Benthic Node Mooring Autonomous Underwater Vehicle (AUV) MBARI

6. 2003.07.155 Data Stream Challenge (MUSE)

7. 2003.07.156 Data Management Challenge Large number of data sources Large number of data sources Large variety of data sources Large variety of data sources Dynamic systems Dynamic systems  Data sources may appear and disappear  Devices & platforms reconfigured often  Interactions from shore and ship No standard data format No standard data format  Data can be instrument ‘native’  New sources coming on-line all the time  Streams or files, automated or manual

8. 2003.07.157 Example: Samples Database

9. 2003.07.158 Example: Video and Images 14 years, up to 300 dives/year 14 years, up to 300 dives/year 14000 video tapes, 10000 hours 14000 video tapes, 10000 hours 47000 frame grabs… => 900,000 annotations 47000 frame grabs… => 900,000 annotations How to manage this valuable repository? How to manage this valuable repository? Advanced annotation system Advanced annotation system Detailed knowledge base of concepts Detailed knowledge base of concepts Easy-to-use querying tool Easy-to-use querying tool

10. 2003.07.159 Video Annotation and Reference System (VARS)

11. 2003.07.1510 Notes About SSDS: The Shore Side Data System A MOOS Development Project A MOOS Development Project Goals: low cost, flexible, expandable, reliable Goals: low cost, flexible, expandable, reliable Future systems beyond MOOS (e.g., MARS) Future systems beyond MOOS (e.g., MARS) Now in 3rd year, deploying initial elements Now in 3rd year, deploying initial elements Key Tenets of SSDS Development Key Tenets of SSDS Development Iterative development—improve it as we go Iterative development—improve it as we go Test with real data—new and archival Test with real data—new and archival Build for change—use modular interfaces Build for change—use modular interfaces

12. 2003.07.1511 Shore Side Data System: Requirements Overview Ingest data in any described format and save it Ingest data in any described format and save it Capture, publish data descriptions (metadata) Capture, publish data descriptions (metadata) Provide standards-based access to data Provide standards-based access to data Raw data, and other common digital formats Raw data, and other common digital formats APIs for common visualization and analysis tools APIs for common visualization and analysis tools User-oriented web interfaces, quick-look plots User-oriented web interfaces, quick-look plots Merge data (different sources & time intervals) Merge data (different sources & time intervals) Support data visualization & quality control Support data visualization & quality control Provide data access security as needed Provide data access security as needed

13. 2003.07.1512 Shore Side Data System: User Requirements Raw data via device ID pages? (sort of limited) Raw data via device ID pages? (sort of limited) Standard plots like OASIS quality controlled ones? Standard plots like OASIS quality controlled ones? Access data from applications via a DODS URLs? Access data from applications via a DODS URLs? Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret And hopefully, Ocean Data View And hopefully, Ocean Data View Access data via returned data files (e.g., ASCII CSV w/headers) opened within desktop applications? Access data via returned data files (e.g., ASCII CSV w/headers) opened within desktop applications? Excel, ArcView, Ocean Data View Excel, ArcView, Ocean Data View Delivery of data directly into an application? Delivery of data directly into an application? Ability to subset data, for example by time window? Ability to subset data, for example by time window? Ability to merge data from different data sets? Ability to merge data from different data sets?

14. 2003.07.1513 Data Management at MBARI: SSDS Efforts Infrastructure/model development Infrastructure/model development Ontologies Ontologies Metadata schema Metadata schema Metadata entry/correction/annotation Metadata entry/correction/annotation User interfaces User interfaces Data processing Data processing Visualizations Visualizations Federated access to MBARI data/metadata Federated access to MBARI data/metadata

15. 2003.07.1514 More MBARI SSDS Tasks Legacy data migration Legacy data migration OASIS, expd etc., Samples, Waypoints, ? OASIS, expd etc., Samples, Waypoints, ? New data sources New data sources MTM II, AUV Sonar, CIMT, … MTM II, AUV Sonar, CIMT, … Outreach (integrating non-SSDS projects) Outreach (integrating non-SSDS projects) Documentation Documentation NEPTUNE NEPTUNE Education Education Operational support Operational support

16. 2003.07.1515 MOOS/SSDS Architecture (shows data flow) MOOS/SSDS Architecture (shows data flow) Devices Deployed Platform Shore Side Data System User Applications (User Tools) Data Tracking Communications Applications/ Interfaces Archiving 101110 110011 110 234 999 Data Presentation Data line 1 more data last data OceanSideShoreSide Portal

17. 2003.07.1516 SSDS Elements Applications Data Presentation Data line 1 more data last data Ingest 110 234 999 223 207 191 Archiving 101110 110011 Arriving Data Data Tracker Data Catalog External Data Stores Data For Analysis Web I/F < Requests Data > Shared Descriptions < Requests Data > Metadata (Re)Processed and New Data Sets Automated Data Flow Internal Interfaces On-Demand Interactions

18. Example SIAM to SSDS Data Flow Portal Mooring SSDS

19. Example SIAM to SSDS Data Flow A device is connected to a platform, such as a Mooring. Portal Device Mooring SSDS

20. <RecordVariable name="time" columnIndex="1" format="double" longName="Time(GMT)" units="milliseconds since Jan 01, 1970"/> Example SIAM to SSDS Data Flow The mooring retrieves the metadata from the device. Portal Device Mooring SSDS

21. Metadata Packet Example SIAM to SSDS Data Flow The metadata is packaged and sent to a portal on shore before any data is sent to shore. Portal DeviceMooring SSDS

22. Example SIAM to SSDS Data Flow Metadata Packet The portal forwards the metadata to SSDS. Portal DeviceMooring SSDS

23. DB Example SIAM to SSDS Data Flow SSDS stores the metadata in a database. This allows applications to query for and use data. Portal DeviceMooring SSDS

24. DB Example SIAM to SSDS Data Flow SSDSPortal DeviceMooring

25. DB 34,56.234,0.0023,... Example SIAM to SSDS Data Flow The device produces a data record. Portal Device Mooring SSDS

26. DB Data Packet 34,56.234,0.0023,... Example SIAM to SSDS Data Flow The data is packaged and sent to SSDS. Portal DeviceMooring SSDS

27. DB VersionID, DeviceID, MetadataID, RecordType, PlatformID, SystemTime, SequenceNumber, DataBuffer(34,56.234,0.0023,…) Serialized Example SIAM to SSDS Data Flow SSDS uses information in the packet to sort and store the data in a ‘raw’ format. Portal DeviceMooring SSDS

28. DB netCDF Example SIAM to SSDS Data Flow Serialized VersionID, DeviceID, MetadataID, RecordType, PlatformID, SystemTime, SequenceNumber, DataBuffer(34,56.234,0.0023,…) The ‘raw’ data is parsed and stored as netCDF for easier access. Portal DeviceMooring SSDS

29. Software applications allow users to discover and obtain data in formats useful to the typical MBARI user. (netCDF, text, etc.) DB netCDF Example SIAM to SSDS Data Flow Serialized netcdf parosci { dimensions: time = UNLIMITED ; // (17761 currently) variables: double time(time) ; time:long_name = "Time (GMT)" ; time:units = "seconds since 1970-01-01 00:00:00" ; double depth(time) ; depth:long_name = "depth" ; depth:units = "UNKNOWN" ; // global attributes: :title = "AUV data" ; :created = "2003-06-12T23:34:58Z" ; :history0 = ": Deployment information for parosci.log" ; :deploymentName = "2003.099.10" ; :instrumentId = "3699" ; } Portal DeviceMooring SSDS MBARI Software

30. 2003.07.1529 DB netCDF Example SIAM to SSDS Data Flow Serialized Software applications also provide simple visual representations of data Portal DeviceMooring SSDS MBARI Software

31. DB netCDF Example SIAM to SSDS Data Flow Serialized Provide internet access Portal DeviceMooring SSDS MBARI Software Web Pages

32. Portal DeviceMooring DB SSDS netCDF Existing netCDF Software Example SIAM to SSDS Data Flow Serialized MBARI Software Web Pages Save development time by using existing software applications

33. 2003.07.1532 SSDS Data Mgt Sequence

34. 2003.07.1533 AUV Data Sequence Diagram

35. 2003.07.1534 Metadata Approach (Credit: Dan Davis) XML suitable for MOOS metadata XML suitable for MOOS metadata Enables use of many other tools/software Enables use of many other tools/software But, it looks a little bit user-unfriendly But, it looks a little bit user-unfriendly Use XML-driven GUI technology to create forms to create and display metadata Use XML-driven GUI technology to create forms to create and display metadata Users don’t have to directly read XML Users don’t have to directly read XML It’s there and easy to access if they want it It’s there and easy to access if they want it Bind XML metadata to each device through its puck Bind XML metadata to each device through its puck

36. 2003.07.1535 Sensor Puckto host computer serial interface During pre-deployment instrument configuration, and test, sensor driver and associated metadata is stored in compact flash memory in puck During pre-deployment instrument configuration, and test, sensor driver and associated metadata is stored in compact flash memory in puck Metadata stored in puck interface

37. 2003.07.1536 Metadata Schema Design

38. 2003.07.1537 Metadata User Form Design User interface designer uses schema to build a form for creation, display, access, of metadata instances User interface designer uses schema to build a form for creation, display, access, of metadata instances There may be different forms for different users (e.g. scientific, system, and operational) to create, and display metadata of interest There may be different forms for different users (e.g. scientific, system, and operational) to create, and display metadata of interest

39. 2003.07.1538 Metadata Form Design

40. 2003.07.1539 Instrument Configuration Metadata forms are used during device configuration to create metadata that is entered into device puck Metadata forms are used during device configuration to create metadata that is entered into device puck Similarly metadata forms are used during configuration of other system elements, such as platforms, and even communication links. This metadata is maintained in system nodes. Similarly metadata forms are used during configuration of other system elements, such as platforms, and even communication links. This metadata is maintained in system nodes.

41. 2003.07.1540 Metadata Form Layout

42. 2003.07.1541 SSDS— Metadata (Object View)

43. 2003.07.1542 The data source. SSDS tracks: Software or hardware source Unique identifier Manufacturer information References to documentation SSDS— Metadata (Device)

44. 2003.07.1543 SSDS— Metadata (Deployment) ‘Deployment’ information. SSDS tracks: Where the data was collected. When it was collected. What other data was used. Relation to other deployments

45. 2003.07.1544 References to the data. SSDS tracks: The data storage location. How to access this data. The deployment that produced this data. SSDS— Metadata (DataContainer)

46. 2003.07.1545 Format and contents of a DataContainer. SSDS tracks: The contents of a data set. The data format (to allow parsing by software). Descriptive info like units, scale, … SSDS— Metadata (Records)

47. 2003.07.1546 Metadata and Access: Catalogs and Repositories View From the Shore View From the Shore Many data registries and models Many data registries and models GDC, OBIS, EarthRef, NVODS, … GDC, OBIS, EarthRef, NVODS, … Many standards Many standards Communications protocols: SOAP, OPeNDAP, OBIS, … Communications protocols: SOAP, OPeNDAP, OBIS, … Metadata formats (MIF, XML, NGDC, NetCDF…) Metadata formats (MIF, XML, NGDC, NetCDF…) Metadata ontologies and efforts Metadata ontologies and efforts NGDC, MarineXML, ESRI, Metadata Wranglers NGDC, MarineXML, ESRI, Metadata Wranglers Conclusion: Watch, Learn, Try (Iterate) Conclusion: Watch, Learn, Try (Iterate)

48. 2003.07.1547 SSDS Data Access Desktop Application: HOOVES Desktop Application: HOOVES Data File Service Data File Service Quick Look Quick Look Metadata Access (and Validation) Metadata Access (and Validation) Metadata Editing Metadata Editing Networked API: Servlet / JSP Pages Networked API: Servlet / JSP Pages Application API (NetCDF): OPeNDAP Application API (NetCDF): OPeNDAP Web Access (NetCDF): Live Access Server Web Access (NetCDF): Live Access Server Archived Files: Direct Access (?) Archived Files: Direct Access (?)

49. 2003.07.1548 HOOVES Help

50. 2003.07.1549 HOOVES Mission View

51. 2003.07.1550 HOOVES Mission Outputs View

52. 2003.07.1551 HOOVES Mission Resources: Overview

53. 2003.07.1552 HOOVES Mission Resources: Vehicle

54. 2003.07.1553 HOOVES Instrument View

55. 2003.07.1554 SSDS Schedule

56. 2003.07.1555 Prime Areas for Collaboration Infrastructure/model development Infrastructure/model development Ontologies Ontologies Metadata schema Metadata schema Metadata entry/correction/annotation Metadata entry/correction/annotation User interfaces User interfaces Data processing Data processing Visualizations Visualizations Federated access to data/metadata Federated access to data/metadata Documentation Documentation

57. 2003.07.1556 IAG Team Kevin Gomes Kevin Gomes John Graybeal John Graybeal Mike McCann Mike McCann Brian Schlining Brian Schlining Rich Schramm Rich Schramm And, a Mystery Guest (To Be Determined) And, a Mystery Guest (To Be Determined) Science Representative to SSDS John Ryan John Ryan