MOOS SSDS Data Access Features A Discussion with MBARI’s Science Data Users

21 Nov Today’s Goals Provide overview of SSDS in MOOS context Provide overview of SSDS in MOOS context Review user interface (UI) products and concepts Review user interface (UI) products and concepts Show you UI options, and let you guide our way Show you UI options, and let you guide our way Offer opinions on good and bad approaches Offer opinions on good and bad approaches Identify long-term goals and ideas Identify long-term goals and ideas Note: We are not presenting a released product Note: We are not presenting a released product It will be a while (6-12 months) before the system is solid It will be a while (6-12 months) before the system is solid Just getting early feedback on concepts from users Just getting early feedback on concepts from users Part of iterative development process Part of iterative development process

21 Nov Today’s Sequence of Events ‘Throwaway Demo’ (“Look, it works, really!”) ‘Throwaway Demo’ (“Look, it works, really!”) Introduce MOOS and SSDS Introduce MOOS and SSDS Demonstrate Our Concept of Operations Demonstrate Our Concept of Operations Show a simulated example using real SSDS software. Knowledge increases! Data is stored! Show a simulated example using real SSDS software. Knowledge increases! Data is stored! Demonstrate Prototype User Interfaces Demonstrate Prototype User Interfaces Tour of (MOOS) HOOVES Tour of (MOOS) HOOVES You Tell Us: What Are the Most Wanted MOOS Data Products You Tell Us: What Are the Most Wanted MOOS Data Products Fill out the survey, join ssdsusers, just talk to us… Fill out the survey, join ssdsusers, just talk to us… We’ll keep improving (read: changing) these products for a while, but we’ll try to give fair warning. We’ll keep improving (read: changing) these products for a while, but we’ll try to give fair warning.

21 Nov Demonstrating Real Data Interface to MOOS Test Mooring data Interface to MOOS Test Mooring data Put together quickly, using assumptions Put together quickly, using assumptions This interface is simple because data is still simple This interface is simple because data is still simple Data was collected through latest MTM run Data was collected through latest MTM run As data sets get more complicated, this kind of interface breaks down As data sets get more complicated, this kind of interface breaks down Subsetting, varied record types, deployment changes all make the data rather messy Subsetting, varied record types, deployment changes all make the data rather messy At end of presentation you can tell us what you need to work ‘in the real world’ At end of presentation you can tell us what you need to work ‘in the real world’

21 Nov MOOS (Shows Data Flow) Devices Deployed Platform Shore Side Data System User Applications (User Tools) Data Tracking Communications Applications/ Interfaces Archiving Data Presentation Data line 1 more data last data OceanSideShoreSide Portal

21 Nov Shore Side Data System Requirements Requirements Ingest data in any described format and save it Ingest data in any described format and save it Capture, publish data descriptions (via metadata) Capture, publish data descriptions (via metadata) Provide standards-based access to data Provide standards-based access to data Raw data, and other common digital formats Raw data, and other common digital formats APIs for use by common visualization and analysis tools APIs for use by common visualization and analysis tools User-oriented web interfaces, with quick-look plots User-oriented web interfaces, with quick-look plots Merge data from different sources & time intervals Merge data from different sources & time intervals Allow straightforward visualization of data Allow straightforward visualization of data Simplify data processing and product archiving Simplify data processing and product archiving Provide for automated generic quality checks Provide for automated generic quality checks Provide data access security as needed Provide data access security as needed

21 Nov John’s Top 10 List: Why Use SSDS? 10.All MOOS packets come via ISI. We’ll format and put the raw data back together for you. 9.All the annoying stuff—backups, permissions, storage, naming—is handled by us (for free!). 8.You can point other people to your data. 7.You can see other people’s data. 6.You can actually search for data (by name!). 5.You can easily get basic (+) plots of your data. 4.You won’t ever have to remember what’s in your data again, it will all be described. Forever. 3.It will plug data in to your favorite application. 2.Every data access tool we build works for you. 1.Everybody else will be doing it.

21 Nov How Does It Work? 1. First, you describe what’s in your data records. (This is the metadata—more about it later. the metadatathe metadata 2. That description is stored with (near) your instrument, and sent to SSDS before any data from your instrument. SSDS tracks this info…. 3. When you send a data record, identify its type. identify its typeidentify its type 4. SSDS automatically routes data of each type to its own ‘data bucket’. 5. SSDS automagically knows all about your data records, because you described them. Now it can do all sorts of useful things with your data: Plot  Print  Search  Merge Plot  Print  Search  Merge Format (on request)  Describe (in files & headers) Format (on request)  Describe (in files & headers) Send to Applications  Point to by variable name Send to Applications  Point to by variable name 1234

21 Nov Metadata Description Example <recordDescription><recordType>1</recordType><bufferStyle>ascii</bufferStyle><bufferLengthType>variable</bufferLengthType><bufferParseType>separated</bufferParseType><bufferItemSeparator>,</bufferItemSeparator> <recordTerminator>\r\n</recordTerminator><variable><columnIndex>1</columnIndex><format>%3s</format><variableName>messageType</variableName><units></units> Message Type Message Type WB indicates house keeping message WB indicates house keeping message </variable><variable><columnIndex>2</columnIndex><format>%i</format><variableName>messageSize</variableName><units>characters</units> Message Size Message Size The number of characters in the message not including the The number of characters in the message not including the message type, size, or checksum characters message type, size, or checksum characters </variable>…

21 Nov Metadata Description Example … 3 %6s date YYMMDD Date in YYMMDD format 4 %4s time HHMM Time in HHMM format 5 %f voltage Volts Voltage …

21 Nov Data Record Identification Each data record sent via ISI is wrapped with ‘record header’ metadata to identify it Each data record sent via ISI is wrapped with ‘record header’ metadata to identify it Contents of record header metadata include: Contents of record header metadata include: Data source ID (all MOOS data sources have ID) Data source ID (all MOOS data sources have ID) Metadata ID (what file has a description to this record) Metadata ID (what file has a description to this record) Record Type ID (which kind of record is this) Record Type ID (which kind of record is this) Timestamp (provided by ISI when record is submitted) Timestamp (provided by ISI when record is submitted) System now knows what to do with each data record (otherwise they’d all be mushed together in a long stream of records) System now knows what to do with each data record (otherwise they’d all be mushed together in a long stream of records) Right now we also ask for and use ‘source platform’ Right now we also ask for and use ‘source platform’ Exact mechanism used to supply this is TBD Exact mechanism used to supply this is TBD

21 Nov Recap: MOOS Data Flow Devices Deployed Platform Shore Side Data System User Applications (User Tools) Data Tracking Communications Applications/ Interfaces Archiving Data Presentation Data line 1 more data last data OceanSideShoreSide Portal 4 2b 2c 3 5a 1 5b 2a

21 Nov SSDS Elements Applications Data Presentation Data line 1 more data last data Ingest Archiving Arriving Data Data Tracker Data Catalog External Data Stores Data For Analysis Web I/F < Requests Data > Shared Descriptions < Requests Data > Metadata (Re)Processed and New Data Sets Automated Data Flow Internal Interfaces On-Demand Interactions

21 Nov Standard Interfaces Applications Data Presentation Data line 1 more data last data Ingest Archiving Arriving Data Data Tracker Data Catalog External Data Stores Data For Analysis Web I/F < Requests Data > Shared Descriptions < Requests Data > Metadata (Re)Processed and New Data Sets DODS LAS XML/ DTDs HTTP Z39.50 SQL netCDF FGDC inGrid Browsers (and Tools) ncBrowse DODS Ferret

21 Nov Science Data Users: What Do You Need (First)? Raw data via device ID pages? (sort of limited) Raw data via device ID pages? (sort of limited) Standard plots the same as OASIS quality control? Standard plots the same as OASIS quality control? Standard plots with multiple plots per page? Standard plots with multiple plots per page? Access data from applications via a DODS URLs? Access data from applications via a DODS URLs? Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret And hopefully, Ocean Data View And hopefully, Ocean Data View Access data via shipped data files (e.g., ASCII CSV w/headers) opened within desktop applications? Access data via shipped data files (e.g., ASCII CSV w/headers) opened within desktop applications? Excel, ArcView, Ocean Data View Excel, ArcView, Ocean Data View Delivery of data directly into an application? Delivery of data directly into an application? Ability to subset data, for example by time window? Ability to subset data, for example by time window? Ability to merge data from different data sets? Ability to merge data from different data sets?

21 Nov Summary Good data descriptions of structured data records will make many things possible Good data descriptions of structured data records will make many things possible Many capabilities will be possible—need your input to determine priorities Many capabilities will be possible—need your input to determine priorities This project is iterative, so we can fix, improve, and add high priority features This project is iterative, so we can fix, improve, and add high priority features We’ll present some examples as food for thought, then open it up for discussion We’ll present some examples as food for thought, then open it up for discussion

21 Nov References What Data is Available? What Data is Available? What Data is Available? What Data is Available? Metadata “Explained” Metadata “Explained” Metadata “Explained” Metadata “Explained” How To Access the Data? How To Access the Data? How To Access the Data? How To Access the Data? How to Display the Data? How to Display the Data? How to Display the Data? How to Display the Data? MOOS Data Architecture MOOS Data Architecture MOOS Data Architecture MOOS Data Architecture How to Access Instrument How to Access Instrument How to Access Instrument How to Access Instrument How Data Access Works How Data Access Works How Data Access Works How Data Access Works How Easy to Use is It? How Easy to Use is It? Are we sure the data is OK? Are we sure the data is OK? What if there’s a problem? What if there’s a problem? Is data distributable? Is data distributable? Is data secure? Is data secure? What aren’t you getting? What aren’t you getting?

21 Nov Example of System In Action [Developer] Get Device ID [Developer] Get Device ID [Developer] Define instrument using XML [Developer] Define instrument using XML [Instrument] Start up, send metadata into system [Instrument] Start up, send metadata into system [SSDS] Receive metadata, update instrument info [SSDS] Receive metadata, update instrument info [Instrument] Send data record(s) into system [Instrument] Send data record(s) into system [SSDS] Receive data, save, make available to users [SSDS] Receive data, save, make available to users [User/Operator] Access data [User/Operator] Access data