1. Monterey Bay 2006 Data System Overview October 6, 2005 Michael Godin Monterey Bay Aquarium Research Institute (MBARI)

2. Agenda The Challenge User & System Requirements Usage of the System System Architecture Development Milestones What do we need before the VE? M. Godin, MBARI, 10/6/2005

3. Data Management Challenge Large number of data sources Large variety of data sources Dynamic systems –Data sources may appear during the campaign No standard data format –Streams or files, automated or manual Large number of data sources Large variety of data sources Dynamic systems –Data sources may appear during the campaign No standard data format –Streams or files, automated or manual M. Godin, MBARI, 10/6/2005

4. User & System Requirements M. Godin, MBARI, 10/6/2005

5. User Requirements Participant s Transfer Data To Server Add new Data Streams to Server Add new Data to Server Recognize any Data Format Quality/Ver sion Control Secure Archive Transfer Data From Server Common Format Data Original Format Data Decision Making Tools State of system information Observatio n state Adaptive sampling tools Model Outputs Model-data comparison Other Users Transfer Data From Server Image Access to Embargoed DataImage Access to Embargoed Data Full Access to non-embargoed dataFull Access to non-embargoed data M. Godin, MBARI, 10/6/2005

6. System Requirements Meeting User Needs System Requirement s Transfer Data To Server Add new Data Streams to Server Add new Data to Server Recognize any Data Format Quality/Vers ion Control Secure Archive Transfer Data From Server Common Format Data Original Format Data Decision Making Tools State of system information Observation state Adaptive sampling tools Model Outputs Model-data comparison Data Access Security M. Godin, MBARI, 10/6/2005

7. Describing Incoming Data File Streams Interfaces for Adding Data Interfaces for Getting Data & Images Supported Interfaces for Using Data Interfaces for System State Usage of the System M. Godin, MBARI, 10/6/2005

8. Describing Incoming Data Streams M. Godin, MBARI, 10/6/2005

9. Interfaces for Adding Data FTP M. Godin, MBARI, 10/6/2005

10. Interfaces for Adding Data FTPWeb M. Godin, MBARI, 10/6/2005

11. Interfaces for Adding Data FTPWebE-mail M. Godin, MBARI, 10/6/2005

12. Dynamic Data Ingest Your Server FTP or HTTP Interfaces for Adding Data FTPWebE-mail Server Pull Your Database Central Server Storage Request Data Response M. Godin, MBARI, 10/6/2005

13. Live Access Server (LAS)Live Access Server (LAS) Interfaces for Getting Data & Images M. Godin, MBARI, 10/6/2005

14. Interfaces for Getting Data & Images Live Access Server (LAS)Live Access Server (LAS) Metadata Oriented Query Assistant (MOQuA)Metadata Oriented Query Assistant (MOQuA) M. Godin, MBARI, 10/6/2005

15. Ocean Data ViewOcean Data View Supported Interfaces for Using Data M. Godin, MBARI, 10/6/2005

16. Ocean Data ViewOcean Data View Space/Time Toolkit?Space/Time Toolkit? AMES CIP?AMES CIP? Supported Interfaces for Using Data M. Godin, MBARI, 10/6/2005

17. Interfaces for System State Network Status (Nagios) Network Status (Nagios) M. Godin, MBARI, 10/6/2005

18. Interfaces for System State Network Status (Nagios) Network Status (Nagios) Last Known Position Of Assets Last Known Position Of Assets M. Godin, MBARI, 10/6/2005

19. Interfaces for System State Network Status (Nagios) Network Status (Nagios) Last Known Position Of Assets Last Known Position Of Assets Processing Stoplight Chart Processing Stoplight Chart M. Godin, MBARI, 10/6/2005

20. Where We Were: 2003 Data System Where We Got to: Post-2003 Data System Where We Want to be: MB2006 Data System Architecture Where We Will be Soon: Virtual Experiment Data System Data System Architecture M. Godin, MBARI, 10/6/2005

21. Original Archive 110 234 999 Where We Were: 2003 Data System Architecture Devices Models & Derived Outputs Principal Investigators’ Processing, QC, & Archives Common Archive 101110 110011 Principal Investigators’ Downloads Derived Data Primary Data Polarbear MBARI Firewall Custom Processing Public Image Requests mbari.org (static images) M. Godin, MBARI, 10/6/2005

22. Original Archive 110 234 999 Where We Got To: Post-2003 Data System Architecture Devices Models & Derived Outputs Principal Investigators’ Processing, QC, & Archives Common Archive 101110 110011 Principal Investigators’ Downloads Derived Data Primary Data Polarbear MBARI Firewall Custom Processing aosn.mbari.org Data Query Data line 1 more data last data Public Image Requests Principal Investigators’ Data Requests Reanalyses M. Godin, MBARI, 10/6/2005

23. Original Archive 110 234 999 Data Stream Descriptions MB2006 Data System Architecture Devices Models & Derived Outputs Dynamic Data Ingest Principal Investigators’ Processing, QC, & Archives Common Archive 101110 110011 Data Query Data line 1 more data last data Static Metadata Dynamic Metadata System State Central Server Principal Investigators’ Data Requests Public Image Requests Decision Maker Queries Derived Data Primary Data M. Godin, MBARI, 10/6/2005

24. Original Archive 110 234 999 Data Stream Descriptions Virtual Experiment Data System Architecture Devices Models & Derived Outputs Dynamic Data Ingest Principal Investigators’ Processing, QC, & Archives Common Archive 101110 110011 Data Query Data line 1 more data last data Static Metadata Dynamic Metadata System State Central Server Principal Investigators’ Data Requests Public Image Requests Decision Maker Queries Derived Data M. Godin, MBARI, 10/6/2005

25. Development Milestones Basic Data Ingest, Processing, Storage, and Serving: October 28, 2005 –Mods to MBARI’s Shore Side Data System –Mods to NOAA/PMEL’s Live Access Server Data Version Control, Asset Map: 11/14/2005 Waypoint–Path comparison: before VE? Post-Virtual Experiment Milestones: –User Access Controls & Public Images –Stop-Light Chart, Network Map –Custom Output Data Formats & Graphics –Saved Queries and Routine Outputs M. Godin, MBARI, 10/6/2005

26. What Do We Need for the VE? Example Data Sets, by 10/21/2005 Desired Data Outputs, by 10/28/2005 Metadata by 11/14/2005 (entry form will be forthcoming…) –Data Import Specifications (how to get data) –Data Reference Information Nice to have by 11/14/2005: –Waypoint files M. Godin, MBARI, 10/6/2005

27. The End M. Godin, MBARI, 10/6/2005