User Working Group 2013 Data Access Mechanisms – Status 12 March

Data Management & Stewardship Preserve NASA’s data for the benefit of future generations Data Access Provide intuitive services to discover, select, extract and utilize data Science Information Services Provide a knowledgebase to help a broad user community understand and interpret satellite ocean data and related information PO.DAAC Functional Areas

Data Management & Stewardship Preserve NASA’s data for the benefit of future generations Science Information Services Provide a knowledgebase to help a broad user community understand and interpret satellite ocean data and related information PO.DAAC Functional Areas Data Access Provide intuitive services to discover, select, extract and utilize data

High-Level Functions Consumers Archive Ingest Web Services & Publishing Direct Data Access High-Level Access Tools Web Portal Visualization Inventory Information Providers Data In Information In Data Out Information Out Data Providers Data Providers

High-Level Functions Consumers Archive Ingest Web Services & Publishing Direct Data Access High-Level Access Tools Web Portal Visualization Inventory Information Providers Data In Information In Data Out Information Out Data Providers Data Providers

 Recommendation 24 – Harden tools: Closed. (Tool improvements have focused on stability, debugging, and updates, along with selected re- architecture to improve maintainability.)  Recommendation 25 – Improve linkages between tools and datasets: Closed. (Links to applicable tools and services now exist on the dataset information pages (access tab) in the beta site.)  Recommendation 27 – Harden HITIDE and add datasets: Partially Closed. (The HITIDE interface has been reworked and functionality has been standardized. Extraction methods are still being worked and datasets will continue to be added.)  Recommendation 28 – Decommission POET: Open. (Decommission is pending the addition of all datasets from POET into THREDDS. Updates and testing are in progress.)  Recommendation 29 – Add datasets in THREDDS: Partially Closed. (Applicable datasets from the recommendation have been made available. Datasets will continue to be added.)  Recommendation 34 – Report on how UWG Homework shaped FY13 activities: Closed. (The UWG homework responses aligned with our development goals, but additionally served as a vetting document for evaluating updates.) Applicable 2012 UWG Recommendations

Purpose of this Presentation: Data Access Mechanisms Status  Provide a functional overview and architectural response Presentation Outline:  Technology mapping and infusion  Current status + Outlook of each functional area:  Direct Data Access  High Level Access Tools  Web Services Outline

 “Technology on purpose”  Process for Infusion and Deprecation  Leveraging ESDSWG Tech Infusion Working Group’s work  Eye to the outside (Other DAACs, other agencies, etc.) Technology Mapping (Infusion Process) Infusion Pool Needs PO.DAAC Vision User Needs ESDIS Goals Standards New Technologies New Approaches Cloud Lessons-Learned Selection

Consumers Archive Ingest Web Services & Publishing Direct Data Access High-Level Access Tools Web Portal Visualization Inventory Information Providers Data In Information In Data Out Information Out Data Providers Data Providers Direct Data Access

Direct Data Access: FTP 2012 Developments: None Outstanding Challenges: Minimal (possible impact with URS) Opportunities: No functionality gaps have been identified  Standard way of doing business  Our most popular means of access  High degree of user-dependency

Direct Data Access: OPeNDAP 2012 Developments: Configured for access to public NetCDF 3/4 and HDF 4/5 datasets Identified performance issue Improved response time by adding cache Outstanding Challenges: Will continue monitoring the performance Opportunities: No functionality gaps identified  Widely used in the Earth Science communities 25

Direct Data Access: THREDDS 2012 Developments: 20 datasets added Created NCML files for time aggregation Tied THREDDS to the OPeNDAP cache Outstanding Challenges: Allow dynamic creation of new variables Opportunities: Will continue to add Datasets  Popular capability in Oceans and Climate communities  Specifically requested by the UWG 29

Direct Data Access: FastCopy 2012 Developments: Retired in August 2012 Outstanding Challenges: Ran on old hardware (retired) Vendor support ended No longer required the interface Opportunities: N/A  File transfer mechanism requested by specific partners  Alternative to FTP

Consumers Archive Ingest Web Services & Publishing Direct Data Access High-Level Access Tools Web Portal Visualization Inventory Information Providers Data In Information In Data Out Information Out Data Providers Data Providers Web Services & Publishing

Web Services: Consolidated Web Services (CWS) 2012 Developments: Refactored existing Web Services Deployed documentation Deployed HTML interface Deployed dataset and granule search Deployed metadata retrieval Outstanding Challenges: Fully decouple imaging and extraction Opportunities: Add L3 imaging and extraction services Add Events service Deploy Imaging and extraction (in test)  New paradigm for PO.DAAC (unsilo’ing) and our Users  Better exposes PO.DAAC’s services  Allows users to quickly write their own clients  Provides flexibility and agility New 25

Web Services: Publish / Subscribe 2012 Developments: Feeds implemented and available for selected datasets Outstanding Challenges: Deploy updates (currently in test) Opportunities: Publish feeds for all active datasets New  New paradigm for users  No longer have to crawl the FTP site  Improves client’s ability to customize their processing 25

Consumers Archive Ingest Web Services & Publishing Direct Data Access High-Level Access Tools Web Portal Visualization Inventory Information Providers Data In Information In Data Out Information Out Data Providers Data Providers High Level Access Tools

High Level Access: Dataminer 2012 Developments: Retired in August 2012 Outstanding Challenges: Code was silo’ed; hard to maintain Still required level-2 extraction Opportunities: Transitioned to HITIDE (HIgh-level Tool for Interactive Data Extraction)  Selected for extraction capabilities  Developed externally  Limited ability to modify software

High Level Access: HITIDE 2012 Developments: User Interface completely revamped Able to search across multiple datasets Re-architected (better utilizes Web Services) Initial release in operations Outstanding Challenges: Improve performance (imaging and extraction) Improve DMAS integration Implement additional capabilities Opportunities: Add co-location search  Fills need for Level 2 extraction  Refactors and expands upon Dataminer's functionality New

High Level Access: LAS 2012 Developments: Deployed and configured (PODAAC-LABS) Several datasets added Proof of concept very successful Outstanding Challenges: Addition of more datasets Opportunities: Expect to upgrade the server to v8.0  Fills need for Level 3 capabilities  De-facto standard in the Ocean Science community  Helps with decommission of POET New

High Level Access: POET 2012 Developments: No functional modifications in 2012 Outstanding Challenges: Retirement currently planned for March of this year Pending UAT and final data additions to THREDDS+LAS Opportunities: N/A  Provides access and extraction of Level 3 data  Getting expensive  LAS is a more broadly accepted alternative 28

End of Presentation