Status of the GHRSST GDAC (Global Data Assembly Center) at the JPL PO.DAAC Ed Armstrong, Jorge Vazquez and Andrew Bingham

28 Nov 2005US GHRSST meeting, RSMAS Overview of GDAC Tasks Ingest project data files L2P, L4, HR-DDS, MMR and MDB files from RDACs Stage on public FTP site for 40 days Deliver data and metadata to LTSRF (NODC) for 30 day old data Maintain metadata in a relational database (MMR) Provide web-base interface for data discovery and metadata query Fill ancillary fields for all L2P products Support GHRSST User Services and Applications Maintain a match-up database of in situ/satellite obs RDAC: MODIS L2P production....data management and distribution center for all GHRSST products

28 Nov 2005US GHRSST meeting, RSMAS Recent Activities Public GHRSST data release in August 2005 Interacted with OBPG and RSMAS to design the MODIS L2P processing architecture New Linux-based web and database severs, and development workstation integrated into PO.DAAC GDAC web pages enhanced Access to MMR improved Input for GDS rev 1.7 including metadata revision IDL and C readers for L2P Implemented FGDC metadata and transfer to LTSRF Ingested NWP and model data streams for L2P ancillary fields Developing code and methodology for realtime ancillary filling Implemented user support through a customer management tool (RightNow) First draft of a user requirements document

28 Nov 2005US GHRSST meeting, RSMAS GDAC System Architecture GDAC (JPL U.S.) RDACs OCEANIDS (Data Management) MMR/MDB L4PC (MIST – U.S.) Archive (NODC – U.S.) NRT Users Medspiration (Europe) RSS (U.S.) BlueLink (Australia) PO.DAAC (U.S.) NGSST (Japan) NAVOCEANO (U.S.) Data Tools

28 Nov 2005US GHRSST meeting, RSMAS GDAC Web Page

28 Nov 2005US GHRSST meeting, RSMAS Master Metadata Repository

28 Nov 2005US GHRSST meeting, RSMAS Data Access FTP: podaac.jpl.nasa.gov/pub/GHRSST Data public since late August 2005 Organised by: datatype / sensor / RDAC / year / DOY Search interface through the MMR Future delivery mechanisms POET (L2 subsetting) OpenDAP Live Access Server Subscription Data casting

28 Nov 2005US GHRSST meeting, RSMAS Data Ingest RDACDatastreamStatus EURL2P TMI L2P SEVRI L2P NAR 16 & 17 & 18 L2P AVHRR LAC (16 & 17) L2P AVHRR GAC (16 & 17) L2P ATSR L2P AMSR-E L4 UHFnd HR-DDS Good No data since 25 Mar REMSSL2P AMSR-E L2P TMI Good, sporadic in Oct NAVOL2P AVHRR LAC (18) L2P AVHRR GAC (18) Expected Dec 2005 ? RSMAS/OBPG/JPL L2P MODIS_AIn preparation NOAA NESDIS L2P GOESUnknown AUS (Bluelink) L2P AVHRR HRPTUnknown

28 Nov 2005US GHRSST meeting, RSMAS Data Latencies Datastream Avg (hr)Std Dev (hr)Max (hr) EUR L2P TMI EUR L2P SEVRI EUR L2P NAR 16 EUR L2P NAR 17 EUR L2P AVHRR LAC 16 EUR L2P AVHRR LAC 17 EUR L2P AVHRR GAC 16 EUR L2P AVHRR GAC 17 EUR L2P AATSR EUR L2P AMSR-E EUR L4 UHFnd

28 Nov 2005US GHRSST meeting, RSMAS Oceanids ingest statistics Granules Volume (GB)

28 Nov 2005US GHRSST meeting, RSMAS Ancillary fields JPL GDAC will fill the L2P ancillary fields of wind, solar insolation, sea ice and aerosols Some RDACs cannot fill these fields in an operational environment GDAC has accepted this additional responsibility at last GHRSST ST meeting Ingest numerical weather prediction and model data, i.e., NCEP, NAAPS, FNMOC sea ice and SST A backup data stream will be implemented for each ancillary field (e.g., ECMWF model output) Ancillary filling methodology and management has been conceptualized. Readers for input fields being built

28 Nov 2005US GHRSST meeting, RSMAS NOAA LTSRF Interface Status After 30 days data are placed on a rolling stage for NOAA to pull FGDC metadata created on the fly through access to the MMR Formatted for the NOAA accession record (daily collection of same product files) Currently NOAA has implemented an automated retrieval of L2P data and FGDC metadata FGDC records for all L2P Medspiration data created through Nov 2005 Other products to follow

28 Nov 2005US GHRSST meeting, RSMAS MMR Status MMR (Master Metadata Repository) Fully transactional MySQL relational database Schema finalized and implemented for FR and DSD metadata in Jan 2005 New database server implemented in Jun 2005 Over 80,000 DSD and FR records ingested since Feb 2005 No operational failures Web search interface improved, access is very fast. Automated translation to FGDC metadata through XML style sheet conversion Other standards can also be supported

28 Nov 2005US GHRSST meeting, RSMAS User services and applications Updated US GDAC User Requirements document GHRSST requirements GDAC system requirements RDAC requirements User requirements E.g., OOSes

28 Nov 2005US GHRSST meeting, RSMAS MODIS L2P Joint effort between Univ. of Miami RSMAS, NASA Ocean Biology Processing Group (OBPG) and JPL This is a new RDAC activity Initially concentrate on Aqua SST RDAC Division of Labor RSMAS: Algorithm and error statistics OBPG: 1 km L2 SST production with some ancillary fields JPL GDAC: L2P integration with error statistics and ancillary fields Current Status OBPG has produced a realtime L2 SST stream for MODIS Aqua (and Terra) RSMAS has developed a MODIS error hypercube JPL has tested throughput and acquisition from OPBG server

28 Nov 2005US GHRSST meeting, RSMAS Future activities... Ancillary fields filling Accept delivery of Linux 8 node cluster Installed in Jan 2006 Implement code for realtime filling Investigating parallelization of processing MODIS L2P Finalize transfer between three parties Implement error statistics and L2P Schedule for Aqua and Terra SST L2P Interface to new RDACs NAVO NOAA NESDIS OBPG Update ICD

28 Nov 2005US GHRSST meeting, RSMAS... Future activities FGDC metadata Operational transfers for all GHRSST products (L2P, L4) to LTSRFs User Services GHRSST product support Applications development Data delivery L2P subsetting in POET OpenDAP Improved search and delivery via MMR New technology infusion GDAC will be a test bed for the Earth Science Data Casting Project a recently funded NASA ROSES activity for XML-based data management and delivery Implement the Matchup Database (MDB) with similar functionality as MMR