1. ICOADS: Data, Products, and Access
Steven J Worley
Scott D. Woodruff
Zaihua Ji
Richard W. Reynolds
8/23/2019

2. New reference document Status of Release 2.2 Access
Outline
New reference document
Status of Release 2.2
Access
8/23/2019

3. New Reference Document
ICOADS Release 2.1 Data and Products
Worley et al. ,Int. J. Climatol. 25: (2005)
Special Issue: Advances in Marine Climatology
Copies are available
Release History
1st, Release 1, 1985 coverage
6th (last), Release 2.1, 2003 coverage
Data Composition
185 million records
Emphasize record tracking - DCK (Table II), SID (Table III)
Development Combination
Delayed mode (DM) processing - GTS plus other sources
Real-time (RT) mode processing - exclusively GTS
60-80 entries each for DCK and SID
NODC WOD = DCK 780, and SIDs = 48,49 (WOA94), 88,89 (WOD98)
8/23/2019

4. New Reference Document
How DM and RT Compare?
Use overlap period 1997  1998
Bars = DM
Lines = RT
DM decreases the number of drifting buoy observation. MEDS, composites and QA checks.
DM increases records for ships, moored buoys, and C-Man
60-80 entries each for DCK and SID
NODC WOD = DCK 780, and SIDs = 48,49 (WOA94), 88,89 (WOD98)
8/23/2019

5. New Reference Document
Data Usage Considerations
Changing observing systems - temporally
decreasing ships since 1989
increasing moored and drifting buoys
Increasing buoys and decreasing ships (1989)
8/23/2019

6. New Reference Document
Data Usage Considerations
Changing observing systems, spatially
45% of possible 2° boxes have 5 samples/month in the 70-80’s
Decreasing low counts due to reduced ships observations
Percentage of ocean area sampled, monthly, ranked by number of samples in 2 degree boxes
Increasing high counts due to high frequency sampling from buoys
8/23/2019

7. New Reference Document
Data Usage Considerations
Other changes
Instrument types and sampling procedures
Codes for measurements and visual observations
Instrument package placement on platforms
Sampling frequency and time of day
More …
Increasing buoys and decreasing ships
Early period all ships and notable gaps
8/23/2019

8. Status of Release 2.2 Focus time period, 1998 - 2004 Processing Status
Reprocessing of from Release 2.1
Add on two years,
Processing Status
Observations
Complete and available from NCAR (12 October)
Further analysis of result to be done
Don’t expect a rerun, but ….
Monthly Summary Statistics (MSG)
Prepare after the observations are fixed (~ 1-2 months)
LMRF (old binary format) will be generated for Release 2.2
We need LMRF to compute MSG, will transition later
Some past users may want this also - access by special request
Warning: LMRF will be discontinued
8/23/2019

9. Status of Release 2.2 Availability First from NCAR
All products
Following soon after from ESRL
The former NOAA/CDC is now part of the NOAA Earth System Research Laboratory (ESRL)
Only Monthly Summary Statistics in netCDF format
Future at NCDC
Observations; retrospective and near real-time
8/23/2019

10. Status of Release 2.2 Data Composition New DM Data, 1998-2004
TAO/TRITON, PIRATA, JAMSTEC Moored Buoy Research Archives
Including RH » DPT
NDBC Moored Buoy Data
DPT in DM and frequency not known in RT stream
MEDS Drifting Buoy QC’d Archive
WMO Pub 47 metadata ( )
Thanks to Liz Kent and Dave Berry
Kent, et al, 2005: WMO Pub. No. 47 Metadata and an Assessment of Observation Heights in ICOADS. Submitted JAOT
8/23/2019

11. Status of Release 2.2 Data Composition
Carry forward RT Data,
NCEP GTS with DM sources filtered out
New RT Data,
NCEP GTS with filtered
8/23/2019

12. Status of Release 2.2 Challenges and Misc. Notes
Meshing DM tropical moored buoys
Sampling frequency, hourly and 10-minute
10-minute reduced to hourly
Clean insert with accurate exclusion of GTS
Daily resolution and matching WMO numbers
Removed buoys with erroneous ID=SHIP from NCEP ship GTS
Eight month period (10/99 - 5/2000)
Fill some gaps in MEDS with NCEP buoy GTS
Made some QA(QC) flag fixes for full period of record
Minor errors transitioning from LMRF to IMMA format
Further checking will be done and documentation prepared
Many details are not told here
Ask questions offline
8/23/2019

13. Status of Release 2.2 Moored buoy coverage NDBC (approximate)
PMEL/NOAA (TAO and PIRATA)
JAMSTEC
Note: new array beginning in Indian Ocean
European buoys not shown
8/23/2019

14. Status of Release 2.2 Resource issue for US ICOADS team
Current personnel
Person Time Commitment
Role
Name
Organization .8
Coordinator
Scott Woodruff
NOAA/ESRL
Programmer
Sandy Lubker .1
Steven Worley
NSF/NCAR .3
Zaihua (Hua) Ji
Science Advisor
Dick Reynolds
NOAA/NCDC
Marine Operations
2.2
Total
Bad News: FY05 lost 20% of
Good News:
Received international contributions (e.g. Kent et al, metadata)
NCDC/CDMP (Elms) supporting data and metadata rescue
8/23/2019

15. t,s,v = subsetting by time, space, and variable
Access
Links to all data and information
(under Data and Metadata)
See Table VI (Worley et al. 2005)
Product
Format
Partner
Subset
Comments
Observations:
Release 2.2
ASCII
NCAR
t,s,v
Binary (LMRF) by request
Release 2.1
NCDC
t,s
Currently not available
Auxiliary
ASCII, other
none
GUI (IMMA), and datasets
NRT
ESRL
NCEP GTS, min. variables, update cycle end of month + 5 days
Monthly Statistics:
binary, ASCII
GUI and bulk download
netCDF, ASCII, other
GUI or LAS
GUI or LAS, min. variables
t,s,v = subsetting by time, space, and variable
8/23/2019

16. Access User profiles from NCAR for the past 14 Months
Who is using ICOADS?
8/23/2019

17. Access User profiles from NCAR for the past 14 months
What quantities of ICOADS are requested?
8/23/2019

18. Access User profiles from NCAR for the past 14 months
What time domains of ICOADS are requested?
8/23/2019

19. Access User profiles from NCAR for the past 14 months
What space domains of ICOADS are requested?
8/23/2019

20. Access Review IMMA structure Mandatory Record Segment
IMMA “core”, basic data, 48 fields
Optional Record Attachments
ICOADS, historical ID (e.g. DCK, SID), QA(QC) flags, etc
IMMT-2/FM 13, non-core fields from IMMT and FM-13
In use
Ship Metadata, WMO Pub. 47 ship metadata
Model QC, e.g. UK model comparison with VOSCLIM
To be activated
Historical (proposed, primarily for pre-1854 data)
Others (e.g. moored/drifting buoys?)
Recommended Attachment
Supplemental Data – original data in original format
8/23/2019

21. Access New interface to IMMA observations @ NCAR Added services
Put online about two weeks ago
Verify and improve linkages throughout ICOADS portal
Builds on previous successes
Provide ASCII output
Previously from LMRF (binary) now IMMA
All data are available for bulk file download (FTP, Web)
Users select time and space domains
User select how QA(QC) flags are applied
Added services
Access to all attachment data
Auxiliary data sources
8/23/2019

22. Access
Highlights from Interface
Filtering application
8/23/2019

23. Access
Highlights from Interface
Core variable selection
8/23/2019

24. Access Highlights from Interface Attachment data – e.g. ship metadata
8/23/2019

25. Access Highlights from Interface Auxiliary Data – two types
New, likely unique
Replacement, destine to eventually replace other data
Note: Can switch off ICOADS request
Just acquire auxiliary data
If you provide IMMA data we can serve it
Incorporate in ICOADS later
New
Replacement
8/23/2019

26. Access New interface - Level of automation
Request submission detected automatically by daemon, every 2 minutes
Processing begins immediately, with max. processes = 3
Processing may take several hours for long temporal requests
IMMA is not efficient for processing speed
Users are notified by and download via Web or FTP
8/23/2019

27. Access Type of Barometer Aneroid (AN) Digital Aneroid (DA)
Sample with ship metadata from WMO Pub 47
Type of Barometer
Aneroid (AN)
Digital Aneroid (DA)
Length of Vessel (m)
Depth of SST measurements (m)
Height of anemometer (m)
8/23/2019

28. Access Accessories to go with the data IMMA documentation
Sample FORTRAN code created dynamically to match data requested
Release ID
Variable definitions
8/23/2019

29. Access Current strategy Future possibility Delayed mode processing
File based
Future possibility
DB with core, attachments, position, date
Enable near real-time processing
Processing, O(minutes)
Push or pull data delivery
Could be easily updated (monthly) with GTS
8/23/2019

30. Conclusions
New reference document with many details about ICOADS (Worley et al. 2005)
Release 2.2, (plus other fixes), observations are released to the public.
Monthly Summary Statistics will be computed soon
Broaden access in different forms from the partners soon
Data services are going well (judged from NCAR), especially internationally.
IMMA permits more comprehensive access; to attachment and auxiliary data
Supporting resources in the US are thin. International collaboration has been helpful and more is welcomed. Project recognition from the science community can help sustain support for ICOADS
8/23/2019

31. Questions?
8/23/2019