1. Himawari‐8 Operational Data Processing and Access
Leon Majewski
25 August 2015

2. Data Ingest and Reception
Multiple access methods and processing chains
HimawariCloud
NHMS access: operational since July 2015
HimawariConnect
ABOM access: Due September 2015
100 Mbit link to JMA (Tokyo)
HimawariCast
Melbourne: Due August 2015
Darwin: Due September 2015
Methods:
Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection
Himawari Cloud is the distribution point for NMHS
Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km)
Adds robustness (fall back methods)
Latency
Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16
That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

3. Himawari-8 HimawariCast Hope JMA HimawariConnect HimawariCloud
Methods:
Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection
Himawari Cloud is the distribution point for NMHS
Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km)
Adds robustness (fall back methods)
Latency
Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16
That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

4. Data Ingest and Reception
Multiple access methods and processing chains
HimawariCloud
NHMS access: operational since July 2015
HimawariConnect
ABOM access: Due September 2015
100 Mbit link to JMA (Tokyo)
HimawariCast
Melbourne: Due August 2015
Darwin: Due September 2015
Methods:
Direct = the direct link between ABOM and JMA – ABOM will be the only NMHS with a direct connection
Himawari Cloud is the distribution point for NMHS
Himawari Cast is a broadcast of lower resolution data (1 x 1km, 13 x 2km)
Adds robustness (fall back methods)
Latency
Important to note that the 00:00Z is available to us at 00:12 and available to users at 00:16
That is, we turn it around in ~3 minutes, some data is available via CMSS in 30 seconds

5. Data Ingest and Reception
Approximate HimawariCloud data availability timeline 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1 2 3 4 5 6 7 8 9 10
ABOM image generation
ABOM product generation
What does this translate to in terms of latency?
Chart is an approximation of the data processing schedule
Darwin observed around 00:06-00:07, data is available 7-8 minutes after obs
Brisbane/Perth observed around 00:07-00:08, data is available 6-7 minutes after obs
Syd/Mel/Hbt/Adel… observed around 00:08-00:09, data is available 5-6 minutes after obs
Forecaster access is 6-10 minutes after scan completion
00:00Z scan completes at 00:10Z
Available to download at 00:12Z-00:15Z
netCDF and GeoTIFF output are generated
Data is distributed to forecasters at 00:16Z-00:19Z 1 2 3 4 5 6 7 8 9
 Observation
 Processing
 Available for download

6. Real time processing: Observations
HSF data is converted to netCDF4
Metadata: CF, ACDD
Native projection (+proj=geos)
Application of calibration coefficients
Scaled radiance (bands 1-6)
Bidirectional reflectance factor (bands 1-6)
Brightness temperature (bands 7-16)
Generation of solar geometry
scaled_radiance
"scaled" means multiplied by a constant factor of proportionality. The scaled_radiance is equal to the measured radiance multiplied by pi and divided by the solar irradiance averaged over the spectral band for normal incidence and an Earth-Sun distance of 1 astronomical unit. The radiant fluxes are integrated across the spectral band. Equivalently, the scaled_radiance is equal to the toa_bidirectional_reflectance_factor multiplied by the cosine of the solar zenith angle and divided by the square of the Earth-Sun distance.
toa_bidirectional_reflectance_factor
"toa" means top of atmosphere. The term "bidirectional" implies single directions for the incident and reflected radiances (entering and emanating, respectively, from solid angles that are differential in theory but very small in practice for satellite observed radiances at the TOA). "bidirectional_reflectance_factor" is the ratio of the reflected radiant flux exiting a surface to the reflected radiant flux from an ideal and diffuse (Lambertian) surface under identical view direction and solid angle and identical single direction illumination. The fluxes are integrated across the spectral band. The toa_bidirectional_reflectance_factor is equal to the measured radiance multiplied by pi, divided by the cosine of the solar zenith angle, and divided by the solar irradiance averaged over the spectral band for normal incidence and the Earth-Sun distance at the time of the measurement.
The brightness temperature is the temperature a black body in thermal equilibrium with its surroundings would have to be to duplicate the observed intensity of a grey body object, such as the Earth, at a given wavelength.

7. Real time processing: Imagery
Generation of imagery from netCDF4
GEOTIFF and JPEG with colour scale applied
Native projection
Simple latitude/longitude grid (+proj=latlong)
Geospatial Data Abstraction Library
Open source toolkit for geospatial data manipulation
Image translation (crop, format) & re-projection
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

8. Real time processing: Products
GEOCAT framework (NOAA & University of Wisconsin)
Cloud mask & properties (Heidinger & Pavolonis)
Fog & Low cloud (Pavolonis)
Volcanic Ash (Pavolonis)
Atmospheric Motion Vectors (Daniels & Le Marshall)
Sea Surface Temperature (Griffin & Majewski)
External processing systems (post-GEOCAT)
Solar Radiation (MINES-ParisTech & Grant)
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

9. Data access: Observations
Operational (Government, commercial & non-commercial)
netCDF: Registered user service
Service Level Agreement
Research use: National Compute Infrastructure
Delayed availability (< 24 hours)
HSF: 30 days online, archive to MDSS
netCDF: online, copy on MDSS
Working with NCI to achieve efficient access
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

10. Data access: Products
Operational (Government, commercial & non-commercial)
Research use: National Compute Infrastructure RDSI
Delayed availability (< 24 hours)
netCDF: online, copy on MDSS
Released as determined to be "valid"
Consistent with NOAA output
Routine validation metrics available
Updates are expected as improvements are made
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

11. Validation Observations JMA GSICS web page JMA Navigation web page
Products
Cloud properties: CloudSat, heritage products
AMV: Radiosondes, NWP model impact
Solar radiation: Solar radiation network
Sea Surface Temperature: drifting buoys
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

12. Future Developments Meteorological product development
Overshooting tops
Convective initiation
Significant rainfall
NWP data assimilation
Atmospheric Motion Vectors
Clear Sky Radiances
Cloud properties
Public viewer release (late September)
End note:
At the start, I mentioned HW8 was a big data challenge.
The volume and velocity components are challenging, but we can do it.
The real challenge is to take advantage of the variety.
New products is one thing
HW8 alone (temporal and spatial)
HW8 products (AMVS and cloud type to see where those ice clouds are headed over the next hour?)
HW8 fused with other data sets (satellite, nwp, …)
But taking advantage of the products is another

13. Leon Majewski Leon.Majewski@bom.gov.au
Thank you…
Leon Majewski

14. Advanced Himawari Imager
Advanced imagers provide us with a "Big Data" challenge
Volume
Imagery: 50TB/year
Products: 100 TB/year
Velocity
10 minute refresh
Variety
Band combinations
New Products
AHI Band
Central Wavelength
Spatial Resolution
Bit Depth
MTSAT Central Wavelength
GOES-R ABI Central Wavelength
[µm]
[m]
[bits]
1 (B)
0.47
1000 11
2 (G)
0.51
3 (R)
0.64
500
0.68 4
0.86
1.37 5
1.60
2000
1.61 6
2.30
2.24 7
3.90 14
3.70 8
6.20
6.80
6.17 9
6.90
6.93 10
7.30 12
7.33
8.60
8.40
9.60
9.61 13
10.40
10.80
10.33
11.20 15
12.40
12.00
12.30 16
13.30
Essentially a big data challenge
we get lots more data due to improved resolutions
we get it faster and more frequently
we've got a lot more scope for products and band combinations
it's a challenge to store, process, delivery, retrieve…
and do it in a robust fashion:
in MTSAT land, if an issue arose, one image per hour would be lost. now 6 per hour – difficult to catch up

15. netCDF4 data format
P1S-ABOM_OBS_B13- PRJ_GEOS141_2000-HIMAWARI8-AHI.nc
Date:
Time coverage: P1S
Product code: ABOM_OBS_B13
Projection code: PRJ_GEOS141_2000
Platform: HIMAWARI8
Sensor: AHI
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

16. netCDF4 data format Variable names: channel_0001_scaled_radiance
Band 1 scaled radiance
channel_0003_brf
Band 3 TOA bidirectional reflectance factor
channel_0013_brightness_temperature
Band 13 TOA brightness temperature
Going back to the first slide, you can see many of the AHI bands line up with the NOAA GOES-R ABI
Consequently many of the algorithms NOAA has developed for ABI are applicable to AHI (with a bit of tweaking)
Note that

17. Imagery: VIS, IR, WV, Combinations