1. Australian Geoscience Data Cube
A Collaboration between Geoscience Australia, CSIRO and NCI
GA/CSIRO Agency Report
Robert Woodcock - CSIRO
Simon Oliver – Geoscience Australia
The Australian geoscience data cube is an innovative project which is transforming the way we analyse large gridded datasets such as earth observation data. It is allowing us to harness the power of high performance computation though the creation of high performance data which in turn allows us to deliver useful information into the hands of decision makers and the public. The Australian Geoscience Data Cube (AGDC) is being developed as a partnership between Geoscience Australia (GA), Australia’s National Computational Infrastructure (NCI), and the Commonwealth Science and Industrial Research Organisation (CSIRO) with the main aim being to support the management and quantitative analysis of massive volumes of Earth observation (EO) and other geoscientific data.
CEOS WGISS41

2. Overview Regional Copernicus Data Access/Analysis Hub RADAR cube
Australian Geoscience Data Cube
Update on progress:
Version 1 User Documentation - development discontinued - focus on v2
Version 2 Development
The Future: Data Cubes and WGISS
CEOS WGISS41

3. Australia’s Regional Copernicus Data Access/Analysis Hub
An integrated ‘Team Australia’ approach to
Support government information requirements.
Support the broader objective of enhancing access to satellite Earth observation data by research, industry and civil society.
Facilitate collaboration between Australians, Europeans and inhabitants of the South-East Asia-South Pacific region in exploitation of Earth observation data.
Benefits for: Australia, the region, EC/ESA/EUMETSAT, and the global satellite EO community.
All Sentinel Products For
Supporting
Consortium
Partners
Collaborators
CEOS WGISS41

4. Australia’s Regional Copernicus Data Access/Analysis Hub
ESA
IntHub
EUMETCAST
Terrestrial
GEANT/AARNET
Partnerships
(pay-per-use)
Master Repository (NCI)
Delivery for:
Australia
Region
Regional Portal
(a la ‘SciHub’)
Original Format Data Files
(HSM Storage)
Agreed Standards
Interoperable
Services
(e.g. Thredds)
Analysis-Ready Data
(Lustre/spinning disk)
R&D for:
Research/Academia
Civil Society
Government
Industry
Australia and Region
Cloud and HPC (Petascale)
Industry Exploitation
Operational Products
Deep National Archives
(Government Systems)
CEOS WGISS41

5. RADAR cube UK Catapult collaborative agreement signed between:
GA / UNSW / CSIRO / UK Catapult
Analysis Ready Data for RADAR:
Sentinel-1
Level-1 Interferometric Wide Swath
Level-1 Extra Wide Swath
ERS-1/2 SAR
ENVISAT ASAR
ALOS-1 PALSAR
CEOS WGISS41

6. AGDC Overview
A series of data structures and tools to enable efficient analysis of large earth observation archives in HPC / Cloud and Desktop environments
Simple Data Structures
Spatially regular tiles
Managed by a relational database
Calibrated and Standardised Unique Observations
Surface Reflectance Observations
Quality Assured Observations
Flagged for cloud, cloud shadow, saturation and other quality indicators
Analysis Ready Data as input
CEOS WGISS41

7. AGDC Programme Partners: GA, CSIRO and the NCI
International collaborators are increasingly involved including the USGS, NASA and CEOS.
In the midst of moving to an updated version reflecting recent advances in technology
Supporting the establishment of other international data cubes, initially with Kenya and Colombia
AGDC is currently supporting a range of remote sensing applications across the water, vegetation and mineral domains
Providing valuable information for environmental monitoring and modelling across all Australian jurisdictions
CEOS WGISS41

8. Landsat Analysis Ready Data preparation
DataCube
Solid science
Taking the data to comparable quality assured measurements
Adapting software systems for embarrassingly (massively) parallel processing, enabling quality assurance, building code workflows that allow processes to be iterated and improved / experiments – this is where workflows are important (I will discuss in a minute how we are tackling this)
CEOS WGISS41

9. Using tidal models to map tidal extents
Tidal Range of >10m
Tidal Zone Extent
Can be attributed with offsets of LAT to lowest observed tide and HAT to highest observed
Tidal Zone Morphology
Fraction of water observations over the time series. Can we attribute this with depths?
CEOS WGISS41

10. Using tidal models to map tidal extents
Time-series analysis - calculating median from acquisitions at low (right) and high (right) tide models
Dr Stephen Sagar
Instead of the median pixel value, this image uses the bottom 10% of tide heights to select pixels, producing a generic low-tide image of the entire Australian coast. This is highly sought-after for coastal studies and navigation charting.
Similarly by taking the top 10% of tide heights to select pixels, we can produce the high-tide image of Australia. Once again this is very valuable for coastal studies.
CEOS WGISS41

11. Mangrove change over 25 years
Here we use the Data Cube to create yearly statistical summaries of greeness in the landscape using the Normalised Difference Vegetation Index. The image shows peak greenness for 1987, 2000 and 2014 as blue, green and red. So areas that are blue were green early in the time series while currently are not. Areas that are red are green in 2014 but not earlier. Areas that are white or shades of grey have not changed but have stayed at the same level of greeness through time. In this area it shows how mangroves have colonised some areas and retreated from others over the last 30 years.
CEOS WGISS41

12. Terrestrial applications
Vicarious calibration sites
Identify climatic zones, spatial and temporal variation, and seasonal suitability for calibration activities
Landsat MODIS blending
Blend Landsat and MODIS scenes to produce Landsat-like data (25m resolution) with MODIS repeat cycle (~ every 4-days)
GEOGLAM Rangeland and Pasture Productivity
Rangeland and pasture cover information for ecosystem health and livestock resources globally
CEOS WGISS41

13. Solutions to assist algal management in lakes and rivers
An increasing global problem
Systems under pressure
Human and animal health impacts
AU$ M per annum in bloom impacts
Tim Malthus, Janet Anstee, Hannelie Botha, Stephen Gensemer, Eric Lehmann, Xavier Ho
Fundamentally physically based approach… driven by colour
Development of specific algal alert indices – here for chlorophyll – have had to have made the ranges ourselves.
Information services
CEOS WGISS41

14. Processing Pathway and Decision Support Tools
From remote sensing data to visualization of algal alerts
Select extents from WOFS data
Source latest AGDC dataset(s)
Apply pixel quality and optical filters
Apply algal alert algorithm(s)
Produce visualisation products
Moving towards operationalisation
Built around the Australian Geospatial DataCube (Landsat 8)
Software framework / Product delivery system
Rapid turnaround of satellite data streams to red/yellow/green alerts
Able to accommodate new sensors and data streams (S2), new algorithms
CEOS WGISS41

15. AGDC v2 Complete rewrite of AGDC v1 (v2 Deployment July 2016)
Reprocessing of Landsat - improved geolocation, spectral & terrain correction
MODIS refinements to ARD preparation (C005 and prep for C006)
AVHRR national archive move to AGDC under investigation
Hybrid NoSQL influenced database model
Ingest support for multiple sensors
Multidimensional storage - NetCDF4-CF1.6
Multidimensional access - XArray and Dask
Analysis and Production Pipeline Provenance capture
Projection support / Geographic / Sinusoidal / Albers Equal Area etc.
Transition planning - Landsat Archive Reprocessing - featuring terrain corrected Landsat as an additional layer
Analytics / Query Language / Quality and Spectral Semantics
Test it out - github.com/data-cube/agdc-v2/tree/develop - example ingest configurations for MODIS MCD43A1->4, USGS Landsat SR, Himawari 8 BRF from Australian BoM, GA ARG25 NBAR/T products, SRTM DEM.
1. MODIS (and AVHRR) dot point(s)
Refining our acquisition, management and processing workflows for MODIS with a view to Collection 6: Australian L1A archive for NRT and swath-based ocean and terrestrial products, and; USGS/LPDAAC C5 and C6 selected products for global and Australasian applications.
Refactoring our national archiving, calibration and “stitching” activities for Australian-acquired AVHRR towards a low-maintenance and sustainable system and resource.
CEOS WGISS41

16. AGDC v2 Data Ingestion Prepare data for analysis in a couple of steps:
Storage configuration - Projection / resampling / grid specification
Data preparation (assuming Analysis Ready Data input) - write configuration file allowing datacube to interpret input data
Ingest
Stack
Analyse
TODO - simplify data preparation - currently using python template
CEOS WGISS41

17. AGDC v2 Projection and Sensor Support
Flexible storage unit representation
Ability to access data in native or lat/lon projection coordinates through API
MODIS MCD43A4 tile in sinusoidal projection, Sentinel-2 L1C and Landsat storage units in Albers Equal Area
South Eastern Australia - At right: Sinusoidal projection - At left: Australian Albers Equal Area Projection
CEOS WGISS41

18. Open Data and Code AGDC Web: http://www.datacube.org.au
AGDC Wiki:
Code repositories are available through GitHub:
CEOS WGISS41

19. OGC Discrete Global Grid Systems (DGGS) SWG
DGGS an important part of the future versions of Datacube
OGC DGGS SWG to begin elaboration of Extension Standards to the DGGS Core Standard
Anticipated Extensions include:
Interoperability interface protocols for OGC Web Services (e.g. WCS, WCPS, WCTiles, etc…) to facilitate DGGS-to-DGGS communication and processing
3D (and higher dimensional) DGGS Specifications
Best Practice Guide
CEOS WGISS41

20. Future: WGISS involvement
AWS - S3 object storage study
The Future: Data Cubes and WGISS
Discrete Global Grid Systems - OGC SWG and implications for AGDC evolution
CEOS WGISS41