Presentation on theme: "Early Access to NCI Climate Data & Analysis Systems Ben Evans"— Presentation transcript:
Early Access to NCI Climate Data & Analysis Systems Ben Evans
NCI: Vision and Role Vision – Provide Australian researchers with a world-class, high-end computing service Aim – To make a real difference to the conduct, ambition and outcomes of leading Australian research Role – Sustain / develop Australian capability computing targeted to Earth Systems Science – Provide comprehensive computational support for data-intensive research – Provide specialised support for key research communities – Develop and maintain a strategic plan to support the advancement of national high-end computing services – Provide access through partner shares, national merit allocation scheme Partners – ANU, CSIRO, Geoscience Australia, Bureau of Meteorology (2012),
Providing capability in modelling & data-intensive science Peak Infrastructure – Vayu—Sun Constellation – commissioned 2010 – 140 Tflops, 240K SPEC, 36 TBytes/800 TBytes – Well balanced; Good performance – Batch oriented, parallel processing Data Storage Cloud – Large/fast/reliable data storage – Persistent disk-resident or tape – Relational databases – National datasets and support for Virtual Observatories – New Storage Infrastructure (Nov, 2011) – Dual site storage – Dual site data services Data Intensive Computing – Data analysis – Data Compute Cloud (2011)
Options for data services Data Cloud spans two physical sites Internal network completed June 2011 Redundant 10 GigE network links to AARNet completed Aug 2011 Data migration from old tape silo to disk mid-Sept Floor replacement and additional pod cooling completed end Sept Storage architecture physical layer acceptance due end Oct 2011
Options for data services HSM (tape) – 1,2 copies at 2 locations Scratch disk – one location, 2 speed options Persistent disk – two locations, 2 speed options Persistent data services – two locations, movable/sychronised VMs Self-managed backup, synchronised data, or HSM Filesystems layered on top of hardware options. Specialised: Databases, Storage Objects, HDFS, … Domain speciality: ESG, THREDDS/OpenDAP, Workflows, Data Archives (OAIS)
NCI: Cloud Computing for Data Intensive Science Special Features for the Data-Intensive Cloud Integration into NCI Environment Fast access to: Storage, Filesystems and Significant Datasets Compute (and some GPU) Cloud Environments – deployed on demand NF software – large repository Integrated data analysis tools and visualisation Networks – general (AARNet, CSIRO, GA, …), – direct (HPC Computing, Telescopes, Gene Sequencers, CT scanners, …, International Federations) More features....
Astronomy Virtual Observatory X-ray Optical Radio An era of cross-fertilisation - match surveys that span the electromagnetic spectrum ANU’s SkyMapper telescope - providing the world’s first digital map of the southern sky. the measurements of brightness and position will form the basis of countless science programs and calibrate other future surveys Less time at the telescope - more time in front of the computer!
Earth Observations Workflow Eg. National Environmental Satellite Data Backbone at NCI – CSIRO, GA Earth Observing (EO) sensors carried on space-borne platforms produce large (multiple TB/year) data sets serving multiple research and application communities The NCI established a single National archive of raw (unprocessed) MODIS data for the Australian region, and to support processing software (common to all users) and specialised tools for applications. LANDSAT is now being processed. The high quality historical archive is complemented by exploiting the NCI network connectivity to download and merge data acquired directly from the spacecraft by local reception stations all round Australia in real time. Data products and tools available through web technologies and embedded workflows. Collaborators: King, Evans, Lewis, Wu, Lymburner
An Earth Systems Virtual Laboratory ESG – internationally significant climate model data Data analysing capability …
Earth Systems Grid and Access to Analysis Mission: The DOE SciDAC-2 Earth System Grid Center for Enabling Technologies (ESG-CET) project is to provide climate researchers worldwide with access to: data, information, models, analysis tools, and computational resources required to make sense of enormous climate simulation datasets. ESGF – Federation of worldwide sites providing data. Core nodes are: PCMDI (LLNL), BADC (UK), DKRZ (MPI), NCAR/JPL. NCI joined to provide an Australian Node. NCI : Support the ESG as the Australian node and subsequent processing – Support publishing of Australian data as a primary node – Store/Replicate priority overseas data – Provide associated compute/storage services through NCI shares.
Methods to get climate data Providing two main methods to access climate data: 1.ESG portal web site 2.Filesystem on the NCI data cloud Providing computational systems for analysing the data: 1.dcc – data compute cloud 2.…
Current Status Status of Publishing - CSIRO-QCCCE mk3.6 Models run at QCCCE facility and data transferred to NCI Processed data and through L1 checks (check) Eg: 7 Tbytes of final output required 90+ Tbytes of temporary space on Vayu and 200 Tbytes on the data cloud Moved data between vayu and dc – time consuming Currently completing rest of data processing. Expect final size to be ~30 Tbytes Status of Publishing – CAWCR ACCESS Preindustrial runs commencing by early October 500 years at 250 year intervals First publish by end of Dec (20Tbytes). Second release around Feb. Status of ESG software Stable release and serving data Federated Identity/authorisation systems are evolving Continual updates (data replication not yet working within ESG stack)
CMIP5 Modelling groups and data being generated CAWCR - Centre for Australian Weather and Climate Research CCCMA - Canadian Centre for Climate Modelling and Analysis CCSM - Community Climate System Model CMA-BCC - Beijing Climate Center, China Meteorological Administration CMCC - Centro Euro-Mediterraneo per I Cambiamenti Climatici CNRM-CERFACS - Centre National de Recherches Meteorologiques - Centre Europeen de Recherche et Formation Avancees en Calcul Scientifique. EC-Earth - Europe FIO - The First Institute of Oceanography, SOA, China GCESS - College of Global Change and Earth System Science, Beijing Normal University GFDL - Geophysical Fluid Dynamics Laboratory INM - Russian Institute for Numerical Mathematics IPSL - Institut Pierre Simon Laplace LASG - Institute of Atmospheric Physics, Chinese Academy of Sciences China MIROC - University of Tokyo, National Institute for Environmental Studies, and Japan Agency for Marine- Earth Science and Technology MOHC - UK Met Office Hadley Centre MPI-M - Max Planck Institute for Meteorology MRI - Japanese Meteorological Institute NASA GISS- NASA Goddard Institute for Space Studies USA NCAR - US National Centre for Atmospheric Research NCAS - -UK National Centre for Atmospheric Science NCC - Norwegian Climate Centre NIMR - Korean National Institute for Meteorological Research QCCCE-CSIRO - Queensland Climate Change Centre of Excellence and Commonwealth Scientific and Industrial Research Organisation RSMAS - University of Miami - RSMAS 24 modelling groups, 25 platforms being described, 44 models, 65 grids, and 223 simulations
Simulations: ~90,000 years ~60 experiments within CMIP5 ~20 modelling centres (from around the world) using ~several model configurations each ~2 million output “atomic” datasets ~10's of petabytes of output ~2 petabytes of CMIP5 requested output ~1 petabyte of CMIP5 “replicated” output Which will be replicated at a number of sites (including ours), arriving now! Of the replicants: ~ 220 TB decadal ~ 540 TB long term ~ 220 TB atmos-only ~80 TB of 3hourly data ~215 TB of ocean 3d monthly data! ~250 TB for the cloud feedbacks! ~10 TB of land-biochemistry (from the long term experiments alone). Slide sourced from Metafor web site early CMIP5 Modelling groups and data being generated
Summary Modeling centers13 Models17 Data nodes13 Gateways5 Datasets15,950 Size177.4 TB Files40,7792 CMIP5 Archive Status – automatically generated Last Update: Monday, 19 September :21AM (UTC)
ProtocolNumber of Centers Number of Models Number of Datasets Size (TB) HTTP131915, GridFTP OPeNDAP Datasets by Access Protocol
Modelling Centre ModelCapacity (TB) CSIRO-QCCCECSIRO-Mk3.6~15 IPSLIPSL-CM5A-LR9.4 INMinmcm46.3 CMIP3All35.4 AllHigh Priority Variables - Lawson 7.14 Datasets on our data cloud
Status of Data Replication
Data Replication Data Replication supported by two methods 1.Bulk-fast transfers by ESG nodes 2.User-initiated data transfers at variable level First method is fast but requires coordination at the sites and international networks Second method can be very slow but relatively “simple” We will provide more details during our session today.
Data Processing Capability We have provided a first iteration of data processing capability – Early Access mode dcc provides data processing directly on the data cloud Upgrades in place as more hardware becomes available – Filesystem upgrade – mid Nov – Data processing – mid Nov (pending order) – Some software license issues being resolved Data processing pipelines being understood/established. Eg CVC data pipeline
Data Replication Issues: Slow links to some sites can get swamped ESG software not ready to manage official replicas Don’t have a clear view of when model data will be available Data may need to be revoked as errors are found. Data capacity being closely managed/prioritised. CAWCR, CoE, CSIRO, BoM and shareholders monitoring for future expansion
Keeping in touch ESG status messages / ESG Federation web site Twitter feeds: Open weekly Townhall Q/A meeting – details to be established. Regular meetings between CAWCR, CoE and NCI on status. Keep your Team leaders advised on your requirements/issues. More developments planned through a VL proposal.