VO Sandpit, November 2009 e-Infrastructure for Climate and Atmospheric Science Research Dr Matt Pritchard Centre for Environmental Data Archival (CEDA)

Slides:

Advertisements

Similar presentations

Computing Infrastructure

Advertisements

BADC Workshop 1: Data & Services from the BADC Royal Met. Soc. Conference – 12 September 2005 Kevin Marsh et al.

VO Sandpit, November 2009 CEDA Storage Dr Matt Pritchard Centre for Environmental Data Archival (CEDA)

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Center for Computational Sciences Cray X1 and Black Widow at ORNL Center for Computational.

Saurabh Bhardwaj Centre for Global Environment Research, Earth Science and Climate Change Division Ongoing Climate

VO Sandpit, November 2009 NERC Big Data And what’s in it for NCEO? June 2014 Victoria Bennett CEDA (Centre for Environmental Data Archival)

(A)ATSR archive at the NERC Earth Observation Data Centre Matt Pritchard, Jack Abolins, Brian Maddison & Chris Mutlow NEODC / CCLRC Rutherford Appleton.

Adam Duffy Edina Public Schools.  The heart of virtualization is the “virtual machine” (VM), a tightly isolated software container with an operating.

LinkSCEEM-2: A computational resource for the development of Computational Sciences in the Eastern Mediterranean Mostafa Zoubi SESAME SESAME – LinkSCEEM.

GRID Activities at ESAC Science Archives and Computer Engineering Unit Science Operations Department ESA/ESAC – Madrid, Spain.

ASKAP Central Processor: Design and Implementation Calibration and Imaging Workshop 2014 ASTRONOMY AND SPACE SCIENCE Ben Humphreys | ASKAP Software and.

Bill Wrobleski Director, Technology Infrastructure ITS Infrastructure Services.

Introduction to DoC Private Cloud

Dr. Ognjen Prnjat European and Regional eInfrastructure management Greek Research and Technology Network Oceanos and synnefo - clouds.

Virtual Network Servers. What is a Server? 1. A software application that provides a specific one or more services to other computers  Example: Apache.

Cluster computing facility for CMS simulation work at NPD-BARC Raman Sehgal.

JASMIN Petascale storage and terabit networking for environmental science Matt Pritchard Centre for Environmental Data Archival RAL Space Jonathan Churchill.

VO Sandpit, November 2009 CEDA Mission: “curation and facilitation” “Managing complex datasets and accompanying information for reuse and repurpose” Sam.

Virtualisation Cloud Computing at the RAL Tier 1 Ian Collier STFC RAL Tier 1 HEPiX, Bologna, 18 th April 2013.

CEMS: The Facility for Climate and Environmental Monitoring from Space Victoria Bennett, ISIC/CEDA/NCEO RAL Space.

JASMIN Overview UKMO Visit 24/11/2014 Matt Pritchard.

JASMIN/CEMS and EMERALD Scientific Computing Developments at STFC Peter Oliver, Martin Bly Scientific Computing Department Oct 2012.

Climate Sciences: Use Case and Vision Summary Philip Kershaw CEDA, RAL Space, STFC.

1 T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES METEOROLOGICAL DATA MANAGEMENT Mustafa Sert October 2011.

Redefining the Desktop Stu Baker AUL for Library Technology

Planning and Designing Server Virtualisation.

VO Sandpit, November 2009 e-Infrastructure to enable EO and Climate Science Dr Victoria Bennett Centre for Environmental Data Archival (CEDA)

12th November 2003LHCb Software Week1 UK Computing Glenn Patrick Rutherford Appleton Laboratory.

NOCS, PML, STFC, BODC, BADC The NERC DataGrid = Bryan Lawrence Director of the STFC Centre for Environmental Data Archival (BADC, NEODC, IPCC-DDC.

Adam Duffy Edina Public Schools.  Traditional server ◦ One physical server ◦ One OS ◦ All installed hardware is limited to that one server ◦ If hardware.

Sandor Acs 05/07/

JASMIN and CEMS: The Need for Secure Data Access in a Virtual Environment Cloud Workshop 23 July 2013 Philip Kershaw Centre for Environmental Data Archival.

S&T IT Research Support 11 March, 2011 ITCC. Fast Facts Team of 4 positions 3 positions filled Focus on technical support of researchers Not “IT” for.

14 Aug 08DOE Review John Huth ATLAS Computing at Harvard John Huth.

RAL Site Report Andrew Sansum e-Science Centre, CCLRC-RAL HEPiX May 2004.

ARGONNE NATIONAL LABORATORY Climate Modeling on the Jazz Linux Cluster at ANL John Taylor Mathematics and Computer Science & Environmental Research Divisions.

ITEP computing center and plans for supercomputing Plans for Tier 1 for FAIR (GSI) in ITEP  8000 cores in 3 years, in this year  Distributed.

Tier1 Hardware Review Martin Bly HEPSysMan - RAL, June 2013.

Virtualisation & Cloud Computing at RAL Ian Collier- RAL Tier 1 HEPiX Prague 25 April 2012.

RAL Site Report HEPiX FAll 2014 Lincoln, Nebraska October 2014 Martin Bly, STFC-RAL.

- Vendredi 27 mars PRODIGUER un nœud de distribution des données CMIP5 GIEC/IPCC Sébastien Denvil Pôle de Modélisation, IPSL.

Scientific Advisory Committee – September 2011COLA Information Systems COLA’s Information Systems 2011.

IPCC TGICA and IPCC DDC for AR5 Data GO-ESSP Meeting, Seattle, Michael Lautenschlager World Data Center Climate Model and Data / Max-Planck-Institute.

1 Accomplishments. 2 Overview of Accomplishments  Sustaining the Production Earth System Grid Serving the current needs of the climate modeling community.

CBM Computing Model First Thoughts CBM Collaboration Meeting, Trogir, 9 October 2009 Volker Friese.

Power and Cooling at Texas Advanced Computing Center Tommy Minyard, Ph.D. Director of Advanced Computing Systems 42 nd HPC User Forum September 8, 2011.

Welcome to the PRECIS training workshop

DiRAC-3 – The future Jeremy Yates, STFC DiRAC HPC Facility.

RAL Site Report HEPiX Spring 2012, Prague April Martin Bly, STFC-RAL.

RAL Site Report HEPiX Spring 2015 – Oxford March 2015 Martin Bly, STFC-RAL.

Support to scientific research on seasonal-to-decadal climate and air quality modelling Pierre-Antoine Bretonnière Francesco Benincasa IC3-BSC - Spain.

Sensors and Instrumentation Computational and Data Challenges in Environmental Modelling Dr Peter M Allan Director, Hartree Centre, STFC.

Using a Simple Knowledge Organization System to facilitate Catalogue and Search for the ESA CCI Open Data Portal EGU, 21 April 2016 Antony Wilson, Victoria.

© 2014 kCura. All rights reserved. vCloud Hybrid Services VMUG

Extending Auto-Tiering to the Cloud For additional, on-demand, offsite storage resources 1.

Introduction to Data Analysis with R on HPC Texas Advanced Computing Center Feb

A41I-0105 Supporting Decadal and Regional Climate Prediction through NCAR’s EaSM Data Portal Doug Schuster and Steve Worley National Center for Atmospheric.

EGI-InSPIRE RI EGI Compute and Data Services for Open Access in H2020 Tiziana Ferrari Technical Director, EGI.eu

© Thomas Ludwig Prof. Dr. Thomas Ludwig German Climate Computing Center (DKRZ) University of Hamburg, Department for Computer Science (UHH/FBI) Disks,

EGI-InSPIRE RI An Introduction to European Grid Infrastructure (EGI) March An Introduction to the European Grid Infrastructure.

High Performance Computing (HPC)

Workshop on the Future of Big Data Management June 2013 Philip Kershaw

HEPiX Spring 2014 Annecy-le Vieux May Martin Bly, STFC-RAL

CoSeC: Computational Science Centre for Research Communities

Welcome! Thank you for joining us. We’ll get started in a few minutes.

JASMIN Success Stories

Climate Data Analytics in a Big Data world

Introduction to RDS Datasets

Data Curation in Climate and Weather

Presentation transcript:

VO Sandpit, November 2009 e-Infrastructure for Climate and Atmospheric Science Research Dr Matt Pritchard Centre for Environmental Data Archival (CEDA)

VO Sandpit, November 2009 Centre for Environmental Data Archival BADC NEODC UKSSDC IPCC DDC

VO Sandpit, November 2009 Centre for Environmental Data Archival CEDA Data ProjectTypeCurrent volume (Tb) NEODCEarth Observation300 BADCAtmospheric Science350 CMIP5Climate Model350 Total1000 Tb = 1 Pb

VO Sandpit, November 2009 EO data Sea Surface Temperature from space

VO Sandpit, November 2009 Centre for Environmental Data Archival CEDA Users

VO Sandpit, November 2009 Centre for Environmental Data Archival CEDA Users

VO Sandpit, November 2009 Centre for Environmental Data Archival CEDA Activities

VO Sandpit, November 2009 e-Infrastructure e-Infrastructure Investment JASMINCEMS

VO Sandpit, November 2009 JASMIN/CEMS Data ProjectJASMINCEMS NEODC Current300 BADC Current350 CMIP5 Current350 CEDA Expansion200 CMIP5 Expansion CORDEX300 MONSooN Shared Data 400 Other HPC Shared Data 600 User Scratch Totals3500 Tb1100 Tb

VO Sandpit, November 2009 JASMIN functions CEDA data storage & services Curated data archive Archive management services Archive access services (HTTP, FTP, Helpdesk,...) Data intensive scientific computing Global / regional datasets & models High spatial, temporal resolution Private cloud Flexible access to high-volume & complex data for climate & earth observation communities Online workspaces Services for sharing & collaboration

VO Sandpit, November 2009 Use cases Processing large volume EO datasets to produce: Essential Climate Variables Long term global climate-quality datasets EO data validation & intercomparisons Evaluation of models relying on the required datasets (EO datasets & in situ ) and simulations) being in the same place

VO Sandpit, November 2009 Use cases User access to 5 th Coupled Model Intercomparison Project (CMIP5) Large volumes of data from best climate models Greater throughput required Large model analysis facility Workspaces for scientific users. Climate modellers need 100s of Tb of disk space, with high-speed connectivity UPSCALE project 250 Tb in 1 year PRACE supercomputing facility in Germany (HERMIT) Being shipped to RAL at present To be analysed by Met Office as soon as available Deployment of VMs running custom scientific software, co-located with data Outputs migrated to long term archive (BADC)

VO Sandpit, November 2009 JASMIN locations JASMIN-West University of Bristol 150 Tb JASMIN-North University of Leeds 150 Tb JASMIN-South University of Reading 500 Tb + compute JASMIN-Core STFC RAL 3.5 Pb + compute

VO Sandpit, November 2009 JASMIN kit

VO Sandpit, November 2009 JASMIN kit

VO Sandpit, November 2009 JASMIN kit

VO Sandpit, November 2009 JASMIN kit JASMIN/CEMS Facts and figures JASMIN: 3.5 Petabytes Panasas Storage 12 x Dell R610 (12 core, 3.0GHz, 96G RAM)Servers 1 x Dell R815 (48 core, 2.2GHz, 128G RAM)Servers 1 x Dell Equalogic R6510E (48 TB iSCSI VMware VM image store) VMWare vSphere Center 8 x Dell R610 (12 core, 3.5GHz, 48G RAM) Servers 1 x Force10 S4810P 10GbE Storage Aggregation Switch

VO Sandpit, November 2009 JASMIN kit JASMIN/CEMS Facts and figures CEMS: 1.1 Petabytes Panasas Storage 7 x Dell R610 (12 core 96G RAM) Servers 1 x Dell Equalogic R6510E (48 TB iSCSI VMware VM image store) VMWare vSphere Center + vCloud Director

VO Sandpit, November 2009 JASMIN kit JASMIN/CEMS Facts and figures Complete 4.5 PB (usable - 6.6PB raw) Panasas storage managed as one store, consisting of: 103 4U “Shelves” of 11 “Storage Blades” 1,133 (-29) “Storage Blades” with 2x 3TB drives each 2, " Disc Drives (3TB Each) 103 * 11 * = 1,104 CPUs (Celeron 1.33GHz CPU w. 4GB RAM) 29 “Director Blades” with Dual Core Xeon 1.73GHz w.8GB RAM) 15 kW Power in / heat out per rack = 180 kW (10-20 houses worth) 600kg per rack = 7.2 Tonnes 1.03 Tb/s total storage bandwidth = Copying 1500 DVDs per minute 4.6PB Useable == 920,000 DVD's = a 1.47 km high tower of DVDs 4.6PB Useable == 7,077,000 CDs = a 11.3 km high tower of CDs

VO Sandpit, November 2009 JASMIN links

Thank you!