We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byZaria Bilton
Modified over 2 years ago
© Crown copyright Met Office Met Office Unified Model I/O Server Paul Selwood
© Crown copyright Met Office I/O Server motivation
© Crown copyright Met Office Some History… I/O has always been a problem for NWP, more recently for climate ~2003 – application level output buffering ~2008 – very simple, single threaded I/O servers added for benchmarking Intercepted low-level “open/write/close” Single threaded Some benefit, but limited Not addressed scaling issues – message numbers
© Crown copyright Met Office Old UM I/O – Restart Files
© Crown copyright Met Office Old UM I/O - Diagnostics
© Crown copyright Met Office Why I/O Server approach? Full parallel I/O difficult with our packing “Free” CPUs available “Spare” memory available Chance to re-work old infrastructure Our file format is neither GRIB or netCDF.
© Crown copyright Met Office Diagnostic flexibility Variables (primary and derived) Output times Temporal processing (e.g. accumulations, extrema, means) Spatial processing (sub-domains, spatial means) Variable to unit mapping Basic output resolution is a 2D field
© Crown copyright Met Office Key design decisions Parallelism over output streams Output streams distributed over servers Server is threaded “Listener” receives data & puts in queue “Writer” processes queue including packing Ensures asynchronous behaviour Shared FIFO queue Preserves instruction order Metadata/Data split Data initially stored on compute processes Data of same type combined into large messages
© Crown copyright Met Office Parallelism in I/O Servers Multiple I/O streams in typical job I/O servers spread among nodes Can utilise more memory Will improve bandwidth to disk
© Crown copyright Met Office Automatic post-processing Model can trigger automatic post-processing Requests dealt with by I/O Server FIFO queue ensures integrity of data
© Crown copyright Met Office How data gets output ComputeI/O ListenerWriter Thread 0 Thread 1
© Crown copyright Met Office I/O Server development Initial version – Synchronous data transmission Asynchronous diagnostic data Asynchronous restart data Amalgamated data Asynchronous metadata Load balancing Priority messages with I/O Server
© Crown copyright Met Office Lots of diagnostic output Which processes are I/O servers “Stall” messages Memory log Timing log Full log of metadata / queue All really useful for tuning!
© Crown copyright Met Office Lots of tuneable parameters… Number and spacing of I/O servers Memory for I/O servers Number of local data copies Number of fields to amalgamate Load balancing options Timing tunings + standard I/O tunings (write block size) etc
© Crown copyright Met Office Overloaded servers
© Crown copyright Met Office I/O Servers keeping up!
© Crown copyright Met Office MPI considerations Differing levels of MPI threading support Best with MPI_THREAD_MULTIPLE OK with MPI_THREAD_FUNNELED MPI tuning Want metadata to go as quickly as possible Want data transfer to be truly asynchronous Don’t want to interfere with model comms (e.g. halo exchange) Currently use 19 environment variables!
© Crown copyright Met Office Deployment July 2011 – Operational global forecasts January 2012 – Operational LAM forecasts February 2012 – High resolution climate work Not currently used in Operational ensembles Low resolution climate work Most research work
© Crown copyright Met Office Global Forecast Improvement QG 00/12 QG 06/18 QU Time 777s559s257s %age 19%28%27% Total saving: over 21 node-hours per day
© Crown copyright Met Office Impact on High Resolution Climate N512 resolution AMIP 59 GB restart dumps Modest diagnostics Cray XE6 with up to 9K cores All “in-run” output hidden Waits for final restart dump Most data buffered on client side
© Crown copyright Met Office Current and Future Developments MPI Parallel I/O servers Multiple I/O servers per stream Gives more memory per stream on server Reduced messaging rate per node Parallel packing Potential for parallel I/O Read ahead Potential for boundary conditions / forcings Some possibilities for initial condition
© Crown copyright Met Office Parallel I/O server improvement Before After
© Crown copyright Met Office Questions and answers
Achieving High Availability with SQL Server using EMC SRDF Prem Mehra – SQL Server Development, Microsoft Art Ullman - CSC.
CEA/DSM/LSCE – Yann Meurdesoif 01/10/ XIOS Yann Meurdesoif(LSCE-IPSL), H. Ozdoba, A. Caubel, O. Marti.
Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China
DC-API: Unified API for Desktop Grid Systems Gábor Gombás MTA SZTAKI.
1 CSE 380 Computer Operating Systems Instructor: Insup Lee University of Pennsylvania Fall 2003 Lecture Notes: Multiprocessors (updated version)
Silberschatz, Galvin and Gagne ©2010 Operating System Concepts Essentials – 8 th Edition Chapter 16: Windows 7.
Advanced data management Jiaheng Lu Department of Computer Science Renmin University of China
Multiple Processor Systems Bits of Chapters 4, 10, 16 Operating Systems: Internals and Design Principles, 6/E William Stallings.
RMS and Scheduling for Future Generation Grids Ramin Yahyapour University Dortmund Leader CoreGRID Institute on Resource Management and Scheduling CoreGRID.
1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 1 Why Parallel Computing? An Introduction to Parallel Programming Peter Pacheco.
Copyright 2004 Bernd Brügge TUM Software Engineering WS TUM System Design II Bernd Brügge Technische Universität München Applied Software Engineering.
1. Best Practices for Oracle Database Performance on Windows Christian Shay Principal Product Manager, Oracle.
1 Operating-System Structures Operating System & its purpose Operating Systems lead to new Hardware Features System Components System Services System Calls.
Multiprocessing and NUMA. What we sort of assumed so far… Northbridge connects CPU and memory to rest of system – Memory controller implemented in Northbridge.
The Client/Server Database Environment CS263 Lecture 12.
Database System Concepts, 6 th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 10: Storage and.
SCSC 311 Information Systems: hardware and software.
National Aeronautics and Space Administration Optimizing Lustre Performance Using Stripe-Aware Tools Paul Kolano NASA Advanced Supercomputing.
1 Computer Networks: A Systems Approach, 5e Larry L. Peterson and Bruce S. Davie Chapter 9 Applications Copyright © 2010, Elsevier Inc. All rights Reserved.
DBMS Storage and Indexing. Disk Storage Disks and Files DBMS stores information on (“hard”) disks. This has major implications for DBMS design! ▫ READ:
Parallel I/O from a Users Perspective HPC Advisory Council Stanford University, Dec. 6, 2011 Katie Antypas Group Leader, NERSC User Services.
1 Making Parallel Processing on Clusters Efficient, Transparent and Easy for Programmers Andrzej M. Goscinski School of Computing and Mathematics Deakin.
1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 2 Parallel Hardware and Parallel Software An Introduction to Parallel Programming Peter Pacheco.
What is an Operating System? A program that acts as an intermediary between a user of a computer and the computer hardware. Operating system goals: Execute.
1 System Busses / Networks-on-Chip EECE Advanced Topics in VLSI Design Spring 2009 Brad Quinton.
1 Process Description and Control Chapter 2. 2 Process A program in execution An instance of a program running on a computer The entity that can be assigned.
1 Advanced Database Application Development Performance Tuning Performance Benchmarks Standardization E-Commerce Legacy Systems.
Version 4.1 CCNA Discovery 2– Chapter 7. Contents 7.1: ISP Services : TCP / IP Protocols 7.2: 7.3: DNS 7.3: 7.4: Application Layer Protocols 7.4.
Database System Concepts, 5th Ed. ©Silberschatz, Korth and Sudarshan See for conditions on re-usewww.db-book.com Chapter 20: Database System.
Vipul Patel Ideas … Please …
© 2016 SlidePlayer.com Inc. All rights reserved.