E-infrastructure shared between Europe and Latin America www.eu-eela.org 1 EELA is a project funded by the European Union under contract 026409 E-Infraestructure.

Slides:

Advertisements

Similar presentations

EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,

Advertisements

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

IST E-infrastructure shared between Europe and Latin America Climate Application Final Report Jose M. Gutierrez Valvanuz Fernandez.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA Project Overview Jesus Casado Project.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim

Toni Saarinen, Tite4 Tomi Ruuska, Tite4 Earth System Grid - ESG.

E-infrastructure shared between Europe and Latin America 1 EELA is a project funded by the European Union under contract E-Infraestructure.

DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The EELA Project Rafael Mayo CIEMAT IBERGRID.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug

Session 7a, 10 May 2007 IST-Africa 2007 Copyright 2007 EELA Project: IST

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA Project Overview Pedro Henrique Rausch.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA: E-Infrastructure shared between Europe.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Pilot Test-bed Operations and Support Work.

E-science grid facility for Europe and Latin America EELA-2 and beyond it Bernard M. Marechal CETA-CIEMAT & UFRJ (SPAIN & BRAZIL) 2 nd EELA-2.

4th EGEE Meeting, PISA October, 2005 E-Infraestructure shared between Europe and Latin America Jesús Casado, CIEMAT

IST E-infrastructure shared between Europe and Latin America Biomedical Applications in EELA Esther Montes Prado CIEMAT (Spain)

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA: THE FIRST EIGHT MONTHS Bernard M.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Aplicaciones GRID en el proyecto EELA Rafael.

GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.

ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.

Luis A. Trejo (ITESM) ICT Day Opportunities to participate in EU ICT research projects San José, 16 February 2010 Mexico participation.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America WP3: Identification and support of Grid.

Computational grids and grids projects DSS,

ESG The Earth System Grid (ESG) Presented by Don Middleton & Luca Cinquini NCAR Scientific Computing Division On Behalf of the ESG Team SCD Executive Committee.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The EELA Project Giuseppe Andronico INFN.

1 Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA at an end EELA-2 before its onset Bernard.

L ABORATÓRIO DE INSTRUMENTAÇÃO EM FÍSICA EXPERIMENTAL DE PARTÍCULAS Enabling Grids for E-sciencE Grid Computing: Running your Jobs around the World.

IST E-infrastructure shared between Europe and Latin America High Energy Physics Applications in EELA Raquel Pezoa Universidad.

November SC06 Tampa F.Fanzago CRAB a user-friendly tool for CMS distributed analysis Federica Fanzago INFN-PADOVA for CRAB team.

IST E-infrastructure shared between Europe and Latin America Climate Application Jose M. Gutierrez Valvanuz Fernandez Antonio.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The EELA Project presented by Jesús Marco.

INFSO-RI Enabling Grids for E-sciencE Status of the Biomedical Applications in EELA Project (E-Infrastructures Shared Between Europe.

Intergrid KoM Santander 22 june, 2006 E-Infraestructure shared between Europe and Latin America José Manuel Gutiérrez

Ames Research CenterDivision 1 Information Power Grid (IPG) Overview Anthony Lisotta Computer Sciences Corporation NASA Ames May 2,

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Applications in EELA Rafael Mayo CIEMAT.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Applications in the EELA Project Rafael.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America 1 EELA: E-Infrastructure shared between Europe and Latin.

The Earth System Grid: A Visualisation Solution Gary Strand.

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) The ESG and NCAR Web Portals Luca Cinquini NCAR, ESG Outline: 1.ESG Data Services.

Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.

The Earth System Grid (ESG) Computer Science and Technologies DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America EELA Infrastructure (WP2) Roberto Barbera.

GRIDS Center Middleware Overview Sandra Redman Information Technology and Systems Center and Information Technology Research Center National Space Science.

INFSO-RI Enabling Grids for E-sciencE OSG-LCG Interoperability Activity Author: Laurence Field (CERN)

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Wrote by Jorge Gomes and presented by Bruno.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America High Energy Physics Applications in EELA.

IST E-infrastructure shared between Europe and Latin America WP1 FINANCIAL PROCEDURES Raquel Muñoz Félix Barrio Tutorial Santiago.

Introduction to EELA Antonio Fuentes Tutorial Grid Madrid, May 2004.

E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA gLite Information System Pedro Rausch IF.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Grid Applications in the EELA Project Geneviève.

E-science grid facility for Europe and Latin America Grid infrastructure for e-Science: a use case from Latin America and Europe Bernard.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America gLite Information System Claudio Cherubino.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

Università di Perugia Enabling Grids for E-sciencE Status of and requirements for Computational Chemistry NA4 – SA1 Meeting – 6 th April.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Applications in the EELA Project Rafael.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The EELA Grid Infrastructure and HEP Applications.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America WP3 General Overview Rafael Mayo CIEMAT.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The EELA Grid Infrastructure Roberto Barbera.

13th EELA Tutorial, La Antigua, 18-19, October E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA

Tutorial on grid, visualization and high performance calculation FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin.

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America The GILDA t-Infrastructure Roberto Barbera.

Co-ordination & Harmonisation of Advanced e-Infrastructures for Research and Education Data Sharing Grant.

IST E-infrastructure shared between Europe and Latin America The GILDA t-Infrastructure and the GENIUS portal Christian Grunfeld,

EGEE is a project funded by the European Union under contract IST GENIUS and GILDA Guy Warner NeSC Training Team Induction to Grid Computing.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.

Grid Computing: Running your Jobs around the World

The GENIUS portal and the GILDA t-Infrastructure

Presentation transcript:

E-infrastructure shared between Europe and Latin America 1 EELA is a project funded by the European Union under contract E-Infraestructure shared between Europe and Latin America A. S. Cofiño (1), M. Carrillo (2), C. Baeza (3), J. Fernández (1), R. M. San-Martín (2), R. Abarca (3) and J. M. Gutiérrez (1) on behalf of the EELA team. (1) Dept of applied mathematics and computing sciences. University of Cantabria. Spain, (2) Servicio Nacional de Meteorología e hidrología. Peru, (3) Universidad de Concepción. Chile GRID distributed computation of nested climate simulations. The project EELA.org

E-infrastructure shared between Europe and Latin America 2 GRID Computing Applications draw computing power from a Computational Grid in the same way electrical devices draw power from an electrical grid

E-infrastructure shared between Europe and Latin America 3 GRID Computing Developed in the mid-90 Use of distributed, heterogeneous, dynamic and, usually, parallel computational resources. Middleware and standard software to build applications (Globus Toolkit, OGSA, …) Several research projects (and comercial products) developing this technology. Applications draw computing power from a Computational Grid in the same way electrical devices draw power from an electrical grid

E-infrastructure shared between Europe and Latin America 4 EELA Goals E-infrastructure shared between Europe and Latin America Goal: To build a bridge between consolidated e-Infrastructure initiatives in Europe and emerging ones in Latin America. Objectives: –Establish a human collaboration network between Europe and Latin America –Setting a pilot e-infrastructure in Latin America –Identifying and promoting a sustainable framework for e-Science in Latin America

E-infrastructure shared between Europe and Latin America 5 EELA structure EELA is structured in four Working Packages: WP1. Project administrative and technical management WP2. Pilot testbed operation and support WP3. Identification and support of Grid-enhanced applications –Task 3.1. Biomed Applications –Task 3.2. High Energy Physics Applications –Task 3.3. Additional Applications:  E-Learning  Climate WP4. Dissemination activities

E-infrastructure shared between Europe and Latin America 6 Partners EU –Spain: CIEMAT, CSIC, UPV, RED.ES, UC –Italy: INFN –Portugal: LIP Latin America –Venezuela: ULA –Cuba: CUBAENERGIA –Chile: UTFSM, REUNA, UDEC –Peru: SENAMHI –Mexico: UNAM –Argentina: UNLP –Brazil: UFRJ, CNEN, –CECIERJ/CEDERJ, RNP, UFF International –CLARA –CERN

E-infrastructure shared between Europe and Latin America 7 Available resources

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Madrid, First EELA Annual Review, GOAL: Predict local impacts of “El Niño” in Latin America A challenging problem for the climate community, with huge socio-economical impact in Latin America. GRID helps to share computing resources, heterogeneous data, as well as know-how in a user-friendly form. A new integrated climate application developed in EELA from the scratch, with no similar counterpart in any other Earth Science/Climate EU Project. WP 3.3. Climate Task Anomalous heatingAnomalous cooling

E-infrastructure shared between Europe and Latin America 9 CAM & WRF The Community Atmosphere Model (CAM) and the Weather Research and Forecasting (WRF) models are state-of-the-art atmosphere (global and regional) models developed at NCAR. Output format: NetCDF CAM: FORTRAN90 system() calls were introduced into the model to make it move data and status info to the Grid catalogs (LFC and AMGA) WRF: The model does not yet interact with the middleware. Although it runs in the Grid testbed, it is not yet coupled to CAM.

E-infrastructure shared between Europe and Latin America 10 SE Challenging computational problem with nontrivial dependent relationships among the applications. A cascade of dynamic dependent jobs is adopted. The cascade of applications interacts with the middleware to: –Prepare and submit dependent jobs. –Store and retrieve the generated data sets (data sharing). –Manage metadata (for the data sets and application status). –Restarting broken experiments CAM WRF (par 1) WRF (par 2) WRF (par n) … SE SOM Computational challenge SOM … SE SST + other forcings

E-infrastructure shared between Europe and Latin America 11 UI submiter Generate CAM.jdl Submit CAM.jdl Insert entry in CAM collection AMGA Resource Broker Computing element Update status Insert runon CAM collection AMGA CAM Upload info Insert entry in WRF collection AMGA Restart checkpoint Insert entry in CHECKPOINT collection AMGA Update history Insert entry in HISTORYCAM collection AMGA CAM Application Workflow

E-infrastructure shared between Europe and Latin America 12 SE LFC CAM AMGA DATA Metadata status information WRF Info & Data Flow File catalog Storage Element Metadata Catalog

E-infrastructure shared between Europe and Latin America 13 LFC CAM AMGA DATA Metadata WRF Info & Data Flow UI Portal SE status information If a submitted job is not running, the User queries AMGA about whether it was successful. If not, the User checks what was the last restart file and restarts the CAM job. While a CAM job is running, the User queries AMGA about the data sets produced by CAM and then triggers the WRF jobs. And so on with the SOMs.

E-infrastructure shared between Europe and Latin America 14 LFC CAM AMGA R-GMA DATA Metadata WRF To be implemented! Info & Data Flow (future) coordinator UI Portal A coordinator application (unattended and event- driven) should care for successful job completion. SE status information Relational Grid Monitoring Architecture

E-infrastructure shared between Europe and Latin America 15 Current status Using GENIUS to interact with the applications (CAM+WRF). In the future a climate specific portal will be developed (JSR168) to run and track scientific experiments.

E-infrastructure shared between Europe and Latin America 16

E-infrastructure shared between Europe and Latin America 17 The CAM namelist needs to be prepared and provided here by the user. (not very user-friendly, yet)

E-infrastructure shared between Europe and Latin America 18 &camexp absems_data = ‘lfn:/grid/eela/.../abs_ems_factors.nc‘ aeroptics = ‘lfn:/grid/eela/.../AerosolOptics.nc‘ bnd_topo = ‘lfn:/grid/eela/.../topo-from-cami.nc‘ bndtvaer = ‘lfn:/grid/eela/.../AerosolMass.nc' bndtvo = 'lfn:/grid/eela/.../pcmdio3.nc‘ bndtvs = 'lfn:/grid/eela/.../sst_HadOIBl.nc' caseid = 'nino82d' iyear_ad = 1982 start_ymd = ncdata = 'lfn:/grid/eela/.../cami.nc‘ mfilt = 1,4,1...

E-infrastructure shared between Europe and Latin America 19

E-infrastructure shared between Europe and Latin America 20

E-infrastructure shared between Europe and Latin America 21

E-infrastructure shared between Europe and Latin America 22

E-infrastructure shared between Europe and Latin America 23 Currently, the regions are prepared off-line and uploaded to the LFC.

E-infrastructure shared between Europe and Latin America 24 All CAM simulations are available for regionalization with WRF, but the coupler CAM -> WRF is NOT yet implemented

E-infrastructure shared between Europe and Latin America 25 However, we have other input data sets for WRF available in the catalog.

E-infrastructure shared between Europe and Latin America 26 The simulations finished provide access to the results

E-infrastructure shared between Europe and Latin America 27 The output file can be downloaded in NetCDF format or accessed via a THREDDS or OpenDAP aware application.

E-infrastructure shared between Europe and Latin America 28 For instance, the toolsUI java application from Unidata can load the OpenDAP (DODS) address and access only the requested portions of the data

E-infrastructure shared between Europe and Latin America 29 NOAA provides user-friendly access to setup regional domains for WRF

E-infrastructure shared between Europe and Latin America 30 This Java Web Start application could in the future be launched from our web portal as starting point to design a regional simulation

E-infrastructure shared between Europe and Latin America 31 Conclusions The EU-funded EELA project aims at establishing an e- infrastructure and scientific collaboration between european and Latin American countries. Within the climate task, we have implemeted a sequence of climate applications CAM+WRF(+SOM) which runs integrated in a common Grid providing regional simulations for a given SST and other forcings. The applications interact with the middleware uploading data and status information to the Grid catalogs (LFC and AMGA) which can easily be accessed through a web portal. There are still computing issues (CAM-WRF coupler, simulation restarts, coordinator application, more user- friendly model setup) to be solved prior to enter scientific production.

E-infrastructure shared between Europe and Latin America 32 CAM Application: User’s Guide to the NCAR Community Atmosphere Model (CAM 3.0): J. R. McCaa, M. Rothstein, B. E. Eaton, J. M. Rosinski, E. Kluzek, M. Vertenstein: Climate And Global Dynamics Division, NCAR, Boulder, Colorado, 2004, 88 pp. W. D. Collins, C. M. Bitz, et al. (2006) “The Community Climate System Model: CCSM3”, Journal of Climate, Special Issue on CCSM, 19(11). WRF Application: Skamarock, W. C., J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, W. Wang and J. G. Powers, 2005: A Description of the Advanced Research WRF Version 2. NCAR Technical note, 2005, 88 pp. Michalakes, J., J. Dudhia, D. Gill, T. Henderson, J. Klemp, W. Skamarock, and W. Wang, 2004: "The Weather Reseach and Forecast Model: Software Architecture and Performance,"Proceedings of the 11th ECMWF Workshop on the Use of High Performance Computing In Meteorology, October 2004, Reading U.K. Ed. George Mozdzynski. References SOM Application (grid version): F. Luengo, A.S. Cofiño, and J.M. Gutiérrez (2004) “GRID Oriented Implementation of Self-Organizing Maps for Data Mining in Meteorology”, Lecture Notes in Computer Science, 2970, 163 – 171.

E-infrastructure shared between Europe and Latin America 33 Thanks for your attention!

E-infrastructure shared between Europe and Latin America 34 Acronyms Survival guide for reading Grid documents AMGA : ARDA Metadata Grid App. ARDA : A Realisation of Distributed Analysis for LHC CE : Computing element CLARA : Cooperación latino-americana de redes avanzadas EGEE : Enabling Grids for E-sciencE GGF : Global grid forum GIIS : Grid index info service GILDA : Grid INFN laboratory for dissemination activities GMA : Grid monitoring architecture GRAM : Grid res. alloc. manager GRIS : Grid resource info service GSI : Grid security infrastructure INFN : Istituto nazionale di Fisica nucleare JDL : Job description language LCG : LHC computing grid LDAP : Lightweight directory access protocol LFC : Logical file catalog LHC : Large Hadron Collider MDS : Monitoring & discovery system NREN : National Research and Education Network OGSA : OpenGrid services architecture PKI : Public key infrastructure RB : Resource broker R-GMA : Relational GMA SE : Storage element VOMS : Virtual organization membership service WS : Web service

E-infrastructure shared between Europe and Latin America 35 The Earth System Grid TOMCAT Servlet engine TOMCAT Servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server LBNL LLNL ISI NCAR ORNL ANL Striped gridFTP client Striped gridFTP client gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP LAS Live Access Server LAS Live Access Server

E-infrastructure shared between Europe and Latin America 36 CAM. Global simulation Requires metadata catalog so it runs In: UC and CIEMAT (EU), UFRJ (LA). WP 3.3. Climate Demo… WRF. Regional Simulation It runs in any site of the production testbed: UFRJ

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Madrid, First EELA Annual Review, The second year will focus on a scientific challenge: High resolution regional simulations for El Niño and La Niña and strong events. Comparison with historical local data, including sensitivity studies of SST. Scientific Challenge CAM WRF (job 1) WRF (job 2) WRF (job n) … SE SOM SST …

FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Madrid, First EELA Annual Review, CAM Application: User’s Guide to the NCAR Community Atmosphere Model (CAM 3.0): J. R. McCaa, M. Rothstein, B. E. Eaton, J. M. Rosinski, E. Kluzek, M. Vertenstein: Climate And Global Dynamics Division, NCAR, Boulder, Colorado, 2004, 88 pp. W. D. Collins, C. M. Bitz, et al. (2006) “The Community Climate System Model: CCSM3”, Journal of Climate, Special Issue on CCSM, 19(11). WRF Application: Skamarock, W. C., J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, W. Wang and J. G. Powers, 2005: A Description of the Advanced Research WRF Version 2. NCAR Technical note, 2005, 88 pp. Michalakes, J., J. Dudhia, D. Gill, T. Henderson, J. Klemp, W. Skamarock, and W. Wang, 2004: "The Weather Reseach and Forecast Model: Software Architecture and Performance,"Proceedings of the 11th ECMWF Workshop on the Use of High Performance Computing In Meteorology, October 2004, Reading U.K. Ed. George Mozdzynski. References SOM Application (grid version): F. Luengo, A.S. Cofiño, and J.M. Gutiérrez (2004) “GRID Oriented Implementation of Self-Organizing Maps for Data Mining in Meteorology”, Lecture Notes in Computer Science, 2970, 163 – 171.