Virtual Laboratories VGL and Friends R.Fraser, T.Rankine, J.Vote, R.Woodcock AuScope Grid Roadshow 2014 CSIRO | MINERAL RESOURCES FLAGSHIP.

Slides:

Advertisements

Similar presentations

Challenges and Achievements Presented by Cameron Kiddle Research Fellow, Grid Research Centre, University of Calgary.

Advertisements

Information on GVL - Genomics Virtual Laboratory Oct 2013 Audience: Service Desk Developed as part of the Australian.

NG-CHC Northern Gulf Coastal Hazards Collaboratory Simulation Experiment Integration Sandra Harper 1, Manil Maskey 1, Sara Graves 1, Sabin Basyal 1, Jian.

Ryan Fraser (CSIRO), Lesley Wyborn (GA), Richard Chopping (GA), Terry Rankine (CSIRO), Robert Woodcock (CSIRO) MINERALS DOWN UNDER Virtual Geophysics Laboratory.

Simon Woodman Hugo Hiden Paul Watson Jacek Cala. Outline 1. What is e-Science Central? 2. Architecture and Features 3. Workflows and Applications.

A PPARC funded project AstroGrid’s Common Execution Architecture Guy Rixon, reporting on behalf of Paul Harrison and the other AstroGrid developers SC4DEVO-1,

Astrophysics, Biology, Climate, Combustion, Fusion, Nanoscience Working Group on Simulation-Driven Applications 10 CS, 10 Sim, 1 VR.

WMU GNL Automation How to make my IT life easier CHRISTOPHER KEYAERT CONSULTANT AT INOVATIV CLOUD AND DATACENTER MANAGEMENT MVP.

Virtual Geophysics Laboratory (VGL) VGL v1.1 Launch Ryan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock February 2013 CSIRO.

Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close R.Fraser, T.Rankine, J.Vote, L.Wyborn, B.Evans, R.Woodcock, C.Kemp July 2013 CSIRO |

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

Mingfang Wu, Stefanie Kethers, Andrew Treloar Getting from managed to reused: Making it easier for researchers to do something useful with data.

Metadata Creation with the Earth System Modeling Framework Ryan O’Kuinghttons – NESII/CIRES/NOAA Kathy Saint – NESII/CSG July 22, 2014.

WPS Application Patterns at the Workshop “Models For Scientific Exploitation Of EO Data” ESRIN, October 2012 Albert Remke & Daniel Nüst 52°North Initiative.

Teaching and Learning with Open Science Grid quarknet.fnal.gov/e-labs/

Sage Bionetworks Mission Sage Bionetworks is a non-profit organization with a vision to create a “commons” where integrative bionetworks are evolved by.

Virtual Geophysics Laboratory Exploiting the Cloud and Empowering Geophysicists Ryan Fraser, Terry Rankine, Lesley Wyborn, Joshua Vote, Ben Evans. Presented.

Repository Development Projects LeMill & Waramu Tallinn University Centre for Educational Tecnology Estonia.

Publishing and Visualizing Large-Scale Semantically-enabled Earth Science Resources on the Web Benno Lee 1 Sumit Purohit 2

Ryan Fraser, Nicholas Carr Connecting RDSI, NeCTAR and ANDS via Provenance - VHIRL.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

NEPTUNE Canada Workshop Oceans 2.0 Project Environment NEPTUNE Canada DMAS Team Victoria, BC February 16, 2009.

National Earth Science Infrastructure Program AuScope Limited Headquarters School of Earth Sciences University of Melbourne Victoria 3010 Tel

Making Connections: SHARE and the Open Science Framework Jeffrey Open Repositories 2015.

Linking AuScope to the broader minerals industry value chain Jonathan Law, Robert Woodcock, Ryan Fraser, Terry Rankine, Guillaume Duclaux.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM The Spatial Information Services Stack – infrastructure for the AuScope Community Earth.

Virtual Geophysics Laboratory Scientific workflows exploiting the cloud Ryan Fraser, Terry Rankine, Lesley Wyborn, Joshua Vote, Ben Evans... Presented.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.

National Spatial Data Infrastructure The Spatial Information Services Stack Dr Robert Woodcock.

IPlant cyberifrastructure to support ecological modeling Presented at the Species Distribution Modeling Group at the American Museum of Natural History.

Distributed Aircraft Maintenance Environment - DAME DAME Workflow Advisor Max Ong University of Sheffield.

GEM Portal and SERVOGrid for Earthquake Science PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce Computer Science, Informatics, Physics.

Unidata TDS Workshop THREDDS Data Server Overview

Technical Session Virtual Geophysics Laboratory MINERAL RESOURCES Josh Vote | Software Developer September 2014.

Using Biological Cyberinfrastructure Scaling Science and People: Applications in Data Storage, HPC, Cloud Analysis, and Bioinformatics Training Scaling.

Large Scale Nuclear Physics Calculations in a Workflow Environment and Data Provenance Capturing Fang Liu and Masha Sosonkina Scalable Computing Lab, USDOE.

Presented by Scientific Annotation Middleware Software infrastructure to support rich scientific records and the processes that produce them Jens Schwidder.

Infrastructures for Social Simulation Rob Procter National e-Infrastructure for Social Simulation ISGC 2010 Social Simulation Tutorial.

GEON2 and OpenEarth Framework (OEF) Bradley Wallet School of Geology and Geophysics, University of Oklahoma

Portable Infrastructure for the Metafor Metadata System Charlotte Pascoe 1, Gerry Devine 2 1 NCAS-BADC, 2 NCAS-CMS University of Reading PIMMS provides.

Presented by Jens Schwidder Tara D. Gibson James D. Myers Computing & Computational Sciences Directorate Oak Ridge National Laboratory Scientific Annotation.

The Astronomy challenge: How can workflow preservation help? Susana Sánchez, Jose Enrique Ruíz, Lourdes Verdes-Montenegro, Julian Garrido, Juan de Dios.

A Practical Approach to Metadata Management Mark Jessop Prof. Jim Austin University of York.

Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close Ryan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock July.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM The NCRIS AuScope Community Earth Model Bruce Simons.

| nectar.org.au NECTAR TRAINING Module 2 Virtual Laboratories and eResearch Tools.

Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM “Building Clients for the AuScope Spatial Information Services Stack (SiSS)” AuScope.

Application of RDF-OWL in the ESG Ontology Sylvia Murphy: Julien Chastang: Luca Cinquini:

IPlant Collaborative Tools and Services Workshop Overview of the iPlant Discovery Environment Sriram Srinivasan.

Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,

USGS ScienceBase Making Connections with Metadata Integration GSA2011 R. Sky Bristol.

Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech 5 th GO-ESSP Community Meeting.

Data Management: Data Processing Types of Data Processing at USGS There are several ways to classify Data Processing activities at USGS, and here are some.

Nci.org.au © National Computational Infrastructure 2016 Virtual Laboratories in Australia Lesley Wyborn (NCI) With contributions from:

Enabling Digital Earth by focussing on ‘accessibility’ rather than ‘delivery’. Ryan Fraser CSIRO.

SharePoint University of the Highlands and Islands SharePoint for Records Management.

CyVerse Workshop Discovery Environment Overview. Welcome to the Discovery Environment A Simple Interface to Hundreds of Bioinformatics Apps, Powerful.

Enhancements to Galaxy for delivering on NIH Commons

Accessing the VI-SEEM infrastructure

Tools and Services Workshop

Joslynn Lee – Data Science Educator

Marketplace & service catalog concepts, first design analysis

VHIRL: Virtual Hazard Impact and Risk Laboratory – “A reuse story”

Virtual Geophysics Laboratory (VGL): Exploiting the Cloud and HPC

Electronic Data Exchange and Evaluation System

ANDS and why we care about provenance

Roadmap and short term activities on interoperability of Data Cubes

Presentation transcript:

Virtual Laboratories VGL and Friends R.Fraser, T.Rankine, J.Vote, R.Woodcock AuScope Grid Roadshow 2014 CSIRO | MINERAL RESOURCES FLAGSHIP

Messy desk 2 | – where’s your analysis?

Let’s map it out… Hardcopy of data SSH Client Maths Package Proprietary Package Transform to a regular grid Crop data to area of interest Reformat data for processing Upload data to HPC resource Configure job and start processing Download results Get field data Visualise data 3 |

Components of a Virtual Laboratory Data Services Toolboxes Compute Services Enablers Virtual Laboratory 4 |

…to this 5 | Virtual Geophysics Laboratory Build “science” from existing libraries Run job Collect and publish results Discover (raw) data Select spatial bounds

Virtual Laboratories – Scientific Workflows Automated - Simplify access and integration Reproducible process Flexible – reusable, support differing licensing Allow scientists to focus on the science and not the tech Exploiting the Cloud and HPC Operational or education tool 6 |

Components of a Virtual Laboratory ToolboxesData Services Compute Services EnablersVirtual Laboratory 7 |

Data discovery 8 |

Data discovery 9 | Some data services support subsetting Some data services support reformatting e.g. CSV, NetCDF, GeoTIFF

Components of a Virtual Laboratory Data Services ToolboxesCompute Services EnablersVirtual Laboratory 10 |

Toolboxes & Compute Services 11 | A variety of different Toolboxes (scientific codes) are available Flexibility in what compute services to utilise

Toolboxes 12 | The steps so far have been building an environment to run a processing script...or build from existing templates Either write your own...

Components of a Virtual Laboratory Data Services ToolboxesCompute Services (OGC, Provenance) Virtual Laboratory Enablers 13 |

Virtual Geophysics Laboratory Build “science” from existing libraries Run job Collect and publish results Discover (raw) data Select spatial bounds Published Record Enablers/Provenance 14 | Standards are used to enable….”enough said” Provenance – information that describes creation of something Process record Audit trail Publish the actual method/process used to create the results!

What have we gained? 15 | Provenance “....documenting their procedures in VGL it is all recorded for us.” Data Access - “Geophysicists no longer have to spend days searching for data from multiple sources, downloading reformatting....” Reproducibility “......collaboration and sharing now possible” Barrier to Entry “......making geophysical processing and modelling tools available to the broader community at a far lower cost. It lowers the skills barrier to entry”

Components of the Virtual Geophysics Laboratory Data Services ToolboxesCompute Services Dynamic Virtual Geophysics Laboratories Magnetics Gravity DEM eScript Under world Petascale Compute Org. Cloud NeCTAR Cloud Amazon Cloud Desktop Service Orchestration VGL Portal Provenance Metadata Scripting Tool eScript Mag.Grav. Org. Cloud VGL Portal DEM Mag.Grav. Petascale Compute Org. Cloud Under world Enablers (OGC, Provenance) 16 |

Repurposing to a Virtual Hazards Laboratory (VHIRL) Dynamic Virtual Hazards Laboratories Magnetics Gravity DEM ANUGA eScript Petascale Org. Cloud NeCTAR Cloud Amazon Cloud Desktop Service Orchestration VHIRL Portal Provenance Metadata Scripting Tool ANUGA Mag.Grav. Petascale VHIRL Portal DEM BathyDEM Amazon Cloud Org. Cloud Landsat Bathymetry eScript Unchanged Data Services ToolboxesCompute Services Enablers (OGC, Provenance) 17 |

VL: Engagement My job (Provenance) Reproducible process My toolbox (Market Place) Analogous to cloud application store Automation My Field(VL core) Flexibility 18 |