We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published bySilas Estes
Modified about 1 year ago
© Geodise Project, University of Southampton, Geodise: Taking the Grid to the Engineer Graeme Pound International Summer School on Grid Computing 2006 Ischia, Italy 9 th -21 st July 2006
© Geodise Project, University of Southampton, Lecture summary Application domain –Engineering design search and optimisation –Computational Fluid Dynamics Design principles –Usability –Integration –Share and reuse Technical solution –Generic toolboxes –Job submission –Data management Application Examples Conclusions
© Geodise Project, University of Southampton, Application Domain
© Geodise Project, University of Southampton, Size Marketability Strength Cost Weight Performance Reliability Safety Environment Structure CONCEPT DETAIL
© Geodise Project, University of Southampton, Modern engineering firms are global and distributed CAD and analysis tools, user interfaces, PSEs, and Visualization Optimisation methods Data archives (e.g. design/ system usage) Knowledge repositories & knowledge capture and reuse tools. Management of distributed compute and data resources How to … ? … improve design environments … cope with legacy code / systems … integrate large-scale systems in a flexible way … produce optimized designs … archive and re-use design history … capture and re-use knowledge Design Challenges
© Geodise Project, University of Southampton, What is Engineering Design Search & Optimisation? Improving designs by exploiting engineering modelling and analysis. Systematically modifying variables in a design to increase, or reduce a quality measure (called the objective function), within certain constraints. Involves computational and data intensive processes
© Geodise Project, University of Southampton, Defining the Objective Function CAD geometry Meshing CFD analysis Post-processing Design Variables x 1 = 0.5, x 2 = 0.25 Objective function y = 42
© Geodise Project, University of Southampton, RSM Construct RSM Evaluate Search Using RSM Best Design Adequate ? RSM Tuning Build Data- Base CFD DoE Initial Geometry CFD … … … … Cluster Parallel Analysis Design of Experiment & Response Surface Modelling
© Geodise Project, University of Southampton, Computational Fluid Dynamics
© Geodise Project, University of Southampton, Application profile – CFD / EDSO CFD analysis may be: –Computationally expensive –Require/produce large volumes of data Evaluation of an engineering objective function may require: –Multiple applications invoked in sequence –Third party proprietary applications with specific hardware requirements –Pre- and post- processing –Automation of interactive tasks into batch processes Optimisation algorithms may be incorporated into larger optimisation strategies Data reuse/analysis may prevent expensive duplicate calculations No two problems are the same
© Geodise Project, University of Southampton, Design Principles
© Geodise Project, University of Southampton, Problem Solving Environments “A PSE is a computer system that provides all the computational facilities needed to solve a target class of problems.” S. Gallopoulos, E. Houstis & J. Rice (1994)
© Geodise Project, University of Southampton, Design Principles An emphasis upon the usability of the problem solving environment Integration of existing Grid resources and applications Facilitate collaboration by promoting the sharing and reuse of components, workflows, results and knowledge
© Geodise Project, University of Southampton, Scripting languages Why use scripting languages? Flexibility High-level functionality Rapid application development Extend the user’s existing PSE Workflows described in a human readable format may be shared and reused
© Geodise Project, University of Southampton, Technical Solution
© Geodise Project, University of Southampton, Geodise Scripting Environments Matlab - technical computing environment –500,000+ users –Data analysis and visualisation toolboxes –High-level scripting –Commercial product Jython - Java implementation of Python –object-oriented Python language –100% pure Java –Active funded development –Open source project Cross platform Scripting complex engineering workflows
© Geodise Project, University of Southampton, Scripting EDSO workflows Matlab User’s Script Mesh CFD CAD Post Process Grid Resources Geodise Archive Globus Compute ASP Condor Compute
© Geodise Project, University of Southampton, Technology overview Computational resources –Globus (v2.4 remains ubiquitous), via Java CoG Kit –Condor –OMII core services –Microsoft Compute Cluster Server 2003 –Any others? XML Web Services –Java, Tomcat, WebSphere application server –ASP.NET, IIS Data management –Oracle 9i or 10g –GridFTP file server
© Geodise Project, University of Southampton, Computational Toolboxes
© Geodise Project, University of Southampton, Certificate Management Functions gd_certinfo Returns information about the user's certificate gd_createproxy Creates a Globus proxy certificate gd_proxyinfo Returns information about the user's proxy certificate gd_proxyquery Queries whether a valid proxy certificate exists gd_destroyproxy Destroys the local copy of the user's Globus proxy certificate
© Geodise Project, University of Southampton, Certificate Management C:\GEODISE >> gd_createproxy
© Geodise Project, University of Southampton, Certificate Management C:\GEODISE >> subject = gd_certinfo subject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound issuer : support.ac.uk start date : Wed Sep 22 16:31:24 BST 2004 end date : Thu Sep 22 16:31:24 BST 2005 subject = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound >> isvalid = gd_proxyinfo subject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxy issuer : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound type : full legacy globus proxy strength : 512 bits timeleft : 11 h, 59 min, 30 sec isvalid = 1 >>
© Geodise Project, University of Southampton, Job Submission Functions gd_jobstatus Gets the status of a Globus GRAM job gd_jobsubmit Submits a compute job to a Globus GRAM job manager gd_jobpoll Queries the status of a Globus GRAM job until complete gd_jobkill Kills a Globus GRAM specified by a job handle gd_chmod Changes file permissions of a file on a Globus resource gd_condorsubmit Submits a job to a Condor pool via a Globus resource gd_submitunique Submits a GRAM job to a unique working directory gd_listjobs Returns all the job handles belonging to the user
© Geodise Project, University of Southampton, Job Submission C:\GEODISE >> RSL = '&(executable="/bin/date")(stdout="date.out")'; >> host = 'grid-compute.oesc.ox.ac.uk'; >> jobhandle = gd_jobsubmit(RSL, host) jobhandle = https://grid-compute.oesc.ox.ac.uk:30001/30705/ / >> isdone = gd_jobpoll(jobhandle) isdone = 1 >>
> host = 'grid-compute.oesc.ox.ac.uk'; >> jobhandle = gd_jobsubmit(RSL, host) jobhandle = https://grid-compute.oesc.ox.ac.uk:30001/30705/1098694366/ >> isdone = gd_jobpoll(jobhandle) isdone = 1 >>">
© Geodise Project, University of Southampton, GridFTP File Transfer gd_getfile Retrieves a remote file using GridFTP gd_putfile Puts a remote file using GridFTP gd_transferfile Performs a third-party file transfer using GridFTP gd_makedir Creates a remote directory using GridFTP gd_listdir Lists the contents of a directory on a GridFTP resource gd_fileexists Tests the existence of files on a Globus resource gd_rmdir Deletes a remote directory using GridFTP gd_rmfile Deletes a remote file using GridFTP gd_rmuniquedir Deletes a remote directory and its contents
© Geodise Project, University of Southampton, File Transfer C:\GEODISE >> exists = gd_fileexists(host, 'date.out') exists = 1 >> gd_getfile(host,'date.out','localfile.txt'); >> type('localfile.txt') Mon Oct 25 09:52:46 BST 2004 >> gd_rmfile(host, 'date.out'); >> exists = gd_fileexists(host, 'date.out') exists = 0 >>
© Geodise Project, University of Southampton, Data Management Challenges Data may be generated from distributed applications on the Grid. Traditionally data stored on file systems with little descriptive information – hard to find and share. Engineers may want to access the data from distributed locations
© Geodise Project, University of Southampton, Need Metadata Globus Server Archive Files Archive Data Structures Database Archive Metadata Locate Files Query Retrieve Files Retrieve Structures Metadata obj=2 airfoil cad x.y.z = 43
© Geodise Project, University of Southampton, Database Toolbox Overview Store data with descriptive information – Standard and application specific metadata. – Query over metadata to easily locate required data. – Data retrieval based on ID rather than location. Familiar interface for engineers – Wrap toolbox as Matlab/Jython functions – easy integration. – Can be used in Matlab/Jython scripts – popular among engineers. Support data aggregation (data groups) concept. Central and local databases (shared vs. personal). Secure Web service access to central database over SSL. – Certificate-based authentication and authorisation.
© Geodise Project, University of Southampton, Database and XML Toolboxes
© Geodise Project, University of Southampton, Architecture
© Geodise Project, University of Southampton, Database Toolbox Storage serviceExample: %Archive data: >> fileID = gd_archive('C:\input.dat'); %Retrieve data: >> gd_retrieve(fileID, 'E:\tmp' ) ans = E:\tmp\input.dat Metadata serviceExample: %Define metadata and archive file: >> m.grids = 1; >> m.turb_model = 'sa'; >> fileID = gd_archive('C:\input.dat', m); Query serviceExample: >> r = gd_query('standard.userID = me & grids < 2'); >> gd_display(r): standard.userID = me standard.ID = input_dat_8a ad2d-4055-aad9-a1 grids = 1
© Geodise Project, University of Southampton, XML Toolbox Serialise/Deserialise Matlab variables to and from XML Read any XML document into a Matlab structure High level functions ( xml_save() and xml_load() ) 1566 downloads from MatlabCentral in 18 months User base in academic & commercial research
© Geodise Project, University of Southampton, Data Transfer between PSEs
© Geodise Project, University of Southampton, OptionsMatlab Matlab interface to the Options design exploration system Reduce barriers to entry State of the art design search and optimisation algorithms User’s objective and constraint functions exposed as Matlab functions Grid-enabled job brokers easily incorporated Composition of complex optimisation strategies
© Geodise Project, University of Southampton, Application Examples
© Geodise Project, University of Southampton, CFD 3D engine nacelle optimisation Negative Scarf Inlet Conventional Inlet 0 12 Goal - reduce ground noise generated by fan when plane takes off. Optimise aerodynamic performance when scarf angle is varied.
© Geodise Project, University of Southampton, GEM: Electromagnetic optimisation Large number of designs, parameters and solutions. Query for a particular data range to post- process. Transmission of light properties through a photonic crystal. Parameters are radius of holes and light wavelength.
© Geodise Project, University of Southampton, GENIE: Earth System Modelling GENIE supports environmental scientists modelling long term climate change. Tuning Earth system model components Optimising a parameterised model over a multi-dimensional state space. Database used for monitoring, sharing, post-processing and fault tolerance. Default Tuned Target
40 Friday, 01 May 2015 Resource Usage 5 client installations 9 Grid resources exploited 352 simulations defined (1000 and 2000 yrs) 3,736 compute tasks submitted 46,992 CPU hours (estimated) 428,000 IGCM-GOLDSTEIN years performed
41 Friday, 01 May 2015 Resource Usage
© Geodise Project, University of Southampton, Conclusions Many alternative possible approaches You should consider: –Profile of the end user –User requirements –Potential modes of use –Available resources User feedback essential for development –User experiences may differ from your preconceptions
Software Reuse and Component-Based Software Engineering CIS 376 Bruce R. Maxim UM-Dearborn.
RMS and Scheduling for Future Generation Grids Ramin Yahyapour University Dortmund Leader CoreGRID Institute on Resource Management and Scheduling CoreGRID.
Designing a Java-based Grid Scheduler using Commodity Services Patrick Wendel Arnold Fung Moustafa Ghanem Yike Guo Discovery NetInforSense Department of.
Page 1 LAITS Laboratory for Advanced Information Technology and Standards Duh 7/10/03 Geospatial Service Workflow Concepts and Tools Liping Di Laboratory.
The PLANETS Testbed DPE, PLANETS and CASPAR 2nd Annual Conference Lisbon, 5–6 September, 2007 Max Kaiser, Austrian National Library
© Geodise Project, University of Southampton, Semantic Web based Content Enrichment and Knowledge Reuse in e-Science.
Manchester Computing Cross Council ICT Conference For e-Science & GRID May 2004 End to End Services to support an e-Science Community Professor M.
Page 1 LAITS Laboratory for Advanced Information Technology and Standards Duh 7/10/03 The GMU Geospatial Grid Technology Development and Application Project.
December 2009 Data Integration in Grid Environments Alex Poulovassilis, Birkbeck, U. of London.
Who We Are Integral Systems builds satellite ground systems for command and control, integration and test, data processing, and simulation. We’ve provided.
Copyright © SDA Software 2003 SDA Application Platform Suite Stay ahead, Design your business, Achieve your goal!
Access Grid Update Robert Olson Argonne National Laboratory SURA/ViDe 5 th Annual Digital Video Workshop.
TICER Summer School, August 24th Ticer Summer School Thursday 24 th August 2006 Dave Berry & Malcolm Atkinson National e-Science Centre, Edinburgh.
Silberschatz, Galvin and Gagne ©2013 Operating System Concepts – 9 th Edition Chapter 2: Operating-System Structures.
Exploiting Reference Architecture to Guide the NASA Earth Science System Enterprise Chris A. Mattmann NASA Jet Propulsion Laboratory University of Southern.
Services and Identity Management Prof. Sasu Tarkoma.
1(#total) CS5038 The Electronic Society Lecture 10: e-Science Lecture Outline Background: Big Science Grid Computing Standards for Grid Computing e-Science.
Regnet Specification : Technical point of view REGNET.
Group Abstractions for Distributed and Grid Computing Systems José C. Cunha CITI – Centre for Informatics and Information Technologies.
Presented to: SOA Brown Bag #12 By: Paul Caron, SWIM T&E Lead Date: December 13, 2011 Federal Aviation Administration System Wide Information Management.
Dr. Daniel Sabbah Vice President of Strategy & Technology IBM Software Group Bringing Grid & Web Services Together Globus World San Francisco, CA Tuesday,
Grid Computing Evolution and Challenges for Resilience, Performance and Scalability Luca Simoncini University of Pisa, Italy July 2, 2005WS on Grid Computing.
The Client/Server Database Environment CS263 Lecture 12.
ERP and E-Business- An Overview Based on the book Enterprise Resource Planning Solutions and Management by Flona Fui-Hoon Nah, Idea Group Publishing 2001.
An Open Grid Services Architecture Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science.
Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager.
Supporting further and higher education 19th APAN Meetings in Bangkok Innovative Uses of Pervasive Broadband Network Is adoption of technology running.
Types of Information systems BY, Ikhlaq Ikhlaq. IS ACTIVE DIFFERENT LEVELS Operational-level systems Operational-level systems Support operational managers.
Digital Preservation and Management. Preserving Digital Resources: Why is it an Issue? Technology obsolescence Digital media life expectancy Variety of.
© 2016 SlidePlayer.com Inc. All rights reserved.