Presentation on theme: "Dynamic Data Driven Application Systems (DDDAS) A new paradigm for"— Presentation transcript:
1Dynamic Data Driven Application Systems (DDDAS) A new paradigm for applications/simulationsandmeasurement methodology… and how it would impact CyberInfrastructure!Dr. Frederica DaremaSenior Science and Technology AdvisorDirector, Next Generation Software ProgramNSF
2(Dynamic Data-Driven Simulation Systems) (serialized and static) What is DDDAS(Symbiotic Measurement&Simulation Systems)(Dynamic Data-Driven Simulation Systems)NEW PARADIGMSimulations(Math.ModelingPhenomenologyObservation ModelingDesign)(serialized and static)OLD(First Principles)TheorySimulations(Math.ModelingPhenomenology)(First Principles)TheoryExperimentMeasurementsField-Data(on-line/archival)UserMeasurementsExperimentField-DataUserFeedback & ControlDynamicLoopChallenges:Application Simulations DevelopmentAlgorithmsComputing Systems Support
3Examples of Applications benefiting from the new paradigm Engineering (Design and Control)aircraft design, oil exploration, semiconductor mfg, structural engcomputing systems hardware and software design(performance engineering)Crisis Management and Environmental Systemstransportation systems (planning, accident response)weather, hurricanes/tornadoes, floods, fire propagationMedicalcustomized surgery, radiation treatment, etcBioMechanics /BioEngineeringManufacturing/Business/FinanceSupply Chain (Production Planning and Control)Financial Trading (Stock Mkt, Portfolio Analysis)DDDAS has the potential to revolutionizescience, engineering, & management systems
4NSF March 2000 Workshop on DDDAS (Co-Chairs: Craig Douglas, UKy; Abhi Desmukh, UMass) Invited PresentationsNew Directions on Model-Based Data Assimilation (Chemical Appl’s)Greg McRae, Professor, MITCoupled atmosphere-wildfire modelingJanice Coen, Scientist, NCARData/Analysis Challenges in the Electronic Commerce EnvironmentHoward Frank, Dean, Business School, UMDSteered computing - A powerful new tool for molecular biologyKlaus Schulten, Professor, UIUC, Beckman InstituteInteractive Control of Large-Scale SimulationsDick Ewing, Professor, Texas A&M UniversityInteractive Simulation and Visualization in Medicine: Applications to Cardiology, Neuroscience and Medical ImagingChris Johnson, Professor, University of UtahInjecting Simulations into Real LifeAnita Jones, Professor, UVAWorkshop Report:
6Surface hydrophone array Source: Schlumberger online oilfield glossary at
7Fire ModelSensible and latent heat fluxes from ground and canopy fire -> heat fluxes in the atmospheric model.Fire’s heat fluxes are absorbed by air over a specified extinction depth.56% fuel mass -> H20 vapor3% of sensible heat used to dry ground fuel.Ground heat flux used to dry and ignite the canopy.Kirk Complex Fire. U.S.F.S. photoSlide Courtesy of Coen/NCAR
8Coupled atmospheric and wildfire models Slide Courtesy of Coen/NCAR
12Compare with Classical (Old) Supply Chain ManufacturingDistributionRetailCustomerPartsSupplierManufacturingDistributionRetailCustomerPartsSupplierManufacturingDistributionRetailCustomerTransportation Supplier
13Some Technology Challenges in Enabling DDDAS Application developmentinterfaces of applications with measurement systemsdynamically select appropriate application componentsability to switch to different algorithms/components depending on streamed dataAlgorithmstolerant to perturbations of dynamic input datahandling data uncertaintiesSystems supporting such dynamic environmentsdynamic execution support on heterogeneous environmentsExtended Spectrum of platforms: assemblies of Sensor Networks and Computational Grid platformsGRID Computing, and Beyond!!!
14What is Grid Computing? coordinated problem solving on dynamic and heterogeneous resource assembliesDATA ACQUISITIONADVANCED VISUALIZATION,ANALYSISCOMPUTATIONAL RESOURCESIMAGING INSTRUMENTSLARGE-SCALE DATABASESExample: “Telescience Grid”, Courtesy of Ellisman & Berman /UCSD&NPACI
15The NGS Program developsTechnology for integrated feedback & control Runtime Compiling System (RCS) and Dynamic Application CompositionDynamic AnalysisSituationApplicationModelDistributedProgrammingModelApplicationProgramCompilerFront-EndApplicationIntermediate RepresentationCompilerBack-EndLaunchApplication (s)PerformanceMeasuremetns&ModelsDynamicallyLink&ExecuteApplicationComponents&FrameworksDistributed Computing ResourcesDistributed Platformcomputing SystemsAdaptableInfrastructureMPPNOWSARtac-combasedatacntlfirealg acceleratorSP….
16Some more Challenges on Applications Development Issues Handling Data Streams in addition to Data SetsHandling different data structures – semantic informationInterfaces to Measurement Systems Interactive Visualization and SteeringStandards for data exchangeCombining Local and Global KnowledgeModel InteractionsApplication control of measurement systemsDynamic Application Composition and Runtime support(Examples from ITR supported efforts)
17Important Point: DDDAS is not just DATA ASSIMILATION!!! Data Assimilation compares/corrects specific calculated points with experiments, rather than dynamically as needData Assimilation does not include the notion of the simulation/application controlling the measurement processRather…Data Assimilation techniques can be used in certain DDDAS cases
18Programming Environments Procedural - > Model BasedProgramming -> CompositionCustom Structures -> Customizable Structures(patterns, templates)Libraries -> Frameworks ->Compositional Systems(Knowledge Based Systems)Application Composition Frameworksand….Interoperability extended to include measurementsData Models and Data ManagementExtend the notion of Data Exchange Standards (Applications and Measurements)
19Additional Considerations/Requirements on Hardware and Software Systems Extended Spectrum of platformsAssemblies of Computational Grid and Sensor Networks platformsSystems Architectures including Measurement SystemsProgramming EnvironmentsApplication, System, and Resource ManagementModels of the Computational InfrastructureSecurity and Fault ToleranceDDDAS will accentuate and create the need for advances in such areas
20Towards Enabling DDDAS “Users shouldn’t Have to be HeroesToday’s Grid Environments:to Achieve Grid Program Performance”and... because heroism is not enoughDynamic Data-DrivenApplication SystemsMeasurement&SimulationSymbiotic--SystemsDynamicCompilersApplication&CompositionNGS ProgramPerformanceEngineering
21Impact to CyberInfrastructure The CyberInfrastructure that will result when thinks of the present paradigm of (disjoint) simulations and measurements will be different than the CyberInfrastructure needed to support DDDASFor example, bandwidth requirements, resource allocation and other middleware and systems software policies, prioritization, security, fault tolerance, recovery, QoS, etc…, will be different when one needs to guarantee data streaming to an executing simulation or control of measurement process
22Why Now is the Time for DDDAS Technological progress has prompted advances in some of the challengesComputing speeds advances (uni- and multi-processor systems), Grid Computing, Sensor NetworksSystems SoftwareApplications Advances (parallel & grid computing)Algorithms advances (parallel &grid computing, numeric and non-numeric techniques: dynamic meshing, data assimilation)Examples of efforts in:ApplicationsAlgorithms
23Agency Efforts NSF NGS: The Next Generation Software Program (1998- ) develops systems software supporting dynamic resource executionScalable Enterprise Systems Program (1999, )geared towards “commercial” applications (Chaturvedi example)ITR: Information Technology Research (NSF-wide, FY00-04)has been used as an opportunity to support DDDAS related effortsIn FY00 1 NGS/DDDAS proposal received; deemed best, fundedIn FY01, 46 ~DDDAS pre-proposals received; many meritorious; proposals received; 8 were awardedIn FY02, 31 ~DDDAS proposals received; 8(10) awardsIn FY02, so far: received 35 (“Small” ITR) proposals ~DDDAS; more expected in the “Medium ITR” category -Gearing towards a DDDAS programexpect participation from other NSF DirectoratesLooking for participation from other agencies!
24“~DDDAS” proposals awarded in FY00 ITR Competition Pingali, Adaptive Software for Field-Driven Simulations
25“~DDDAS” proposals awarded in FY01 ITR Competition Biegler – Real-Time Optimization for Data Assimilation and Control of Large Scale Dynamic SimulationsCar – Novel Scalable Simulation Techniques for Chemistry, Materials Science and BiologyKnight – Data Driven design Optimization in Engineering Using Concurrent Integrated Experiment and SimulationLonsdale – The Low Frequency Array (LOFAR) – A Digital Radio TelescopeMcLaughlin – An Ensemble Approach for Data Assimilation in the Earth SciencesPatrikalakis – Poseidon – Rapid Real-Time Interdisciplinary Ocean Forecasting: Adaptive Sampling and Adaptive Modeling in a Distributed EnvironmentPierrehumbert- Flexible Environments for Grand-Challenge Climate SimulationWheeler- Data Intense Challenge: The Instrumented Oil Field of the Future
26“~DDDAS” proposals awarded in FY02 ITR Competition Carmichael – Development of a general Computational Framework for the Optimal Integration of Atmospheric Chemical Transport Models and Measurements Using AdjointsDouglas-Ewing-Johnson – Predictive Contaminant Tracking Using Dynamic Data Driven Application Simulation (DDDAS) TechniquesEvans – A Framework for Environment-Aware Massively Distributed ComputingFarhat – A Data Driven Environment for Multi-physics ApplicationsGuibas – Representations and Algorithms for Deformable ObjectsKarniadakis – Generalized Polynomial Chaos: Parallel Algorithms for Modeling and Propagating Uncertainty in Physical and Biological SystemsOden – Computational Infrastructure for Reliable Computer SimulationsTrafalis – A Real Time Mining of Integrated Weather Data
27Measured Response Partners A Homeland Security Simulation (Briefed WH 5/14/02)Alok Chaturvedi, DirectorShailendra Mehta, co-DirectorPurdue e-Business Research CenterPartnersInstitute for Defense AnalysesOffice of VP IT, Purdue UniversityResearch and Academic Computing, Indiana UniversitySimulex, Inc
28Explore, Experiment, Learn, Analyze, Test, & Anticipate Parallel WorldsReal WorldEnvironmentExplore, Experiment, Learn, Analyze, Test, & AnticipateImplement, AssessBehaviormodeling,demographics,and calibrationData collection,association,trends, and parameterestimationTimeCompressionNear exact replicaof the “real” worldSEAS architectureSupports millions ofArtificial agentsDecision Support LoopSyntheticThe user(s) can seamlessly switch between real and virtual worlds through an intuitive user interface.SCMERPCRMDataWarehouseSimulation Loop
29Reproduction Model Interventions: Get in contactwith infectedInfectedw/o SymptomsSusceptibleExposedentering incubation periodUninfectedImmunizedend ofincubation periodmortality not due to infectionImmunerecoveredInfectedw/ SymptomsMortalitySuccumb to the diseaseInterventions:Screen, Isolate (camp or shelter), Treat, Vaccinate
30Mobility Models Regular Movement Event Traffic Morning and Evening RushEvacuationPanic Fleeing
31New Infections T6 Intervention No Intervention T2 Intervention
32Towards a National Grid for HLS Data FusionBio sensorhumanMEMSThe virtual worldNanoSensorelectronicReal World
33A Data Intense Challenge: The Instrumented Oilfield of the Future NSF ITR ProjectA Data Intense Challenge:The Instrumented Oilfield of the FuturePI: Prof. Mary Wheeler, UT AustinMulti-Institutional/Multi-Researcher CollaborationSlide Courtesy of Wheeler/UTAustin
34Highlights of Instrumented Oilfield Proposal IT Technologies:Data management, data visualization, parallel computing, and decision-making tools such as new wave propagation and multiphase, multi- component flow and transport computational portals, reservoir production:THE INSTRUMENTED OILFIELDMajor Outcome of Research:Computing portals which will enable reservoir simulation and geophysical calculations to interact dynamically with the data and with each other and which will provide a variety of visual and quantitative tools. Test data provided by oil and service companies
35Economic Modeling and Well Management Production ForecastingWell ManagementReservoirPerformanceSimulation ModelsVisualizationDataAnalysisMultiple RealizationsFieldMeasurementsData Management and ManipulationReservoir MonitoringField ImplementationData Collections from Simulations andField Measurements
36A Data Intense Challenge: The Instrumented Oilfield of the Future ITR ProjectA Data Intense Challenge:The Instrumented Oilfield of the FutureIndustrial Support (Data):British Petroleum (BP)ChevronInternational Business Machines (IBM)LandmarkShellSchlumberger
38Dynamic Contrast Enhanced Imaging Dynamic image quantification techniquesUse combination of static and dynamic image information to determine anatomic microstructure and to characterize physiological behaviorFit pharmacokinetic models (reaction-convection-diffusion equations)Collaboration with Michael Knopp, MD
39Dynamic Contrast Enhanced Imaging Dynamic image registrationCorrect for patient tissue motion during studyRegister anatomic structures between studies and over timeNormalizationImages acquired with different patterns spatio-temporal resolutionsImages acquired using different imaging modalities (e.g. MR, CT, PET)
40Clinical Studies using Dynamic Contrast Imaging 1000s of dynamic images per research studyIterative investigation of image quantification, image registration and image normalization techniquesAssess techniques’ ability to correctly characterize anatomy and pathophysiology“Ground truth” assessed byBiopsy resultsChanges in tumor structure and activity over time with treatment
41Knopp M, OSU Radiology / dkfz prior to therapy13701370after 2 cycles142114211421Diese Datei und alle darin enthaltenen Bilder sind Eigentum von Michael V. Knopp.Jede Benutzung, Veröffentlichung oder Weitergabe gleich welcher Art ist nicht statthaft, wenn dies nicht ausdrücklich und schriftlich genehmigt wurde.Fragen und Mitteilungen anThis is protected information. Any copies or distribution without the express written approval of Dr. M. Knopp is prohibitedafter 4 cycles14381438Knopp M, OSU Radiology / dkfz
42Software SupportComponent Framework for Combined Task/Data ParallelismUse defines sequence of pipelined components -- “filter group”User directive tells preprocessor/runtime system to generate and instantiate copies of filtersMany filter groups can be simultaneously activeIntegration proceeding with Globus/Network Weather Service
44Adaptive Software Project Cornell UniversityCS department (Keshav Pingali)Civil and Environmental Engineering (Tony Ingraffea)Mississippi State UniversityUniversity of Alabama, BirminghamMechanical and Aerospace (Bharat Soni)College of William and MaryOhio State UniversityClark-Atlanta University
45Cracks: They’re Everywhere! SCOPE of ASPCracks: They’re Everywhere!Implement a system for multi-physics multi-scale adaptive CSE simulationscomputational fracture mechanicschemically-reacting flow simulationUnderstand principles of implementing adaptive software systems
47Problem description Regenerative cooling nozzle from NASA Simplified geometryChemically-reacting flow in interior of pipeNozzle is cooled by fluid-flow in eight smaller channels at periphery of pipeProblem:simulate flowsdetermine crack growthcouple the multi-physics modelsWhen successful add the ability to inject monitoring measurements
48Understanding fracture Wide range of length and time scalesMacro-scale (1in- )components used in engineering practiceMeso-scale ( microns)poly-crystalsMicro-scale ( Angstroms)collections of atoms10-310-610-9m
49Chemically-reacting flows MSU/UAB expertise in chemically-reacting flowsLOCI: system for automatic synthesis of multi-disciplinary simulations
51Adaptive Sampling and Adaptive Modeling PoseidonRapid Real-TimeInterdisciplinary Ocean Forecasting:Adaptive Sampling and Adaptive Modelingin a Distributed EnvironmentNicholas M. Patrikalakis, Henrik Schmidt, MITAllan R. Robinson, James J. McCarthy, Harvard
52Ocean Science Issues Data driven simulations via data assimilation Simulation driven adaptive sampling of the oceanInterdisciplinary ocean science: interactions of physical, biological, acoustical phenomenaExtend state-of-the-art via feedback from acoustics to physical&biological oceanographyApplication in fisheries management, but also in oil-slick containment
54Development of a General Computational Framework for the Optimal Integration of Atmospheric Chemical Transport Models and Measurements Using AdjointsGreg Carmichael (Dept. of Chem. Eng., U. Iowa)Adrian Sandu (Dept. of Comp. Sci., Mich. Inst. Tech.)John Seinfeld (Dept. Chem. Eng., Cal. Tech.)Tad Anderson (Dept. Atmos. Sci., U. Washington)Peter Hess (Atmos. Chem., NCAR)Dacian Daescu (Inst. of Appl. Math., U. Minn.)
55Application: The Design of Better Observation Strategies to Improve Chemical Forecasting Capabilities. Example flight path of the NCAR C-130 flown to intercept a dust storm in East Asia that was forecasted using chemical models as part of the NSF Ace-Asia (Aerosol Characterization Experiment in Asia) Field Experiment Will help to Better Determine Where and When to Fly and How to More Effectively Deploy our Resources (People, Platforms, $s)Shown are measured CO along the aircraft flight path, the brown isosurface represents modeled dust (100 ug/m3), and the blue isosurface is CO (150 ppb) shaded by the fraction due to biomass burning (green is more than 50%).
56Project Goal: To develop general computational tools, and associated software,for assimilation of atmospheric chemical and optical measurements into chemical transport models (CTMs).These tools are to be developed so that users need not be experts in adjoint modeling and optimization theory.
57Approach:Develop novel and efficient algorithms for 4D-data assimilation in CTMs;Develop general software support tools to facilitate the construction of discrete adjoints to be used in any CTM;Apply these techniques to important applications including:(a) analysis of emission control strategies for Los Angeles;(b) the integration of measurements and models to produce a consistent/optimal analysis data set for the AceAsia intensive field experiment;(c) the inverse analysis to produce a better estimate of emissions; and(d) the design of observation strategies to improve chemical forecasting capabilities.
58Data Assimilation for Chemical Models Solid lines represent current capabilities Dotted lines represent new analysis capabilities Future: enable DDDAS capabilities
59General Software Tools Framework to Facilitate the Close Integration of Measurements and Models The framework will provide tools for: 1) construction of the adjoint model; 2) handling large datasets; 3) checkpointing support; 4) optimization; 5) analysis of results; 6) remote access to data and computational resources.
60Modeling Uncertainty Stochastically-excited structures Irreducible versus epistemic uncertaintyStochastically-excited structuresBoundary conditions, geometry, propertiesSensitivity/failure analysisGaussian and non-Gaussian processesPolynomial Chaos vs. Monte CarloStochastic spectral/hp element methods“…Because I had worked in the closest possible ways withphysicists and engineers, I knew that our data can never be precise…”Norbert WienerSlides Courtesy of Karniadakis/Brown
61Partially Correlated non-Uniform Random Inflow DeterministicStochasticPressureVorticity: Regions of Uncertainty
62Non-uniform Gaussian Random BC Exponential correlationStochastic input:2D K-L expansion4th-order Hermite-Chaos expansion15-term expansionUmean along centerlineVmean along centerline
63Non-uniform Exponential Random BC Exponential correlationStochastic input:2D K-L expansion4th-order Laguerre-Chaos expansion15-term expansionUmean along centerlineVmean along centerline
64Uncertainty Ignorance Research Opportunities in UncertaintyUUncertainty analysis is a fertile and much needed area for inter-disciplinary researchEEstimates of uncertainties in model inputs are desperately neededUncertainty Ignorance
65What about Industry &DDDAS Industry has history offorging new research and technology directions andadapting and productizing technology which has demonstrated promiseNeed to strengthen the joint academe/industry research collaborations; joint projects / early stagesTechnology transferestablish path for tech transfer from academic research to industryjoint projects, students, sabbaticals (academe <----> industry)Initiatives from the Federal Agencies / PITACCross-agency co-ordinationEffort analogous to VLSI, Networking, and Parallel and Scalable computingIndustry is interested in DDDAS
66(emphasis on multidisciplinary research) Research and Technology Roadmap(emphasis on multidisciplinary research)ientgraoApplication Composition System}•Distributed programming models.•Application performance Interfaces.•Compilers optimizing mappings on complex.isystemsntDApplication RunTime System}EProvidingEenhanced•Automatic selection of solution methods.g•Interfaces, data representation & exchange.M.capabilities•Debugging toolsrOforSaApplicationstMeasurement System}i•Application/system multi-resolution models.o•Modeling languages..•Measurement and instrumentationnY1Y2Y3Y Y5ExploratoryDevelopmentIntegration & Demos
67DDDAS: http://www.cise.nsf.gov/dddas http://www.dddas.org NGS: DDDAS has potentialfor significant impact toscience, engineering, and commercial world,akin to the transformation effectedsince the ‘50sby the advent of computersDDDAS:NGS:
68backup slides Following is a List of Presentations of DDDAS projects at theInternational Conference on Computational SciencesJune 2-6, 2003, Melbourne Australia
69Dynamic Data Driven Application Systems WORKSHOP (June 2 & June 3) Agenda (Titles of presentations and speakers)Mon June 2Session 1 (3:30pm- 4:15pm) Introduction: Dynamic Data Driven Application SystemFrederica Darema, NSFGuest Talk: Bayesian Methods for Dynamic Data Assimilation and Process Design in the Presence of UncertaintiesGreg McRae, MITSession 2 (4:30pm- 6:00pm)Computational Science Simulations based on Web ServicesKeshav Pingali, Cornell U.Driving Scientific Applications by Data in Distributed EnvironmentsJoel Saltz, The Ohio State UniversityDDEMA: A Data Driven Environment for Multiphysics Applications John Michopoulos, NRL
70Dynamic Data Driven Application Systems WORKSHOP Tues June 3Session 3 (9:30am- 10:30am)Computational Aspects of Chemical Data Assimilation into Atmospheric ModelsGregory Carmichael, U of IowaVirtual Telemetry for Dynamic Data-Driven Application Simulations Craig C. Douglas, University of Kentucky and Yale UniversitySession 4 (11:00am- 12:30pm)Tornado Detection with Support Vector MachinesTheodore B. Trafalis, University of OklahomaA Computational Infrastructure for Reliable Computer Simulations Jim Browne, UTAustin Discrete Event Solution of gas Dynamics within the DEVS Framework: Exploiting Spatiotemporal HeterogeneityJames Nutaro – U of Arizona
71Dynamic Data Driven Application Systems WORKSHOP Tues June 3 (cont’d)Session 5 (2:30pm- 3:30pm)Data Driven Design Optimization Methology: A Dynamic Data Driven Application SystemDoyle Knight, Rutgers U.Rapid Real-Time Interdisciplinary Ocean Forecasting Using Adaptive Sampling and Adaptive Modeling and legacy Codes: Component Ecapsulation using XMLConstantinos Evangelinos, MITSession 6 (4:00am- 5:30pm)Generalized Polynomial Chaos: Algorithms for Modeling and Propagation of UncertaintyDongbin Xiu, Brown UniversityDerivation of Natural Stimulus Feature Set Using A Data Driven ModelJohn Miller, Montana State U.Simulating Seller’s Behavior in a Reverse Auction B2B ExchangeAlok Chaturvedi, Purdue U.