Presentation is loading. Please wait.

Presentation is loading. Please wait.

VV&A of DIME/PMESII TOOLS Dean Hartley January 21, 2008.

Similar presentations

Presentation on theme: "VV&A of DIME/PMESII TOOLS Dean Hartley January 21, 2008."— Presentation transcript:

1 VV&A of DIME/PMESII TOOLS Dean Hartley January 21, 2008

2 Building and Using a Model Real World Proxy for or Perception of Real World Data Conceptual Model Coded Model Hard Data Soft Data CM Creation Model Creation Data Use Risk 1 Risk 2 Risk 3 Risk 4 Risk 5 Risk 6 Model Use Risk7 Risk 7 Most PMESII values are not observable

3 Verifying and Validating a Model Real World Proxy for or Perception of Real World Data Conceptual Model Coded Model Model Verification CM Validation Data Use Data V&V Data Verification Data V&V Model Validation Theory, Data & SME Validation

4 Changed-Tool Processes* Entrenched VV&A Standard Model-Creation Processes Design Walkthroughs, Code Review, Debugging, etc. Standard Model-Acceptance Processes Alpha testing, Beta testing, Acceptance testing, etc. Multi-model-System Processes* Developmental testing Periodic Supplemental Tests Tests with different data sets, Tests of different functionality Periodic testing New-Tool Processes* Changed-Environment Processes* * e.g., Interface testing, Special input testing, Full system testing Formal Process Accredi tation Triggered testing

5 U DB TU DB TU DB T U DB TU DB TU DB TU DB T Entrenched VV&A as Part of the Lifecycle Understand the Requirements Design the System Build the System Test and Evaluate PPP P D T TTTT D Use the System A A Accreditation =A Triggered testing =T Periodic testing =P Developmental testing=D Legend P A A A A A A

6 Simplified Entrenched VV&A Process Flow Initiate VV&A Process Define D Tests Execute D Tests Evaluate D Tests Support Accreditation Define P Tests Execute P Tests Evaluate P Tests Define T Tests Execute T Tests Evaluate T Tests Type of V&V Development Triggered Periodic Code Changes Time Passes Trigger Event M&S Use Use the Model Evaluate Use Final D Test? YesNo Manage Residual Risk Revisit Initiation Process

7 Verification Metrics Example

8 Validation Metrics for Conceptual Models Labels for individual model sub-components and for entire ensemble of models “5” label meets the most stringent standards, “4” next most stringent, etc. In general, the label for an ensemble will be lower than the labels for its components Expectations for PMESII components and ensembles shaded in yellow LabelComponentEnsemble 5 Expresses fully validated theory, e.g., Newtonian physics with caveats on operations near light speed or in regimes subject to quantum effects Expresses multiple Level 5 theories with fully engineered interfaces, e.g., fly-out model of ground-to-air rocket, involving, chemical reactions of propellants, ballistics, air flow, electronics, etc. 4 Expresses well researched theory involving considerable data checking and peer review, e.g., economic theory earning Nobel prize Expresses multiple Level 4 or Level 5 theories with well researched interfaces, e.g., economic model ensemble used by Federal Reserve in setting U.S. interest and discount rates 3 Expresses theory supported by data and published in peer-reviewed literature Expresses multiple Level 3, 4 or 5 theories with considerable peer-reviewed interfaces, e.g., some U.S. combat models 2 Expresses theory with rational basis, accepted by some experts as plausible (SWAG) Expresses multiple Level 2 - 5 theories with plausible interfaces (SWAG) 1Expresses a codified theory (WAG) Expresses multiple Level 1 - 5 theories with codified interfaces (WAG) 0 Uncodified, mental model of uncertain consistency and completeness Uncodified processes for connecting models of uncertain consistency and completeness

9 Coded Model Validation Preliminaries

10 Coded Model Validation Display Inter-Model Connections 0.00 1.00 2.00 3.00 4.00 5.00 Model 1-2 Model 1-3 Model 1-4 Model 1-5 Model 1-6 Model 2-3 Model 2-4 Model 2-5Model 2-6 Model 3-4 Model 3-5 Model 3-6 Model 4-5 Model 4-6 Model 5-6 PMESII System

11 Experimental Design Create Experimental Design DIME/PMESII System SME Input Outputs Exploratory Space Analysis Risk Analysis Save Select Input Sets and Output Sets Filter to select preset Data Tests Time Past Now Modified

12 Embedded Results-Traceback System Internal, Time-Coded Data Automated Results- Traceback Program Extract data during model execution to support later traceback of the results Time Now DIME/PMESII System Modified Embedded Instruments

13 Model Comparison SME Calibrate DIME/PMESII System Time Past Now DIME/PMESII System Modified DIME/PMESII System Modified Prior Situation Current Situation Future Situation Future Validation? Data Extraction Extrapolation Yes No Output Comparison ISSM

14 Dean Hartley Hartley Consulting Questions? From the hills of East Tennessee

15 Backup Dean Hartley January 21, 2008

16 CM Risks and Mitigations (1 of 2)

17 CM Risks and Mitigations (2 of 2)

18 VV&A Process Maturity Metric Probable Current Level Probable Achievable Level Red font indicates additions to the Youngblood VPMM

Download ppt "VV&A of DIME/PMESII TOOLS Dean Hartley January 21, 2008."

Similar presentations

Ads by Google