Model validity, testing and analysis

Conceptual and Philosophical Foundations Model Validity and Types of Models –Statistical Forecasting models (black box) –Descriptive Policy models (transparent) Philosophical Aspects - Philosophy of Science - Logical Empiricim and Absolute Truth - Conversational justification & relative truth (‘purpose’) - Statistical significance testing (Barlas and Carpenter 1990 and Barlas 1996)

Two aspects of model validity Structure Validity –Primary importance –Special place in System Dynamics Behavior Validity –Role in system dynamics –The special type of behavior validity in system dynamics –Ex ante versus ex post prediction (Barlas 1996 and 1989)

Overall Nature and Selected Tests of Formal Model Validation

Logical Sequence of Formal Steps of Model Validation

Structure Validity (Simulation Verification) Direct Structure Tests –Crucial, yet highly qualitative and informal –Distributed through the entire modeling methodology Indirect Structure Tests (Structure-oriented behavior) –Crucial and partly quantitative and formal –Tool: SiS software

Indirect Structure Testing Software: SiS Based on automated dynamic pattern recognition Extreme condition pattern testing Also in parameter calibration and policy design (Kanar 1999; Kanar and Barlas 1999; Bog et al 2004)

Indirect Structure Testing Software (SiS) Basic Dynamic Patterns

Indirect Structure Testing Software (SiS) List of dynamic behavior pattern classes

Software Implementation Our Software (SiS) Main ISTS Algorithm Simulati on Software Integrator General Picture of the Processes in Validity Testing mode General Picture of the Processes in “Parameter Calibration” mode

Sample Model Used with SiS

Validity Testing with Default Parameters Simulation Output (with default base parameters) Likelihood Values of simulation behavior correctly classified as the GR2DB pattern

Validity Testing by Setting Parameters Fig1 : Simulation Output (with base parameters) Fig2 : Simulation Output (with changed parameters) Likelihood Values of simulation behavior in Fig2 compared to the NEXGR pattern

Parameter Calibration with Specified Pattern The ranges and number of values tried for each parameter Simulation Output (with base parameters)

Result of the Parameter Calibration  Best parameter set is 41  Best Likelihood Result: Best Parameter Set:  1. advertising effectiveness: 0.25  2. customer sales effectiveness: 6.0  3. sales size: 1.0 Simulation Output as Desired (after automated parameter calibration)

Parameter Calibration with Input Data A view of the SiS interface during parameter calibration

Result of the Parameter Calibration  Best parameter set is 21  Best Likelihood Result: Best Parameter Set:  1. advertising effectiveness: 5.0  2. customer sales effectiveness: 0.0 Fig1 : Simulation Output (with base parameters) Fig2 : Simulation Output (after parameter calibration to match the input pattern)

Behavior Validity Two types of patterns –Steady state –Transient Major pattern components –Trend, periods, amplitudes,...

Behavior Validity Testing Software: BTS II

Uses of BTS II and SiS in Model Analysis Analysis: Understanding the dynamic properties of the model BTS II can assist in quantifying, measuring and assessing dynamic pattern components SiS can assist in deeper structural analysis (related to qualitative pattern modes)

Uses of BTS II and SiS in Policy Design BTS II can assist in numerical performance improvement policies SiS can assist in more structural dynamic pattern improvement Parameter calibration can be extended to cover automated policy design

Implementation Issues More tools User friendliness More thorough (field) testing of the tools Better integration with simulation software...

Policy Implementation Issues Validity of the policy recommendation (Robustness, timing, duration, transition...) Finally, ‘validity of the implementation’ itself –Validated model means just a reliable laboratory; implementation validity does not automatically follow; it is a whole area in itself

Concluding Observations Validity as a process, rather than an outcome Continuous (prolonged) validity testing Validation, analysis and policy design all integrated From validity towards quality Quality ‘built-in versus inspected-in’ Group model building Testing by interactive gaming

Back to philosophy... A gradual, continuous, multi-method, qualitative and quantitative, formal and informal process of establishing confidence in a model. We should use any formal test/tool compatible with this philosophy, but never assume that tools themselves would be sufficient without proper philosophy

DISCUSSION

Reference Yaman Barlas Boğaziçi University Industrial Engineering Department Bebek Istanbul, Turkey SESDYN Group: