A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation Lieutenant Colonel Wayne Stilwell United States Army 7 September 2006 Dr. Donald E. Brown, Advisor Dr. William T. Scherer, Chair Dr. Stephanie Guerlain Dr. Paul Reynolds COL (Dr.) George F. Stone, III

Degree Requirements Course work completed in May 2005 Course work completed in May 2005 Seminar in Honolulu, Hawaii 2-10 FEB 2006: Project Albert 11 th International Workshop for Agent-Based Simulation Seminar in Honolulu, Hawaii 2-10 FEB 2006: Project Albert 11 th International Workshop for Agent-Based Simulation 13th PAIW in Netherlands NOV th PAIW in Netherlands NOV 2006 Will lead a team of researchers into command agent simulation calibration experimentation Will lead a team of researchers into command agent simulation calibration experimentation Article submitted to: Article submitted to: Journal of Defense Modeling and SimulationJournal of Defense Modeling and Simulation A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation Planned Articles: Planned Articles: IEEE Journal (Proof adapted from Luenberger 1973)IEEE Journal (Proof adapted from Luenberger 1973) MORS (Replication of a live experiment with agent-based simulation)MORS (Replication of a live experiment with agent-based simulation) Command Agent Calibration - 13 th PAIWCommand Agent Calibration - 13 th PAIW

CAVP CAVP is an iterative, information engineering-based process that calibrates CASS agent parameters to a range of acceptable outputs. CAVP is an iterative, information engineering-based process that calibrates CASS agent parameters to a range of acceptable outputs. CAVP relies on: CAVP relies on: Empirical data or Expert opinion on real systemEmpirical data or Expert opinion on real system Response surface methods (to include ERSM), data mining tools such as classification trees and linear regression, NOLH design of experiments, and expert opinion of reasonable inputResponse surface methods (to include ERSM), data mining tools such as classification trees and linear regression, NOLH design of experiments, and expert opinion of reasonable input

Problem Statement Simulation validation techniques do not currently offer an ability to: Simulation validation techniques do not currently offer an ability to: Measure the influence of non-linear relationships that contribute to the outcome of a dynamic systemMeasure the influence of non-linear relationships that contribute to the outcome of a dynamic system Reduce the complexity of higher order interactionReduce the complexity of higher order interaction Calibrate multiple simulation inputs to desired outputsCalibrate multiple simulation inputs to desired outputs Validate a CAS via the entire component comparison before white-box validationValidate a CAS via the entire component comparison before white-box validation

Complex Adaptive System Agent-based Agent-based Heterogeneous Heterogeneous Dynamic Dynamic Feedback Feedback Organization Organization Emergence Emergence Non-linear interactionNon-linear interaction Non-reductionismNon-reductionism Emergent behaviorEmergent behavior Hierarchical StructureHierarchical Structure Decentralized ControlDecentralized Control Self OrganizationSelf Organization Non-equilibrium OrderNon-equilibrium Order AdaptationAdaptation Collectivist DynamicsCollectivist Dynamics MOUT as a Complex Adaptive System

Literature Review Key Authors

Calibration Definition The process of adjusting parameter values in the simulation model to better represent the underlying system The process of adjusting parameter values in the simulation model to better represent the underlying system Calibration implies the existence of a standard to judge against. Calibration implies the existence of a standard to judge against.

Validation Definition (DMSO) The quality of being inferred, deduced, or calculated correctly enough to suit a specific purpose. The quality of being inferred, deduced, or calculated correctly enough to suit a specific purpose. The degree of validity is the level of trust a simulation user can place in the output of the model. The degree of validity is the level of trust a simulation user can place in the output of the model.

Classic Validation Concept White Box first: Validate each module according to its components White Box first: Validate each module according to its components Black Box next: Compare the total system output to actual system output Black Box next: Compare the total system output to actual system output

Literature Review Key Points Aggregation based simulations are an improvement over differential equations-based simulations when modeling complex phenomena Aggregation based simulations are an improvement over differential equations-based simulations when modeling complex phenomena CAS require more sophisticated validation methodologies than are currently available to improve the value of decisions CAS require more sophisticated validation methodologies than are currently available to improve the value of decisions Behavioral input of each agent creates emergent behavior, requiring more extensive validation techniques Behavioral input of each agent creates emergent behavior, requiring more extensive validation techniques Statistical approaches like the ERSM can provide the basis for an improved validation method. Statistical approaches like the ERSM can provide the basis for an improved validation method.

ERSM (Schamburg and Brown )

NOLH

The CAVP Determine CAS for investigation Determine CAS for investigation Examine extant system output Examine extant system output Determine Measures of Performance (MOP) Determine Measures of Performance (MOP) Develop Inputs Develop Inputs Construct the CASS Construct the CASS Determine a NOLH DOE Determine a NOLH DOE Compare MOP using Metrics of Evaluation (MOE) Compare MOP using Metrics of Evaluation (MOE) Conduct Global Convergence Optimization on responses not in tolerance Conduct Global Convergence Optimization on responses not in tolerance Declare calibration state; If not calibrated, use CART to determine causality of inputs Declare calibration state; If not calibrated, use CART to determine causality of inputs

CAVP Proof

Iterative Composite Mapping

The Experiment Recreate live soldier firefight CAS (blank rounds with sensors) via a CASS 4 varying scenarios 4 varying scenarios 5 Measures of Performance (Blue casualties, red casualties, blue rounds fired, red rounds fired, time in seconds) 5 Measures of Performance (Blue casualties, red casualties, blue rounds fired, red rounds fired, time in seconds) Metric of Evaluation (distance function) Metric of Evaluation (distance function) Used MANA as simulation of choice Used MANA as simulation of choice NOLH-based DOE, 33 design points, 200 iterations per design point. NOLH-based DOE, 33 design points, 200 iterations per design point. Heterogenous soldiers Heterogenous soldiers Expert opinion on input ranges for 10 control variables Expert opinion on input ranges for 10 control variables Exogenous variables held steady throughout experiment Exogenous variables held steady throughout experiment

Analysis

Input Parameters VariableAbbreviationReasonable RangeReason for Range Blue Speed BLSPEED Blue may choose to stalk quietly (slow speed), or move rapidly Blue Accuracy BLACC 5-60Blue soldiers are well trained, but may fire less accurately when rapidly acquiring targets in a constrained space. Blue Detection Range BLUEDET Blue soldiers have good sensors and vision, but urban terrain limits detection maximum. Blue Classification Range BLUECL 30-80Blue soldiers are well trained,, but may have trouble distinguishing between a non- uniformed enemy and a civilian. Blue Concealment BLUECONC 15-45Blue can hide their movements in some respects, but must enter doorways and expose themselves to fire at critical points. Red Speed REDSPEED 10-60Red is relatively stationary, but can move quickly within the room if necessary. Red Accuracy REDACC 5-35Red soldiers are not as accurate as blue soldiers. However, the close range of the engagements may reduce that disadvantage. Red Detection Range REDDET Red soldiers are inside of buildings and can see out slightly better than blue soldiers can see in. Red Classification Range REDCL 45-80Red soldiers can recognize blue soldiers as far as maximum vision allows Red Concealment REDCONC 40-75Red can better hide and conceal themselves on the defense and inside of urban terrain.

MOP Target Values Response (E j )Target (μ E j )=T σ γ (Lower)=L (μE j -1σ)=L γ (Upper)=U (μE j +1σ)=U Time Scenario 1 (E 1 ) Blue Casualties (E 2 ) *.38* Red Casualties (E 3 ) *.82* Blue Rounds Fired (E 4 ) Red Rounds Fired (E 5 )

NOLH Design of Experiment

Sample Result d(C ij, E j )Υ j

Results Scenario 1 TIMERED RDS BLUE RDS RED CASBLUE CAS d(C ij, E j )Υ j

CART Analysis of Red Rounds, Scenario 1 VariableScore BLSPEED100.00|||||||||||||||||||||||||||||||||||||||||| BLCONC32.12||||||||||||| REDSPEED8.39||| REDACC5.27| BLACC2.97 REDCL2.88 BLCL1.70 REDDET0.24 BLDET0.00 REDCONC0.00 Scenario #Regression Equation of Significant Factors P-Value Scenario 1time = blue speed.00 Scenario 2time = blue speed.00 Scenario 3time = blue speed.00 Scenario 4time = blue speed.00 Classification TreeRegression Equation

Conclusions MANA will calibrate three of the five MOPs MANA will calibrate three of the five MOPs MANA is not valid for use unless rate of fire can be changed in the simulation MANA is not valid for use unless rate of fire can be changed in the simulation If the simulation can be changed, it may come into calibration for the final two MOPs, and then could become a valid CAS If the simulation can be changed, it may come into calibration for the final two MOPs, and then could become a valid CAS

Contributions A process to calibrate agent input against an error tolerance for complex adaptive system simulations A process to calibrate agent input against an error tolerance for complex adaptive system simulations A simulation validation methodology that uses a reverse order from classical validation methodologies A simulation validation methodology that uses a reverse order from classical validation methodologies A composite-mapping process that efficiently searches a problem space and guides a simulation developer towards more effective simulations A composite-mapping process that efficiently searches a problem space and guides a simulation developer towards more effective simulations

CAV Process Conclusions Relates CASS output back to agent inputs and can effectively calibrate a simulation Relates CASS output back to agent inputs and can effectively calibrate a simulation Determines the calibration state of agents, and determines the validation state of the CAS Determines the calibration state of agents, and determines the validation state of the CAS Can guide the modeler and inform the simulation development process Can guide the modeler and inform the simulation development process