1. ABS VV&A Framework Study Phase II Pythagoras COIN – Application of the Validation Framework Lisa Jean Moya WernerAnderson, Inc. lmoya@werneranderson.com Phase II Workshop 3 9 July 2008 7/9/2008 1 P-COIN Validation Briefing Wkshp3 Moya

2. Scenario 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 2

3. Analysis Context Can Pythagoras be used to model population dynamics? In a Disaster Relief/Humanitarian Assistance mission for the stated scenario, is it better to base the MAGTF ashore or afloat? Alternative selection drivers – Do no harm: create no increase in insurgency activity. – Improve the political situation: create an improvement in GOVT and Pro-GOVT sectors. Measures – Box & Whisker plot comparisons of the percent of population by population segment in each insurgency sector at end state (18 months) 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 3

4. Conceptual Model of Civilian Population 4 FARCPro-FARCNeutralPro-GoCGoC Insurgency Behavior Orientation Civilian Population Segments FARC = Revolutionary Armed Forces of Colombia GoC = Govt of Colombia Natural Drift Salience Influencing events 7/9/2008P-COIN Validation Briefing Wkshp3 Moya

5. 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 5

6. 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 6

7. Areas of Interest Core – MAGTF influence on Insurgency orientation Cases – MAGTF/No MAGTF – Ashore/Afloat Dynamic Influences – Natural Drift – Salience Background – Population segments 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 7

8. Areas of Interest Core – MAGTF influence on Insurgency orientation Cases – MAGTF/No MAGTF – Ashore/Afloat Dynamic Influences – Natural Drift – Salience Background – Population segments 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 8 First order assessment

9. Pythagoras-COIN Building Blocks 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 9

10. Salience as a Dynamic Influence 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 10

11. Salience as a Dynamic Influence 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 11

12. As the simulation might progress … 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 12 An extreme example to demonstrate the issue

13. Implications Influence changers only applied w.r.t. initial state; changes in orientation do not change the influence → Dynamic effects of salience and natural drift are not accounted for → Secondary and tertiary effects of MAGTF arrival not accounted for … Dampening on the insurgency orientation ! Risk is that the simulation does not model the desired population dynamics ! Risk is that the dynamics of the MAGTF arrival are not adequately captured 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 13

14. Data 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 14 "Humanitarian exchange." Wikipedia, The Free Encyclopedia. 3 Jul 2008, 10:26 UTC. Wikimedia Foundation, Inc. 9 Jul 2008.http://en.wikipedia.org/w/index.php ?title=Humanitarian_exchange&oldid= 223273181 Data imprecise – Mitigated by “tolerance” and multiple runs Data processing – Need to verify that process results in expected directional & magnitude shifts Data is perishable – Natural drift data has an embedded perishibility Would an actual model use require a “warm-up period” on the Markov Chain? – Other influencing events might significantly change the data values

15. Uses of the Model Markov assumptions in referent descriptions – No long term effects – May need a “warm-up” period Outside the salience or along with salience? Data precision – No exact results; results in the distribution Data perishability – Need to collect new data after significant events – Including MAGTF departure Were the dynamics captured … – Could add other influencing events Could add additional dynamics Q: Can we apply the influencers more robustly? Q: Would changing our initial starting agents help? 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 15

16. Analysis Results Given in box & whiskers plots at end state with data table No statistical comparisons No “hard” description of better Point estimate in time (18 months) 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 16

17. MAGTF Influence Data Expect: Sea better than shore – Urban Middle Class & Urban Poor “drive” result Catholic Church drive more right with Sea vs Shore Displaced Persons “wash”? Salience causes Urban Poor and Middle Class to be like Military; Military to be like Catholic Church – in opposition to the direct MAGTF influence … what would we expect? 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 17

18. Second Order Effects 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 18 What do we expect in the interactions

19. Earlier iteration (Military) 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 19 Not clear from documentation what is being reported These multiple influences appear to be captured – Presuming no population segment strays “too far” from initial state – Except … Military does! No MAGTF Afloat Ashore

20. 20 Analysis Conclusions (IPR#5) “Which COA is better?” cannot be answered with much confidence “What is the chance that ashore is better than afloat?” can be answered with greater confidence – More pro-Government sentiment if Marines stay afloat – Lower pro-FARC sentiment if Marines stay afloat – Marine arrival has a polarizing effect (fewer neutrals) – Marine arrival in either case increases anti-Government sentiments of the Illicit Organizations and the Military Afloat seems to usually do less harm. – There is no factor in our influence estimation that BOTH reduces the negative impact of Ashore AND increase the negative impact of Afloat

21. 21 Analysis Conclusions (IPR#5) (cont) Because the current Markov chain will eventually return to the same steady state, regardless of MAGTF action, once the MAGTF leaves, we need to consider: – Does the MAGTF commander care about leaving a lasting impression? – At what point in time do we measure ‘better’? – Pythagoras could change the final steady state as a function of one or more population segments exceeding or falling below some target value. However, this data was not collected

22. Validation Conclusions 1.The P-COIN simulation fails to capture the dynamic effects intended in the conceptual model of the insurgency in Colombia provided to the P-COIN developer. That is, P- COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF. 2.The data supporting the P-COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming-up” the Markov chains supporting the data used to build the P-COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change. 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 22

23. Validation Conclusions (cont) 3.The P-COIN model should not be used to evaluate long term effects on the population resulting from the influencing event of the MAGTF arrival. 4.This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs. 5.There is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure. However, item 1 implies that P-COIN also provides little insight into the ashore or afloat question in its current implementation. 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 23 Recommend: Applying influencers more robustly

24. What Would Be Useful Better documentation on the P-COIN instantiation Time series data A descriptive walk-thru of results charts (meaning & implications) Verification cases (isolated effects) to ensure dynamics have expected direction (first derivative) and order of magnitude – Descriptions of why we believe it is correct Referent – Better explanations of expected resulting effects from data values Most had to be inferred Order of magnitude differences unknown Expected interaction effects would be “spectacular” 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 24

25. Can the Results Be Trusted? Without trusting the dynamics – Take caution but … – Recommendation is innocuous Under current political circumstances – No … new data is required Can Pythagoras model population dynamics – Probably … more care is required in the instantiation 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 25

26. Levels of Validation Process 7/9/2008P-COIN Validation Briefing Wkshp3 Moya 26 Adapted from Harmon & Youngblood (2005) p. 186 Subjective validation Objective requirements Objective results Objective referent Automated validation Initial (level 0) Initial (level 0) Subjective (level 1) Subjective (level 1) Complete (level 2) Complete (level 2) Accurate (level 3) Accurate (level 3) Confident (level 4) Confident (level 4) Automated (level 5) Automated (level 5) Tier 0, “I have no idea.” Tier 1, “It works; trust me.” Tier 3, “It does the right things; its rep ns are complete enough.” Tier 4, “For what it does; its rep ns are accurate enough.” Tier 5, “I’m confident that this sim n is valid.” SME Ind nt Observer Formal Proof None SME opinion Single source Multiple sources Rigorously derived Correlated with statistical estimates of uncertainties Referent None Validated Analyzed Conceptual Model