Validation Issues in Computational Social Simulation Presented by: Jessica Glicken Turnley Galisteo Consulting Group, Inc. GALISTEO Consulting Group Inc. Human Complex Systems Conference May 2005 Lake Arrowhead, CA

GALISTEO Consulting Group Inc. 2 Validation and verification Target System / Problem Modeling process Validation –does output correspond to target system? what does the model tell you about problem? Output Verification – does each run of the code give you the same output? –

GALISTEO Consulting Group Inc. 3 Output  Comparing the output to the target system - is the model a ‘true’ representation of the target? Need to confirm WHAT is being modeled – rules or behavior Because we are dealing with point data in a complex system that changes over time, the data may be accurate but not true  How good is our theory? Are there enough instances of our target system type to ensure cross-instance consistency of results?  How do we deal with the ‘small ‘n’’ problem? x x x Time

GALISTEO Consulting Group Inc. 4 Output (cont’d)  To what extent does the model address the problem of concern? Was the appropriate theoretical frame used?  What would happen if a different one were used? Were the right data used?  Are we dealing with a ‘wicked problem’ – does the fact of our modeling the problem change the problem?

GALISTEO Consulting Group Inc. 5 The modeling process DataConceptual model Conceptual model formalized into computational form CodeOutput SURROGATE TARGET SYSTEM How is the target system conceptualized? Validation Domain

GALISTEO Consulting Group Inc. 6 Data  Must data be measurable? Observable? Positivist predeliction  What is the nature of the posited social reality? Behavior? The ‘rules’ that shape that behavior?  Etic or emic formulations of the rules?  What uncertainty is inherent in the data? Is it epistemic or aleatory uncertainty?

GALISTEO Consulting Group Inc. 7 Data Is there ‘interpretation-free’ social data?  Paradigmatic preconceptions (cultural narratives) give coherence to what we observe From ethnomethodology to post-modernism  (Wittgenstein) – data acquires meaning (significance) through its embeddedness in a system Is the meaning the significance the modeler assigns the same significance that players assign?

GALISTEO Consulting Group Inc. 8 The modeling process Is there theory-free empirical data? Are observables the only legitimate (or useable) data? What is your confidence in your data? DataConceptual model Conceptual model formalized into computational form CodeOutput

GALISTEO Consulting Group Inc. 9 Conceptual model  Model (by definition) represents only PART of the selected system Data set IS NOT THE SAME AS the target system  Data can be quantitative or qualitative Which part of the system is represented?  Partially determined by target problem  Partially determined by modeler’s paradigmatic preconceptions The rationale for the part selected is framed in terms of theory

GALISTEO Consulting Group Inc. 10 Conceptual model  Theoretical frames help define model structure Is terrorism best understood as…  an ideological expression?  a realpolitik struggle?  a consequence of sociological/ environmental factors?  a vehicle for communication?  Model structure and theoretical frame define salience of data types

GALISTEO Consulting Group Inc. 11 Social/Cultural EE Evans-Pritchard Radclaffe-Brown Structural-functionalism Form follows function White, Marx (cultural) materialism Exploitation of environment by man determines society/culture Benedict Culture and personality Personality traits ascribed to group Durkheim, Levi-Strauss Structuralism Culture is the externalization of mental structures in behavior Geertz Symbolic anthro ‘Reality’ is symbolically mediated - ‘Culture as text’ Derrida Post- modernism ‘Reality’ is irrelevant. It is only accessed through individual history/perspective Wilson Sociobiology Asserts dominance of biology (genes) over culture Note: Names of key theorists are in italics ANTHROPOLOGY

GALISTEO Consulting Group Inc. 12 Biological/ Physical Archeology Human evolution Population diversity Environmental determinism Genomics UnilinearMultilinear Linguistics Chomsky Universal grammar – function of the brain Sapir, Whorf Cultural linguistics – cultural forms in language guide perception of ‘reality’ De Sassure Language is a system of symbols arbitrarily connected to the world Wilson Sociobiology Asserts dominance of biology (genes) over culture

GALISTEO Consulting Group Inc. 13 The modeling process Is there theory-free empirical data? Are observables the only legitimate (or useable) data? DataConceptual model Conceptual model formalized into computational form CodeOutput What is the selection logic for collecting/using some data and not others?

GALISTEO Consulting Group Inc. 14 Formalizing the conceptual model  Model purpose and theoretical frame will help determine modeling approach Stocks and sinks? Agents? Social networks?  Chosen approach may raise data issues Data unavailability – historically not collected because no way to use it Not all relevant data will be directly compatible with modeling approach

GALISTEO Consulting Group Inc. 15 Formalizing the conceptual model Translating narrative into computationally manipulable structures  Forces rigor Rigor should extend from model structure to data and output labels and other aspects of communication  Requires consistency  BUT… Data must be compatible with computational capabilities

GALISTEO Consulting Group Inc. 16 The modeling process Is there theory-free empirical data? Are observables the only legitimate (or useable) data? DataConceptual model Conceptual model formalized into computational form CodeOutput How does the computational format constrain data type and selection? What is the goodness of fit between the formalized and conceptual (textual) model? What is the selection logic for using some data and not others?

GALISTEO Consulting Group Inc. 17 The modeling process Is there theory-free empirical data? Are observables the only legitimate (or useable) data? DataConceptual model Conceptual model formalized into computational form CodeOutput How does the computational format constrain data type and selection? What is the goodness of fit between the formalized and conceptual (textual) model? What is the selection logic for using some data and not others? How accurately does the code reflect the formalized conceptual model?

GALISTEO Consulting Group Inc. 18 Output  Comparing the output to the target system - is the model a ‘true’ representation of the target? Need to confirm WHAT is being modeled – rules or behavior Because we are dealing with point data in a complex system that changes over time, the data may be accurate but not true  How good is our theory? Are there enough instances of our target system type to ensure cross-instance consistency of results?  How do we deal with the ‘small ‘n’’ problem? x x x Time

GALISTEO Consulting Group Inc. 19 Output (cont’d)  To what extent does the model address the problem of concern? Was the appropriate theoretical frame used?  What would happen if a different one were used? Were the right data used?  Are we dealing with a ‘wicked problem’ – does the fact of our modeling the problem change the problem?

GALISTEO Consulting Group Inc. 20 The modeling process Is there theory-free empirical data? Are observables the only legitimate (or useable) data? DataConceptual model Conceptual model formalized into computational form Code Output How does the computational format constrain data type and selection? What is the goodness of fit between the formalized and conceptual (textual) model? What is the selection logic for using some data and not others? How accurately does the code reflect the formalized conceptual model? How well does the output represent the target system? To what extent does the modeling process address the problem of concern?

GALISTEO Consulting Group Inc. 21 Validation questions  What ‘social reality’ does the model represent? Nature of the data Boundaries of the system Observer bias  What is selection logic for data inclusion? Social theory drives determination of salience  What is the level of confidence in the data? Is the uncertainty aleatory or epistemic? Is it effectively represented and communicated?

GALISTEO Consulting Group Inc. 22 Validation questions (cont’d)  Is the modeling approach compatible with problem definition and theoretical approach?  How does the computational format constrain data type and selection? How do these constraints impact the output?  Are non-observable dimensions of human experience (meaning, intent, affect) able to be usefully captured computationally?

GALISTEO Consulting Group Inc. 23 More questions…  If we are using models as a research method…. What is the difference between the types of learnings we can get from experiments vs. models?  (?Computational models create predictions which we match to data, vs. experiments which test predictions developed from data?) How are models and experiments the same?  What is the difference between a model and a simulation?

GALISTEO Consulting Group Inc. 24 And yet more…  In some domains, validation means comparison of model output to experimental results, where experiments are surrogates for the ‘real world’ How do we deal with…  Our inability to do experiments on human organizations?  The ‘poverty’ and/or ‘un-naturalness’ of the social experimental environment when compared to the ‘real world’? How do you integrate several ‘simple’ systems into a complex social interaction?

Answers, thoughts and comments to: Jessica Glicken Turnley