1. Simulating Civilization Change with Nexus Cognitive Agents Deborah Duong, Agent Based Learning Systems Gerald Pearman, Augustine Consulting AHFE 12

2. Purpose: To show the benefits of Symbolic Interactionist Simulation for the Simulation of Social Aggregation Purpose and Agenda Agenda 2 Coevolution and Symbolic Interactionist Simulation The Sociological Dynamical System Simulation SISTER: Symbolic Interactionist Simulation of Trade and Emergent Roles Nexus Network Learner Modeling Corruption Social Impact Model

3. Coevolving Agents Genetic Algorithms, Neural Networks, or Reinforcement Learning Algorithms in Agents can co-evolve Agents learn to optimize a function in an environment composed mostly of agents also learning to optimize a function. Moving fitness landscapes –Agents apply selective pressure upon each other. –Selective pressure causes the optimal way to achieve the goal to change over time. Evolutionary Stable Strategies –Maynard-Smith’s theory that a co-evolutionary system converges when no species (or inducing agent) can make a change that will make it better off (Nash Equilibrium) Agents Differentiate into a System –Species (or inducing agents) can be both cooperative and competitive 3 Social Impact Model

4. Symbolic Interactionist Simulation Coevolutionary reinforcement learning algorithm. Autonomous agents each have an inductive mechanism. –For example, an entire Genetic Algorithm or Neural Network. –Agents only experience through their senses (no direct knowledge transfers from other agent minds). Agents choose to interact with other agents based on signs that the other agents display. –Agents induce both the signs to display and the signs that they read. Social system emerges –Signs come to mean behaviors. –Behaviors interlock into a system of expectations. –Value function becomes reward function as society evolves (Adam Smith’s invisible hand). 4 Social Impact Model

5. The First Symbolic Interactionist Simulation: The Sociological Dynamical System Simulation A System of IAC Networks as the Basis for Self Organization in a Sociological Dynamical System Simulation - Duong, 1991; Duong and Reilly, Behavioral Science 1995. Employer agents hire from a pool of employee agents, periodically laying off employees. Employees who are less talented are laid off in greater proportions than those who are more talented. Employees seek employment. Employees can choose to wear three different kinds of signs –One of two signs they can not change. This is their “skin color”. –One of three signs they have to pay for with money from employment. This is their “suit”. –One of three signs they can change arbitrarily. This is their “fad”. Employees induce what sign they should wear based on what has gotten them employed in the past. 5 Social Impact Model

6. The Sociological Dynamical System Simulation (SDSS) Employees have a hidden, unchanging, talent level that employers can not see until after the employee is employed. Employers seek talented employees. Employers hire based on the signs the employees wear, inducing how much talent they have from their signs. The two “races” of employees are equally talented, but the employers do not know this. Employers and Employee Agents both have the same kind of induction mechanism. –Each Agent has their own Interactive Activation and Competition (IAC) Neural Network to induce the signs they read and display. 6 Social Impact Model

7. Interactive Activation and Competition An Employer’s IAC 7 Social Impact Model The IAC is a Constraint Satisfaction (Hopfield) Neural Network. Nodes within each pool are mutually inhibited. A central pool contains memory instances. Each instance has positive connections to its features. To guess the talent of an applicant, the employer “turns on” the applicants features and sees which talent node turns on. Simulates schema, or mental groupings of features that go together

8. SDSS: Emergent Phenomena 8 Social Impact Model Status Symbols –Employee agents learn to buy expensive suits and Employer agents learn to seek expensive suits. –Because the less talented get laid off more, they have less money, and talented employees learn to differentiate themselves. Racism and Social Class –One race gets into a vicious cycle: because of schema, by accident one race gets associated with less talent (even though they are equally talented). –Many talented in a race could not afford suits because they were never given opportunity. –One race would have less money than the others as a result Meaning attributed to Fads

9. SISTER: Symbolic Interactionist Simulation of Trade and Emergent Roles “SISTER: A Symbolic Interactionist Simulation of Trade and Emergent Roles” - Duong 1995, Duong and Grefenstette, JASSS 1/2005. Trade is good for “farmer” agents. –Agents need each of four food groups, and as much as they can get of each. –Agents can make more food if they concentrate their efforts on fewer of them. Agents have efforts to spend on making or trading food as they wish. Agents can trade if they have corresponding trade plans. Agents have a sign to display to attract trade. To learn to trade, agents induce what signs to display and what signs to trade with based on whatever gets them the most of each food in the four food groups at the end of the day. 9 Social Impact Model

10. Coevolving Genetic Algorithms Each agent has an entire genetic algorithm that tells it: –Where to place efforts –What sign to trade with, what and how much to trade –What sign to display 10 Social Impact Model

11. SISTER: Emergent Phenomena 11 Social Impact Model Division of Labor As agents learn to trade, their utility increases, and their sign comes to mean a role Price Goods become valued at standard ratios Money In a third of the runs, one good is traded for the purpose of trading again Different types of stores Central bargain stores and local convenience stores

12. SISTER: Results 12 Social Impact Model

13. Nexus SISTER is a “theoretical” symbolic interactionist simulation –SISTER embodies the formation of social patterns of behavior –Its scenarios are not realistic Nexus is a “data-driven” symbolic interactionist simulation –In order to do analysis, we must start from a scenario in the real world. –Being realistic and theoretically correct at the same time is difficult. –Nexus attempts to mirror the virtuous and vicious cycles of the real world that created its input data. 13 Social Impact Model

14. What is Nexus Network Learner ? 14 One of the two Nexus Cognitive Agent models that Debbie Duong wrote at the OSD/CAPE/Simulation Analysis Center. –Nexus Network Learner models Social Role Networks –Nexus Schema Learner models Cognitive Dissonance A Simulation of Social Role Networks in which Agents learn: –To choose role partners to perform transactions with: -Choice based on signs, social markers and communications on past transaction behaviors. –Transaction behaviors and signs. -Choice based on signs and social markers. –Based on Cultural Values. Social markers, roles, transaction behaviors, signs, role- based communications and cultural values are all input to the program. –Population data determines the initial population tendencies. –Utility and motivation determines how they change.

15. How Does Nexus Network Learner Work? 15 Artificial Intelligence Technologies represent Cognition. –Rule Based. -An ontology of roles with crisp rules for roles. –Represents general social structures, that can be used in many scenarios. –Defines utility of transactions. –Machine Learning. -Bayesian networks initialize social markers, signs/transaction behaviors, and role choice behaviors. -The Bayesian Optimization Algorithm (BOA) changes those behaviors based on the utility of transactions. –BOA can be seeded with initialization data and injected data. –A form of Evolutionary Computation using reinforcement Learning optimizes (satisfices) utility. –As conditionals change, the equilibrium point moves (in accord with the New Institutional Economics).

16. What Happens in Nexus Network Learner? 16 Individual Agents Choose Network Partners. –Ontology tells who may choose and how many. -Example: an “Employer” may choose an average of 5 employees with a standard deviation of three. –Bayesian network tells how the choices are ranked. –Passive role may have an option to reject offer. -Example: an “Employee” may reject an employer because a role relation has told her he steals paychecks. –Ontology may include a chance occurrence of natural attrition. Individual Agents engage in transactions. –Account distributions send funds through networks according to rules in ontology and transaction behaviors in Bayesian networks. –Probability of observing, reporting, and knowing about behaviors are role-based. –Agents may go to jail, and not be allowed to participate in transactions for a time. Every N cycles, they judge their learned strategies by utility based on transactions that their valued role partners engaged in. –Ontology determines culturally valued individuals and transactions. –After testing all strategies agents recombine them.

17. Performing Tests with Nexus Network Learner 17 A wide variety of tests relevant to Irregular Warfare (IW) may be performed. For example, new network formations and behaviors may be tested based on many different things… –The effect of different utility functions. -For example, make agents care only for self rather than larger social network. –The effect of different penalties. -For example, a penalty attribute that encodes different jail terms or different chances of getting caught. –The effect of different exogenous resources. -For example, test resource rents or foreign aid. –The effect of different abilities to observe. -For example, the effect of a media agent. –The effect of removing different agents. -For example, measure how long it takes to replace a terrorist leader Monte Carlo methods reveal if new structures are the result of different CONOPS. –Bayesian Networks make Nexus Stochastic

18. How Nexus Agents Learn 18 As each agent learns, all the agents coevolve, making them very adaptive. –Every agent has its own private learning algorithm. –Their behaviors effect the larger social structure and the larger social structure effects their behaviors. -Micro-Macro Integration is modeled. –They can adapt to data from other simulations and to initial country data as well. The learning algorithm in each agent makes the adaptation to data flexible. –BOA (Bayesian Optimization Algorithm) can start learning from initial data. -In the calibration phase. agents to adapt to initial data, so that they generate it though their perceptions and motivations. -Thus they “explain” the data, going from correlation to cause. –This greater ability to ingest data also allows them to meld with other simulations in a composition. -Together, composed simulations create a coherent picture of the social environment. -Conflicts are resolved through mutual adaptation.

19. Use Case: Modeling Corruption with Nexus Network Learner 19 Social Impact Model

20. Interpretive Social Science Used in Nexus From economics: The New Institutional Economics (NIE) (North) –Institutions (Social and Legal Norms and Rules) underlie economic activity and constitute economic incentive structures –Institutions come from the efforts of agents to understand their environment, so as to reduce uncertainty, given their limited perception –When some uncertainties are reduced, others arise, causing economic change –To find the leverages to corruption, NIE would look at actor’s definition of their environment, and how this changes incentives and thus institutions From sociology: Symbolic Interactionism (Mead) –Roles and Role Relations (such as in trade roles and trade relations) are learned, created during the display and interpretation of signs (such as gender, ethnicity, and other demographic characteristics) –Institutions (social and legal norms and rules) are commonly accepted interpretations of symbols, that start out as a subjective perception and engrained in society as an objective rule [1][1] See http://coase.org/niereadinglist.htm for an extended reading listhttp://coase.org/niereadinglist.htm [2][2] See Duong, Deborah Vakas, “The Generative Power of Signs: Tags for Cultural. Reproduction” Handbook of Research on Agent-Based Societies: Social and Cultural Interactions, Goran Trajkovsky and Samuel Collins, eds., 2008. http://www.scs.gmu.edu/~dduong/GenerativePowerOfSigns.pdf and also Blumer, Herbert (1969). Symbolic Interactionism: Perspective and Method. Berkeley: University of California Press.http://www.scs.gmu.edu/~dduong/GenerativePowerOfSigns.pdf [3][3] Duong, Deborah Vakas and John Grefenstette. “SISTER: A Symbolic Interactionist Simulation of Trade and Emergent Roles”. Journal of Artificial Societies and Social Simulation, January 2005. http://jasss.soc.surrey.ac.uk/8/1/1.html. http://jasss.soc.surrey.ac.uk/8/1/1.html

21. Nexus Main Components Nexus models individuals and their interactions. Individuals have various roles on the three different networks and dynamically interact with other agents through these roles. –Retailer – Customer –Government Employer – Government Employee –Head Of Household– Dependent Role Networks are Input to Nexus Only commodity is Money –Agents pass money to other agents’ accounts –External support is in the form of injections of funds to certain individuals (that have certain roles) –Utility of agents (their “happiness”) is raised when they spend the money on things they need Trade Network Bureaucratic Network Kinship Network Role Interactions Individual Determined Traits Situational Traits Behavioral Traits Transaction-based Utility Experience Cognition Institutions Emerged Learned Attributes External Control User-defined Policies External Support

22. Nexus Main Components: Individuals They want their kin to be happy, and can think about how to adjust their behaviors towards that goal, based on experience of what met that goal in the past They have the demographic characteristics, both determined and situational, of the modeled country –Determined Traits: Gender, Ethnicity, Age, etc. –Behavioral Traits: Tendency to Steal or Bribe, based on other traits and on learning during the run –Situational Traits: Employment, Are they under penalty, etc. They actively seek role relationships, following socio-cultural rules about who proposes the relationship, what sort of person is chosen –For example, a husband chooses a wife, or an employer chooses an employee –They judge others based on characteristics they can see or they have heard rumors about There role responsibilities include the distribution of funds to accounts they are responsible for They are happy when funds flow through certain accounts, for example, from the household budget to the grocery store income. They have differing length legal penalties, as well as social stigma

23. Nexus Main Components: Role and Role Relations There are eight types of corruption relations possible in the three networks (example actions provided): –Nepotism: Hiring Kin/ Trade Network –Commission for Illicit Services: Bribing/Government Network –Misappropriation: Stealing/ Trade or Gov Network –Rig Election: Elected Officials bribing for Employment –Gratuity: Bribing/ Trade Network –Unwarranted Payment: Accepting Bribes/Government Network –Levy Toll Sidelining: Stealing/Government Network –Scam: Stealing From Customer in Trade Sector. There are many other types of role and role relations (64) in the model: –Each role has a corresponding role –Roles are dynamic (such as an agent can move from a government employee to unemployed) Bureaucratic Network Gov Employee Gov Employer Attempts to Bribe Accepts/Rejects Bribe Observer

24. Nexus Main Components: Kinship Network Active RolesCorresponding Passive Role FatherChild Head Of HouseholdHome Receiver HusbandWife Brother Dependent (Provider) MaternalAunt MaternalCousin MaternalGrandparent Mother Parent PaternalCousin PaternalGrandparent PaternalUncle Sibling Sister Spouse Provider (Dependent) Three main active roles, from which 14 more are derived Derived Roles are used to model Residence –Utility (Satisfaction) calculated based on Residence in Anthropology –Matrilineal, Patrilineal of Neolocal Support account goes from Provider to Dependent Derived Roles

25. Nexus Main Components: Trade Network Derived Roles are ten different income levels Accounts include personal salary, employee salaries, money for office purchases Active RolesCorresponding Passive Role HeadOfCorporationCorporateReceiver CustomerRetailer EmployerEmployee PurchaserVendor Service ProvideeService Provider

26. Nexus Main Components: Bureaucratic Network Active RolesCorresponding Passive Role TaxpayerTaxman GovernmentEmployerGovernmentEmployee GovernmentPurchaserGovernmentVendor HeadOfGovernmentGovernmentReceiver Service ProvideeService Provider Derived Roles are ten different pay grades Accounts include corporate and income taxes, government salaries, government office money, government money for purchases

27. Experiments 27 Social Impact Model

28. Experiment: Stuck in Stealing Mode 28 Comparison of the evolution of an African society which initiates in a strong vicious cycle of stealing to one with more moderate levels of stealing. If they started out stealing excessively, they never learned not to, never attempted to find service providers who wouldn’t steal from them If the stealing is in more moderate amounts, agents learn to find service providers that do not steal from them within two years Agents in a stealing vicious cycle never use bribing to accomplish goals Agents with moderate stealing used bribes, but after fifteen years, employers and service providers stopped bribing Convergence occurs at the fifteen year mark After 15 years in both the excessive stealing scenario and the ‘normal’ stealing scenario, we see homogeneous responses across strategies. For instance, agents generally provided the same response for a particular parameter (say ‘Bribe-ForServices’) after 15 years as opposed to more heterogeneous responses for the same parameter after two years. Implication: Diversity of Behavior is needed for flexibility

29. Experiment: Reverse Engineer Terrorist Networks In the US Army 2011 Tactical Wargame Nexus Network Learner was used to simulate terrorist networks in Afghanistan. Added onto the above dynamic networks were a Pashtoon Tribal network, a terrorist network, and a coalition forces network. In these networks, if agents representing host nation citizens obtained a role they behaved in that role in the expected frequency. They also noted others behaviors according to that role, and had a chance of discussing that behavior with coalition forces according to the role and the popularity of coalition forces. Intelligence Community Wargame players were able to use the behavioral messages to reconstruct the terrorist networks. The dynamic and changing nature of networks made this task overly challenging to those who lack Intelligence training 29

30. Summary 30 Social Impact Model

31. Symbolic Interactionist Simulation 31 Symbolic Interactionist Simulation is a form of Reinforcement Learning by Coevolution. Agents learn associated rules in the form of actions to take with other agents based on signs displayed and read. Symbolic Interactionist Simulation can be Theoretical (SISTER) or Data-Driven (Nexus). Symbolic Interactionist Simulation can model the motivation based vicious and virtuous cycles of behavior that determine social structure

32. Questions and Comments 32 Social Impact Model POC: Deborah Duong dduong@agentBasedLearningSystems.com