1. SICSA student induction day, 2009Slide 1 Multithreading RePast Models Alex Voss 1, Jing-Ya You 2, Eric Yen 2, Simon Lin 2, Ji-Ping Lin 3, Andy Turner 4 1 School of Computer Science, University of St Andrews 2 Academia Sinica Grid Computing, Academia Sinica, Taiwan 3 Center for Survey Research, Academia Sinica, Taiwan 4 School of Geography, University of Leeds Workshop on Future Directions in Agent Based Modelling Leeds, UK, June 2010

2. SICSA student induction day, 2009Slide 2 Overview Something about the model we want to build on migration Quite a bit about how we tweaked the model to make use of multiple CPUs/cores A bit about what we will do next and questions we want to explore My interest in this…

3. SICSA student induction day, 2009Slide 3 Migration in Taiwan Migration has been an important factor in Taiwanese social development and influenced by outside factors since the 1600s Aim is to test existing theories of migration constructively and to investigate recent developments such as increased outward migration to China. Timely as Taiwan is running another census in 2010. Based on work conducted by Ji-Ping Lin of the Academia Sinica Center for Survey Research on Migration using the 1990 and 2000 Taiwan Population and Housing Census

4. SICSA student induction day, 2009Slide 4 SimTaiwan: Migration in Taiwan Based on Taiwan 2000 Population and Housing Census Dataset is individual-level but with restricted variables Held at Academia Sinica Need to identify additional datasets to complement census Issues with data protection Need to scale up to ca. 22 million individuals They are heterogeneous agents with quite a large number of attributes and history.

5. SICSA student induction day, 2009Slide 5 SimTaiwan Tests Four different model implementations: 1.Naïve single-threaded model 2.Improved single-threaded model 3.Initial multi-threaded model 4.Improved multi-threaded model Test runs with each of these models to measure: 1.Wallclock and CPU time 2.Memory usage 3.Code hotspots 4.Worker thread activity (where applicable)

6. SICSA student induction day, 2009Slide 6 Test Code and Parameters Simplified model with only fertility and mortality, same for all measured models 250k male and 250k female random initial population, running for 365 ticks (=days) Measurements taken using JProfiler 6.0.4 –CPU sampling (5 sec intervals) –Memory allocations recording JVM Parameters: -Xmx8192M –Xss128M Hardware: Dell PowerEdge R610 with 2xXeon E5504 @ 4x2GHz (8 cores total) and 16GB RAM

7. SICSA student induction day, 2009Slide 7 Naïve Serial Version More time spent in RePast scheduling code than in model code because events are scheduled for each individual agent every step.

8. SICSA student induction day, 2009Slide 8 Improved Serial Version Event scheduled on DemographicsContext, code iterating through individual agents Wallclock time down from 5:32 to 2:23 Opens up opportunities for parallelising code as well…

9. SICSA student induction day, 2009Slide 9 Initial Parallel Version Need to partition data to allow multiple worker threads to exploit multiple CPUs & cores PartitionedContext keeping agents in separate HashSets that can return independent Iterators for use by multiple threads. ThreadPoolExecutor with configurable number of worker threads (here 8) Initial version brings only modest / no improvement, wallclock time in some runs > improved serial code Max. CPU utilisation ~ 200% (top)

10. SICSA student induction day, 2009Slide 10 Initial Parallel Version (II) Worker threads blocking a lot on monitors placed around RePast constructs. Main issue seems to be that use of RandomHelper is not thread- safe Simulation schedule relatively minor issue Some contention around simulation objects

11. SICSA student induction day, 2009Slide 11 Improved Parallel Version Overloading some of RePast’s code to make it thread safe. Reducing scope of monitor objects used and pulling code parts that are safe out of synchronized sections Introducing thread-local variable containing a per thread random number generator: protected static ThreadLocal uniform = new ThreadLocal () { @Override protected Uniform initialValue() { RandomEngine generator = new MersenneTwister((int)System.currentTimeMillis()); return new Uniform(generator); } };

12. SICSA student induction day, 2009Slide 12 Improved Parallel Version (II) Monitor contention is eased signficantly Wallclock running time down to 1:03 and max. CPU utilisation up to ~ 600% Time spent in serial code for analysis and production of charts is now significant

13. SICSA student induction day, 2009Slide 13 What have we learned/developed? Advice on structuring RePast code –Parallelise using PartitionedContext –Iteration instead of scheduling events RePast does put some barriers in the way but should be possible to overcome Speed-up initially not as much as hoped for but was overcome by introducing thread-local random number generators Can we factor this work into development of RePast? Or present as tutorial?

14. SICSA student induction day, 2009Slide 14 Next Step: Debugging/Profiling on the Grid Tests to establish optimum number of partitions and threads vs no. of agents Verification of the code and sensitivity analysis Repeated runs to uncover rare events & need to repeat runs to obtain comparable average figures Availability of high-memory machines will become an issue once we scale up to full 22 million agents; –48GB server available at ASGC –Upgrade / purchase of server at St Andrews planned

15. SICSA student induction day, 2009Slide 15 Questions What will happen when we make the model more complex? What decisions about the model (will) affect the degree of parallelism and running times? How many CPU cores can we effectively utilise? –Need machine with more cores as well as more memory –This is now becoming affordable thanks to AMD Commodity computing is what we are interested in – less skills involved (?) and availabilty for social scientists

16. SICSA student induction day, 2009Slide 16 My Interests… Not about building the most sophisticated model or the highest performance one but… about making ABM framework(s) (RePast) usable for social scientists interested in population-level phenomena, addressing the practical issues of developing and using agent-based models in anger cf. challenges outlined by Peter McBurney today