1. Presented by Parallel Discrete Event Simulation (PDES) at ORNL Kalyan S. Perumalla, Ph.D. Oak Ridge National Laboratory

2. 2 Perumalla_PDES_0611 PDES: Selected application areas  Network simulation  Internet protocols, security, P2P designs, …  Traffic simulation  Emergency planning/ response, environmental policy analysis, urban planning, …  Social dynamics simulation  Operations planning, foreign policy, marketing, …  Sensor simulations  Wide area monitoring, situational awareness, border surveillance, …  Organization simulations  Command and control, business processes, … Emergencies Current and future defense systems Protection and awareness systems Global and local events

3. 3 Perumalla_PDES_0611 High Performance PDES kernel requirements  Global time synchronization  Total time-stamped ordering of events  Paramount for accuracy  Fast synchronization  Scalable, application-independent, time-advance mechanisms  Critical for real-time and as-fast-as-possible execution  Support for fine-grained events  Minimal overhead relative to event processing times  Application computation is typically only 5 µs to 50 µs per event  Conservative, optimistic, and mixed modes  Need support for the principal synchronization approaches  Useful to choose mode on per-entity basis at initialization  Desirable to vary mode dynamically during simulation  General-purpose API  Reusable across multiple applications  Accommodates multiple techniques  Lookahead, state saving, reverse computation, multicast, etc.

4. 4 Perumalla_PDES_0611 µsik—unique PDES “micro-kernel”  Some recent results of fine-grained PDES benchmark (phold)  Among the largest/fastest scalability results in parallel discrete event simulation 0 10 20 30 40 50 60 70 1282565121024 Aggregate events/s (M) Number of processors LP = 1 thousand, MSG = 1 million LP = 1 million, MSG = 100 million LP = 1 million, MSG = 1 billion Unique mixed-mode kernel  The only scalable mixed-mode kernel in the world  Supports conservative, optimistic, and mixed modes in a single kernel Used in a variety of applications  DES-based vehicular traffic models  DES-based plasma physics models  DES-based neurological models  Largest internet simulations

5. 5 Perumalla_PDES_0611 µsik scaled to over 10 4 processors  Some recent results of fine-grained PDES benchmark  On Blue Gene Watson (BGW) at IBM TJ Watson Research Center  Well-known PHOLD benchmark, with 1 million logical processes, 10 million pucks  The largest and fastest scalability results in PDES recorded to date

6. 6 Perumalla_PDES_0611 µsik micro-kernel internals Processable PpPp User LPs Kernel LPs Micro-kernel Future event list Processed event list Local virtual time When update kernel Q’s? New LP added or deleted LP executes an event LP receives an event LP = Logical process KP = Kernel process ECTS = Earliest committable time stamp EPTS = Earliest processable time stamp EETS = Earliest emittable time stamp PEL = Processed event list FEL = Future event list LVT = Local virtual time EPTS Q Commitable PcPc ECTS Q Emittable PePe EETS Q LP PEL→t LVT FEL→t KP

7. 7 Perumalla_PDES_0611 libSynk: µsik’s synchronization core OS/Hardware libSynk µsik Process µsik Process µsik Process TM TM RedTM Null RM RM Bar FM MyrFM TCP FM FM ShMFM MPI XY Implies X uses Y Network

8. 8 Perumalla_PDES_0611 µsik micro-kernel capabilities  μsik is currently able to support the following:  Lookahead-based conservative and/or optimistic execution  Reverse computation-based optimistic execution  Checkpointing-based optimistic execution  Resilient optimistic execution (zero rollbacks)  Constrained, out-of-order execution  Preemptive event processing  Any combinations of the above  Automated, network-throttled flow control  User-level event retraction  Process-specific limits to optimism  Dynamic process addition/deletion  Shared and/or distributed memory execution  Process-oriented views  It accommodates addition of the following:  Synchronized multicast  Optimistic dynamic memory allocation  Automated load-balancing

9. 9 Perumalla_PDES_0611 SensorNet: Parallel simulation/ immersive test-bed  Seamless integrated testbed to incorporate a variety of important simulations, stimulations, and live devices  Achieves unified capabilities and significant fidelity for test and evaluation of CB sensor device-based designs, concepts, and operations TOSSIM SimDriver SimComm Tiny Viz Script Interpreter External Event Buffer Tython Reflected Simulation Script Python Environ- mental Model Proxy SCIPUFF Weather Model Plume Models Comm. Models Plume Models Sensor Net Simulator Weather Simulator Operations Simulator Live Devices CB Sensor Network Testbed Framework CurrentEnvisioned

10. 10 Perumalla_PDES_0611 SensorNet: Simulation-based analysis for plume tracking  Environmental phenomenon exhibits high variability  Phenomenon drives the sensor network’s computation and communication  Trace gathered at base station of sensed phenomenon reflects high variability  Communication effects induce unpredictable gaps in series  Accurate, integrated simulation of phenomenon and communication captures complex interdependencies 1 Base Station Source Release 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 Sensor Network Layout -96.0-91.2-86.4 Longitude 40.6 42.6 44.6 46.6 48.5 50.5 Turbulent Winds Surface Dosage CHEM at T = 17.0 h 40.6 42.6 44.6 46.6 48.5 50.5 Light Winds Surface Dosage CHEM at T = 17.0 h 40.6 42.6 44.6 46.6 48.5 50.5 No Winds Surface Dosage CHEM at T = 17.0 h Kg  s/cm 3 1.00E-02 1.00E-03 1.00E-04 1.00E-05 1.00E-06 1.00E-07 1.00E-08 Latitude 300400500 Time (s) 0 10 20 No Winds Winds Turbulent 30 40 600 Mean Concentration (10-6 kg  s/m 3 ) 0 10 20 30 40 No Winds Winds Turbulent

11. 11 Perumalla_PDES_0611 SCATTER: Ultra-Scale PDES-based mobility simulations  Scalable tool for transportation and energy/event/emergency research  Regional scale: multiple states  10 6 –10 7 intersections  Current tool capabilities  At most 10 4 intersections  Faster than real time is very useful Higher accuracy Lower accuracy Fidelity Speed Faster Slower Desirable Network flow methods OREMS CORSIM MITSIM TRANSIMS SCATTER Good for emissions estimates, traffic analysis,... Good for rough estimates of evacuation delay,... Our approach: SCATTER  DES models  vs time-stepped  Parallel execution  vs sequential  Scalability to high- performance computing  10 2 –10 3 CPUs  Important behaviors  Kinetic + non-kinetic

12. 12 Perumalla_PDES_0611 SCATTER: Benchmark performance µs/event Number of vehicles Nodes = 9 Nodes = 1089 Event processing speed Real time/simulated time Number of vehicles Nodes = 9 Nodes = 1089 Speedup over real-time Speedup over sequential run Number of processors Parallel Speedup Very low event processing overhead (ms)! Significant speedup with parallel execution! Significantly faster than real time with 1 million vehicles!

13. 13 Perumalla_PDES_0611 Contacts Kalyan S. Perumalla Oak Ridge National Laboratory (865) 241-1315 perumallaks@ornl.gov 13 Perumalla_PDES_0611