1. Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006

2. Outline The problem A prototype automated output Analyser Findings from prototype Analyser The AutoSimOA Project Current work - zCollecting and characterising real and artificial models

3. The Problem Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts. Simulation software generally have very limited facilities for directing/advising on simulation experiments. Main exception is directing scenario selection through ‘optimisers’. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

4. The Problem Despite continued theoretical developments in simulation output analysis, little is being put into practical use. There are 3 factors that seem to inhibit the adoption of output analysis methods: Limited testing of methods Requirement for detailed statistical knowledge Methods generally not implemented in simulation software (AutoMod/AutoStat is an exception) A solution would be to provide an automated output ‘Analyser’.

5. A Prototype Analyser Simulation model Warm-up analysis Run-length analysis Replications analysis Use replications or long-run? Recommendation possible? Recommend- ation Output data Analyser Obtain more output data Masters Project (3 students). The Analyser looked at: Warm-up Run-length Number of replications Scenario analysis could be added.

6. A Prototype Analyser A prototype Analyser has been developed in Microsoft Excel. At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.

7. Illustration: Warm-up Load Analyser into Excel. Enter name of SIMUL8 model. Specify initial number of replications and run-length to use.

8. Illustration: Warm-up

9. Illustration: Replications

10. Findings from Prototype Analyser It is possible to link an Automated Analyser in Excel to a simulation software tool. This was just a proof of concept. Key issues to address: More thorough testing of output analysis methods for their accuracy and their generality. Adaptation of methods to sequential procedures and to minimise the need for user intervention.

11. A 3 year, EPSRC funded project (GR EP/D033640/1) in collaboration with SIMUL8 Corporation. The AutoSimOA Project Objectives To determine the most appropriate methods for automating simulation output analysis To determine the effectiveness of the analysis methods To revise the methods where necessary in order to improve their effectiveness and capacity for automation To propose a procedure for automated output analysis of warm-up, replications and run-length Only looking at analysis of a single scenario

12. The AutoSimOA Project CURRENT WORK: 1.Literature review of warm-up, replications and run-length methods. 2.Development of artificial data sets (Auto- Regressive; Moving average; M/M/n/p Queues…) 3.Collection of ‘real’ simulation models.

13. Use models / data sets:  Provide a representative and sufficient set of models / data output for use in discrete event simulation research.  Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..

14. Group A …Group N Group B Auto Correlation Spread round mean Normality Trend Cycling/Seasonality Terminating Non-terminating Steady state In/out of control Transient

15. Model characteristics  Deterministic or random  Significant pre- determined model changes (by time)  Dynamic internal changes i.e. ‘feed- back’  Empty-to-empty pattern  Initial transient (warm-up)  Out of control trend ρ≥1  Cycle  Auto-correlation  Statistical distribution Output data characteristics

16. ARTIFICIAL MODELS  Create simple models where theoretical value of some attribute is known. E.g. M/M/1: mean waiting time.  Create simple models where value of some attribute is estimated but model characteristics can be controlled. E.g. Single item inventory management system: Number-in-stock.  Construct output, which closely resembles real model output, with known value for some specific attribute. E.g. AR(1) with Normal errors Create different output types Transient Steady state Steady state cycle Trend + Initial transient (warm-up)

17. Example artificial models: 1. Auto-Regressive (2) series Exponential Under Damped oscillations Mean shift2 Initial Bias Functions:

19. mean 1.8 Example artificial models: 2. E 4 ~ Erlang(4) / M / 1 Queue Traffic Intensity = 0.8

20. REAL MODELS Models created in “real circumstances” that cover each general type of model and output encountered in real life modeling. e.g. Call centre: percentage of calls answered within 30 secs e.g. Production Line Manufacturing Plant: through-put / hour e.g. Fast Food Store: average queuing time e.g. Swimming Pool complex: average number in system Transient Steady State Cycle Steady State With or without warm-up Trend

21. Example ‘ real ’ models: 1. Argos – Number of customers in queue to pay Stochastic model with changing arrival rates. Empty to empty; transient; autocorrelated; non-normal output.

22. Example ‘ real ’ models: 2. Leggings Manufacturing Plant – Through-put / hour Stochastic model. Steady state with warm-up; not autocorrelated; normal output.

23. Example ‘ real ’ models: 3. Sanitory Towel Packing Plant – Through-put / hour Stochastic model with changing productivity in work stations. Steady state daily cycle. Series of means of each cycle: autocorrelated; non-normal output.

24. Use this representative and sufficient set of models/output when The AutoSimOA Project determining the most appropriate methods for automating simulation output analysis determining the effectiveness of the analysis methods revising the methods where necessary in order to improve their effectiveness and capacity for automation In order to propose a procedure for automated output analysis of warm-up, replications and run-length.

25. Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006