1. Automated Analysis of Simulation Output Data and the AutoSimOA Project
Stewart Robinson and Katy Hoad and Ruth Davies
Warwick Business School
Simulation Group Seminar, 5 May 2006

2. Outline The problem An automated output Analyser: Warm-up analysis
Replications analysis
Run-length analysis: batch means method
Example (demonstration)
Discussion
The AutoSimOA Project

3. The Problem
Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts.
Simulation software generally have very limited facilities for directing/advising on simulation experiments.
Main exception is directing scenario selection through ‘optimisers’.
With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

4. The Problem
Despite continued theoretical developments in simulation output analysis, little is being put into practical use.
There are 3 factors that seem to inhibit the adoption of output analysis methods:
Limited testing of methods
Requirement for detailed statistical knowledge
Methods generally not implemented in simulation software (AutoMod/AutoStat is an exception)
A solution would be to provide an automated output ‘Analyser’.

5. An Automated Output Analyser
Simulation
model
Warm-up
analysis
Run-length
Replications
Use replications
or long-run?
Recommendation
possible?
Recommend-
ation
Output data
Analyser
Obtain more output data
For this project the Analyser looked at:
Warm-up
Run-length
Number of replications
Scenario analysis could be added.

6. An Automated Output Analyser
A prototype Analyser has been developed in Microsoft Excel.
At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.

7. Warm-up Analysis
The Analyser uses 3 procedures from which the user can select the desired warm-up period: MSER-5, Batch Means Bias Detection, Welch’s Method.
The 3 procedures were chosen on the basis of:
Accuracy
Reliability
Generality
Ease of implementation (in Excel)
Requires minimum user intervention
Varied (e.g. not all graphical procedures)

8. Warm-up Analysis Adaptation of Welch’s Method Smoothness Criterion
ith jump:
Average jump:
Increase window size until average jump is reduced to 10% of its value in the raw data.

9. Warm-up Analysis Adaptation of Welch’s Method
Convergence Criterion (average difference rule)
Suppose the moving average plot becomes smooth at observation Xj. Then Cj should have a low value:
Obtain a value for Cj such that Cj/M<L
M is the difference between the max and min Xi for i>=j
Tests showed that a value of L= gave convergence close to that chosen by visual inspection.

10. Replications Analysis
Option to run normal streams or mixed normal and antithetic streams.
Set significance level and confidence interval half width (%).

11. Run-Length Selection: Batch Means Method
Three procedures are used for selecting the batch size:
Fishman’s algorithm (Fishman, 1978)
Law and Carson’s algorithm (Law and Carson, 1979)
ABATCH algorithm (Fishman and Yarberry, 1997)

12. Example The Analyser is applied to an M/M/1 queuing model in SIMUL8:
Arrival rate = 1
Service rate = 0.67
Queue limit: 100
Output statistic: customers in the system
Demonstration!

13. Example
Run Length:
Batch Means example

14. Example Run Length: Batch Means example results 95%
Confidence Interval
Algorithm
Batch mean
St. dev.
Lower
Upper
Size of half width
Relative width
Batch size
Batches
Data points used
Fishman
97.54
1.44
97.10
97.98
0.442
0.453% 16 43
688
Law and Carson
97.48
1.21
96.91
98.05
0.566
0.581% 30 20
600
ABATCH
97.51
0.88
97.06
97.96
0.450
0.462% 48 17
816

15. Discussion
It is possible to link an Automated Analyser in Excel to a simulation software tool.
At present this is just a proof of concept.
Key issues to address:
More thorough testing of output analysis methods for their accuracy and their generality.
Adaptation of methods to sequential procedures and to minimise the need for user intervention.

16. The AutoSimOA Project
A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
Objectives
To determine the most appropriate methods for automating simulation output analysis
To determine the effectiveness of the analysis methods
To revise the methods where necessary in order to improve their effectiveness and capacity for automation
To propose a procedure for automated output analysis of warm-up, replications and run-length
Only looking at analysis of a single scenario

17. The AutoSimOA Project Programme of work: Milestone Timescale (months)
Literature review of warm-up, replications and run-length methods * 3
Development of artificial data sets and collection of simulation models 2
Testing of warm-up methods 6
Testing of replications methods
Testing of run-length methods
Development, testing and revision of candidate methods
Develop and test automated procedure (including prototype software) 5
Dissemination
Total 36

18. The AutoSimOA Project CURRENT WORK:
Literature review of warm-up, replications and run-length methods *
Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…)
and collection of ‘real’ simulation models
Produce output data
Analyse and categorise output: Auto-correlation; Normality; M.A.D…..

19. Example artificial models:
1. Auto-Regressive (2) series
Exponential
Under Damped oscillations
Mean shift 2
Initial Bias Functions:

21. Example artificial models:
2. E4 ~ Erlang(4) / M / 1 Queue
mean 1.8
Traffic Intensity = 0.8

22. Example ‘ real ’ models:
1. Coventry Train Station – Queuing Times

23. Example ‘ real ’ models:
2. Argos – Queuing Times

24. Example ‘ real ’ models:
3. Tesco petrol Station – Queuing Times

25. Example ‘ real ’ models:
4. Café Library – Queuing Times

26. Auto Correlation
Spread round mean
In/out of control
Terminating
Group B
Non-terminating
Trend
Normality
Transient
Steady state
Seasonality