1. APPENDIX D R ANDOM N UMBER G ENERATION Organization of chapter in ISSO* – General description and linear congruential generators Criteria for “good” random number generator – Random variates with general distribution Different types of random number generators *Note: These slides cover some topics not included in ISSO Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall

2. D-2 Uniform Random Number Generators Want a sequence of independent, identically distributed U(0, 1) random variables However, random number generators (RNGs) produce a deterministic and periodic sequence of numbers What qualities should the generators have?

3. D-3 Criteria for ‘Good’ Random Number Generators Long period Good distribution of the points (low discrepancy) Able to pass some statistical tests Speed/efficiency Portability – can be implemented easily using different languages and computers Repeatability – should be able to generate the same sequence over again

4. D-4 Generating Random Numbers Given a transition function, f, the state at step n is given by The output function, g, produces the outputs as The output sequence is Want the sequence period to be close to 2 b, where b corresponds to the number of bits

5. D-5 Types of Random Number Generators Linear – most commonly used Combined – can increase period and improve statistical properties Non-linear – structure is less regular than linear generators but more difficult to implement

6. D-6 Linear Congruential Generators U(0,1) numbers via linear congruential generators (LCG) are calculated by These are the most widely used and studied random number generators The values a, c, and m should be carefully chosen

7. D-7 Linear Congruential Generators Some values for a and m (assuming c = 0) –a = 23, m = 10 8 +1 (original implementation) –a = 65534, m = 2 29 (poor because of high order correlations) –a = 5 15, m = 2 47 (long period, good distribution, but lower order bits should not be trusted) –a = 16807, m = 2 31 –1 (this has been discussed as the minimum standard for RNGs)

8. D-8 0500100015002000 0.42 0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58 Empirical Mean Number of Samples m = 9, a = 4, c = 1 m = 27, a = 26, c = 5 m = 482, a = 13, c = 14 m = 2 31 – 1, a = 4, c = 1

9. D-9 Lattice Structure (Exercise D.2) 00.20.40.60.81 0 0.2 0.4 0.6 0.8 1 UkUk Uk–-1Uk–-1 30 points 00.20.40.60.81 0 0.2 0.4 0.6 0.8 1 UkUk Uk–-1Uk–-1 96 points

10. D-10 Fibonacci Generators These are generators where the current value is the sum (or difference, or XOR) or the two preceding elements Lagged Fibonacci generators use two numbers earlier in the sequence

11. D-11 Multiple Recursive Generators Multiple recursive generators (MRGs)are defined by where the a i belong to {0,1,…,m – 1} and For prime m and properly chosen a i ’s, the maximal period is m k -1

12. D-12 Combining Generators Used to increase period length and improve statistical properties Shuffling: uses the second generator to choose a random order for the numbers produced by the final generator Bit mixing: combines the numbers in the two sequences using some logical or arithmetic operation (addition and subtraction are preferred)

13. D-13 Nonlinear Generators Nonlinearity can be introduced by using a linear transition function with a nonlinear output function An example is the explicit inversive generator where

14. D-14 Random Number Generators Used in Common Software Packages Important to understand the types of generators used in statistical software packages and their limitations MATLAB: –Versions earlier than 5: a linear congruential generator with –Versions 5 & 6: a lagged Fibonacci generator combined with a shift register random integer generator with period EXCEL: u n = fractional part (9821×u n –1 + 0.211327); period SAS (v6): LCG with period

15. D-15 Inverse-Transform Method for Generating Non-U(0,1) Random Numbers Let F(x) be the distribution function of X Define the inverse function of F by Generate X by Example: exponential distribution

16. D-16 Accept  Reject Method Let p X (x) be the density function of X Find a function  (x) that majorizes p X (x) –, q is a density function Generate X by –Generate U from U(0,1) (*) –Generate Y from q(y), independent of U –If, then set X=Y. Otherwise, go back to (*) Probability of acceptance (efficiency) = 1/c Related to Markov chain Monte Carlo (MCMC) method (see Exercise 16.4)

17. D-17 pX(x)pX(x) q(x) = U(0,1) 0 0.2 0.4 0.6 0.8 1.0 1.2 0.5 1.0 1.5 2.0 2.5

18. D-18 U ~ U(0,1): 0.9501, 0.2311, 0.6068, 0.4860, 0.8913,  Y ~ q(y)  U(0,1): 0.7621, 0.4565, 0.0185, 0.8214, 0.4447,  : 0.7249, 0.8131,  X ~ P X (x): 0.7621, 0.4565,  reject accept

19. D-19 References for Further Study L’Ecuyer, P. (1998), “Random Number Generation,”in Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice (J. Banks, ed.), Wiley, New York, Chapter 4. Neiderreiter, H. (1992), Random Number Generation and Quasi-Monte Carlo Methods, SIAM, Philadelphia.