Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Surprising Consequences of Randomness LS 829 Mathematics in Science and Civilization Feb 6, 2010 6/25/20151LS 829 - 2010.

Similar presentations


Presentation on theme: "The Surprising Consequences of Randomness LS 829 Mathematics in Science and Civilization Feb 6, 2010 6/25/20151LS 829 - 2010."— Presentation transcript:

1 The Surprising Consequences of Randomness LS 829 Mathematics in Science and Civilization Feb 6, 2010 6/25/20151LS 829 - 2010

2 Sources and Resources Statistics: A Guide to the Unknown, 4 th ed., by R.Peck, et al. Publisher: Duxbury, 2006 Taleb, N. N. (2008) Fooled by Randomness The Hidden Role of Chance in the Markets and Life, 2nd Edition. Random House. Mlodinow, L (2008) The Drunkard’s Walk. Vintage Books. New York. Rosenthal, J.S. (2005) Struck by Lightning Harper Perennial. Toronto. www.stat.sfu.ca/~weldon 6/25/20152LS 829 - 2010

3 Introduction Randomness concerns Uncertainty - e.g. Coin Does Mathematics concern Certainty? - P(H) = 1/2 Probability can help to Describe Randomness & “Unexplained Variability” Randomness & Probability are key concepts for exploring implications of “unexplained variability” 6/25/20153LS 829 - 2010

4 AbstractReal World MathematicsApplications of Mathematics Probability Applied Statistics Surprising FindingsUseful Principles Nine Findings and Associated Principles 6/25/20154 LS 829 - 2010

5 Example 1 - When is Success just Good Luck? An example from the world of Professional Sport 6/25/20155LS 829 - 2010

6 6/25/20156LS 829 - 2010

7 6/25/20157LS 829 - 2010

8 Sports League - Football Success = Quality or Luck? 6/25/20158LS 829 - 2010

9 6/25/20159LS 829 - 2010

10 Recent News Report “A crowd of 97,302 has witnessed Geelong break its 44-year premiership drought by crushing a hapless Port Adelaide by a record 119 points in Saturday's grand final at the MCG.” (2007 Season) 6/25/201510LS 829 - 2010

11 Sports League - Football Success = Quality or Luck? 6/25/201511LS 829 - 2010

12 Are there better teams? How much variation in the total points table would you expect IF every team had the same chance of winning every game? i.e. every game is 50-50. Try the experiment with 5 teams. H=Win T=Loss (ignore Ties for now) 6/25/201512LS 829 - 2010

13 5 Team Coin Toss Experiment My experiment … T T H T T H H H H T TeamPoints 316 212 58 14 40 But all teams Equal Quality (Equal Chance to win) Experiment Result -----> Win=4, Tie=2, Loss=0 but we ignore ties. P(W)=1/2 5 teams (1,2,3,4,5) so 10 games as follows 1-2,1-3,1-4,1-5,2-3,2-4,2-5,3-4,3-5,4-5 6/25/201513LS 829 - 2010

14 Implications? Points spread due to chance? Top team may be no better than the bottom team (in chance to win). 6/25/201514LS 829 - 2010

15 Simulation: 16 teams, equal chance to win, 22 games 6/25/201515LS 829 - 2010

16 Sports League - Football Success = Quality or Luck? 6/25/201516LS 829 - 2010

17 Does it Matter? Avoiding foolish predictions Managing competitors (of any kind) Understanding the business of sport Appreciating the impact of uncontrolled variation in everyday life 6/25/201517LS 829 - 2010

18 Point of this Example? Need to discount “chance” In making inferences from everyday observations. 6/25/201518LS 829 - 2010

19 Example 2 - Order from Apparent Chaos An example from some personal data collection 6/25/201519LS 829 - 2010

20 Gasoline Consumption Each Fill - record kms and litres of fuel used Smooth ---> Seasonal Pattern …. Why? 6/25/201520LS 829 - 2010

21 Pattern Explainable? Air temperature? Rain on roads? Seasonal Traffic Pattern? Tire Pressure? Info Extraction Useful for Exploration of Cause Smoothing was key technology in info extraction 6/25/201521LS 829 - 2010

22 Jan 12, 2010STAT 10022 Intro to smoothing with context …

23 Optimal Smoothing Parameter? Depends on Purpose of Display Choice Ultimately Subjective Subjectivity is a necessary part of good data analysis 6/25/201523LS 829 - 2010

24 Summary of this Example Surprising? Order from Chaos … Principle - Smoothing and Averaging reveal patterns encouraging investigation of cause 6/25/201524LS 829 - 2010

25 3. Weather Forecasting 6/25/201525LS 829 - 2010

26 Chaotic Weather 1900 – equations too complicated to solve 2000 – solvable but still poor predictors 1963 – The “Butterfly Effect” small changes in initial conditions -> large short term effects today – ensemble forecasting see p 173 Rupert Miller p 178 – stats for short term … 6/25/201526LS 829 - 2010

27 Conclusion from Weather Example? It may not be true that weather forecasting will improve dramatically in the future Some systems have inherent instability and increased computing power may not be enough the break through this barrier 6/25/201527LS 829 - 2010

28 Example 4 - Obtaining Confidential Information How can you ask an individual for data on Incomes Illegal Drug use Sex modes …..Etc in a way that will get an honest response? There is a need to protect confidentiality of answers. 6/25/201528LS 829 - 2010

29 Example: Marijuana Usage Randomized Response Technique Pose two Yes-No questions and have coin toss determine which is answered Head 1. Do you use Marijuana regularly? Tail 2. Is your coin toss outcome a tail? 6/25/201529LS 829 - 2010

30 Randomized Response Technique Suppose 60 of 100 answer Yes. Then about 50 are saying they have a tail. So 10 of the other 50 are users. 20%. It is a way of using randomization to protect Privacy. Public Data banks have used this. 6/25/201530LS 829 - 2010

31 Summary of Example 4 Surprising that people can be induced to provide sensitive information in public The key technique is to make use of the predictability of certain empirical probabilities. 6/25/201531LS 829 - 2010

32 5. Randomness in the Markets 5A. Trends That Deceive 5B. The Power of Diversification 5C. Back-the-winner fallacy 6/25/2015LS 829 - 201032

33 5A. Trends That Deceive People often fail to appreciate the effects of randomness 6/25/201533LS 829 - 2010

34 The Random Walk 6/25/2015LS 829 - 201034

35 Trends that do not persist 6/25/201535LS 829 - 2010

36 Longer Random Walk 6/25/2015LS 829 - 201036

37 Recent Intel Stock Price 6/25/2015LS 829 - 201037

38 Things to Note The random walk has no patterns useful for prediction of direction in future Stock price charts are well modeled by random walks Advice about future direction of stock prices – take with a grain of salt! 6/25/2015LS 829 - 201038

39 5B. The Power of Diversification People often fail to appreciate the effects of randomness 6/25/201539LS 829 - 2010

40 Preliminary Proposal I offer you the following “investment opportunity” You give me $100. At the end of one year, I will return an amount determined by tossing a fair coins twice, as follows: $0 ………25% of time (TT) $50.……. 25% of the time (TH) $100.……25% of the time (HT) $400.……25% of the time. (HH) Would you be interested? 6/25/201540LS 829 - 2010

41 Stock Market Investment Risky Company - example in a known context Return in 1 year for 1 share costing $1 0.00 25% of the time 0.50 25% of the time 1.00 25% of the time 4.00 25% of the time i.e. Lose Money 50% of the time Only Profit 25% of the time “Risky” because high chance of loss 6/25/201541LS 829 - 2010

42 Independent Outcomes What if you have the chance to put $1 into each of 100 such companies, where the companies are all in very different markets? What sort of outcomes then? Use coin- tossing (by computer) to explore 6/25/201542LS 829 - 2010

43 Diversification Unrelated Companies Choose 100 unrelated companies, each one risky like this. Outcome is still uncertain but look at typical outcomes …. One-Year Returns to a $100 investment 6/25/201543LS 829 - 2010

44 Looking at Profit only Avg Profit approx 38% 6/25/201544LS 829 - 2010

45 Gamblers like Averages and Sums! The sum of 100 independent investments in risky companies is very predictable! Sums (and averages) are more stable than the things summed (or averaged). Square root law for variability of averages Variation -----> Variation/  n 6/25/201545LS 829 - 2010

46 Summary - Diversification Variability is not Risk Stocks with volatile prices can be good investments Criteria for Portfolio of Volatile Stocks –profitable on average –independence (or not severe dependence) 6/25/2015LS 829 - 201046

47 5C - Back-the-winner fallacy Mutual Funds - a way of diversifying a small investment Which mutual fund? Look at past performance? Experience from symmetric random walk … 6/25/201547LS 829 - 2010

48 Implication from Random Walk …? Stock market trends may not persist Past might not be a good guide to future Some fund managers better than others? A small difference can result in a big difference over a long time … 6/25/201548LS 829 - 2010

49 A simulation experiment to determine the value of past performance data Simulate good and bad managers Pick the best ones based on 5 years data Simulate a future 5-yrs for these select managers 6/25/201549LS 829 - 2010

50 How to describe good and bad fund managers? Use TSX Index over past 50 years as a guide ---> annualized return is 10% Use a random walk with a slight upward trend to model each manager. Daily change positive with probability p Good managerROR = 13%pap=.56 Medium managerROR = 10%pap=.55 Poor managerROR = 8% paP=.54 6/25/201550LS 829 - 2010

51 6/25/201551LS 829 - 2010

52 Simulation to test “Back the Winner” 100 managers assigned various p parameters in.54 to.56 range Simulate for 5 years Pick the top-performing mangers (top 15%) Use the same 100 p-parameters to simulate a new 5 year experience Compare new outcome for “top” and “bottom” managers 6/25/201552LS 829 - 2010

53 START=100 6/25/201553LS 829 - 2010

54 Mutual Fund Advice? Don’t expect past relative performance to be a good indicator of future relative performance. Again - need to give due allowance for randomness (i.e. LUCK) 6/25/201554LS 829 - 2010

55 Summary of Example 5C Surprising that Past Perfomance is such a poor indicator of Future Performance Simulation is the key to exploring this issue 6/25/201555LS 829 - 2010

56 6. Statistics in the Courtroom Kristen Gilbert Case Data p 6 of article – 10 years data needed! Table p 9 of article – rare outcome if only randomness involved. P-value logic. Discount randomness but not quite proof Prosecutor’s Fallacy P[E|I] ≠ P[I|E] 6/25/201556LS 829 - 2010

57 Lesson from Gilbert Case Statistical logic is subtle Easy to misunderstand Subjectivity necessary in some decision- making 6/25/2015LS 829 - 201057

58 Example 7 - Lotteries: Expectation and Hope Cash flow –Ticket proceeds in (100%) –Prize money out (50%) –Good causes (35%) –Administration and Sales (15%) 50 % $1.00 ticket worth 50 cents, on average Typical lottery P(jackpot) =.0000007 6/25/201558LS 829 - 2010

59 How small is.0000007? Buy 10 tickets every week for 60 years Cost is $31,200. Chance of winning jackpot is = …. 1/5 of 1 percent! 6/25/201559LS 829 - 2010

60 Summary of Example 7 Surprising that lottery tickets provide so little hope! Key technology is simple use of probabilities 6/25/201560LS 829 - 2010

61 Nine Surprising Findings 1.Sports Leagues - Lack of Quality Differentials 2.Gasoline Mileage - Seasonal Patterns 3.Weather - May be too unstable to predict 4.Marijuana – Can get Confidential info 5A. Random Walk – Trends that are not there 5B. Risky Stocks – Possibly a Reliable Investment 5C. Mutual Funds – Past Performance not much help 6.Gilbert Case – Finding Signal amongst Noise 7. Lotteries - Lightning Seldom Strikes 6/25/201561LS 829 - 2010

62 Nine Useful Concepts & Techniques? 1.Sports Leagues - Unexplained variation can cause illusions - simulation can inform 2.Gasoline Mileage - Averaging (and smoothing) amplifies signals 3.Weather – Beware the Butterfly Effect! 4.Marijuana – Randomized Response Surveys 5A. Random Walks – Simulation can inform 5B. Risky Stocks - Simulation can inform 5C.Mutual Funds - Simulation can inform 6.Gilbert Case – Extracts Signal from Noise 7.Lotteries – 14 million is a big number! 6/25/201562LS 829 - 2010

63 Role of Math? Key background for –Graphs –Probabilities –Simulation models –Smoothing Methods Important for constructing theory of inference 6/25/201563LS 829 - 2010

64 Limitation of Math Subjectivity Necessary in Decision-Making Extracting Information from Data is still partly an “art” Context is suppressed in a mathematical approach to problem solving Context is built in to a statistical approach to problem solving 6/25/2015LS 829 - 201064

65 The End weldon@sfu.ca Questions, Comments, Criticisms….. 6/25/201565LS 829 - 2010


Download ppt "The Surprising Consequences of Randomness LS 829 Mathematics in Science and Civilization Feb 6, 2010 6/25/20151LS 829 - 2010."

Similar presentations


Ads by Google