Benford’s Very Strange Law John D. Barrow

Simon Newcomb 1888:"We are probably nearing the limit of all we can know about astronomy" appeared in The American Journal of Mathematics (1881) 4, 39-40 re-discovered by Benford in 1938, and he published an article called "The Law of Anomalous Numbers" in Proc. Amer. Phil. Soc 78, pp 551-72. 1835-1909 ‘Note on the Frequency of Use of the Different Digits in Natural Numbers’, 1881

Log Tables Yield…

Newcomb’s ‘Law’ "That the ten digits do not occur with equal frequency must be evident to anyone making much use of logarithmic tables, and noticing how much faster the first pages wear out than the last ones. The first significant figure is oftener 1 than any other digit, and the frequency diminishes up to 9." The law of probability of the occurrence of numbers is such that all mantissae [fractional part] of their logarithms are equally probable.

Newcomb said this law was “evident” Data on first digits are evenly spread on a logarithmic scale But it will not be on a linear scale. They become increasingly sparse  Newcomb said this law was “evident” P(d)  [log(d+1) – log(d)]/[log(10) – log(1)] = log(1 + 1/d)

Probability of the First Digit Being Equal to d P(d)= log10[1 + 1/d], d = 1, 2,.. Ignore signs and take first digit after decimal point eg for -3.1526 it is 1

A Big Surprise P(1) = 0.30 P(2) = 0.18 P(3) = 0.12 P(4) = 0.10 You might have thought P(1) = P(2) = P(3) = ….P(9) = 0.11.. But… P(1) = 0.30 P(2) = 0.18 P(3) = 0.12 P(4) = 0.10 P(5) = 0.08 P(6) = 0.07 P(7) = 0.06 P(8) = 0.05 P(9) = 0.05

P(d)= log10[1 + 1/d] first-digit distribution Rediscovered by Frank Benford at GEC in 1938 1883-1948 P(d)= log10[1 + 1/d] first-digit distribution then becomes known as “Benford’s Law” Grad U Michigan 1910 worked for general Electric from 1948 publ his law 1938 ‘The Law of Anomalous Numbers’ (1938)

Benford gathered 20,000 pieces of data and studied First-digit frequencies Data 1 2 3 4 5 6 7 8 9 River areas 31.0% 16.4 10.7 11.3 7.2 8.6 5.5 4.2 5.1 Base ball 32.7 17.6 12.6 9.8 7,4 6.4 4.9 5.6 3.0 magazines 33.4 18.5 12.4 7.51 7.1 6.5 Powers of 2 30 17 13 10 20 tables 30.6 9.4 8.0 4.7  half -lives 29.6 17.8 1.7 10.5 9,9 4.8 5.2 BenfordLaw 30.1 12.5 9.7 7.9 6.7 5.8 4.6

Random street addresses

http://thinkorthwim.com/wp-content/uploads/2009/11/benfordslaw.jpg

Picking Raffle Tickets P(1) goes up as be go to 19 tickets, then falls http://blog.stevemould.com/how-to-cheat-with-frank-benford/

Take an average over all Possible numbers of tickets P(1) depends on the number of tickets Number of tickets P(1) Take an average over all Possible numbers of tickets  The average is 30.1% Number of tickets S. Mould

Universal distribution P(x) for numbers with units Means it must be scale invariant P(kx) = f(k)P(x) Since  P(x)dx = 1 we must have  P(kx)dx = 1/k so 1/k =  P(kx)dx = f(k) P(x)dx = f(k) Means f(k) = 1/k d/dk of P(kx) = f(k)P(x) xdP(kx)/d(kx)  d(kx)/dk = -P(x)/k2 Put k = 1 Means P(x) = 1/x In reality we won’t go to zero or infinity so don’t worry about 0 1/x dx being infinite

Other Digits The fraction in 1.4 to 1.3 is By the same kind of analysis we can determine the probability that the second digit will have a certain value. It's only necessary to consider a single order of magnitude, since the pattern is repeated on each order. For example, in the base 10, the probability of the second digit being "3" is equal to the sum of the probabilities of the first two digits being "1.3", "2.3", "3.3", ... or "9.3" for numbers in the range from 1 to 10. This is indicated by the shaded regions in the logarithmic scale: The fraction in 1.4 to 1.3 is Now just find the fractions in 2.2 to 2.3 etc and add all the answers together

Probabilities for Successive Significant Digits P(first digit is d) = log[1 + 1/d], d = 1,2,3,…9. P(second digit is d) = 9k=1 log[1 + (10k+d)-1], d = 0,1,2…9. (Newcomb) The joint distribution of all digits can be found and they are not independent P(first = d1, …,kth = dk) = log[1 + (i=1k di  10k-i)-1] Eg for 0.314; P(3,1,4) = log[1 + (314)-1] = 0.0014.. http://www.mathpages.com/home/kmath302/kmath302.htm Unconditional probability that second digit is 1 is P(second digit =1) = 0.109, But conditional probability that it is 1 given that the first is 1 is 0.115 Dependence falls off fast as distance between digits increases Distn of the nth digit approaches a uniform distribution on 0,1,2,…,9 very fast as n  , so P  1/10 for occurrence of each 0,1,2…,9 as log(1 + 1/n) 1/n

Invariances Pick Out Benford Scale invariance – no preferred units Base invariance wrt base of arithmetic b P(d) = logb(1 + 1/d) But why should there be a distribution like this at all? Scale invar implies base invar but not vice versa

Do All First-Digit Distributions Follow Newcomb-Benford? http://www.mathpages.com/home/kmath302/kmath302.htm

Random number generator Figure 34-2 shows two examples of Benford's law. The histogram on the left is for 14,414 numbers taken from the income tax returns of U.S. corporations. The pattern here is obvious and very repeatable. The leading digit in these numbers is a 1 about 30.1% of the time, a 2 about 17.6% of the time, and so on. Mathematicians immediately recognize that these values correspond to the spacing on the logarithmic number line. That is, the distance between 1 and 2 on the log scale is log(2) - log(1) = 0.301. The distance between 2 and 3 is log(3) - log(2) = 0.176, and so on. Benford showed us that this logarithmic pattern of leading digits is extremely common in nature and human activities. In fact, even the physical constants of the universe follow this pattern– just look at the tables in a physics textbook. US tax return data Random number generator

Not Everything Follows Benford Continued fraction digits are mostly 1’s in general but they are not Benford-Newcomb-like a = k + x = integer + fractional part For almost all real numbers: P(k) = ln[1 + 1/k(k + 2)]/ln[2] P(1) = 0.41, P(2) = 0.17, P(3) = 0.09, P(4) = 0.06, P(5) = 0.04 Steeper than Benford: P(k)  1/k2 as k  ln(1+x)  x

First digits are Benford-Newcomb distributed so long as Data measure same phenomena (eg all prices or areas) There is no built in max or min values The numbers are not assigned (like phone nos) The underlying distribution is fairly smooth More observations of small items than large ones Data spans several whole numbers on the log scale: * The distribution must be broad rather than narrow * Red area is relative Prob first digit is 1 Ratios of areas proportional to widths Eg incomes. populns Blue area is relative Prob first digit is 8 Consider the probability distributions shown below, plotted on a log scale.[12] In each case, the total area in red is the relative probability that the first digit is 1, and the total area in blue is the relative probability that the first digit is 8. For the left distribution, the size of the areas of red and blue are approximately proportional to the widths of each red and blue bar. Therefore the numbers drawn from this distribution will approximately follow Benford's law. On the other hand, for the right distribution, the ratio of the areas of red and blue is very different from the ratio of the widths of each red and blue bar. Rather, the relative areas of red and blue are determined more by the height of the bars than the widths. The heights, unlike the widths, do not satisfy the universal relationship of Benford's law; instead, they are determined entirely by the shape of the distribution in question. Accordingly, the first digits in this distribution do not satisfy Benford's law at all. Thus, real-world distributions that span several orders of magnitude rather smoothly like the left distribution (e.g. income distributions, or populations of towns and cities) are likely to satisfy Benford's law to a very good approximation. On the other hand, a distribution that covers only one or two orders of magnitude, like the right distribution (e.g. heights of human adults, or IQ scores) is unlikely to satisfy Benford's law well.[ Broad Ratios of areas not proportional to widths Eg human heights, IQ scores Narrow

Different Types of Data Benford-like ? yes yes no yes

Winning Lotteries The Massachusetts Numbers Game – State Lottery 1. Bet on a 4-digit number 2. A 4-digit number is generated randomly 3. All winners share the jackpot A Possible Strategy To avoid sharing the prize. Assume entrants pick numbers from their experience (ie not at random) and obey Benford’s law. So pick numbers that are least probable by the Benford-Newcomb law. So start with 9’s and 8’s Evidence (Hill 1988) that numbers ‘randomly’ chosen by people tend to start with low digits

Generalised Benford’s Laws A random process with probability distribution P(x)  1/x gives Benford data for first digits: P(d)= log[1 + 1/d] Random processes with P(x)  1/xa and a  1 give P(x) = C dd+1 x-a dx = (101-a – 1)-1[(d+1)1-a – d1-a] For a = 2: P(d = 1) = 0.56, P(d = 2) = 0.185, P(d = 3) =0.09, P(d = 9) = 0.012 For prime numbers from 1 to N a(N) = 1/[logN – c] c = 1.10 + 0.05 large N Replace 10 by b to get other bases Perone et al

A Well-defined Approach to Uniformity by the Primes Christian Perone a = 1.10

Detecting Fraud ‘Natural’ distributions and their combinations should follow Benford Maybe ‘Doctored’ or ‘artificial’ constructions do not ??? Mark Nigrini Univ. Cincinnati PhD thesis (1992) ‘The detection of income evasion through an analysis of digital distributions’ Data from the lines of 169,662 IRS model files follow Benford's law closely. Fraudulent data taken from a 1995 King’s County, New York, District Attorney's Office study of cash disbursement and payroll in business don’t follow Benford's law. The fraudulent or concocted data appear to have far fewer numbers starting with 1 and many more starting with 5 or 6 than do true data.

Benford's law can be used to test for fraudulent or random-guess data in income tax returns and other financial reports. Here the first significant digits of true tax data taken by Mark Nigrini from the lines of 169,662 IRS model files follow Benford's law closely. Fraudulent data taken from a 1995 King’s County, New York, District Attorney's Office study of cash disbursement and payroll in business do not follow Benford's law. Likewise, data taken from the author's study of 743 freshmen's responses to a request to write down a six-digit number at random do not follow the law. Although these are very specific examples, in general, fraudulent or concocted data appear to have far fewer numbers starting with 1 and many more starting with 6 than do true data.

Forensic Accounting with Newcomb-Benford Robert Burton, the chief financial investigator for the Brooklyn District Attorney recalled in an interview that he had read an article by Dr. Nigrini that fascinated him. "He had done his Ph.D. dissertation on the potential use of Benford's Law to detect tax evasion, and I got in touch with him in what turned out to be a mutually beneficial relationship," Mr. Burton said. "Our office had handled seven cases of admitted fraud, and we used them as a test of Dr. Nigrini's computer program. It correctly spotted all seven cases as "involving probable fraud." One of the earliest experiments Dr. Nigrini conducted with his Benford's Law program was an analysis of President Clinton's tax return. Dr. Nigrini found that it probably contained some rounded-off estimates rather than precise numbers, but he concluded that his test did not reveal any fraud. He feels your pain

President Clinton’s Tax Returns over 13 Years