1. Power Law and Its Generative Models Bo Young Kim 2010-03-16

2. Contents 1.Recall The Definition of Power Law 2.Recall Some Properties of Power Law 3.Generative Models for Power Law - Power Laws via Preferential Attachment - Power Laws via Multiplicative Processes 2Applied Algorithm Lab.

3. 1.Recall The Definition of Power Law 2.Recall Some Properties of Power Law 3.Generative Models for Power Law - Power Laws via Preferential Attachment - Power Laws via Multiplicative Processes 3Applied Algorithm Lab.

4. 1. Recall The Definition of Power Law X: a nonnegative random variable Def Power Law X is said to have a power law distribution if Pr[X≥x]~cx -α for constants c>0, α>0 Def f(x)~g(x) ⇔ lim x f(x)/g(x) = 1 What does this mean? In a power law distribution, asymptotically the tails fall according to the power α. (heavier tail than exponential distribution) 4Applied Algorithm Lab.

5. 1.Recall The Definition of Power Law 2.Recall Some Properties of Power Law 3.Generative Models for Power Law - Power Laws via Preferential Attachment - Power Laws via Multiplicative Processes 5Applied Algorithm Lab.

6. 2. Recall Some Properties of Power Law E.g. The Pareto distribution Pr[X≥x]=(x/k) -α ln(Pr[X≥x])=-α(ln(x)-ln(k)) * Linear Log-log plot (complementary cumulative distribution function) - X has a power law distribution - Then a log-log plot behavior is a straight line. (asymptotic sense) 6Applied Algorithm Lab.

7. 2. Recall Some Properties of Power Law “Scale Invariance” - Let f(x) := P[X≥x] - f(x) ~ cx -α - f(kx) ~ c(kx) -α = k -α (cx -α ) = k’f(x) ∝ f(x) (k’=k -α ) - Scaling by a constant simply multiplies the original power law relation by the constant k’. - If we change the measurement unit(=scale), it retains the same power law form w/ the same exponent.  We cannot decide what scale we’re observing. (like Fractals) 7Applied Algorithm Lab.

8. 2. Recall Some Properties of Power Law Web follows power law. [4] Recall (Rank exponent) - d v : outdegree of a node v - r v : the rank of a node v d v =k*r v R (R,k: constant) Designing random graph models that yield Web-like graphs? i.e. that yields power law distributions for the indegree and outdegree? 8Applied Algorithm Lab.

9. 1.Recall The Definition of Power Law 2.Recall Some Properties of Power Law 3.Generative Models for Power Law - Power Laws via Preferential Attachment - Power Laws via Multiplicative Processes 9Applied Algorithm Lab.

10. Generative Models for Power Law - Power Laws via Preferential Attachment Def Preferential Attachment Process (=Yule Process) Any process s.t. some quantity (some form of wealth) is distributed among a number of individuals according to how much they already have, so that those who are already wealthy receive more than those who are not. ”The rich get richer” 10Applied Algorithm Lab.

11. The Chinese Restaurant Process - A Chinese restaurant has infinitely many tables - Each table can seat infinitely many customers - At each time step, customer X t comes into the restaurant. When X t+1 comes into here… (CRP1) Sits at an already occupied table k w/ prob. N k /(t+α) (N k : # of customers at table k  Σ k N k =t) (CRP2)or, sits at the next unoccupied table w/ prob. α/(t+α) Generative Models for Power Law - Power Laws via Preferential Attachment 11Applied Algorithm Lab.

12. When X t+1 comes into here… (CRP1) Sits at an already occupied table k w/ prob. N k /(t+α) (N k : # of customers at table k  Σ k N k =t) (CRP2)or, sits at the next unoccupied table w/ prob. α/(t+α) Generative Models for Power Law - Power Laws via Preferential Attachment 12Applied Algorithm Lab.

13. CPR rule: Next customer sits at a table w/ prob. Proportional to # of customers already sitting at it(and sits at new table w/ prob. Proportional to α)  Customers tend to sit at most popular tables  Most popular tables attract the most new customers, and become even more popular The concentration parameter α: how likely customer is to sit at a fresh table Generative Models for Power Law - Power Laws via Preferential Attachment 13Applied Algorithm Lab.

14. Generating Power law distribution via Preference Attachment (Most models are variations of this form) Let’s say “Web Page Process” Start w/ a single page This single page has a link to itself At each time step, a new page appears, w/ outdegree 1 Generative Models for Power Law - Power Laws via Preferential Attachment (WPP1) The link of new page points to a page chosen u.a.r. w/ prob. α<1 (WPP2) The link of new page points to page chosen proportionally to the indegree of the page w/ prob. 1- α 14Applied Algorithm Lab.

15. X j (t): # of pages w/ indegree j when ∃ t pages in the system Pr[X j increase] = αX j-1 /t+(1-α)(j-1)X j-1 /t Pr[X j decrease] = αX j /t+(1-α)jX j /t Generative Models for Power Law - Power Laws via Preferential Attachment (WPP1) The link of new page points to a page chosen u.a.r. w/ prob. α<1 (WPP2) The link of new page points to page chosen proportionally to the indegree of the page w/ prob. 1- α 15Applied Algorithm Lab.

16. Pr[X j increase] = αX j-1 /t+(1-α)(j-1)X j-1 /t Pr[X j decrease] = αX j /t+(1-α)jX j /t  dX j /dt = {α(X j-1 -X j )+(1-α)((j-1)X j-1 -jX j-1 )}/t Intuitively appealing, BUT how continuous DE describes a discrete process?  This can be justified formally using martingales [Kumar et al 00] & theoretical frameworks of Kurtz, Wormald [Drinea et al. 00, Kurtz 81, Wormald 95]. Generative Models for Power Law - Power Laws via Preferential Attachment 16Applied Algorithm Lab.

17. dX 0 /dt=1-αX 0 /t Suppose in the steady state limit: X j (t)=c j *t (portion c j )  c 0 =dX 0 /dt=1-αX 0 /t=1-αc 0 ⇔ c 0 = 1/(α+1) Substitute this assumption for dX j /dt = {α(X j-1 -X j )+(1-α)((j-1)X j-1 -jX j-1 )}/t  c j (1+α+j(1-α))=c j-1 (α+(j-1)(1-α))  We can determine c j exactly. Focusing on the asymptotic, for large j c j /c j-1 =1-(2-α)/(1+α+j(1-α))~1-{(2-α)/(1-α)}*(1/j) Generative Models for Power Law - Power Laws via Preferential Attachment 17Applied Algorithm Lab.

18. We have c j ~cj^(- ) for some constant c, giving a power law. Note c j ~cj^(- ) implies WTS: Σ j≥k c j behave the tail of power law distribution (Proof) For some constant c’. So, we’re done. Generative Models for Power Law - Power Laws via Preferential Attachment 18Applied Algorithm Lab.

19. 1.Recall The Definition of Power Law 2.Recall Some Properties of Power Law 3.Generative Models for Power Law - Power Laws via Preferential Attachment - Power Laws via Multiplicative Processes 19Applied Algorithm Lab.

20. Pareto: income distribution obeys power law [Champernowne 53] offered an explanation for this behavior. Partition income in the following manner: 1 st range: between m and γm for some γ>1 2 nd range: between γm and γ 2 m … persons in class j: their income is between γ j-1 m and γ j m P ij : prob. of a person moving from class i to class j At each time step, P ij depends only on the value (j-i).  Under this assumption, Pareto distribution can be obtained. Generative Models for Power Law - Power Laws via Multiplicative Processes 20Applied Algorithm Lab.

21. E.g. γ=2, P ij =2/3 if j-i=-1 P ij =1/3 if j-i=1 Special case: i=1  P 11 =2/3 The equilibrium property of being in class k: 1/2 k X: a person’s income  Pr[X≥2 k-1 m]=1/2 k-1 Pr[X ≥ x]=m/x for x= 2 k-1 m This is a power law distribution. Generative Models for Power Law - Power Laws via Multiplicative Processes 21Applied Algorithm Lab.

22. References [1]M. Mitzenmacher, A Brief History of Generative Models for Power Law and Lognormal Distributions, Internet Mathematics, vol 1, No. 2, pp. 226-251, 2004. [2]Mark Johnson, Chinese Restaurant Processes(CG168 notes), cog.brown.edu/~mj/classes/cg168/.../ChineseRestaurants.pdf [3]The lecture notes of C. Faloutsos, Carnegie Mellon University, 15-826 Multimedia Databases and Data Mining, Spring 2008 http://www.cs.cmu.edu/~christos/courses/826.S08/FOILS- pdf/195_powerLaws.pdfhttp://www.cs.cmu.edu/~christos/courses/826.S08/FOILS- pdf/195_powerLaws.pdf [4]Bruno Bassetti, Mina Zarei, Marco Cosentino Lagomarsino, and Ginestra Bianconi., Statistical mechanics of the “Chinese restaurant process”: Lack of self-averaging, anomalous finite- size effects, and condensation, Phys. Rev. E 80, 066118 (2009) [4 pages] [5]http://en.wikipedia.org/wiki/Power_law, http://en.wikipedia.org/wiki/Chinese_restaurant_process, http://en.wikipedia.org/wiki/Preferential_attachmenthttp://en.wikipedia.org/wiki/Power_lawhttp://en.wikipedia.org/wiki/Chinese_restaurant_process http://en.wikipedia.org/wiki/Preferential_attachment 22Applied Algorithm Lab.