Download presentation

Presentation is loading. Please wait.

Published byTara Geary Modified over 2 years ago

2
Social Media?

4
Properties of Social Media : Scale Twitter (Chirp 2010) – More than 100M user accounts – more than 600M search queries a day – 55M tweets a day Facebook – More than 400M active users – More than 25 billion pieces of content (web links, news stories, blog posts, notes, photo albums, etc.) shared each month.

5
Properties of Social Media: Immediacy Need to share breaking news Search : Content vs. Peer recommendation

6
Properties of Social Media: Duplication Duplication of content – Blogs: Copy-Paste – Twitter: “Re-tweet” – Groups: Cross-posting – Email: Signature lines, Inline Replies

7
Properties of Social Media: Semi-structuredness Informal but structured – Informal != low quality (eg. Wikipedia) Structure – Metadata – Connectivity

8
Suggested Reading “Towards a PeopleWeb”, Raghu Ramakrishnan & Andrew Tomkins, IEEE Computer, Aug 2007 “Important properties of users and objects will move from being tied to individual Web sites to being globally available. The conjunction of a global object model with portable user context will lead to a richer content structure and introduce significant shifts in online communities and information discovery.”

9
Properties of Social Media Scale Immediacy Heterogeneity Duplication Semi-structuredness

10
Properties of Social Media Graphs Small-world property – Six-degrees of separation – Facebook : 5.73 (Bunyan, 2009) – MS Messenger: ~7 (Leskovec & Horvitz, 2007) Mathematically – Low Average Path length – High Clustering coefficient

11
Network Evolution and Path Size

12
Properties of Social Media Graphs Power law degree distribution (asymptotically) Property of most real world networks Existence of “hubs” Scale free networks

13
Probabilistic Modeling of Networks Erdos-Renyi Model – Choose a pair of nodes uniformly at random and add an edge. – G(n, p) – Not Scale Free (small avg. Path but low clustering coefficient) – Scale Free networks don’t evolve by chance

14
Probabilistic Modeling of Networks Preferential Attachment (Barabasi and Albert, 99) – Rich become richer – Stochastic process: Using Polya’s urn

15
Why model? Study network evolution, degeneration – Develop algorithms Detect communities Who are the movers and shakers? Detect diffusion of ideas across networks Detect anomalies

16
Crawling Social Networks HTML (Slashdot) RSS/Atom feeds (blogs) API driven (Twitter, Facebook, …) – Data liberation

17
Twitter twitter = new TwitterFactory().getInstance(twitterID,twitterPassword); List statuses = twitter.getFriendsTimeline(); System.out.println("Showing friends timeline."); for (Status status : statuses) { System.out.println(status.getUser().getName() + ":" + status.getText()); } http://twitter4j.org

18
Storage and Indexing Graph stores can be more efficiently designed traditional RDBMS or flat files (document IR) A family of “triple stores” or graph databases (#NoSQL movement) – Neo4J – CouchDB – Hypertable – …

19
Data is becoming more and more connected (Eifrem, OSCON 2009)

20
Social Media Graphs (Eifrem, OSCON 2009)

21
Social Media Graphs : Representation Nodes Relationship between nodes Properties on Both Storing in Flat Files vs. Graph Databases Neo4J, disk based solution – works well for sizes up to a few billion (Single JVM)

22
Processing Large Scale Graph Data Better representation Parallel computation – MapReduce – BSP

23
Parallelism via Map-Reduce A paradigm to view input as (key, value) pairs and algorithms process these pairs in one of two stages – Map: Perform operations on individual pairs – Reduce: Combine all pairs with the same key – Functional programming origins – Abstracts away system specific issues – Manipulate large quantities of data

24
Parallelism via Map-Reduce Input is a sequence of key value pairs (records) Processing of any record is independent of the others Need to recast algorithms and sometimes data to fit to this model – Think of structured data (Graphs!)

26
Input: Collection of documents Output: For each word find all documents with the word def mapper(filename, content): foreach word in content.split(): output(word, filename) def reducer(key, values): output(key, unique(values)) Example: Building inverted indexes

27
Map-Reducing graph data Note: By design the mappers cannot communicate with each other. The graph representation should be such that that all information (e.g. neighborhood) needed for processing a node should be locally available. The adjacency list representation is perfectly suited. Key: vertex in the graph Value: neighbors of the vertex and their associated values

28
Computing PageRank (MapReduce) PageRank update with dampening parameter α where P is the transition probability matrix. One map-reduce per iteration

29
MapReduce: PageRank Iteration Map(Key k, Value v) { r_old = k.rank; r = 0; foreach node n in v.getNeighbors() { r += p(n, k)*r_old + dampening_factor } v.rank = r; Emit(k, v); }

30
Processing Large Scale Graph Data MapReduce is not the best model for large scale graph processing – Simple graph concepts (Pagerank, BFS, …) are not easy to program – MapReduce does not preserve data locality in consecutive operations

31
A New Paradigm to Process Large Scale Graph Data Bulk Synchronous Parallel Developed in the 80s by Leslie Valiant Introduced by Google for Graph computation “Pregel: a system for large-scale graph processing” (Malewicz et al, PODC 2009)

32
Bulk Synchronous Parallel Sequence of steps – “SuperSteps” Each SuperStep S – Execute a user defined Compute() function on every vertex in parallel – Input to Compute(): All messages from SuperStep S – 1 – Output of Compute(): Messages to other vertices 1B vertices 80B Edges 2000 Workers Bellman-Ford: 200s (Malewicz et al, PODC 2009)

33
Why “SuperStep”? Internally consists of three stages

34
Computing PageRank (BSP version) Compute() { r_old = r; r = 0; for each incoming message m { r += m.p* r_old + dampening_factor; } if(r – r_old < epsilon) done() }

35
Suggested Reading “Truly, Madly, Deeply Parallel”, Robert Matthews, New Scientist, Feb 1996 “Pregel: a system for large-scale graph processing” (Malewicz et al, PODC 2009)

36
Social Network Analysis Retrieving information from structure Example: Community Discovery Many practical applications One approach: “Edge Betweenness” – betweenness(e) = # triangles(e)/max(e) – iteratively prune edges with low betweenness

37
Book Networks, Crowds, and Markets: Reasoning About a Highly Connected World By David Easley and Jon Kleinberg http://www.cs.cornell.edu/home/kleinber/networks-book/

38
Recap … Properties of social media – Scale – Immediacy – Heterogeneity – Duplication – Semi-structuredness Properties of social media graphs – Small-worldness – Scale free property – Evolution models Crawling – API driven Indexing & Retrieval – Graph databases Processing large scale social networks – MapReduce – Bulk Synchronous Parallel IR from structure – Social Network Analysis

39
Social Media Related Projects Twitter

Similar presentations

OK

Design Patterns for Efficient Graph Algorithms in MapReduce Jimmy Lin and Michael Schatz University of Maryland Tuesday, June 29, 2010 This work is licensed.

Design Patterns for Efficient Graph Algorithms in MapReduce Jimmy Lin and Michael Schatz University of Maryland Tuesday, June 29, 2010 This work is licensed.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on central limit theorem formula Ppt on event driven programming language Ppt on arithmetic progression for class 11 Ppt on summary writing exercises Ppt on grammar translation method Ppt on hlookup in excel Ppt on seven segment display diagram Ppt on bionics contact Ppt on sources of water pollution Ppt on campus recruiting systems