1. Small-World File-Sharing Communities Adriana Iamnitchi, Matei Ripeanu and Ian Foster,

2. Selecting tradeoffs Understand users’ requirements … AND users’ behavior –Patterns exist and can be exploited: Zipf request popularity distribution (web caching) Breslau, Cao, Fan, Phillips, and Shenker. Web caching and Zipf-like distributions: Evidence and implications. Infocom, 1999 Power-law node degree distribution in graphs (search) Adamic, Lukose, Puniyani, Huberman. Search in Power-law Networks. In Physical Review E, 64 46135, 2001

3. “No 24 in B minor, BWV 869” “Les Bonbons” “ Yellow Submarine” “Les Bonbons” “Yellow Submarine” “Wood Is a Pleasant Thing to Think About” “Wood Is a Pleasant Thing to Think About” System view: The Data-Sharing Graph G m T (V, E):  V is set of users active during interval T  An edge in E connects users that asked for at least m common files within T

4. Three Data-Sharing Graphs The DØ Experiment –High-energy physics data Grid –70+ institutions, 18 countries The Web: Boeing traces –Web requests from Boeing sites to the Web The KaZaA music-sharing network –Peer-to-peer system UsersRequestsDuration traces D0 3172,757,015180 days Web 60,82616,527,19410 hours Kazaa 14,404976,1845 days

5. Small average path length Large clustering coefficient The DØ Collaboration Small World! CCoef = # Existing Edges # Possible Edges 6 months of traces (January – June 2002) 300+ users, 2 million requests for 200K files

6. Small-World Graphs Small path length, large clustering coefficient –Typically compared against random graphs Think of: –“It’s a small world!” –“Six degrees of separation” Milgram’s experiments in the 60s

7. Other Small Worlds Word co-occurrences Film actors LANL coauthors Internet Web Food web Power grid D. J. Watts and S. H. Strogatz, Collective dynamics of small-world networks. Nature, 393:440-442, 1998 R. Albert and A.-L. Barabási, Statistical mechanics of complex networks, R. Modern Physics 74, 47 (2002).

8. Web Data-Sharing Graphs 7200s, 50files 3600s, 50files 1800s, 100files 1800s, 10file 300s, 1file

9. DØ Data-Sharing Graphs 7days, 1file 28 days, 1 file

10. KaZaA Data-Sharing Graphs 7day, 1file 28 days 1 file 2 hours 1 file 1 day 2 files 4h 2 files 12h 4 files

11. Motivation : Understand patterns (user behavior) to choose the right tradeoffs Understand what?! –The data-sharing graph Data-sharing communities –The D0 Experiment (physicists in a data grid) –Web –Kazaa It’s a small world! So what? –News or consequence of known patterns? –Relevance for mechanism design Outline

12. Exploiting Patterns: Intuition  Exploit the data-sharing graph (as overlay) Exploit its small-world properties –Large clustering coefficient: optimize existing (e.g., centralized) mechanisms –… and small average path length: design a new file location mechanism D0 Web

13. Information Dissemination: Local Answers D0 Web Kazaa Hit rate due to previous information dissemination within clusters (experience sharing): up to 70% of requests will find answers locally

14. The Message Be imaginative when looking for patterns There are small-world patterns in file-sharing communities …that can be exploited for mechanism design –Location of files –Data placement