Presentation on theme: "Crowd Crawling: Towards Collaborative Data Collection for Large-scale Online Social Networks Cong Ding, Yang Chen*, and Xiaoming Fu University of Göttingen."— Presentation transcript:
Crowd Crawling: Towards Collaborative Data Collection for Large-scale Online Social Networks Cong Ding, Yang Chen*, and Xiaoming Fu University of Göttingen *Duke University
Significance of social network data crawling Understanding user behaviors Improving SNS architectures Handling privacy/security issues and so on...
Current data collection methods (1) ISP-based measurement [Schneider IMC09] Only ISP companies can do that
Current data collection methods (2) Cooperate with SNS companies [Yang IMC11] Most research groups do not have chance
Current data collection methods (3) Crawl data by a single group (and share them to others) [Gjoka INFOCOM10] Suffering request rate limiting
Shortages of crawling by a single group Waste computing and network resources Introduce overhead to service providers (and may lead stricter rate limiting) Lack of ground truth for the research community
A new thought Why not collect data collaboratively?
System overview Coordinator Crawlers
System design Fetching UIDs (BFS, etc.) Handling crawling failure (timeout) Bypassing request rate limiting (massive IP addresses) Data fidelity (redundant crawling)
Implementation A proof-of-concept prototype (without the data fidelity part) to crawl in Weibo 472 PlanetLab servers as crawlers
Evaluation In 24 hours, we have crawled 2.22M users data from Weibo, including user profiles, all the posts, all the social connections Comparison: Fu et al. (PLOS ONE 2013) get 30K users data in 6 days Guo et al. (PAM 2013) get 1M users data in 1 month Crowd Crawling Fu et al.Guo et al. #UIDs/day2.22M5K33K
Conclusion and Discussion Data sharing may violate some providers terms of services o Twitter does not allow to share data (even for research) o Weibo allows to share data among researchers Unlimited data sharing might cause ethical issues o The data should be anonymized We will publish the data crawled in the evaluation