1. Crawling Chinese Android Markets Xiang Pan Biyan Zhou George Liu

2. Overview Background & Purpose Goals Work Division Proposed Algorithm & Examples Problem & Solution Precaution Measures Results & Future Improvements

5. Background & Purpose Existing malicious activates – Ex: NickiBot (Spyware) Runs in background forever, difficult to detect Can record phone call, monitor phone logs and SMS, detect location and send information to remote server Purpose of our project – Collect a sizable amount of android applications from less popular Chinese markets for analysis

6. Goals Create a robust crawler that can be tailored for different markets with minimal effort Analyze at least 5 markets to collect suspicious applications Exanimate the precaution measures of these markets

7. Work Division NameJob Description Oct 31 2011 – Nov 4 2011 Biyan1.Analyzed downloading pattern for AppChina android market. 2. Implemented crawler for AppChina android market using the common template. Xiang 1.Designed a template for all the crawlers to be implemented. This crawler can download the applications and capture their information, such as author, version and so on, from the market. To strengthen the speed and stability of the crawler, it is designed to download with multi-threads. 2.Completed common code for crawlers. E.g. Download functionality given url. 3.Implemented crawler for android market. Nov 7 2011 – Nov 11 2011 Biyan1. Analyzed downloading pattern for EOEMarket android market. 2. Implemented crawler for EOEMarket android market using the common template. George1. Analyzed downloading pattern for NDUOA android market. 2. Implemented crawler for NDUOA android market using the common template. Xiang1. Analyzed downloading pattern for MUMAYI android market. 2. Implemented crawler for MUMAYI android market using the common template. Nov 14 2011 – Nov 18 2011 Biyan1. Implemented multi-threading functionality. George1. Analyzed downloading pattern for FPWAP android market. 2. Implemented crawler for FPWAP android market using the common template. Xiang1. Analyzed downloading pattern for ANZHI android market. 2. Implemented crawler for ANZHI android market using the common template.

8. Proposed Algorithm Manually inspect each market for overall data structure – Meta data HTML – Downloading URL (redirection via JScript) Select appropriate unique application attribute (id, names… etc) Correctly parse meta data using regular expressions Store meta data and the application in a user specified location

9. Example #1

10. Example #2

11. Problem & Solution Different HTML structures for meta data of applications in the same market – Only capture one set of data (the most frequently used one) Slow download speed – Utilize multithread download technique, split a single application to multiple parts Wrong Application ID results in termination of downloading – Using try catch structure when a specified file doesn’t exist

12. Precaution Measures Market NameRegistering ProcessAccount Verification (restrict access) Uploading ProcessApplication Verification IDCaptchaFinancial requirement Application description Tencent’s Appstore Yes NoYes AppChina NoYesNo Yes HiMarket NoYesNo Yes ANZHI NoYesNo Yes N-Duo No Yes EOEMarket Service unavailable MUMAYI NoYesNo YesYes-informally iAndroid YesNo Yes FeiPeng No YesYes(receive applications by emails) d.cn No YesYes(receive applications by emails)

13. Results & Future Improvements Created a robust and easy to use crawler Collected over 70 GB (~30,000) of suspicious applications Exanimated 10 different markets for precaution measures Create simple GUI to improve usability Automatic authentication Circumvent market’s cap for daily traffic on a given IP Maintain a Database for these application

14. Q&A