Presentation is loading. Please wait.

Presentation is loading. Please wait.

AppInsight: Mobile App Performance Monitoring in the Wild Lenin Ravindranath, Jitu Padhye, Sharad Agarwal, Ratul Mahajan, Ian Obermiller, Shahin Shayandeh.

Similar presentations


Presentation on theme: "AppInsight: Mobile App Performance Monitoring in the Wild Lenin Ravindranath, Jitu Padhye, Sharad Agarwal, Ratul Mahajan, Ian Obermiller, Shahin Shayandeh."— Presentation transcript:

1 AppInsight: Mobile App Performance Monitoring in the Wild Lenin Ravindranath, Jitu Padhye, Sharad Agarwal, Ratul Mahajan, Ian Obermiller, Shahin Shayandeh Microsoft Research

2

3

4 > One Million Apps > 300,000 Developers

5 “… Too slow - killing the usefulness when you really need to go.” But its slow

6 “Slower than a snail.” “Slow and unresponsive like mud” “Consistently 3 seconds behind where I touch.” “So slow. Did an intern write this app??” “Very very slow compared to even browsing web.” “Sluggish and freezes my HTC phone.” “Loading GPS data is *** slow”

7 “Slower than a snail.” “Slow and unresponsive like mud” “Consistently 3 seconds behind where I touch.” “So slow. Did an intern write this app??” “Very very slow compared to even browsing web.” “Sluggish and freezes my HTC phone.” “Loading GPS data is *** slow” Performance problems are inevitable in the wild Hard to emulate in the lab Diverse environmental conditions – Network connectivity, GPS signal quality, etc Variety of hardware and OS versions Wide range of user interactions

8 What is the user-perceived delay? Where is the bottleneck? Monitor Performance in the Hands of Users Significant barrier for most app developers Little platform support Only option is to instrument your app – Manage your own logging infrastructure

9 AppInsight Automatic App Instrumentation – Zero developer effort – Binary Instrumentation Readily deployable – No changes to the OS or runtime Low overhead What is the user-perceived delay? Where is the bottleneck?

10 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

11 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

12 App Instrumenter App Instrumented Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer App Store

13 Instrumentation impacts app performance – They are already slow enough!!! Limited Resources – Network, Battery, Memory App Instrumentation is Challenging

14 Highly interactive, UI centric – Single UI thread that should not be blocked Most tasks are performed asynchronously – Asynchronous APIs for Network, Sensor, IO etc. – Computation performed on background threads App Instrumentation is Challenging Highly Asynchronous Programming Pattern Tracing async code is challenging

15 Internet 53% “Mitt Romney might vote for Obama as well” “We recommend Mitt Romney for president” “I would definitely trust Mitt Romney with my money.” “If I had a nickel for every time Mitt Romney said something stupid I'd be in his tax bracket” Mitt Romney Mitt Rating Mitt Rating Mitt Rating Mitt Rating Tweets

16 ClickHandler() { tweets = HttpGet(url); rating = ProcessTweets(tweets); display.Text = rating; } ProcessTweets(tweets) {... } Click Handler Start Click Handler End LogStart(); LogEnd(); Thread User Perceived Delay 53% Mitt Rating Mitt Rating Hypothetical Synchronous Code

17 ClickHandler() { AsyncHttpGet(url, DownloadCallback); } DownloadCallback(tweets) { rating = ProcessTweets(tweets); UIDispatch(DisplayRating, rating); } DisplayRating(rating) { display.Text = rating; } ProcessTweets(tweets) {... } Display Rating UI Thread Background Thread UI Dispatch ClickHandler End ClickHandler Start Mitt Rating Mitt Rating System Async Get Call ProcessTweets Download Callback 53% User Click Asynchronous Code

18 Mitt Rating Mitt Rating 53% User Perceived Delay System UI Thread Background Thread Display Rating UI Dispatch Async Get Call Download Callback ClickHandler Start User Transaction Transaction time

19 UI Thread Background Thread Transaction time User Transaction Get Rating 53%

20 UI Thread Background Thread User Transaction 7 edges Mitt Rating Mitt Rating 53%

21 Apps are highly asynchronous Where is the bottleneck? Focus development efforts 30 popular apps 167,000 transactions – On average, 19 asynchronous calls per user transaction – On average, 8 parallel threads per user transaction Up to 7000 edges

22 UI Thread Background Thread Twitter Thread WakeupThread Blocked Fire Display User Transaction Callback Tweets Posts Twitter Facebook Process Posts Process Tweets Facebook Mitt Rating Mitt Rating 47% User Click

23 UI Thread Background Thread User Click Twitter Thread WakeupThread Blocked Fire Callback Twitter Facebook Process Posts Process Tweets Facebook Critical Path Optimizing the critical path reduces the user perceived delay User Transaction Display

24 AppInsight What is the user-perceived delay? Where is the bottleneck? Automatically instruments the app to track user transactions and critical path Automatically instruments the app to track user transactions and critical path

25 App Instrumenter App Instrumented Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer App Store

26 Low Overhead User Transaction UI Thread Background Thread Should not impact app performance Limited Resources Capturing User Transaction

27 User Transaction UI Thread Background Thread

28 UI Thread Background Thread User Click Event Handler UI Manipulations Capture User Transaction

29 UI Manipulations Capture User Transaction UI Thread Background Thread

30 UI Manipulations Thread Execution Capture Click Handler Start Click Handler End Update UI Start End Callback Start Callback End User Transaction UI Thread Background Thread

31 UI Manipulations Thread Execution Capture User Transaction UI Thread Background Thread

32 UI Manipulations Thread Execution Async Calls and Callbacks Async Get Call UI Dispatch Call UI Dispatch Callback Capture Download Callback User Transaction UI Thread Background Thread

33 UI Manipulations Thread Execution Async Calls and Callbacks Capture User Transaction UI Thread Background Thread

34 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Capture User Transaction UI Update User Transaction UI Thread Background Thread

35 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Capture User Transaction UI Thread Background Thread

36 UI Thread Background Thread User Transaction UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Fire Thread Blocked Thread Wakeup Capture

37 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture

38 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture User Transaction UI Thread Background Thread

39 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture Click Handler Start Click Handler End Update UI Start End Callback Start Callback End User Transaction UI Thread Background Thread

40 Trace every method – Prohibitive overhead Enough to log thread boundaries Log entry and exit of Upcalls Capturing Thread Execution Upcalls System App Click Handler Start Click Handler End Update UI Start End Callback Start Callback End UI Thread Background Thread

41 ClickHandler() { AsyncHttpGet(url, DownloadCallback); } DownloadCallback(tweets) { rating = ProcessTweets(tweets); UIDispatch(DisplayRating, rating); } DisplayRating(rating) { display.Text = rating; } ProcessTweets(tweets) {... } Event Handlers are Upcalls Function pointers point to potential Upcalls – Callbacks are passed as function pointers Identify Upcalls

42 Instrument Upcalls Rewrite app – Trace Upcalls ClickHandler() { Logger.UpcallStart(1); AsyncHttpGet(url, DownloadCallback); Logger.UpcallEnd(1); } DownloadCallback(response) { Logger.UpcallStart(2); rating = ProcessTweets(tweets); UIDispatch(DisplayRating, rating); Logger.UpcallEnd(2); } DisplayRating(rating) { Logger.UpcallStart(3); display.Text = rating; Logger.UpcallEnd(3); } ProcessTweets(tweets) {... } Low Overhead 7000 times less overhead Click Handler Start Click Handler End Update UI Start End Callback Start Callback End UI Thread Background Thread

43 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture Async Get Call UI Dispatch Call UI Dispatch Callback Download Callback User Transaction UI Thread Background Thread

44 Async Get Call Callback Async Calls and Callbacks Log Callback Start – We capture start of the thread Log Async Call – Any call that accepts a function pointer Match Async Call to its Callback ClickHandler() { AsyncHttpGet(url, DownloadCallback); Logger.AsyncCall(5); } DownloadCallback(tweets) { Logger.UpcallStart(2);.... }

45 Matching Async Call to its Callback A method could be a callback to many async calls – Replicate the callback for each Async call [Hirzel’01] Async call called in a loop (e.g. download list of URLs) – Cannot be matched correctly Solution: Detour Callbacks

46 System App AsyncHttpGet Async Call DownloadCallback DownloadCallback(response) { } Callbacks

47 Detour Callbacks DetourCallback(response) { DownloadCallback(response); } class DetourObject { } MatchId = 3 obj.DetourCallback MatchId = 3 AsyncHttpGet(url, GetCallback); AsyncHttpGet(url, obj.DetourCallback); obj = new DetourObject(DownloadCallback, MatchId++); System App AsyncHttpGet Async Call DownloadCallback DownloadCallback(response) { } while(...) { }

48 Detour Callbacks DetourCallback(response) { DownloadCallback(response); } class DetourObject { } MatchId = 4 obj.DetourCallback MatchId = 4 obj = new DetourObject(DownloadCallback, MatchId++); System App AsyncHttpGet Async Call DownloadCallback(response) { } AsyncHttpGet(url, obj.DetourCallback); while(...) { }

49 Detour Callbacks DetourCallback(response) { DownloadCallback(response); } class DetourObject { } MatchId = 5 obj.DetourCallback MatchId = 5 obj = new DetourObject(DownloadCallback, MatchId++); System App AsyncHttpGet Async Call DownloadCallback(response) { } AsyncHttpGet(url, obj.DetourCallback); while(...) { }

50 Detour Callbacks Event Subscriptions Delayed Callbacks – Track Object Ids Matching Async Call to its Callback Low Overhead Async Get Call UI Dispatch Call UI Dispatch Callback Download Callback UI Thread Background Thread

51 UI Manipulations Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture User Transaction UI Thread Background Thread

52 App Instrumenter App Instrumented App Marketplace Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

53 App Instrumenter App Instrumented App Marketplace Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

54 Critical Path Aggregate Analysis Exception Path Analysis

55 Critical Path Bottleneck path in the user transaction – Optimizing the critical path reduces user perceived delay UI Update User Manipulation Critical Path Multiple UI updates Timers, Sensors

56 Critical Path Aggregate Analysis Exception Path Analysis

57 Aggregate Analysis Group similar transactions – Same transaction graph Outliers – Points to corner cases Highlight common critical paths – Focus development effort Root causes of Performance Variability – Highlight what really matters in the wild

58 Critical Path Aggregate Analysis Exception Path Analysis

59 UI Thread Background Thread Exception at parseXML() DownloadCallback() Stack trace parseXML() … DownloadCallback() AsyncHttpGet(url) ThreadCallback() ClickHandler() ThreadStart() User Manipulation Exception Path Failures in the wild

60 Critical Path Aggregate Analysis Exception Path Analysis

61 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

62 Developer Feedback Web based Interface Long User Transactions Critical Path Aggregate Analysis – Outliers – Common Case – Factors affecting Crashes - Exception Path

63 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer Deployment

64 30 Windows Phone apps 30 Users Over 4 months of data 6750 app sessions 33,000 minutes in apps 167,000 user transactions Deployment

65 User Transactions and Critical Path Apps are highly asynchronous # edges in the critical path << # edges in the transaction – Top 2 edges responsible for 82% of transaction time – App developer can focus on optimizing these edges 15% of the user transactions take more than 5 seconds Key reason why a system like AppInsight is needed Up to 7000 edges

66 High variability in the wild Aggregate Analysis 29% transaction groups has multiple critical paths Network, GPS state, Device model, User – all affects performance Key reason why performance monitoring in the wild is important

67 AppInsight Overhead Compute Network Memory Binary Size Battery <1% 1.2% 2% 4% 0.02% Negligible Overhead Impact on app performance is minimal – Our users reported no cases of performance degradation Low resource consumption

68 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer

69 App Instrumenter App Instrumented App Store Downloads Analysis Developer Feedback Developer Feedback Server Traces Developer Improved

70 App: My App Problem: UI Hog AppInsight: Aggregate analysis showed high performance variability – Attributed to UI thread Abnormal latencies only at the start of the session System loading DLLs in the critical path Developer Case Study

71 App: Popular App Problem: Slow Transactions AppInsight: Aggregate analysis showed 3G latencies significantly affecting critical paths – Known issue but no quantitative data Current caching policy insufficient – Quantitative data Developer Case Study

72 App: Professional App Problem: Slow Transactions AppInsight: Custom instrumentation in the critical path – Affecting user perceived delay

73 AppInsight Extremely low overhead Zero developer effort Readily deployable Extremely low overhead Zero developer effort Readily deployable Automatically tracks User Transactions Identifies Critical Paths and Exception Paths Aggregate Analysis highlights factors affecting performance Helps app developers understand performance bottlenecks in the wild

74 1.Who cares about Windows Phone. Can you do it for Android and iPhone? 2.This was done in Don’t you read related work? 3.Are you one of those who don’t care about user privacy? 4.Sounds too good. Are there any limitations? 5.Are you going to release it? Or is it a one off paper? 6.I totally don’t get why you are doing this!? FAQ

75 AppInsight can miss certain causal relationships Does not track data dependencies – Two threads communicating through a shared variable One thread polling – Uses heuristics to warn the developer Will miss implicit casual relationships – Resource contention Does not track that state left behind – Dependencies resulting from such state Limitations


Download ppt "AppInsight: Mobile App Performance Monitoring in the Wild Lenin Ravindranath, Jitu Padhye, Sharad Agarwal, Ratul Mahajan, Ian Obermiller, Shahin Shayandeh."

Similar presentations


Ads by Google