1. Paruj Ratanaworabhan, Ben Livshits, and Ben Zorn JSMeter: Characterizing the Behavior of JavaScript Web Applications 1 In collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock

2. Why Measure JavaScript? Standardized, de facto language for the web –Support in every browser, much existing code Major Microsoft investment –Web Application Companion based on Script# Benchmarks are driving browser marketing: “browser wars” –Are benchmarks representative? –Limited understanding of its behavior in real sites Who cares? –Users, web site designers, JavaScript engine developers 2

3. 3 ZDNet 29 May 2008 ghacks.net Dec. 2008

4. Artificial Benchmarks versus Real World Sites 4 Maps 7 V8 programs: richards deltablue crypto raytrace earley- boyer regexp splay 8 SunSpider programs: 3-draytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic string-tagcloud JSMeter Goals of this Talk Show behavior of JavaScript benchmarks versus real sites Consider how benchmarks can mislead design decisions Goals of this Talk Show behavior of JavaScript benchmarks versus real sites Consider how benchmarks can mislead design decisions 11 real sites

5. 5 Instrumentation Framework \ie\jscript\*.cpp Source-level instrumentation custom jscript.dll custom trace files website visits Offline analyzers custom trace files

6. Visiting the Real Sites Attempted to use each site in “normal” way: 6 amazonSearch a book, add to shopping cart, sign in, and sign out bingType in a search query and also look for images and news bingmapSearch for a direction from one city to another cnnRead front page news ebaySearch for a notebook, bid, sing in, and sign out economistRead front page news, view comments facebookLog in, visit a friend pages, browse through photos and comments gmailSign in, check inbox, delete a mail, and sign out googleType in a search query and also look for images and news googlemapSearch for a direction from one city to another hotmailSign in, check inbox, delete a mail, and sign out

7. Understanding JavaScript Behavior CodeEvents JavaScript Objects 7

8. Code Behavior CodeEvents JavaScript Objects 8 Function size Instructions/call Code locality Instruction mix Function size Instructions/call Code locality Instruction mix

9. Total Bytes of JavaScript Source 9 Real Sites V8 SunSpider Code|Objects|Events

10. Ratio of Bytecode / Source 10 Real Sites V8 SunSpider Code|Objects|Events

11. Static Unique Functions Executed 11 Real Sites V8 SunSpider Code|Objects|Events

12. Total Bytecodes Executed 12 90395 Real Sites V8 SunSpider Code|Objects|Events

13. Total Function Calls 13 Real Sites V8 SunSpider Code|Objects|Events

14. Bytecodes / Call 14 function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a } Real Sites V8 SunSpider Code|Objects|Events

15. Fraction of Code Executed 15 Most code not executed Real Sites V8 SunSpider Code|Objects|Events

16. Hot Function Distribution 16 Real SitesV8 Benchmarks 80% of time in < 10 functions80% of time in 100+ functions 80% Code|Objects|Events

17. Opcode Distribution 17 Green = SunSpider | Blue= Real Web Apps | Red = V8 Real Apps Outliers Code|Objects|Events

18. Object Allocation Behavior CodeEvents JavaScript Objects 18 Allocation by types Lifetime distribution Live heap composition Allocation by types Lifetime distribution Live heap composition

19. Total Bytes Allocated 19 Real Sites V8 SunSpider Code|Objects|Events

20. Heap Data by Type 20 Real Sites V8 SunSpider Many functions Rest are strings Few benchmarks allocate much data Code|Objects|Events

21. Object Type Distribution 21 Real Apps Code|Objects|Events economist is an outlier (arrays)

22. Live Heap Over Time (gmail) 22 Functions grow steadily Code|Objects|Events GC reduces size of heap Objects grow steadily too

23. Live Heap over Time (ebay) 23 Heaps repeatedly created, discarded Heap contains mostly functions Code|Objects|Events Heap drops to 0 on page load

24. Two Search Websites 24 Bing (Web 2.0) Google (Web 1.0) Code|Objects|Events

25. Two Mapping Websites 25 Bing Maps Google Maps Code|Objects|Events

26. Object Lifetimes (V8) 26 earley splay Code|Objects|Events Lives forever Dies immediately Lives until program ends

27. Object Lifetimes (gmail) 27 Strings are shortest lived Functions are longest lived Many objects live a major fraction of page lifetime This is as old as an object can be Code|Objects|Events

28. Event Handlers in JavaScript CodeEvents JavaScript Objects 28 Number of events Sizes of handlers Number of events Sizes of handlers

29. Event-driven Programming Model Single-threaded, non-preemptive event handlers Example of some standard event-handler attributes: –onabort, onclick, and onmouseover Very different from batch processing of benchmarks Handler responsiveness critical to user experience 29 Code|Objects|Events

30. Total Events Handled 30 Real Sites V8 Almost no events Code|Objects|Events

31. Average Bytecodes / Event Handled 31 Code|Objects|Events

32. Median Bytecodes / Event Handled 32 Code|Objects|Events

33. Maximum Bytecodes / Event Handled 33 Code|Objects|Events

34. Fraction of Execution in Handlers 34 Code|Objects|Events

35. Distribution of Time in Handlers 35 Code|Objects|Events

36. A Simple Experiment Observation –Real web apps have lots of code (much of it cold) –Benchmarks do not Question: What happens if the benchmarks have more code? –We added extra, unused to code to 6 SunSpider benchmarks –We measured the impact on the benchmark performance 36

37. Performance Impact of Cold Code 37 ChromeIE 8 Cold code has non-uniform impact on execution time Without cold code, Chrome12x faster than IE8 With 2M cold code, Chrome 4.7x faster than IE8 Which comparison is more meaningful? Cold code makes SunSpider on Chrome up to 4.5x slower

38. Impact of Benchmarks What gets emphasis –Making tight loops fast –Optimizing small amounts of code Important issues ignored –Garbage collection (especially of strings) –Managing large amounts of code –Optimizing event handling –Considering JavaScript context between page loads 38

39. Conclusions JSmeter is an instrumentation framework –Used to measure and compare JavaScript applications –High-level views of behavior promote understanding Behavior in benchmarks significantly differs from real sites –Misleads designers, skews implementations Next steps –Develop and promote better benchmarks –Design and evaluate better JavaScript runtimes –Promote better performance tools for JavaScript developers 39

40. Additional Resources Project –http://research.microsoft.com/en-us/projects/jsmeter/http://research.microsoft.com/en-us/projects/jsmeter/ Papers –"JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages)"JSMeter: Measuring JavaScript Behavior in the Wild –"JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages)"JSMeter: Characterizing Real-World Behavior of JavaScript Programs", –“An Analysis of the Dynamic Behavior of JavaScript Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010) 40