1. IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop IBM, INTEL, 2008 SPECweb2005 in the Real World: Using Internet Information Server (IIS) and PHP 01/27/2008 Sam Warner Steve Worley

2. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Analysis of various systems Using IIS as web server Using PHP to server dynamic content SPECweb2005/PHP more representative to real world Results comparison PHP to published JSP results Dual-socket results and Multi-socket results Abstract

3. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Introduction SPECweb® benchmark has brought ability to show the relative performance of various systems manufactured in the Web Serving market SPECweb® can be used for Research efforts at universities Within computer equipment manufactures to evaluate alternative technologies In the purchase process by Information Technology (IT) professionals SPECweb® has two alternatives to enable performance evaluation Java Server Pages (JSP) All current published results Top results are unrealistic at approximately 40000 users Hypertext Preprocessor (PHP) This paper provides results comparison not found in published results Help insure the benchmark remains useful in IT customer purchase process

4. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Overview of the SPECweb2005 Benchmark Standard Performance Evaluation Corporation benchmark for evaluating the performance of World Wide Web Servers Supersedes SPECweb99® and SPECweb99_SSL® Three workloads representing market segment that are part of the benchmark SPECweb2005_Banking SPECweb2005_Ecommerce SPECweb2005_Support Each workload variant enables measuring the maximum number of simultaneous user sessions Still achieving specific QOS metrics Still achieving error rate requirements

5. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Overview of the SPECweb2005 Benchmark Benchmark score Individual sub-metric scores indicate the total number of simultaneous user sessions the server can support Overall score is the geometric mean of the three sub-metrics normalized to a reference platform score A score of 100 represents the same overall performance of the reference platform A score of 20,000 represents a score 200 times that of the reference platform The benchmark does not require a specific choice of web server software Only requirement is that the web server support HTTP 1.1 and SSL (HTTPS) Implementation of the web server will lead to differences in observed performance on the same system under test Current performance using the JSP kit achieves performance which is roughly 370x the reference platform

6. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Top 15 SPECweb2005/JSP Scores

7. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Overview of Top Fifteen Results All fifteen submitted with one of two web severs Sun Java System Web Server 7.0 Accoria Rock Web Server V1.4 13 of the top 15 results Rock Web Server is undeniably fast No measurable market share from market analysis firms All results available at SPEC website http://www.spec.org

8. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop SPECweb2005 and the Real World IT customers use of performance benchmarks Seen as way to evaluate the complex combination of Number of users supported Content development language Web server software Web server hardware Dont have much time to understand the subtleties of a benchmark Forces OEMs to be constrained by competitive pressures to produce leading results Causes the publication of SPECweb2005 to diverge from real world IT performance

9. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop SPECweb2005 and the Real World Trend of web server connections supported Range of 20-2000 sustained customer connections As processing capacities have improved, so has Concerns and processing for security Complexity in formulating the responses for web server requests SPECweb99 Results ranged from 400 to 25000 for the five years of publication Meanwhile, web server connection rates remained in the 1000s SPECweb2005/JSP Two and four socket systems have begun to achieve levels approaching 50,000 simultaneous users for a duration of ½ hour. In contrast to research papers where published connection loads tend to range from 2000 to 3000 users SPECweb2005/PHP By comparing to SPECweb2005/JSP attempt is to bridge gap between real world loads and published results IT industry may be better served by comparisons to connection loads seen in their environments

10. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop SPECweb2005/JSP Scores since Inception

11. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Analysis of Currently Published SPECweb2005 Results Request for Information and Request for Proposal processes have led many OEMs to develop publications for SPECweb2005. Forces OEMs to use software combinations that enable leading performance results Competition Great way to insure that the final purchased product meets the needs of the customer If benchmark closely matches the end-user environment, benchmark produces an informed purchase Top 15 results show connection rates that are 10-20x higher than connection loads that are normally seen by web servers.

12. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Comparison of Web Servers Of top fifteen SPECweb2005 results, thirteen use the Accoria Rock web Server v1.4 Proprietary Non-open source No measurable market share Web Servers discussed in the paper

13. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Web Servers Currently Used in Submissions Choice of Web Server up to OEM Results prepared as market collateral Market conditions dictate the decision of web server Best possible combination of hardware and software If less than ideal component used then consumer education required Results response to RFI Choice of highly relevant results to customer or industry leading results Lower cost to educate customer by simply achieving highest possible result Net result is that neither Apache or IIS, both recognized in their widespread use, have been used in any submissions in the last two years

14. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Comparison of JSP to PHP performance

15. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop JSP use in submissions Presently all of the results submitted have used the JSP kit While JSP is widely used, it is not the primary content development language used To achieve leading industry results an OEM needs to use a software stack that has the lowest overhead to generate a response. The current implementations of SPECweb2005 in JSP has lower overhead than PHP

16. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Analysis of Real World Workloads Industry results on SPECweb2005/JSP have improved by ~50x since inception Publications from universities show that true connection loads trend at levels 1/10 th of those observed in publications. SPECweb99 this same disconnect Using the SPECweb2005/PHP kit on equivalent systems we find that the change in connection loads observed is not as high. Intel® Xeon® X5355 JSP shows 18,000-20,000 connections PHP shows 1000 connections

17. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Web Server Statistics Aversa and Bestavros Paper TCP connection load of between 496 and 663 requests served Response time between 0.92s and 0.26s Grottke paper Connection rates of 390 connections/second (c/s) Cao paper Connection rates of between 0.18 c/s and 34 c/s Connection rates of between 2.41 c/s and 230 c/s Seems reasonable to look at a scenario where the top connection rates are in the 2000-3000 connections/second range

18. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Discussion of use of IIS According to various web server market analysis companies approximately 35% of web servers on the internet use IIS Tends to be used in websites with higher traffic volumes No web server market analysis companies are able to measure the market share represented by Rock Web Server One is left to wonder how the IT industry correlates the performance reported from SPECweb2005 submissions to their own environment

19. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Web Server Market Share

20. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Discussion of use of PHP Variety of market analysis companies have analyzed the percentage of the market share for the various content development languages. These research firms show levels from 35% to 75% for PHP Depends on whether the analysis includes all known websites or newer websites being actively developed The trend is that PHP development has increased from ~25% to ~50% of content development from 2001 to 2007

21. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop SPECweb2005/PHP Results and Analysis The key value in a benchmark is its ability to accurately predict for the market segment intended the value of the new hardware or software component. With the SPECweb2005/PHP kit there appears to be more similarity Hardware requirements to achieve peak results are decreased dramatically. With the SPECweb2005/PHP results are closer to the default OEM configurations

22. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Dual-Socket Results System chosen was Intel® Xeon® 5355 processor dual socket platform Support had the highest connection load Banking produced the lowest connection load Processor utilization 95% processor utilizations for Support and Ecommerce 80% processor utilization for Banking Banking workload using PHP has software serialization due in part to the session state having higher processing and IO requirements that limit the processor utilization achievable

23. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Dual-Socket Response Times Response times for Ecommerce where nearly equivalent to SPECweb2005/JSP results Response times for Support were approximately 0.83x that of JSP Response times for Banking were lower at 0.47x that of JSP Response times are nearly the same across the three workloads.

24. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Multi-Socket Results System chosen was Intel® Xeon® 7140M processor Equivalent characteristics observed to two-socket system Connection rates improved slightly for Ecommerce and Banking Slight degradation on the Support workload Processor Utilization Support and Ecommerce higher than 90% Banking slightly higher than 70% (Due to high software serialization)

25. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Comparison of Configurations to Published Results Major determining factor for choice of JSP kit is the path length for JSP to server the dynamic content is much shorter than for PHP to perform the same task Significant contrast in hardware requirements required for a top performing JSP result and a top performing PHP result Network JSP requires 5-18 subnets of 1Gb PHP drops to two 1Gb subnets Disk JSP requires 10 times the storage

26. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop Conclusions By using the SPECweb2005/PHP kit a new system can be evaluated at a level of performance with respect to connection loads and response times that more closely matches a real world scenario. The amount of hardware required to analyze a new system with the SPECweb2005/PHP kit is much smaller than using the SPECweb2005/JSP kit

27. IBM, INTEL, 2008 Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop www.cs.ualberta.ca/~amaral/SPEC08