2 Session 7 (4 Hours) Here are some things that we’ll cover: What is Software TestingWorst Software PracticesHeuristics of Software TestingTesting on Different Platforms and ServersWeb ServersPerformance, Security and review of specific functionality of most popular Web ServersWeb Capacity Testing – Load and Stress
3 What is Software Testing and Why is it Important? A brief history of Software engineering and the SDLC.The software industry has evolved through 4 eras, 50’s –60’s, mid 60’s –late 70’s, mid 70’s- mid 80’s, and mid 80’s-present.Each era has its own distinctive characteristics, but over the years the software’s have increased in size and complexity. Several problems are common to almost all of the eras and are discussed below.The Software Crisis dates back to the 1960’s when the primary reasons for this situation were less than acceptable software engineering practices. In the early stages of software there was a lot of interest in computers, a lot of code written but no established standards.
4 A brief history of Software engineering and the SDLC (continue) Then in early 70’s a lot of computer programs started failing and people lost confidence and thus an industry crisis was declared. Various reasons leading to the crisis included:Hardware advances outpacing the ability to build software for this hardware.The ability to build in pace with the demands.Increasing dependency on software’sStruggle to build reliable and high quality softwarePoor design and inadequate resources.
5 Software Industry Crisis This crisis though identified in the early years, exists to date and we have examples of software failures around the world. Software is basically considered a failure if the project is terminated because of costs or overrun schedules, if the project has experienced overruns in excess of 50% of the original or if the software results in client lawsuits. Some examples of failures include failure of Air traffic control systems, failure of medical software, and failure in telecommunication software. The primary reason for these failures other than those mentioned above is due to bad software engineering practices adopted.
6 Worst Software Practices Some of the worst software practices include:No historical software-measurement data.Rejection of accurate cost estimates.Failure to use automated estimating and planning tools.Excessive, irrational schedule pressure and creep in user requirements.Failure to monitor progress and to perform risk management.Failure to use design reviews and code inspections.
7 How to Avoid Worst Software Practices? To avoid these failures and thus improve the record, what is needed is a better understanding of the process, better reporting system, better estimation techniques for cost time and quality measures.But the question is, what is a process? Process transform inputs to outputs i.e. a product.A software process is a set of activities, methods and practices involving transformation that people use to develop and maintain software.
8 Software Testing RoleHaving talked about the Software process overall, it is important to identify and relate the role software testing plays not only in producing quality software but also maneuvering the overall process.The computer society defines testing as follows: “Testing -- A verification method that applies a controlled set of conditions and cause for the purpose of finding errors. This is the most desirable method of verifying the functional and performance requirements. Test results are documented proof that requirements were met and can be repeated. The resulting data can be reviewed by all concerned for confirmation of capabilities.”There may be many definitions of software testing and many which appeal to you from time to time, but its best to start by defining testing and then move on depending on the requirements or needs.
9 Heuristics of Software Testing: TestabilitySoftware testability is how easily, completely and conveniently a computer program can be tested.Software engineers design a computer product, system or program keeping in mind the product testability. Good programmers are willing to do things that will help the testing process and a checklist of possible design points, features and so on can be useful in negotiating with them.Here are the two main heuristics of software testing.1. Visibility2. ControlDefinition of Heuristics - wondering, curious
10 Heuristics of Software Testing: Visibility Visibility is our ability to observe the states and outputs of the software under test.Features to improve the visibility areAccess to CodeDevelopers must provide full access (source code, infrastructure, etc) to testers. The Code, change records and design documents should be provided to the testing team. The testing team should read and understand the code.Event loggingThe events to log include User events, System milestones, Error handling and completed transactions. The logs may be stored in files, ring buffers in memory, and/or serial ports. Things to be logged include description of event, timestamp, subsystem, resource usage and severity of event. Logging should be adjusted by subsystem and type. Log file report internal errors, help in isolating defects, and give useful information about context, tests, customer usage and test coverage.The more readable the Log Reports are, the easier it becomes to identify the defect cause and work towards corrective measures
11 Heuristics of Software Testing: Visibility Features to improve the visibility areError detection mechanismsData integrity checking and System level error detection (e.g. Microsoft Appviewer) are useful here. In addition, Assertions and probes with the following features are really helpful: Code is added to detect internal errors. Assertions abort on error. Probes log errors.Resource MonitoringMemory usage should be monitored to find memory leaks. States of running methods, threads or processes should be watched (Profiling interfaces may be used for this.). In addition, the configuration values should be dumped. Resource monitoring is of particular concern in applications where the load on the application in real time is estimated to be considerable.
12 Heuristics of Software Testing: Control Control refers to our ability to provide inputs and reach states in the software under test.The features to improve controllability are:Test PointsAllow data to be inspected, inserted or modified at points in the software. It is especially useful for dataflow applications. In addition, a pipe and filters architecture provides many opportunities for test points.Custom User Interface controlsCustom UI controls often raise serious testability problems with GUI test drivers. Ensuring testability usually requires:Adding methods to report necessary informationCustomizing test tools to make use of these methodsGetting a tool expert to advise developers on testability and to build the required support.Asking third party control vendors regarding support by test tools.
13 Heuristics of Software Testing: Control The features to improve controllability are:Test InterfacesInterfaces may be provided specifically for testing e.g. ExcelExisting interfaces may be able to support significant testing e.g. InstallSheild, AutoCAD, Tivoli, etc.Fault injectionError seeding---instrumenting low level I/O code to simulate errors---makes it much easier to test error handling. It can be handled at both system and application level, Tivoli, etc.Installation and setupTesters should be notified when installation has completed successfully. They should be able to verify installation, programmatically create sample records and run multiple clients, daemons or servers on a single machine.
14 Different Platforms and Servers Many problems that current Web sites face have nothing to do with development, but rather with deployment.The challenge of building Web sites with reliability, scalability, stability, and manageability needs to be addressed.As Web sites begin to handle more business-critical applications, the systems management and operational issues associated with Web development become crucial.Reliability – uptime (+ recoverability – can continue to proceed even after some errors: temp lost of network connectivity)Scalability – amount of operations/clicks/users per second -> building farmStability – Won’t crash on big load or short load spark.Manageability – Updates/upgrades is not a disaster. Don’t have to reboot, do many operations one by one, role back is available, etc
15 Different Platforms and Servers A new model for Web development has evolved to address these development and deployment issues. (The Netscape Application Server has popularized Web application servers. ) The application servers form a clear level of separation between the Web server and data access layers.Web sites built using the application server model consist of at least three back-end layers:• Web server• Application server• Data layersLayers separation: Few cooks for big party. Imagine that everyone will prepare the same dish from end to end many times or somebody working with meat, another person cutting veggies, etc
16 Application Server : Middle Tier The logic exists in the middle tier, with application servers handling all data manipulation and HTML page-creation functions.The application server approach offers a number of natural advantages, particularly for applications that over time will grow in complexity in terms of business logic or number of users.
17 WebServer: Performance Measurements There have been a number of server-side technologies used to increase the power of the server beyond its ability to deliver standard HTML pages; these include CGI scripts, SSL security, ASPs, JSPs, Flash and many more.When testing the Web server, there are three important performance measurements:Response timeTransaction rateConcurrency
18 The Response TimeThe response time is the total time to send the request to the server and receive the complete response back.The response time is closest to the performance that the remote user sees.The lower the response time, the better the server.
19 The Transaction RateThe transaction rate is the total number of requests that can be processed per second.Because modern Web servers are multitasking and/or multithreading, it is possible for a Web server to support a much higher transaction rate than response time.The higher the transaction rate, the better the server.
20 ConcurrencyTo see how well the server multitasks among multiple simultaneous requests, a concurrency statistic measurement is used. Concurrency is when the average number of simultaneous connections to the server are fired at once.Unless the server is overloaded, this number will usually be close to the level specified by the requirements. The higher the number, the better the performance of the server because the system will not overload.
21 Web Server Testing Features (PERFORMANCE) FEATURE DEFINITIONTransactions: The number of times the test script requested the correct URL.Elapsed time: The number of seconds it took to run the request.Bytes transferred: The total number of bytes sent or received, less HTTP headers.Response time: The average time it took for the server to respond to each individual requestTransaction rate: The average number of transactions the server was able to handle per secondTransferance: The average number of bytes transferred per secondConcurrency: The average number of simultaneous connections the server was able to handle during the test session.Status code nnn: This indicates how many times a particular HTTP status code was seen.PAGE 150
22 Web Server SecurityBecause Web sites can share information, tight security and encryption have become important issue.The most common form of security supported by Web servers is basic authentication, in which users need to provide a user ID and password.Most Web servers support such basic authentication, but some servers go a step further and allow access restriction by IP address or host name.
23 Web Server: SSL and Encryption Encryption can be used to protect against wire sniffers.Web servers use SSL to support encryption.All commercial Web servers support SSL, but some support more key-exchange and encryption algorithms.SSL creates a secure, encrypted channel between the server and browser by using certificate authentication.ADD slide with description of SSL and Encrypt…
24 What is SSL?SSL is an acronym for Secure Sockets Layer, a global standard security technology developed by Netscape in It creates an encrypted link between a web server and a web browser. The link ensures that all data passed between the web server and browser remains private and secure and is recognized by millions of consumers by a secure padlock which appears in their browser.
25 Web Server Security: Optional Reading The SSL Protocol:The TLS Protocol. RFC2246:HTTP Over TLS. RFC2818:
26 SSL Certificate: Browser Compatibility Internet Explorer 5.01 and aboveNetscape 4.77 and aboveFirefox 0.1 and aboveMozilla 0.6 and aboveAOL 5 and aboveOpera 8 and aboveSafari 1.2 and above
27 Browsers Global Usage Statistic Worldwide % 1 Microsoft IE % 2 Mozilla Firefox % 3 Apple Safari 1.75 % 4 Opera 0.77 % 5 Netscape 0.26 % USA % 1 Microsoft IE % 2 Mozilla Firefox % 3 Apple Safari 3.55 % 5 Netscape 0.76 %
28 Web Servers List The most important and popular web servers: Apache web server - the HTTP web serverApache TomcatMicrosoft Windows Server 2003 Internet Information Services (IIS)lighttpd (pronounced "lighty")JigsawKloneSun Java System Web ServerXitami web serverZeus web server
29 Apache web server-the HTTP web server Free and the most popular web server in the world developed by the Apache Software Foundation.Apache web server is an open source software and can be installed and made to work on almost all operating systems including Linux, Unix, Windows, FreeBSD, Mac OS X and more. About 60% of the web server machines run the Apache web server
30 Apache TomcatThe Apache Tomcat has been developed to support servlets and JSP scripts. Though it can serve as a standalone server, Tomcat is generally used along with the popular Apache HTTP web server or any other web server. Apache Tomcat is free and open source and can run on different operating systems like Linux, Unix, Windows, Mac OS X, Free BSD.
31 Microsoft Windows Server: Internet Information Services (IIS) The IIS (ver 6.0) for Windows Server 2003 operating system has been developed by Microsoft.It offers higher levels of performance and security than its predecessor, the Windows 2000 server.It is the second most popular server on the web. (Latest version 7.5)Read more about Web Servers here.
32 Web Capacity: Load and Stress Testing Load and stress testing are critical components of Web testing. This type of testing requires many simultaneous users to make requests during peak activity that will put a large load on the Web server's processor.The key to a successful Web site is to have the hardware configured correctly so that it will be powerful enough to meet the demands required.
33 Basic setup for a load testing Load and stress testing are essential to ensure that these demands are met. By performing load testing, you will be able to find performance bottlenecks in your design and setup during the early stages of development.Figure below illustrates a basic setup for a load testing environment
34 Load Testing for the Web Test The performance of the load or stress test Web site should be monitored with the following in mind: The load test should be able to:Support all browsers.Support all Web servers.The tool should be able to simulate at least 1000 users or playback machines.
35 Load Testing for the Web Test (continue) The tool should be able to run on Windows, Linux, Solaris, and most Unix variants.After the tests are run, you should be able to report the transactions, URL, and number of users who visited the site.The test cases should be assembled in a like fashion to set up test suites.There should be a way to:simulate various users at different connection speeds.test the different servers and port addresses.account for the user's cookies.test for the back-end process, including Active Server Pages, applets, servlets, plug-ins, ActiveX components, ISAPI, and cgi-bin.
36 Load TestingLoad testing is a simulation of how a browser will respond to intense use by many individuals. The following are two different types of load tests:Single session. A single session should be set up on a browser that will have one or multiple responses. The timing of the data should be put in a file. After the test, you can set up a separate file for report analysis.Multiple session. A multiple session should be developed on multiple browsers with one or multiple responses. The multivariate statistical methods may be needed for a complex but general performance model.
37 Memory leaks under stress testing Memory leaks are often found under stress testing.A memory leak occurs when a test leaves allocated memory behind and does not correctly return the memory to the memory allocation scheme.The test seems to run correctly, but after several iterations, available memory is reduced until the system fails.Perfmon lab
38 Stress Test Environment (Different Type of the Test Bed) Slide: 1 of 4 As you set up your testing environment for a stress test, you need to make sure you can answer the following questions:• Will my test be able to support all the users and still maintain performance?• Will my test be able to simulate the number of transactions that pass through in a matter of hours?• Will my test be able to uncover whether the system will break?
39 Stress Test Environment. Slide: 2 of 4 • Will my server crash if the load continues over and over? The test should be set up so that you can simulate the load; for example: • If you have a remote Web site, you should be able to monitor up to four Web sites or URLs. • There should be a way to monitor the load intervals. • The load test should be able to simulate the SSL (Secure Server).
40 Stress Test Environment. Slide: 3 of 4 • The test should be able to simulate when a user submits the Form Data (GET method). • The test should be set up to simulate and authenticate the keyword verification. • The test should be able to simulate up to six addresses and an alert should occur when there is a failure.Load valid and invalid data simultaneously.
41 Stress Test Environment. Slide: 4 of 4 • Remember when stressing your Web site to give a certain number of users a page to stress test and give them a certain amount of time in which to run the test. • The key here is to continue to increase the stress level by increasing the number of users until the system performance begins to decrease.
42 Key Features to measure a Stress Test Some of the key data features that can help you measure this type of stress test, determine the load, and uncover bottlenecks in the system are:Amount of memory available and usedThe processor time usedThe number of requests per secondThe amount of time it takes WebPages to be set upServer timing errors
43 My Testing Tools recommendations There are several types of testing tools that can simulate hundreds of users at server connection speeds.I would recommend SilkPerformer from Micro Focus and LoadRunner from Mercury HP.On the next 2 slides I will mention a few key features of these tools.
45 SilkPerformerReal-world simulations. SilkPerformer accurately emulates the most realistic e-business conditions by simulating a nearly infinite number of simultaneous users and traffic scenarios with a single script. It can also simulate multiple combinations of protocols and computing environments using a single recorder to capture and replay scripts.End-to-end reliability. SilkPerformer lets you determine your site's scalability from the earliest stages of development right through final production.Firewall support. SilkPerformer maintains firewall integrity while monitoring all application and database servers across any wide area network or Internet infrastructure.Agent health control. To ensure valid test results, SilkPerformer continuously monitors the CPU utilization, memory requirements, and responsiveness of each agent.
47 LoadRunnerNonintrusive, real-time performance monitors obtain and display performance data from every tier, server and system component, and diagnostics probes gather code-level data to isolate bottlenecks at the SQL statement or method level. This combination of end user, system-level and code-level visibility dramatically reduces time to problem resolution.LoadRunner supports performance testing for a wide range of application environments and protocols, including web, SOA and web services, Ajax, RDP, database, legacy, Citrix, Java, .NET and all major ERP and CRM applications, including PeopleSoft, Oracle, SAP and Siebel. HP LoadRunner has more than 40 non-intrusive monitors tailored for these systems and provides diagnostics for J2EE, .NET, Siebel, Oracle and SAP. HP LoadRunner offers one set of rules for all your enterprise load testing requirements.HP LoadRunner integrates with the leading J2EE, Microsoft Visual Studio and Microsoft .NET environments.
48 Web Testing Tools: Selenium vs. iMacros A comparison of features, methods and commands.Selenium but not iMacrosSafari supportOpera supportGenerate codeFREEiMacros but not SeleniumFull AJAX supportAutomate/Test Flash, Flex, Silverlight & Java appletsAutomate Web Page DialogsUse image recognition to find page elementsTake Screenshots, save websitesAutomate and test file up- and downloadsData Extraction supportFree and commercial editions
49 Essential features both applications have Fully documented commands and specificationTest scripts can be easily editedSingle-step debuggingUnicode supportSchedulingRemote ControlActive user forums
50 LAB Exercise: iMacros Installation & Use Download iMacros Trial (full) version…or fromInstall the applicationSpend next 20 min to quickly learn at least 1 functionality of the iMacrosTry to create and save one CreatePrepare your questions or concerns related to this exercise for group discussionLet’s discuss your strategy of quick learning process …See who is done job differently and successfully