1. Testing Web Applications (Adapted from Introduction to Software Testing Chapter 7.2 Engineering Criteria for Technologies) By Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/

2. Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 2 Section 7.2 Outline 1.Overview 2.Static Hyper Text Web Sites 3.Dynamic Web Applications 1.Client-side testing 2.Server-side testing 4.Web Services Overview Most of these ideas were developed after 2000 Few are widely used Most adapt graph-based testing from Chapter 2

3. Overview Web application failures Oct 2004: Paypal waived transaction fees for an entire day because of a service outage after an upgrade Aug 2006 and July 2008: Amazon failure caused businesses to lose information and revenue Sep 2011: 2 hours downtime of Target’s web site and its 1 day intermittently functioning delayed and cancelled customers’ orders Mar 2012: The BodyShop BOGO turned to extra discount September 2012: Bank of America, JPMorgan Chase, Citigroup, US Bank, Wells Fargo, and PNC suffered online attacks 3 Increasingly use of web applications Introduction to Software Testing (Ch 7.2) © Ammann and Offutt

4. Web Applications 4 Web applications are constructed from diverse, distributed, and dynamically generated web components request response web browser client web app 1 web server web app 2 web app n      Introduction to Software Testing (Ch 7.2) © Ammann and Offutt

5. Issues in Testing Web Software n A web application is a program that is deployed on the web –Usually uses HTML as the user interface –Web-deployment means they are available worldwide –They accept requests through HTTP and return responses –HTTP is stateless – each request/response pair is independent n Web applications are usually very competitive n A web service is a web-deployed program that accepts XML messages wrapped in SOAP –Usually no UI with humans –Service must be published so other services and applications can discover them Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 5

6. Web Software n Composed of independent, loosely coupled software components –All communication is through messages –Web application messages always go through clients –The only shared memory is through the session object – which is very restricted –The definition of state is quite different n Inherently concurrent and often distributed n Most components are relatively small n Uses numerous new technologies, often mixed together Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 6

7. 7 Deploying Software n Bundled : Pre-installed on computer n Shrink-wrap : Bought and installed by end-users n Contract : Purchaser pays developer to develop and install, usually for a fixed price n Embedded : Installed on a hardware device, usually with no direct communication with user  Web : Executed across the Internet through HTTP Introduction to Software Testing (Ch 7.2)

8. © Ammann and Offutt 8 General Problem n Web applications are heterogeneous, dynamic and must satisfy very high quality attributes n Use of the Web is hindered by low quality Web sites and applications n Web applications need to be built better and tested more Introduction to Software Testing (Ch 7.2)

9. © Ammann and Offutt 9 Problem Parameters n HTTP is a stateless protocol –Each request is independent of previous request n Servers have little information about where a request comes from n Web site software is extremely loosely coupled –Coupled through the Internet – separated by space –Coupled to diverse hardware devices –Written in diverse software languages Introduction to Software Testing (Ch 7.2)

10. © Ammann and Offutt 10 Separation of Concerns in Web Apps n Presentation layer n Data content layer n Data representation layer n Data storage layer Permanent data storageHTML, output and UIComputation, data accessIn-memory data storage Introduction to Software Testing (Ch 7.2)

11. Differences in Testing Web Software n Traditional graphs do not apply –Control flow graph –Call graph n State behavior is hard to model and describe n All inputs go through the HTML UI – low controllability n Hard to get access to server-side state (memory, files, database) – low observability n Not clear what logic predicates can be effectively used n No model for mutation operators on web software Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 11

12. © Ammann and Offutt 12 New Essential Problems of Web Apps 1. Web site applications feature distributed integration and are extremely loosely coupled Internet and diverse hardware / software 2.HTML forms are created dynamically by web applications UI created on demand and can vary by user and time 3.Users can change the flow of control arbitrarily back button, forward button, URL rewriting, refresh 4.Dynamic integration of new software components new components can be added during execution Introduction to Software Testing (Ch 7.2)

13. © Ammann and Offutt 13 Problem 1: Loosely Coupled How can we ensure the reliability of this type of software? Traditional software Connected by calls and message passing High and moderate coupling server client server Web-based software Connected with HTTP and XML Loose, extremely loose, distributed coupling Introduction to Software Testing (Ch 7.2)

14. © Ammann and Offutt 14 Extremely Loose Coupling n Tight Coupling : Dependencies among the methods are encoded in their logic –Changes in A may require changing logic in B n Loose Coupling : Dependencies among the methods are encoded in the structure and data flows –Changes in A may require changing data uses in B n Extremely Loose Coupling (ELC) : Dependencies are encoded only in the data contents –Changes in A only affects the contents of B’s data Introduction to Software Testing (Ch 7.2)

15. © Ammann and Offutt 15 WebPics How you’ns doin’ Jeff Offutt! Search Recommended Movies XXXXXX Examine queue View account (Warning: Queue empty) WebPics Huan ying guang ling, Li Nan! Search Recommended Movies ACDB Examine queue View account Frequent customer bonus Problem 2: Dynamic Flow of Control How can we ensure the reliability of this type of system? Introduction to Software Testing (Ch 7.2)

16. © Ammann and Offutt 16 Dynamic Execution of Web Apps n Parts of the program are generated dynamically n Dynamic web pages are created when users make requests n Different users will see different programs ! n The potential control, ala the traditional control flow graph, cannot be known ahead of time The potential flow of control cannot be known statically Introduction to Software Testing (Ch 7.2)

17. © Ammann and Offutt 17 Problem 3: User Control Flow How can we ensure the reliability of this type of software? n Users can make unexpected changes to the flow of control –Operational transitions are NOT based on an HTML link or program statement –Back button, forward button, refreshing, caching, URL rewriting n Web software platforms offer new control flow mechanisms –Forward, redirect, asynchronous message passing (Ajax) n State is stored in the server and in the HTML in the client’s browser (cookies, URL rewriting, hidden form controls) n These transitions can cause unanticipated changes to the state of the web application Introduction to Software Testing (Ch 7.2)

18. © Ammann and Offutt 18 Problem 4: Dynamic Integration n Software modules can dynamically integrate with others if they use the same data structures n EJBs can be inserted into web applications, which can immediately start using them n Web services find and bind to other web services dynamically Introduction to Software Testing (Ch 7.2)

19. © Ammann and Offutt 19 Section 7.2 Outline 1.Overview 2.Static Hyper Text Web Sites 3.Dynamic Web Applications 1.Client-side testing 2.Server-side testing 4.Testing Web Services Static Hyper Text Web Sites

20. Testing Static Hyper Text Web Sites n This is not program testing, but checking that all the HTML connections are valid n The main issue to test for is dead links n We should also evaluate –Load testing –Performance evaluation –Access control issues n The usual model is that of a graph –Nodes are web pages –Edges are HTML links Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 20

21. Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 21 Section 7.2 Outline 1.Overview 2.Static Hyper Text Web Sites 3.Dynamic Web Applications 1.Client-side testing 2.Server-side testing 4.Testing Web Services Dynamic Web Applications

22. Testing Dynamic Web Applications n The user interface is on the client n Some software is on the client (scripts such as JavaScript) n Most software is on the server n Client-side testing does not access source or state on the server n Server-side testing can use the source or the server state Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 22

23. Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 23 Section 7.2 Outline 1.Overview 2.Testing Static Hyper Text Web Sites 3.Testing Dynamic Web Applications 1.Client-side testing 2.Server-side testing 4.Testing Web Services Client-side testing

24. Client-Side (Black-Box) Testing n The UI and the software are on separate computers n The inputs to web software are defined by the HTML form elements –Text boxes, buttons, dropdown lists, links, etc n Techniques for generating values –Supplied by the tester –Generated randomly –User session data – data collected from previous users of the software n Choosing values –Bypass testing – values that violate constraints on the inputs, as defined by client-side information n The problem of finding all screens in a web application is undecidable Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 24

25. Test Value Selection n Challenge –How to automatically provide effective test values ? n Semantic Domain Problem (SDP) –Values within the application domain are needed –Enumeration of all possible test values is inefficient n Possible solutions –Random values (ineffective) –Automatically generated values (very hard!) –User data (incomplete) –Study application and construct a set of values (feasible) –Tester-supplied inputs (feasible but expensive) Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 25

26. Web Application Input Validation Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 26 Sensitive Data Bad Data Corrupts data base Crashes server Security violations Check data Malicious Data Can “bypass” data checking Client Server

27. © Ammann and Offutt 27 User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Username should be plain text only. Age should be between 18 and 150. Invalid data, please correct … Alan<Turing500 Introduction to Software Testing (Ch 7.2)

28. Server-Side (White-Box) Testing n If we have access to the source on the server, we can try to –Model the web software : Atomic Section Modeling –Mutate the web software : Web Mutation Testing n Many testing criteria on non-web software rely on a static control flow graph –Edge testing, data flow, logic coverage, … –Also slicing, change impact analysis, … n The standard control flow graph cannot be computed for web applications ! n Existing testing techniques do not support new features used in web applications Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 28

29. Mutation Testing Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 29 n Fault-based testing n Efficient at finding faults in integration n Source code must be accessible n Purpose: –Evaluate the quality of test cases / test suites –Provide guidelines / test requirements in test case generation n Need to understand faults n Define rules to mimic the faults (so-called mutation operators) n More detail in AO 5.1

30. Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 30 Mutation Testing Mutation operators = rules specifying variations of the program Mutants = modified versions (representing faults) Killed mutants, Live mutants, Equivalent mutants Mutation score = 0 ≤ killed mutants ≤ 1 non-equivalent mutants mutation scores Tests (test inputs) subject (program) Apply mutation operators (mutate source) mutants Run tests on mutants Run tests on subject Distinguishable result? Killed mutants Live mutants Compute mutation scores Yes No

31. Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 31 Mutation Testing n Mutation scores determine the quality of the tests n Mutants are test requirements / guidelines in test case generation mutation scores Tests (test inputs) subject (program) Apply mutation operators (mutate source) mutants Run tests on mutants Run tests on subject Distinguishable result? Killed mutants Live mutants Compute mutation scores No Yes

32. Applying Web Mutation Testing n Analyze faults occurring in web applications n Define rules to mutate source code (i.e., web mutation operators) –Imitate mistakes/faults that developers could have made –Force good tests –Imitate mistakes/faults that developers do not normally make or faults that are hard to detect Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 32

33. Web-specific features Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 33 n Users’ ability to control web applications (browser features) n Identifying web application resources with URLs n Communication among web components depending on requests through the HTTP n Communication via data exchanges between web components n Control connections (include, redirect, forward) n State management techniques (cookies, URL rewriting, hidden form controls) n State scope of web components

34. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 34 n Unintended transitions via web browser features component simple link transition form link transition redirect transition operational transition index login browserecord_add submit a form post (userid, password) redirect record_insert submit a form post (name, category, content) redirect back

35. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 35 n Incorrect/inappropriate URLs –Link to or submit a form to an incorrect/non-existing URLs –Include an incorrect/non-existing web resources –Forward or redirect to an incorrect/non-existing URLs component simple link transition form link transition redirect transition operational transition index login browserecord_add submit a form post (userid, password) redirect record_insert submit a form post (name, category, content) redirect back userlogin.jsp

36. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 36 n Inappropriate transfer modes of HTTP requests component simple link transition form link transition redirect transition operational transition submit a form post (userid, password) index login browserecord_add redirect record_insert submit a form post (name, category, content) redirect back get

37. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 37 n Mismatch/missing parameters component simple link transition form link transition redirect transition operational transition submit a form post (userid, password, courseID) index login browserecord_add redirect record_insert submit a form post (name, category, content) redirect back

38. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 38 n Inappropriate usage of control connections –Redirect vs forward component simple link transition form link transition redirect transition operational transition submit a form post (userid, password) index login browserecord_add redirect record_insert submit a form post (name, category, content) redirect back forward

39. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 39 n Omit necessary state information or submit inappropriate state information –Cookies –URL rewriting –Hidden form controls n Initialize a session when it should not be Server send a request response a request

40. Example Web Faults Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 40 n Inappropriate scope setting –Allow web components to be accessed when they should not be application Client 1 Client 2 session request page forward page request forward page request Scope and Handling State in JSPs, slides from SWE 642, reproduced with permission from J. Offutt

41. Example Web Mutation Operators Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 41 n Simple link replacement … … … … … n Simple link deletion n Form link replacement … … … … method = “get”> method = “post”> … n Transfer mode replacement

42. Example Web Mutation Operators Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 42 n Hidden input replacement … <input type = “hidden” name = n1 value = v1> <input type = “hidden” name = n1 value = null> … … n Hidden input deletion … <input type = “hidden” name = n1 value = v1> <!-- input type = “hidden” name = n1 value = v1 --> … …

43. Example Web Mutation Operators Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 43 n Server-side include replacement … … … n Server-side include deletion … …

44. Example Web Mutation Operators Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 44 n Redirect transition replacement … … … n Get session replacement … session = get.session (true); session = get.session (false); …

45. Example Web Mutation Operators Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 45 n Parameter deletion … <jsp:param name=pname value =pval /> <!-- jsp:param name=pname value =pval --> … … n Scope replacement … <jsp:useBean id = ID 1 scope = "session" class = ”jBean 1 ” > <jsp:useBean id = ID 1 scope = ”page" class = ”jBean 1 ” > …

46. webMujava: Generation and Mutants Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 46

47. 47 Case Study and Results Subject: STIS MutantsHand-seeded faults JSP file#Tests#FaultsLiveKilledScore#FaultsFoundRatio about7160 1.0044 browse135514410.7520160.80 categories11396330.8526210.81 category_edit6130 1.0017140.82 index48081.00430.75 login0170 1.0019110.58 logout37250.71320.67 navigation_bar5100 1.0022 page_footer24041.0022 page_header23120.67551.00 record_add48081.0099 record_delete24041.000n/a record_edit6216150.7121140.67 record_insert49091.0099 record_search22021.0033 update_search23031.0033 Total80219291900.871471180.80

48. Conclusions n The Web provides a new way to deploy software n Web applications: –offer many advantages –use many new technologies –introduce fascinating new problems n Web Software Engineering is just beginning n Two very useful techniques: –Bypass testing : Easy to automate – no source needed –Atomic sections : A fundamental model –Mutation testing : Applicable to interaction faults n This is a very active research area Introduction to Software Testing (Ch 7.2) © Ammann and Offutt 48