Presentation is loading. Please wait.

Presentation is loading. Please wait.

Query-based Test Generation for Database Applications David Chays, Adelphi University John Shahid, Polytechnic University Phyllis G. Frankl, Polytechnic.

Similar presentations


Presentation on theme: "Query-based Test Generation for Database Applications David Chays, Adelphi University John Shahid, Polytechnic University Phyllis G. Frankl, Polytechnic."— Presentation transcript:

1 Query-based Test Generation for Database Applications David Chays, Adelphi University John Shahid, Polytechnic University Phyllis G. Frankl, Polytechnic University DBTest '08, Vancouver, BC

2 Outline Issues in Testing DB Applications AGENDA Overview Generating Test Generation Queries Conclusions and Future Work

3 Relational databases Data is viewed as a collection of relations – relation schema – relation (relation state) Table E ssnname 001-00-0356Johnson 012-34-5678Smith 036-54-5555Jones 051-88-9911Blake Tables, tuples, attributes, constraints for example, create table E (ssn char(11) primary key, name char(25) not null)

4 DB Application SQL: declarative language for defining and manipulating databases. Includes statements to create and modify schema and to select, insert, delete, update table elements that satisfy certain conditions Application typically written in high level language host program with SQL statements embedded or dynamically generated SQL statements passed to DBMS for execution and results returned to host program Communication via host variables

5 Aspects of Correctness Does the DBMS perform all operations correctly? Is concurrent access handled correctly? Is the system fault-tolerant? Is the system secure?...  Does the application program behave as intended?

6 Traditional vs. DB programs function imperative nature function Semi-declarative nature input output input DB state output DB state

7 DB Application Testing Goal Select “ interesting ” DB states along with user inputs that exercise “ interesting ” behavior Cover wide variety of situations that could arise in practice Do so in a way that facilitates checking of output to user and resulting DB state

8 Current Practice Testing is largely manual Limited tool support – Database state generation – DBUnit – extension of JUnit with support for importing XML descriptions of database state and for checking results of individual tests supplied by tester AGENDA goal: More thorough automation of entire test process for DB applications

9 AGENDA ParserState GeneratorInput Generator State ValidatorOutput Validator AGENDA DB

10 AGENDA System Overview Inputs database schema application source code Sample values, divided into data groups test heuristics info about expected behavior of test cases Constraints on expected database state and outputs Outputs Initial database state Test cases Hooks for validation of resulting DB state and output Error reports when tests violate constraints

11 AGENDA System Overview Inputs database schema application source code Sample values, divided into data groups test heuristics info about expected behavior of test cases Constraints on expected database state and outputs Outputs Initial database state Test cases Hooks for validation of resulting DB state and output Error reports when tests violate constraints

12 Test Procedure Analyze schema and sample values files Generate an initial DB Tester specifies global consistency properties (Optional) – Update the initial DB to satisfy the global constraints Source analysis, including extracting SQL queries from the application Generate test templates Tester specifies expected behavior for transaction and/or specific test templates Generate inputs for the given transaction Generate log tables and auxiliary tables to record selected changes, along with rules on these tables to check for violations of expected behavior Run the application on test cases, automatically checking some aspects of specification

13 DB State Generation: Will Live Data Suffice? May not reflect sufficiently wide variety of situations May be difficult to find the situations of interest May violate privacy or security constraints

14 Generating Synthetic Data DB state is a collection of relation states, each of which is a subset of the Cartesian product of some domains. – Generating domain elements and gluing them together isn ’ t enough, since constraints must be honored. We attempt to generate interesting data that obey integrity constraints – Use schema and user supplied info, similar to Category-Partition method

15 Student Registration Application Tables : Person (id, name, passwd, type) Class (crsCode, credit, enrollment, maxEnrollment, profId) Transcript (sid, code, grade) who’s registered for what and their grades Constraints in schema: primary keys (possibly composite) foreign key Transcript.sid references Person.id Transcript.code references Class.crsCode

16 Sample data values and groups Name: Deng David Phyllis Gleb Eric Wang Id: --choice_name: Student 112 252 334 121 013 311 ---- --choice_name: Faculty 888 887 CrsCode: --choice_name: Undergrad CS101 EL101 CS110 ---- --choice_name: Grad EL501 CS608 CS912

17 DB table generation Tool generates tuples for insertion – select data group or NULL, guided by schema, sample values, and heuristics – select value from data group, obeying constraints – keep track of values used Outputs sequence of SQL insert statements

18 Heuristics to guide DB state generation The user is prompted to select the desired heuristic(s): – NULL values – duplicate values – boundary values – all groups The tester may select more than one heuristic. Issues: bookkeeping and controlling combinatorial explosion

19 Sample output of State Generator SidCodeGrade 311CS912 252CS101C 311EL101A 311CS608 112CS912B IdNamePasswdType 311Dengdeng1231 112Ericeric1231 887Phyllispf1232 252Wangw1231 888Glebg122 CrscodeCreditEnrollmentMaxEnrollProfId CS91245160887 CS6083250887 EL10133120888 CS905530 888 CS1013060888 Table Person Table Class Table Transcript

20 Sample output of State Generator SidCodeGrade 311CS912 252CS101C 311EL101A 311CS608 112CS912B IdNamePasswdType 311Dengdeng1231 112Ericeric1231 887Phyllispf1232 252Wangw1231 888Glebg122 CrscodeCreditEnrollmentMaxEnrollProfId CS9124260887 CS6083150887 EL10131120888 CS9055030888 CS1013160888 Table Person Table Class Table Transcript

21 Sample output of State Generator SidCodeGrade 311CS912 252CS905C 311EL501A 311CS608 112CS912B IdNamePasswdType 311Dengdeng1231 112Ericeric1231 887Phyllispf1232 252Wangw1231 888Glebg122 CrscodeCreditEnrollmentMaxEnrollProfId CS91245160887 CS6083250887 EL50133120888 CS905530 888 CS1103060888 Table Person Table Class Table Transcript

22 Input Generator Each combination of data groups can serve as a test template for test cases. For each input parameter (input host variable) find info about data groups for the associated attribute along with candidate values in the Agenda DB. Selection of arbitrary elements of data groups does not work – SELECT sid FROM transcript WHERE code = :hv1 and grade = :hv2 – Choosing hv1 = CS912, hv2 = ‘ A ’ leads to no rows that satisfy the where clause – OK for testing robustness, but one also wants test cases that test more typical behavior Original approach: Combination of sample values from relevant data groups + automatic generation and execution of SELECT statements to extract related attribute values New approach: more general, flexible, hopefully more effective

23

24 Que ry 1 Que ry n Que ry 2. Que ry 3.............. Query parameters Parameters in Where clause Parameters in SET or VALUES clause Parameters associated with Unique attributes Parameters associated with referential attributes Split parameters into two non exclusive sets Choose values from the current table Split parameters into two non exclusive sets Chooses values not in the current table Set of values Chooses values in the referenced table Set of values Query parameters Parameters in Where clause Parameters in SET or VALUES clause Parameters associated with Unique attributes Parameters associated with referential attributes Split parameters into two non exclusive sets Choose values from the current table Split parameters into two non exclusive sets Chooses values not in the current table Set of values Chooses values in the referenced table Set of values Cross product of the set of values............. Filter out those sets in which the same parameter is given more than one value Test Cases

25 Sample Code (Class Registration) BEGIN DECLARE SECTION; 1)int enroll, maxenroll ; 2)char sid[20], code [10]; END DECLARE SECTION; void register(char[] sid, char[] code) { 3) EXEC SQL SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code 4) if (enroll < maxEnroll) { 5) EXEC SQL UPDATE Class SET enrollment = :enroll + 1 WHERE crsCode = :code ; 6) EXEC SQL INSERT INTO TRANSCRIPT (sid, code) VALUES (:sid, :code); } // end if 7) COMMIT; } // end register transaction

26 Approach Test template – id: student – course code: undergrad General approach: generate a query to select values of these parameters that correspond to tuples in the populated DB representing these data groups Use auxiliary data: parameter_value_recs table

27 Table parameter_value_recs parameter_namegroup_namevalue codeUndergradCS101 codeUndergradEL101 codeUndergradCS110 codeGradEL501 codeGradCS608 codeGradCS912 sidStudent112 sidStudent252 sidStudent334 sidStudent121 sidStudent013 sidStudent311 sidFaculty888 sidFaculty887

28 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT

29 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1

30 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM (

31 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’)

32 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE EXISTS ( SELECT * FROM

33 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE EXISTS ( SELECT * FROM Class

34 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE EXISTS ( SELECT * FROM Class WHERE crsCode = temp1 )

35 SELECT enrollment, maxEnrollment INTO :enroll, :maxEnroll FROM Class WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE EXISTS ( SELECT * FROM Class WHERE crsCode = temp1 )

36 UPDATE Class SET enrollment = :enroll + 1 WHERE crsCode = :code SELECT temp1 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE EXISTS ( SELECT * FROM Class WHERE crsCode = temp1 )

37 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT

38 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2

39 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM (

40 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’)

41 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’) CROSS_JOIN (SELECT value AS temp2 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’)

42 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’) CROSS_JOIN (SELECT value AS temp2 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE NOT EXISTS (SELECT sid, code FROM Transcript WHERE sid = temp1 and code = temp2)

43 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’) CROSS_JOIN (SELECT value AS temp2 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE NOT EXISTS (SELECT sid, code FROM Transcript WHERE sid = temp1 and code = temp2) AND EXISTS (SELECT id FROM Person WHERE id = temp1)

44 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’) CROSS_JOIN (SELECT value AS temp2 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) WHERE NOT EXISTS (SELECT sid, code FROM Transcript WHERE sid = temp1 and code = temp2) AND EXISTS (SELECT id FROM Person WHERE id = temp1) AND EXISTS (SELECT crsCode FROM Class WHERE crsCode = temp2)

45 INSERT INTO Transcript (sid, code) VALUES (:sid, :code) SELECT temp1, temp2 FROM ( (SELECT value AS temp1 FROM parameter_value_recs WHERE parameter_name = ‘:sid’ AND group_name = ‘Student’) CROSS_JOIN (SELECT value AS temp2 FROM parameter_value_recs WHERE parameter_name = ‘:code’ AND group_name = ‘Undergrad’) ) W HERE NOT EXISTS (SELECT sid, code FROM Transcript WHERE sid = temp1 and code = temp2) AND EXISTS (SELECT id FROM Person WHERE id = temp1) AND EXISTS (SELECT crsCode FROM Class WHERE crsCode = temp2)

46 What is the result? SidCodeGrade 311CS912 252CS101C 311EL101A 311CS608 112CS912B IdNamePasswdType 311Dengdeng1231 112Ericeric1231 887Phyllispf1232 252Wangw1231 888Glebg122 CrscodeCreditEnrollmentMaxEnrollProfId CS9124260887 CS6083150887 EL10131120888 CS9055030888 CS1013160888 Table Person Table Class Table Transcript Test Cases (112, CS101) (112, EL101) (252, EL101) (311, CS101)

47 Status of Test Generation Query Generator Implemented – Automatically generates test generation queries given info about application unit’s SQL statements and parameters Parsing application for useful test information – Working on automating this for Java programs

48 Initial evaluation 5 transactions of TPC-C Benchmark Produced Type A test cases that caused all 5 transactions to commit Current work: How effective are these tests? – Seeded faults in the transactions – If transaction commits, check specification violation – If transaction does not commit, this means the test case was not Type A or there was a bug Initial results on error-seeded versions – Some transactions identified as buggy but most committed with no specification violation

49 Conclusions Interplay between input parameters and DB state, and SQL statements under test Integration, flexibility – Test template – Type A inputs Non-empty result sets Honoring constraints in updates

50 Conclusions Testing DB applications is hard – Huge database states Powerful DBMS could facilitate testing – Catalog information – Active DB techniques (Rule/Trigger) – Integrity Constraints – DBMS automatically log all the SQL statements AGENDA leverages DBMS capabilities to provide integrated approach to automation of test generation and result checking Preliminary empirical studies show promising results on effectiveness and efficiency of our techniques

51 Ongoing and Future Work Improve State Generator Experiment – Evaluate effectiveness and performance Complex manipulation of host variables by host program – Symbolic execution Type A and type B test cases Other kinds of test templates Further automate and integrate Java static analysis tools

52 Related Work Willmor and Embury, “An intensional approach to the specification of test cases for database applications”, ICSE 2006 Binnig et al, “Reverse query processing”, ICDE 2007 Binnig et al, “Qagen: generating query-aware test databases”, ACM SIGMOD 2007. Emmi et al, “Dynamic test input generation for database applications”, ISSTA 2007.

53 Previous work on AGENDA “A Framework for Testing Database Applications”, ISSTA 2000 Earlier version of AGENDA System was demonstrated at ASE03/ICSE03 “An AGENDA for testing relational database applications”, Journal of Software Testing, Verification and Reliability, Mar 2004. “Testing Database Transaction Concurrency”, International Conference on Automated Software Engineering 2003 “Testing Web Database Applications”, TAV-WEB workshop, ACM SIGSOFT Notes, Sept 2004 “Testing Database Transaction with AGENDA”, International Conference on Software Engineering 2005

54 Questions? chays@adelphi.edu


Download ppt "Query-based Test Generation for Database Applications David Chays, Adelphi University John Shahid, Polytechnic University Phyllis G. Frankl, Polytechnic."

Similar presentations


Ads by Google