1. Functional Testing with the Java Stack Test Runner

2. Disclaimer This is a training not a presentation.
Please be prepared to:
Learn
Ask questions
Participate in the lab
Prerequisites:
Knowledge of Maven (required)
Java Stack ALM Training (recommended)

3. What we’ll cover… In this training, we will discuss
What is functional testing?
How is it unique from other forms of testing?
What is the Java Stack Test Runner?
How does it integrate with Maven and AnthillPro?
How do I invoke Test Runner from my IDE?
Where do I find my functional test results?

4. What we won’t cover… Topics that we will not have time to discuss are:
Testing frameworks and APIs
i.e. TestNG, Selenium, Spock, etc.
Testing methodologies
Writing tests
Test case management tools

5. First, some background…

6. Stages of Testing
What
When
1. Unit Testing
After code compilation
2. Integration testing
After the build artifact is packaged.
3. Functional testing*
After the build artifact is deployed and started.
4. Load testing
After deployment to staging environment
5 End-user testing
After code is “feature complete”.
Testing generally falls into one of three categories based on the granularity of the test itself.
We will discuss functional testing and how it differs from unit and integration testing.

7. Unit Testing Scope Purpose Responsible Party Failure:
Narrow (single class or method)
Purpose
Validates small units of code
Responsible Party
Code developer
Failure:
Indicates code is broken
Typically fails the build
* Has direct dependencies on application code
Unit Testing
- SCOPE: narrow
- PURPOSE: to validate a specific unit of code. For example:
a. A single method or function.
- RESPONSIBILITY: the developer of the code unit to be tested.
- WHEN: Typically after code compilation.
- TEST FAILURES
a. An indication that the code is broken.
b. Will typically fail the build.

8. Integration Testing Scope Purpose Responsible Party Failure:
Medium (spans two integrated systems)
Purpose
Validate code that integrates with another system. (e.g. a database or web service)
Responsible Party
Code developer.
Failure:
Indicates failure of systems to communicate as expected.
Does not typically fail the build.
* Has direct dependencies on application code.
Integration Testing
- SCOPE: medium
- PURPOSE: to validate a system of your code as it integrates
with one or more other systems external to your code. For
example:
a. A database
b. An external web service
c. An operating system specific API.
- RESPONSIBILITY: the developer(s) of the system whose integration
is to be tested.
- WHEN: Typically after unit test validation and packaging. (You
don't want to run integration tests on broken code because these
tests are meant to test the integration of two or more systems,
not the code itself.)
- TEST FAILURES
a. As the integrated systems are typically outside the
direct control of the system being tested, a test
failure could mean one of many things:
- A configuration issue.
- A down system
- etc.
b. A test failure can, but does not typically, fail the
build.
c. A test failure indicates that the application may run,
but with some broken functionality.

9. Functional Testing Scope Purpose Responsible Party Failure:
Broad (end-to-end feature testing)
Purpose
Validate functional requirements and/or user stories
Responsible Party
QA resource or feature developer
Failure:
Indicates failure to meet user story requirements
Results recorded in test case management and/or bug tracker
* Has little or no direct dependencies on application code.
Functional Testing
- SCOPE: broad
- PURPOSE: to validate a functional requirement or user story.
This type of testing has also been called manual user testing,
automated testing, or application testing. A tester or test
agent represents a user or user agent executing some user story
against a deployed and running application. For example:
a. Access Login Page => Enter credentials => Click submit
=> Verify login success.
b. Access a REST or SOAP endpoint => Enter authentication
credentials => Request or post a resource
=> Validate the response.
- RESPONSIBILITY: Either a dedicated QA resource or the
developer(s) that wrote the application. For example, a QA
resource will typically perform functional tests against a
user-interactive web application, whereas a developer will
typically be responsible for testing REST or SOAP APIs exposed
by the application. In other words, functional testing is not
solely the role of QA engineers!
WHEN: After application deployment and startup, typically within
a dedicated test environment.
- TEST FAILURES
a. Failed results indicate a regression or change in
functionality or a failure to meet user story
requirements.
b. Failed results are often tracked and entered into a

10. Functional Testing and Application Life-cycle Management (ALM)

11. ALM Build Tools Maven Project definition Dependency management
Plug-in configuration and management
Build profiles
AnthillPro
Manages build processes
Creation of build artifact(s)
Deployment to server environment
Functional test execution
Maintains build life history
Provides ability to re-execute a process on any build life
ALM Functional Testing and Build Tools
For maximum efficiency and productivity, unit, integration and functional
tests should be capable of being executed in an automated fashion. In ICS,
most projects use the following build tools:
A. MAVEN: a life-cycle based tool for project build configuration.
Provides a default life-cycle for compiling source code, running
unit and integration tests, and deploying source artifacts to a
shared repository.
B. AnthillPro: a build management tool
- UNIT TESTS: the Maven Surefire plug-in is designed for executing
unit tests and reporting on their results.
* Invoked during the "test" phase, just after the compile
phase.
* By default, unit test class names are expected to have
the "Test" prefix or suffix. (The "TestCase" suffix is
also allowed.)
- INTEGRATION TESTS: The Maven Failsafe plug-in is designed for
executing integration tests and reporting on their results.
* Invoked during the "integration-test" phase, just after
the "test" and "package" phases.
* By default, integration test class names are expected to
have the "IT" prefix or suffix. (The "ITCase" suffix is
- FUNCTIONAL TESTS: Maven's default life-cycle provides no phase for
the execution of functional tests

12. ALM Tools: Maven Life-cycles
The default Maven build life-cycle (simplified):
compile
test (unit tests)
package
integration-test (integration tests)
install (to local Maven repository)
deploy (to remote Maven repository)
Where do we plug in functional testing?
ALM Tools & the Maven Life-cycle
A simplified view of the default Maven life-cycle is as follows:
compile
test
package
integration-test
install
deploy
Where does functional testing hook into this life-cycle?

13. ALM Tools: Maven Life-cycles
The woes of having “one build to rule them all”:
Functional Testing ≠ Integration Testing
Confuses build with server deployment
Must re-build to re-deploy or re-test
Complicates build management
Less productive use of build agents
ALM Tools & the Maven Life-cycle
A. In the past, we have typically deployed the application to be tested
during a "pre-integration-test" phase, and then ran functional tests
during the "integration-test" phase. This has a number of drawbacks:
1. It complicates the build process by introducing application
deployment as part of integration testing.
2. It confuses functional tests with integration tests.
3. An application could not be re-deployed without re-building.
4. Functional tests could not be run separate from a build.
5. It unnecessarily mingles the "primary" intent of the build with
the "secondary" processes of deployment and functional testing,
resulting in various complicated configurations for build management
tools, such as AntHill Pro.
6. The majority of the build process's time is spent on the secondary
tasks of deploying the application and executing functional tests,
leading to inefficient use of build agents in cases where only a
build, deployment, or functional test execution was necessary.

14. ALM Tools: Maven Life-cycles
The Java Stack ALM build life-cycle (simplified):
alm-db (run database scripts)
alm-deploy (deploy application to server)
alm-test (run functional tests)
alm-promote (run all previous phases)
This life-cycle is available to the artifact produced by the “alm” module of a Java Stack project.
The Java Stack ALM life-cycle for Maven builds
In cooperation with the Development Platforms Build team, the Java Stack
team defined a new life-cycle for secondary process execution:
alm-db - for running database scripts
alm-deploy - for application deployment
alm-test - for functional testing
alm-promote - for executing all previous phases
The ALM life-cycle is made available to Maven project modules whose pom has
the "alm" packaging type. Web application projects, web service projects,
and standalone QA projects created from the Java Stack Starter will each
contain an ALM module for the purpose of deploying to and/or running
functional tests against a local or remote test environment.

15. ALM Tools: Maven Life-cycle
Advantages of a second build life-cycle:
Clean separation of configuration:
One configuration for build
One configuration for deployment and/or functional testing
Separate processes:
One process for building artifact
One process for deployment and/or functional testing
Simplified build management
More productive use of build agents
What are the advantages of running functional tests in a separate life-cycle?
1. Separate configuration.
The configuration for building an application's artifact is not
mingled and confused with the configuration for deploying the
artifact to an environment and running functional tests against that
environment.
2. Separate execution.
The process for building the application's artifact is executed
separately from the process that deploys the artifact to a test
environment and that then runs tests against it.
3. Simplified build management
4. More productive use of build agents—that is, separating build tasks
from deployment tasks allows each build agent to run in less time.
So each build process is more targeted and focused on what is
actually needed, rather than doing twenty minutes of extra work
you didn’t need it to do.

16. ALM Tools: Test Runner
In Stack 3.2, as part of its ALM tooling, the Java Stack introduced the “test-bundle” artifact as a part of its suite of testing plug-ins, collectively known as “Test Runner”: Why another artifact? Why another tool?
In the past, functional tests could only be packaged in a non-executable
jar file and could not be easily executed outside of Maven. This made
executing functional tests from past code revisions cumbersome and
time-consuming. For this reason, the Java Stack has introduced a new kind
of artifact for packaging functional tests: the test bundle.

17. ALM Tools: The Test Bundle
The “test” jar
Not an executable jar
Difficult to execute without help from Maven.
Relied on maven to supply its dependencies.
Difficult to execute tests from past code revisions.
The “test-bundle” jar
An executable jar
Can be executed with or without Maven.
Comes packaged with all its dependencies.
Each test artifact represents a snapshot of the current code revision.
The Test Bundle: A new kind of artifact

18. The “failsafe:test” goal The “stack-test:functional-test” goal
ALM Tools: Test Runner
The “failsafe:test” goal
Supports TestNG
Supports test suites
Supports environment configuration
Produces JUnit and TestNG reports.
Executes tests from the target/classes folder
The “stack-test:functional-test” goal
Supports TestNG
Supports test suites
Supports environment configuration
Produces JUnit and TestNG reports.
Executes “test-bundle” artifact.

19. ALM Tools: Functional Test Goal
Common configuration for stack-test:functional-test:
testBundle – groupId and artifactId of QA test bundle
testEnv – the test environment properties to load
skipFTs – whether to skip functional tests
suiteXmlResources – TestNG suites to load
includes – test class patterns (e.g. **/*FT.class)
systemPropertyVariables – custom system properties
argLine – JVM arguments (e.g. memory options, debugger, etc.)
For the additional options, see:

20. ALM Tools: The Test Bundle
Simple to upgrade existing QA projects.
1. Upgrade the project to Stack 3.2
Then change the QA pom’s packaging type from to
2. Add QA module as a dependency to the ALM module.
<packaging>jar</packaging>
<packaging>test-bundle</packaging>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>example-qa</artifactId>
<version>${project.version}</version>
<type>test-bundle</type>
</dependency>
1. Simple to upgrade existing QA modules.
Any existing Java Stack QA module created prior to Java Stack 3.2 can be
changed into a test bundle by simply changing the packaging type from
"jar" to "test-bundle".
<packaging>test-bundle</packaging>
2. Can be used to package and run TestNG tests against any deployed web
application regardless of the technology: Java, OpenWeb, XQuery, .NET,
you name it!
3. Binds to the ALM module's "alm-test" phase so it can be run
automatically after application deployment as part of a secondary
process.
Test Runner was designed specifically for functional testing.
Test Runner was designed with the QA engineer in mind.
It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language
Java Web Applications
Web Services
.NET web application
XQuery web applications
Open Web applications
Etc.
Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

21. ALM Tools: The Test Bundle
3. Bind the functional-test goal to the alm-test phase:
<plugin>
<groupId>org.lds.stack.test</groupId>
<artifactId>stack-test-maven-plugin</artifactId>
<executions>
<execution>
<id>test</id>
<goals><goal>functional-test</goal></goals>
<phase>alm-test</phase>
<configuration>
<testBundle>
<groupId>org.lds.training.testrunner.lab1</groupId>
<artifactId>testrunner-lab1-qa</artifactId>
</testBundle>
</configuration>
</execution>
</executions>
</plugin>
1. Simple to upgrade existing QA modules.
Any existing Java Stack QA module created prior to Java Stack 3.2 can be
changed into a test bundle by simply changing the packaging type from
"jar" to "test-bundle".
<packaging>test-bundle</packaging>
2. Can be used to package and run TestNG tests against any deployed web
application regardless of the technology: Java, OpenWeb, XQuery, .NET,
you name it!
3. Binds to the ALM module's "alm-test" phase so it can be run
automatically after application deployment as part of a secondary
process.
Test Runner was designed specifically for functional testing.
Test Runner was designed with the QA engineer in mind.
It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language
Java Web Applications
Web Services
.NET web application
XQuery web applications
Open Web applications
Etc.
Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

22. ALM Tools: The Test Bundle
4. Modify build profiles to fit your needs
<profile>
<id>test</id>
<properties>
<skipFTs>false</skipFTs>
<testEnv>test</testEnv>
</properties>
</profile>
1. Simple to upgrade existing QA modules.
Any existing Java Stack QA module created prior to Java Stack 3.2 can be
changed into a test bundle by simply changing the packaging type from
"jar" to "test-bundle".
<packaging>test-bundle</packaging>
2. Can be used to package and run TestNG tests against any deployed web
application regardless of the technology: Java, OpenWeb, XQuery, .NET,
you name it!
3. Binds to the ALM module's "alm-test" phase so it can be run
automatically after application deployment as part of a secondary
process.
Test Runner was designed specifically for functional testing.
Test Runner was designed with the QA engineer in mind.
It is meant to test functionality, not code and therefore can be used to test applications written on any platform or language
Java Web Applications
Web Services
.NET web application
XQuery web applications
Open Web applications
Etc.
Can be executed as part of build, a secondary process in Anthill, or independently from the command-line.

23. Lab #1 A Simple QA Test Bundle Project https://tech. lds

24. Lab #2 Running Functional Tests in AnthillPro https://tech. lds

25. Credits
The Apache Maven Project: Urban Code, for it’s AnthillPro documentation: The Dark Lord Sauron, for the quote, “One ring to rule them all”. (The Lord of the Rings, by J.R.R. Tolkien)

27. <code snippet block>