1. Automated Testing Saving Time & Money
Blayn Parkinson
E-Learning & training Support Assistant
University of York (UK)

2. About me
Blayn Parkinson E-Learning & training Support Assistant University of York (UK) I have used Blackboard for 3 years.

3. About my institution
University of York 15,330 Students Using Blackboard since Central e-Learning team 6 members (some departments have their own provision)

4. What we are going to learn today
The benefits of using automated testing strategies.
Limits to automated testing
Suggestions on future collaboration in this area
Limits are usually due to time and financial restrictions, most problems can be resolved if enough of both are thrown at the problem.
We would like to see a greater collaboration between BB and the user community to develop a solution to enable institutions to deply automated testing solutions to their live (test/dev) environments.

5. OUR challenge
Thorough testing cycle (each software not deployed without testing).
3 months: April – June (One window a year to achieve this).
Lack of agility in responding to new releases, due to the testing time that we need to devote to a release.
Needed to lock in to a release at a particular time.
Spread the load across the team in terms of manual testing.
Why consider automated testing?
- We had looked at automated testing a number of years ago, but were not that convinced it would add much value. Tests didn’t seem to run very well on Blackboard and the test playback was often hindered by the frameset.
- We decided to have another look at this, and the reasons behind that were:
- Functional testing - is necessary (we do not deploy software changes without testing them first).
- is time consuming, to test in different combinations of browser and OS takes a long time
- can be ambiguous: human testers, when asked to perform a test can go about the same test in different ways. This could result in a problem being undetected.
- What automated testing offers - it is consistent, it always performs the same test
- it is easily repeatable, you can keep on performing the same tests over and over again
- tests can be reused across different browsers and platforms

6. OUR challenge
Thorough testing cycle (each software not deployed without testing).
3 months: April – June (One window a year to achieve this).
Lack of agility in responding to new releases, due to the testing time that we need to devote to a release.
Needed to lock in to a release at a particular time.
Spread the load across the team in terms of manual testing.
Why consider automated testing?
- We had looked at automated testing a number of years ago, but were not that convinced it would add much value. Tests didn’t seem to run very well on Blackboard and the test playback was often hindered by the frameset.
- We decided to have another look at this, and the reasons behind that were:
- Functional testing - is necessary (we do not deploy software changes without testing them first).
- is time consuming, to test in different combinations of browser and OS takes a long time
- can be ambiguous: human testers, when asked to perform a test can go about the same test in different ways. This could result in a problem being undetected.
- What automated testing offers - it is consistent, it always performs the same test
- it is easily repeatable, you can keep on performing the same tests over and over again
- tests can be reused across different browsers and platforms
Functional testing
is necessary
is time consuming
can be ambiguous

7. OUR challenge
Thorough testing cycle (each software not deployed without testing).
3 months: April – June (One window a year to achieve this).
Lack of agility in responding to new releases, due to the testing time that we need to devote to a release.
Needed to lock in to a release at a particular time.
Spread the load across the team in terms of manual testing.
Why consider automated testing?
- We had looked at automated testing a number of years ago, but were not that convinced it would add much value. Tests didn’t seem to run very well on Blackboard and the test playback was often hindered by the frameset.
- We decided to have another look at this, and the reasons behind that were:
- Functional testing - is necessary (we do not deploy software changes without testing them first).
- is time consuming, to test in different combinations of browser and OS takes a long time
- can be ambiguous: human testers, when asked to perform a test can go about the same test in different ways. This could result in a problem being undetected.
- What automated testing offers - it is consistent, it always performs the same test
- it is easily repeatable, you can keep on performing the same tests over and over again
- tests can be reused across different browsers and platforms
Functional testing
is necessary
is time consuming
can be ambiguous
Automated testing is
consistent
easily repeatable
reusable

8. Organising tests
Once we had selected Selenium as our automated testing framework of choice we then had to consider how tests would be stored, understandable and how results would be recorded.
Pre automated testing tests were stored in Google spreadsheets, and a different set of spreadsheets for each test run. Selenium tests are stored in separate files and it’s not always obvious what a test is doing by looking at the code.
These are open to interpretation on what/how to achieve this when left to individuals (example link to Discussion Boards)

9. Recording Scripts Command Speed Target (Locator) Tests Value
There are many testing frameworks around, many of which integrate with an IDE, however we wanted a simple IDE to create tests in and this is what is offered by the Firefox Selenium IDE plugin.
ThoughtWorks Selenium is the testing framework most web browser automated testing products are based upon and the IDE has good export facilities should we want to export a test script out to run in Junit, or Ruby.
The plugin ide is simple. Tests cases are lists here. Each test is run line by line and is made up of a command (open, click, type), a target (html element), and if appropriate an input value. You can also set the speed at which the test should run. You don’t want it to run too quickly, otherwise the page may not have loaded completely and the test could fail.

10. Example script 2_LoginTestStudentAccount open
type
id=user_id
studtesta
id=password
********
clickAndWait
css=input[name="login"]
How breaking up the tests into individual discreet actions helps with reuse and maintenance
If an element or value needs to change across multiple documents find/replace using Notepad ++ batch tool can do this.

11. Test Manager
The solution was to create a database to store test and results in that we named ‘Test Manager’. The system has a no frills GUI via which we can commission tests, stored test scripts and access results.
There is a web service that is available to the tests which interprets results and updates the system accordingly.
This part of the process doesn’t use Selenium itself, it’s just our way of organising the test scripts created using Selenium and making sense of the results.

12. Test Manager

13. Constructing good tests
A good test
Mirror user behaviour
Independent
Report accurately
HTML present?
Text present?
Screenshot
Writing a good automated test.
A good test should be realistic and mirror user behavior – mirror what a user would do and expect to interact with. For example, if you were testing a specific tool that would normally be reached by navigating the left hand menu, then the test should navigate the left hand menu too, not just go to the page containing the tool you actually want to test.
Independent of other tests. If a test fails for whatever reason you don’t want this to impact all the other tests in the suite and prevent them from executing. Where possible tests should be self contained and not dependent on the success of other tests to run.
Report accurately. So careful planning about how the test will determine success is important. Using multiple verification commands is best with the test script analysing the page for expected html or text, or taking a screen shot for inspection later

14. Source Control
TEST SCRIPT (A1.3)
ADD BLOG POST (ROLE STUDENT)
BLOCK 1
Login as test student
BLOCK 2
Navigate to the test course site
BLOCK 3
Navigate the left hand menu to access the blog tool
NEW SCRIPT
Enter the blog and click the Add New Entry button
BLOCK 4
Add content to the content editor
BLOCK 5
Click the submit button
BLOCK 6
Take a screenshot for human inspection/verification
Verify text on the page is present
BLOCK 7
Logout of the VLE
When developing scripts we noticed that there were some actions that cropped up in nearly every test script. For example: logging in, logging out, navigating the left hand menu, or using the Content Editor. To save having to recode these every time a test script was developed we created BLOCKS of reusable code. This speeded up the development of tests scripts.
The example on screen is a test scripts for testing the blog tool, and adds a new post to the blog with the student role. As you can see when this script was developed it only required 2 new set of actions, ‘entering the blog and clicking the add new entry button’, and ‘verifying the success of the text by searching for text on the page’. The rest of the test script is made up of blocks of code that have already been written.

15. Test Results Web Service
Running the Process
Test Manager Results
Execute
Test Suite
Test Script ID
This diagram illustrates how tests are executed and results returned. Selenium will execute a suite of test scripts. Each test will run one at a time, line by line. During the test if any of the verification tests fail the test will fail. If however the tool or feature under test is working then the test will get to the last line of the test which contacts the Test Results Web Service and reports back. When contacting the web service the test will pass a result and it’s ID. The Test Manager then imports incoming results against a results set.
Pass
Test Results Web Service
Pass pending check
Fail

16. Test Cases & Runs Case = Browser/Platform combination
Run = Results against a particular case

17. Test results Green Tick = Success Red Cross X = Failure
The no entry sign = not tested

18. Pros and Cons Saved time Maintenance burden Reusable
Human intervention
To start wrapping up, these are the pros and cons and we see them. Automated testing has saved time and allowed staff who would be performing functional testing, to spend that time looking at new features. The test are reusable, across different browsers.
The cons, there is a maintenance burden keeping the scripts up and running that should be factored in, it’s not a write once operation. Human intervention is still required in some cases to review screen shots taken by tests, and reviewing the results.

19. Taking this forward Automate more Maintenance Continuous testing
Load testing
Blackboard client assisted testing?
We have automated about half our tests so we still have some more to automate.
We need to evaluate what kind of a maintenance burden keeping these tests up and running will bring.
Can we use a continuous testing tool, such as Jenkins, to run tests continuously. This would effectively be providing a service monitoring and services availability capability
Can the tests be exported to Junit and then incorporated into Jmeter to perform realistic load testing of our VLE.
Finally could Blackboard not provide automated testing facilitates that perform functional tests on our installations? They already use automated tests in a development environment, but with the power of local data, then surely they could identify far more bugs.

20. Our Results 50% of tests automated (from 2013 testing cycle)
2014 Upgrade locking into SP16
How much maintenance is needed, BB switch from iframe’s to HTML5 will impact on this.
Can we extend the current % of automated tests
Review this summer of complete process

21. Do This Next
Think about how the adoption of automated testing processes might benefit your institution. Jointly (with Blackboard) could we come up with a solution to better help the testing processes within our own institutions.
Blackboard to design and develop scripts (that they can use), they manage/curate the release of those scripts which institutions can then use to test against their own systems.

22. Thank you!
Blayn Parkinson E-Learning & training Support Assistant University of York (UK) Link to the paper: