1 © Andy Redwood Golden Rules for Quality Testing Andy Redwood

2 What’s In?  Risk Assessed Testing  How to save money  Capability Maturity  Testing Requirements  Test Estimating  Quick wins with Strategies  Marrying documentation and traceability  Philosophy and psychology of testing  Test only what you need to  Can you learn from your defects?  Reusing Test Assets  Test Environments  Measures and metrics  Use of Tools  Importance of sign off A selection of these -

3 Golden Rule Number 1 Testing is a Risk Management exercise. Is anybody worried?  why expend valuable resources searching for defects that are not important or viable to find? Ensure all the Requirements are risk-assessed.  to the best method that works for you or your organisation.  prioritise all the testing to target the most risk with the least test effort.  when there are only minor risks outstanding, an informed decision can be made on when to stop testing.

4 Golden Rule Number 2 Training in Testing reduces long term costs. Budget for Test Training.  less than 1% of an organisations training budget dedicated to training in testing is not unusual. Training in Best Practice.  formal training in Test Management, Test Design, Static and Dynamic techniques.  ISEB Foundation and Practitioners Certification. Spread the Message. Integrate testing with other disciplines.

5 Golden Rule Number 3 The Testing Culture in an organisation can only mature at the rate of its capability. Don’t try too much too soon. Establish all the perceived problems. Create a balanced economy of scale. What can be achieved will vary from one organisation to another.

6 Golden Rule Number 4 The greater the number of late changes to Requirements, the more rework is required. Firm Requirements?  requirements are not just changing but new ones are being created, even when you’re in Test Execution? Adopt the “Bus to Bus Stop” rule.  requirements get on the Bus.  nothing else gets on and nothing gets off (ring-fenced scope). requirements may change seats (change control).  new requirements get on the next Bus coming along behind (good release management).

7 Golden Rule Number 5 Testing estimates should be based on Business Risk. Do you guess?  collect testing measures and metrics.  use previous project information to guide you.  weight the risk with a ‘time taken to test’.  improve with each proceeding project.  predict with ever increasing accuracy. Warning.  ‘similar’ is not the ‘same as’.  you will need to evaluate any new ‘hotspots’.  please don’t estimate by habit!

8 Golden Rule Number 6 Drafting a Test Strategy from a facilitated workshop is a Quick Win. Strategy Workshops.  confirm the status known testing requirements very early in the life cycle.  unknown activities surface early, creating contingency within the programme.  my last 100 Test Strategy workshops have raised an average 36 previously unknown high risk issues.  some of these would not otherwise have been spotted until system test execution.

9 Golden Rule Number 7  demonstrate the relationship – ‘what’ vs ‘how’.  documentation is what we as testers must test back to.  in many cases this relationship is implied and not explicit.  it’s not in the methodologies. Linking Development documents to Test Documents improves understanding.

10 Golden Rule Number 8 All test conditions should have a parent requirement. Traceability.  between Requirements and Test Conditions within a Test Repository.  all Requirements are tested.  aids Test Progress reporting. Eye of a Bird.  scan back and forth looking for requirements that have no conditions, also look for test conditions that do not map to a requirement.  you may identify missing business processes or find an item that requires a Business decision.

11 Golden Rule Number 9 Always test the Test Environment. A few common phrases.  “The log-ons were all wrong”.  “The interface link was down”.  “We were pointing at the wrong files”. Why is this?  lack of ‘pipe-cleaning’ of the test environment prior to starting the real testing.  insufficient check-lists.  not reusing previous knowledge.

12 Golden Rule Number 10 Tests should be executed to prove that the item under test doesn’t work, rather than that it does work. Is this the mindset?  still a radical concept in some quarters?  psychological conflict with reporting a high number of errors and making good progress to plan.  high number of errors found in test must remain an important test objective.  ensures a high quality and robust systems in Production.  tests should be designed to challenge high-risk attributes at the earliest opportunity.

13 Golden Rule Number 11 It is better to test the errors out in early Project phases as opposed to having a failure in Production. Origination.  evidence suggests that 50% of all error found in system testing originate before coding (I.e. in requirements definition or design). Cost.  it can cost hundreds of times more to fix a problem in production than in the Requirements phase.

14 Golden Rule Number 11 It is better to test the errors out in early Project phases as opposed to having a failure in Production. Focus.  on static testing techniques, for example, walkthroughs, reviews, workshops and formal inspections will raise early incidents. Warning.  it is still radical for testing to be perceived as a full life- cycle activity.

15 Golden Rule Number 12 Each test phase should attempt to test something different with as little duplication as possible.  duplication costs. Some duplication is unavoidable.  testing is more efficient and effective if each test phase is defined with minimal duplication, within the Test Strategy or Approach document.  details for each test phase will appear in phase specific test plan documents.  question if a particular test phase is actually required.  base this decision on risk and viability.  prime objective for undertaking the phase, in some cases, has already been proven in previous phases.

16 Golden Rule Number 13 An incident is not a error or a fault, but is anything that was not expected. The Nature of Incidents.  incidents can occur in any Programme phase.  incidents need not be testing specific.  an incident that results from human error or a component fault will be logged as a problem. Evaluation of incidents.  categorise them under strategically defined incident types.  allow sufficient time to conduct root cause analysis.  gather meaningful measures, establish trends, calculate metrics.

17 Golden Rule Number 14 All incidents should be traceable to a Test Condition and a Requirement. This is fundamental if you are to -  mitigate all the risks (against requirements).  know you can stop testing.  evaluate data on types of incidents raised against requirements (or test conditions).  use data to make judgement decisions on likely stability and appropriate amounts of testing required.  point at the high-risk areas in the production system to monitor in the first few weeks post implementation.

18 Golden Rule Number 15 Test Assets and Project Histories must be considered for Re-use. Project Histories.  previous documentation can  reduce analysis, design and build costs.  allow better understanding of the existing functionality.  allow more accurate project estimates.  enable earlier test preparation.  testware is worth an average 40% of every project budget.

19 Golden Rule Number 15 Test Assets and Project Histories must be considered for Re-use. Re-use of Assets.  test packs run as regression tests –  provide the status of changes since the previous release.  can be constructed at all test levels to enable maximum flexibility for testing large or small projects.  form a baseline from which to commence preparation of new tests.

20 Golden Rule Number 16 If you can’t measure, you will not know if what you achieve is good or bad. Why collect?  measures lead to metrics lead to trends.  metrics form Management Information.  appropriate MI allow informed decisions.  measures make perceptions factual. Does it matter what you collect?  objective of the measure may vary.  only consistent measures will allow meaningful metrics.

21 Golden Rule Number 17 Measures point to root causes. Fix the root cause and you improve the quality.  fix to ensure the error will not recur will reduce costs and improve quality of delivery. Warning.  care is required, as it is easy to mistake analysis for the root cause for a witch-hunt.  the root cause could be anywhere; it is unlikely to be in the phase in which the incident was identified.  the important thing is to fix it and learn from it, not apportion any blame.

22 Golden Rule Number 18 Tools can only automate existing process; they require support. Are you ready?  Existing process needs to be in a state that allows it to be automated. A good way to do it!  run tests manually follow by independent automated test runs - less risk, more control. The Cost…  “the cost of the tool is not the cost of the tool”. Help, I need somebody, help…  user groups, limited (sometimes free) on-site vendor support.  help from the many consultancies.

23 Golden Rule Number 19 If you don’t ask a 3 rd Party Supplier the question, you may never be told the answer. Some of the most common questions.  how did you test this package before you handed it over to us?  do you have any test documentation I could look at?  what errors do you know to be outstanding that are in this release?  does this release include changes for any other client and if so, how does it affect us?  what’s the turnaround time for fixing our priority one problems? Want more?  71 weighted questions

24 Golden Rule Number 20 Sign off Criteria should never be a surprise to Management. Review Points.  when criteria are agreed in advance and reviewed at set points in the life cycle, sign off into Production is the norm, not the exception.  nothing escapes into production with high-risk caveats or with insufficient testing. Who Signs?  list all interfacing managers that need to always be informed.  recommend if sign off is ‘ok to go’ or ‘what the deviation is’ from the standard approach.  managers will make proactive decisions rather than have to wait until it has finished to react.

25 © Andy Redwood More Questions?