Test Management Challenges

Topics Drivers Value Contribution Focus Areas –Information provision –Estimation –Supplier management –End to end view of testing –Offshore –Automation –Environments –Non-functional testing –Staff assessment and selection

Drivers Business view of information technology’s contribution –Frustration –Lack of confidence –Not in line with business objectives Greater intolerance of inappropriate levels of quality –Can be too high quality as well as too poor, depends on business goals –Delivering the wrong system on time is also poor quality

Drivers - 2 Higher number of parties involved with their own agenda’s –Internal development teams –Product suppliers –Outsourced partner –Systems integrator –Niche suppliers Different types of development e.g. Agile –Beware as lots of different “flavours” –Test driven development is a good thing and not really new –Some Agile proponents believe no integration or acceptance testing is required Complex architecture e.g. SOA “Service Oriented Architecture” –High degree of retesting required –Not well understood in the market generally yet –Design wise makes sense if technical infrastructure can support it

Drivers - 3 Different geographic locations and cultures –Additional management challenge –Method of working needs adjustment to fully benefit –Account of different cultures in planning and estimating Multiple skill types required within the team –An effective testing team has ALL the skills within IT life cycle –Test managers usually manage progress during stormy weather –Non-functional testing requires development, database, infrastructure, design skills –Automation requires development skills –Requirements verification requires analysis skills –Plus all the testing skills!

Value Contribution Information provision –During the strategy, discover what information is valuable this time around –It’s the stake holders view that matters Finding smarter ways to work –Maybe automation –Effective reporting techniques –Workshops to free blockages or identify risks Taking an overall view to reduce gaps and duplication –If an area of testing is “out of scope” who does own it? –Who is doing what? –Are we using the same standards?

Value Contribution - 2 Contract negotiation and management –Risk assessment and audit of suppliers –Acceptance criteria and process within contract Breadth of skills –Yes ensure your team has testing skills, via training, on the job coaching and experience –Also think about negotiating, project management, and communications –Where can different types of testers come from e.g. business, other areas in IT

Focus Areas Where it frequently goes wrong!

Information Provision What can we provide to demonstrate realisation of business goals What risks are we monitoring to see if they are realised or mitigated against Remember risks can go up with testing as well as be reduced Have set of standard metrics for improvement And a set of specific metrics for this project Think about how information is provide, need to be in the shoes of the receiver

Example Risks Coverage goals Business goals Information Drivers Objective Project Phase Reqs DesignBuildIntegSystestUATTrialProd. Objectives for each project phase are easily identified

Example

Project Percentage of overall Effort Analysis & Design41%7%22%7%18%14% Code & Unit Test41%44%33%49%52%40% System11%26%33%2%20%29% UAT7%22%12%43%9%16% Total Devt percentage82%52%55%56%70%55% Total Test percentage18%48%45%44%30%45% Number of Defects Found Unit20 System UAT Implementation05 Production112 Total Defects Found Total Time to Fix Defects Average Defect Fix Time (HRS) Total Time to Find Defects Average Defect Find Time (HRS)

Estimation Think top down as well as bottom up Don’t forget coverage by type of testing Capture actual against planned so you improve Test support activities – environments, test data, data migration Skill types required, and where they will come from Plan using average skill level not your best people Add risk factors to account for number of parties involved and scale of integration required How much change are you planning, don’t say none! Test effort is only a part of where the time goes on a project

Supplier Management If it’s a product supplier, has the produce been developed with testing in mind What is their track record in support What does their business model look like How do they test internally Be specific about acceptance criteria, for ALL project phases If you can’t document them, the supplier can’t meet them – they are wrong! Include acceptance process to manage exceptions – there will always be some Include test deliverables within contract scope – make them control documents Witness their testing Sample their test plans, test cases and results

Example Proactive and progressive acceptance Design & Build System Test Local Int Legacy PilotBetaRollout User Testing Local Int New Full Integration Baseline and Key Acceptance Milestones Acceptance Criteria TransitionTransformation

End to end view of testing Have a test strategy that goes from project inception through to deployment Clearly define who owns which element of test activities Clearly define the governance and reporting structure Who is going to monitor adherence to strategy and assist with reworking strategy as new information comes available

Example External Influences Build & TestBusinessOperationsSupport & MaintenanceWarranty Outsource Own & Implement PI Strategy And Test Strategy Unit, System Integration Data Environments Much more NF testing Manage UAT Usability testing Support business procedures Manage Performance Security,Recovery Reliability Opps procedures DR Basic Monitoring Test fixes, Regression Causal analysis Automation coverage Business continuity monitoring Initiation Senior Test Authority Owns Information Provision End-to-End Supports Stakeholder Involvement/Project Assurance/Governance Consultants Suppliers Products Business Partners

Offshore Consider your quality plan – how will you know that things have been done to your standards and plan Where does management control lie Where are the physical environments Where are your subject matter experts It’s another supplier 24/7 testing isn’t the goal 24/7 project progress is Consider your strategy carefully

Strategy Considerations Reduced Cost. Clearly this is a driving factor behind every organisation to consider offshore working. Not only for you, or UK organisations but also throughout parts of Europe and the USA. No degradation of quality. The geographic distance, different time zones and management of offshore testing cannot be allowed to affect the quality of the delivered projects. In fact, improvements in quality are frequently included in objectives for offshore testing. No increased timescales. Time to market is a critical business objective, which must not be compromised by the benefits of offshore working. Flexibility. The selected location(s) for offshore working must be prepared to deliver the level of flexibility you require, in particular regarding the timing of management communications and support for error rectification. Demand Fluctuation. The selected location(s) must be able to demonstrate how they will manage the increasing market demand from other organisations for their services, and how they will continue to deliver your required level of service. Growth Potential. The infrastructure of the offshore location(s) must support the development of relevant skills required for all aspects of testing identified as appropriate for offshore working. Maximise skills availability. Different offshore locations are tending to specialise in the development of particular skills relevant to testing and you must be able to benefit from this advantage. Risk Managed. With the political tensions arising in different geographies, it’s important that your operation is not compromised by offshore working locations.

Automation Never seen a business case made on a single project so it’s an investment Needs a business case to plan and verify return on that investment Skills required are different to traditional tester – depends on approach as to how much technical skill required Managers have to have an appreciation to be able to manage the automation team – or they’ll automate what ever they feel like Needs a strategy for automation coverage and goals

Automation Maturity Capture Replay – use of tools out of the box with little expert knowledge Capture Replay - simple modifications to automatically generated code - achieve a greater level of automation reuse Hand crafted test tool script language code to make scripts more robust to environmental conditions Use of Automation harnesses to separate the skills of the test tools expert and test analyst Test Process Automation - tools are combined to automated the complete test process (maximum asset reuse) Shelf ware !

Environments It’s not just the setting up of the technical environment It’s the management of the components within it If production support providing this service, you’ll always take second place so plan for it or ring fence resources Have someone on your team in test support role so they can effectively communicate your requirements for provision and service level for management

Non-Functional Testing Consider all types during strategy Getting requirements is always harder – needs proactive approach Special skills are required for each type of non-functional testing, may need to bring in specialists from other areas of IT

Staff selection and assessment Interviews are the least effective element of selection process Plan interviews by role Think about practical tests and aptitude tests prior to interviews Use role requirements with skill levels rather than just read CV’s The more senior or specialist the role, the harder it is to find the right person, and you’ll need the right skills to recognise them in your candidates There is a lot of legislation now on recruitment, so make sure you’re up to date If you’re taking staff from another area, still measure them against the same criteria as it may change your plans depending on their skill level, best to identify this up front and manage it.

Example Candidate Selection Process ApplicationAptitude Test CVTelephone Interview Practical Test Interpersonal Skills Test Structured interview Personality Test Final Structured Interview Testers Specialist Non- functional Roles Team Leader Test Manager (teams of 20 plus) Test Manager (teams of 50 plus) Test Architect Screen out candidatesRecruitment Decision YNNYYYYYN YYYYYYNNN YYYYYYYNN YYYYYYYNN Y Y YY YY YY YY Y Y Y Y Y YY N

Thank You