1. Continuous Evolution of Performance Test Assurance
Cliff Zattoni
FCA Capacity, Availability & Performance Manager

2. Team Setup Capacity Team x 3 Hybrid outsourced Tower model
Internal governance practise
Service Design + Transition

3. Challenges Outsourced partners in Support don’t take part in testing
Governance is heavily geared towards process and not technical
Leverage is not financial
Captive audience
Regulatory change
Performance Monitoring

4. For assuring the quality of Performance Testing – support side
Our Current Model
For assuring the quality of Performance Testing – support side
NFR engagement with Business and Service Design
Risk assessment
Execution analysis and tuning
Service Technical Debt clinic
Service Transition board

5. Evolution Insourcing of core capacity function
Creation of Service Design
Service Design Principles
NFRs – Re-launch
Release & Environments management re-launch
BAU Testing
Application Performance Monitoring
Empower teams to self-service
Better business engagement

6. Continuous Evolution of Performance Test Assurance
Isabelle Cosar
FCA Non-Functional Test Assurance Manager

7. NFT Setup NFT Team: OAT and Performance Test Assurance Specialists
Performance Delivery testers from supplier
OAT delivery (Infrastructure partners)
Test Leads / Managers (Coordination)

8. For assuring the quality of Performance Testing
Our Current Model
For assuring the quality of Performance Testing
NFT Team supports NFRs definition and review
Risk Assessments & Scoping
Templates, processes & guidelines
Test Assurance Board
Review PT deliverables
Provide consultancy and guidance

9. Recent Evolution(Last 12 months)
New unique supplier consultancy to deliver DEV & testing: new Performance testers and better capability
Key Core systems moving to a Continuous Delivery Model (Scrum etc..) with a central team
New Performance testers, better capability
More Continuous Improvement focus
Piloting of practices and toolset required for a continuous delivery pipeline

10. What’s the next evolution ?
With moving to continuous delivery pipeline then DevOps
Impact of CDM & DevOps on performance Testing
Have your performance test approach evolved?
What challenges did you face ?
Test in Production? Put live first then continue to test in production?
More frequent testing in BAU/Prod to establish consistent baselines?
Less emphasis on process, documentation and sign offs?
NFRs evolve as testing becomes less waterfall driven?
Impact on Performance Test Assurance:
How has your performance test assurance evolved?
Have you been able to maintain the quality of performance testing deliverables when moving to CDM with shorter delivery cycles? New model of governance?

11. - Less testing & QA ? With more confidence in the quality due to smaller and shorter delivery cycles, fixing forward quicker or quicker rollback.
Increase in the risk appetite with more tolerance of failures?
Who drives the risk appetite / what is the Operations involvement?
Better tooling to mitigate risk of less testing?
Challenges ?
Technical challenges (People skills..)
Cultural challenges (Scare of change, of risks)