1. Dr. Teri Stokes, GXP International,
GCP/QA SIAC Jan. 28,2010
Conducting QA Audits
of Software Suppliers
Dr. Teri Stokes, GXP International,
Concord, MA – USA
www. GXPInternational.com
Phone:

2. View Quality Policy
Senior Management Policy and resource allocation set the tone and provide the direction for building quality into software and in a culture of compliance to software verification practices.
Ref. ISO 9001:2000 ideas.
Ref. IEEE 730 Std. Software QA Plans
Slide 2

3. ISO 9001:2000 Ideas
Management Responsibility – plan, resource, monitor, report, review, revise – stay committed
Quality Manual & Quality Mgt. System
Controlled documents & records
Design & development – specified input & output; reviews; verification; design/code change control; validation
Control of production & services
Control of purchases
Slide 3

4. IEEE 730 Std. SW QA Plans Purpose Reference Documents
Management – organization & tasks
Documentation – SDLC documents & review
Standards, practices, conventions, metrics
Reviews & audits
Testing & methods used
Problem reporting & corrective action
SQAP
Slide 4

5. IEEE 730 Std. SW QA Plans 9. Tools, techniques & methodologies
10. Code control
Media control
Supplier control
Records collection, maintenance & retention
Training
Risk management
SQAP
Slide 5

6. SW Supplier - Quality Mgt. System (QMS)
1. Corporate policy and procedures
2. Project/Group level plans
Directives
Corporate HQ, FDA, EU, ISO, IEEE...
Quality Policy
Corp. Management
SQAP for IT platforms
Corp. QA
Q Manual & SOPs
Project MVP
SW Project’s
Software Development Life Cycle – (SDLC)
CSV Package
Users’ CSV Package
SW Engineering Platform
CSV Package
Per system
SDLC SOPs
SW Engr. &
Pjt. Mgt. Stds.
Audit Reports
Audits of either internal or external software suppliers will focus on the quality management system (QMS) and its documented processes used by the supplier. The ISO 9003 standard for software suppliers is a good document to use as an audit reference for quality systems. The FDA has regulations 21CFR Section 820 that applies to Medical Device Quality Management Systems and it too can be used.
There should be a CSV package for the software engineering environment itself and then separate CSV packages for each version of each software application they produce. An independent reporting structure for the supplier’s own QA function is needed to allow them to perform internal audits of the software development life cycle (SDLC) that report issues to a separate management chain for response.
System SOPs
3. System level
CSV Packages & procedures
SW QC
User SOPs
Project Teams
Slide 6

7. The SQC Approach to Product Testing
1. Corporate policy and procedures
2. Project/Group level plans
Directives
Corporate HQ, FDA, EU, ISO, IEEE...
CSV Policy
Company Management
CSV SOP on Testing
SQAP for Prod. Services
SQAP for GXP
Product Development
SQAP for IT platforms
Services PQ Pkg.
SDLC OQ Package
Pltfm. IQ Pkg.
OQ Pkg. Plan
Local SOPs
Dept.
Work Process
Per product
Master Test Plan & Sub Test Plans – U,F,S,M,S,I,R*
SDLC SOPs
SQC SOPs
Test Cases, Scripts,
Data & Result Logs
3. System level
CSV Packages & procedures
Pgmr. Trng.
Test Summary
Report (per Plan)
Product Teams
OQ Package Summary Report
Slide 7

8. SDLC Process for a Product
SDLC Process – Waterfall, Agile, Sprint, Iteration
MRS to FRS
SDD to Milestone
Code Review
Unit Test
Smoke Test
SQC Testing
Trace Matrix,
Master Test Plan
Architecture Doc.
Trace Matrix, Coding Standards,
Milestone Definition
Check that SDD & coding stds. are met. Identify errors & deviations.
Developer Tests to Milestone SDD
Dev. Peer Tests to SDD
Project regression test to ensure a safe SQC Build
Function
Module
System
Interface
Regression
Release
This slide shows an example of developing Test Scripts for a Test Case in the Clinical Data Management (CDM) work process. The CDM work process is shown in 7 steps from Case Report Form (CRF) design through to submitting data for the New Drug Application.
In this example, the Test Case scenario is to take a clinical trial protocol and perform the CDM work process for a clinical study from start to finish. Each major Workflow step needs one or more Test Scripts depending on the complexity of the work activities performed with the system.
For each Test Script there are one or more detailed Step Procedures that tell the tester exactly what to do with the system, what the expected results should be, and has log space for Tester recording of system response to Test actions. Making color coded copies of Test Script and Step Procedures is important to help all participants keep track of the many documents produced. These two key document types have the testing record (green Step Procedure Logs) and the final Test Pass/Fail conclusion (yellow Test Script).
Slide 8

9. Release & Support for a Product
Product Release & Support
MRS to FRS
SQC Testing
Alpha Test
Beta Test
GMR* Test
Bug Fix Testing
Function
Module
System
Interface
Regression
Release
Trace Matrix,
Master Test Plan Summary Report
Internal check that all sections are working together
Trusted customer check out of product in pre-release test mode
Final formal testing by SQC to authorize commercial product release
Function
Module
System
Interface
Regression
Release
This slide shows an example of developing Test Scripts for a Test Case in the Clinical Data Management (CDM) work process. The CDM work process is shown in 7 steps from Case Report Form (CRF) design through to submitting data for the New Drug Application.
In this example, the Test Case scenario is to take a clinical trial protocol and perform the CDM work process for a clinical study from start to finish. Each major Workflow step needs one or more Test Scripts depending on the complexity of the work activities performed with the system.
For each Test Script there are one or more detailed Step Procedures that tell the tester exactly what to do with the system, what the expected results should be, and has log space for Tester recording of system response to Test actions. Making color coded copies of Test Script and Step Procedures is important to help all participants keep track of the many documents produced. These two key document types have the testing record (green Step Procedure Logs) and the final Test Pass/Fail conclusion (yellow Test Script).
* GMR = Gold Master Release
Slide 9

10. Examples of SDLC Tools Code Management: Subversion (SVN)
Concurrent Versions System (CVS)
MS SourceSafe
Traceable Iterations:
SharePoint
Slide 10

11. Popular Test Automation Tools
Tool Name
Company Name
Latest Version
HP Professional HP
10.5
IBM Rational Functional Tester
IBM Rational
Rational Robot
2001
Selenium
OpenSource Tool
1.0.1
SilkTest
Microfocus
2009
Test Complete
AutomatedQA
7.51
Test Partner
Micro Focus
6.3
Watir
1.6.5
(Ref. Wikimedia)
Slide 11

12. Automated Testing: Audit Concerns
Evidence of system output from tests
Traceable cases/scripts back to approved requirements
Full testing coverage for all approved requirements
Testing records per iteration
Slide 12

13. Automated Testing: Audit Concerns
Comprehensive Software Quality Control (SQC) testing for product release
Traceable loop back for repeat testing of failed cases/scripts after bug fix
Traceable bug/enhancement change history
Slide 13

14. Tracking Bugs & Modifications
Bug Tracking Systems
Years, where available, indicate the date of first stable release.
Client-server
Free/Open-source
Debbugs (1994); Bugzilla (1998); MantisBT (2000); BugTracker.NET (2002); Flyspray (2003); Trac (2006); Redmine (2006)
Proprietary
FogBugz (2000); JIRA (2004)
Distributed
Fossil (2006); DisTract (2007)
Hosted
SourceForge (1999); GNU Savannah (2000); Launchpad (2004); Assembla (2005); CodePlex (2006); Google Code (2007); GitHub (2008)
(Ref. Wikipedia)
Slide 14

15. SDLC Documents for Audit
SDLC SOP.DEV.001 SDLC– Product Development
Product Requirements Document
FRS
PRD
OQ Trace Matrix
SDD, Arch. Doc., Code Stds.
SW QA & Verification Plan
Code Reviews/ Unit Tests/ Smoke Test Regression
N Sprint Reports
SW QA & Verification Report
Master Test Plan
OQ Test Report
Product Docs.
Release Test docs.
QMS, IT, and Training Records
Client Audit
Internal QA Audit
Slide 15

16. OQ Package for Product X
Prepared by SDLC Project Team
Standard SW supplier’s CSV package
OQ Verification Plan
Master Test Plan & Sub Test Plans – U,F,S,M,S,I,R*
MRS, FRS, SDD, Code,
SDLC Docs., Support Contract (SLA), (Escrow)
Code & Tools Mgt. Logs, SDLC, Controlled SW Devel. Platform, Tools IQ
Test Cases, Scripts,
Data & Result Logs
Trace
SW Upgrade Plan, Bug Tracking, Audit Logs, QMS with SQC & QA
SW Engr. SOPs, Pgmr. Trng, Product Manuals, User Help Desk
Test Summary
Report (per Plan)
1. System Control
3. Testing Control
2. Human Control
OQ Verification Summary Report
Slide 16
*U,F,S,M,S,I,R = Unit, Function, Smoke, Module, System, Interface, Regression

17. Audit Points for Test Documentation
Master Test Plan was approved with or after Verification Plan approval date and before Sub Plans and Test Script approval dates.
Test Plan describes system to be tested, gives specific strategy for testing, and identifies tasks and roles responsible.
All issues arising are recorded, tracked, and resolved.
Repeat testing is performed using a new copy of a script and the run number is identified.
Test plans and scripts must reflect both Business and GxP risks, but any processes impacting on GxP must be rigorously tested.
The need for TESTABLE specifications must be particularly noted when you write the User Requirements Specification.
Slide 17

18. Audit Points for Test Documentation
Test Script author is not tester or approver for same script.
All printouts are labelled with tester ID, date, and test step ID.
All log entries are made in indelible blue or black ink.
No abbreviations (P/F, Y/N), ditto or check marks were used.
Signature page identifies names, initials, signatures, and Tester/Witness roles.
Test Script identifies requirements being tested.
Test plans and scripts must reflect both Business and GxP risks, but any processes impacting on GxP must be rigorously tested.
The need for TESTABLE specifications must be particularly noted when you write the User Requirements Specification.
Slide 18

19. Audit Points for Test Documentation
Test Summary Report clearly describes what happened, how problems were handled, who was responsible, and how the Test Plan was followed or what deviations were made and why.
Test Summary Report should show that test execution was consistent with Test Plan strategy.
OQ testing uses Software Design Specification for structural testing and FRS for functional testing.
IQ testing uses IT Dept. SOPs and work instructions to test their suitability.
Test plans and scripts must reflect both Business and GxP risks, but any processes impacting on GxP must be rigorously tested.
The need for TESTABLE specifications must be particularly noted when you write the User Requirements Specification.
Slide 19

20. Software Supplier’s Escrow Practices
Over Time, Changes Happen!
A “Trusted Third Party” keeps custody of software and documentation as per contract in case of need.
Slide 20

21. Corrective and Preventive Action (CAPA)
Help Desk tickets, customer complaints, product returns, and service reports should all contribute to ongoing product and process improvement. Does supplier record, track, respond, and review for trends and feed back into product and process improvement?
Slide 21

22. Software Product Development & Support
9. RETIRE
Application
Life Cycle
1.SYSTEM IDEA
8. MAINTAIN
Fix & Modify
7. OPERATE
Decommission
& Replace
Needs Analysis Report
GCP Work Process
Work process
2. SYSTEM PLAN
6. COMMISSION
SLA - Service Level Agreement
User Group
PRS - Product Req. Spec.
Validate & retest after fixes
Note the Development/Delivery aspects of this overall life cycle.
More detail is on the next slide.
3. DESIGN
5. TEST
Fit ?
FRS - Functional Req. Spec. SDD - SW Design Description
Req. & Design Review, Code review, Peer, Unit, Integration, System & Regression testing
Platform System
Software Development Life Cycle (SDLC)
SDLC
4. BUILD
Software supplier
Program to SDD using SW Eng. SOPs
IT/IS dept.

23. Check for Basic IT SOPs Available in Data Center & Remote User Sites
Data Center Operations
IT Training & Work Instructions
System Maintenance & Help Desk Support
Backup & Retrieval, Disaster Recovery Plan
& Remote User Sites
System & DB Security
Infrastructure Change Control
Slide 23

24. View IT Data Center Security SOP
Physical and logical security are both important.
Slide 24

25. View IT Backup & Recovery SOP
Backup and Recovery are not just for disasters!
Slide 25

26. View IT Disaster Recovery SOP
When Disaster Strikes There Is No Time For Planning!
Slide 26

27. View Business Continuity Plan
Working from Home
Alternate Sites
Manual Processes
When the systems are down, people need a plan to keep the business going!
Slide 27

28. Supplier’s Business Continuity Plan
Engineering & Services Systems
1. Threat: Fire, flood, theft, quake, virus, employee,
power surge/loss, network
4. Emergency Service: Maintain operations, protect source documents
5. Repair: Perform tasks as per severity – reboot, reinstall, rebuild, relocate
2. Response: Meeting place, assess damage, users switch to manual operations
Every audit and inspection of a GCP application will ask for the Disaster Recovery Plan for the system. Usually the IT/IS organization provides this as part of the platform service it provides. The platform responsibility, however, is only to have the system ready and available to users with platform components and software application accessible. The recovery and checking of the data for the work process are the responsibility of the user team after a disaster.
6. Restore Operations: Test system, update CM logs, activate switch back process
3. Contingency : Alternate site, supplier contacts, cross trained personnel
Software Supplier
Who, What, When, Where, Why, How?
Slide 28

29. View BCP Incident Report
Flood or “Bug” Issues
Problem Employee
Power Problems
BCPs also address theft, fraud, and vandalism!
Slide 29

30. Reference Documents
Stokes Article: Audits & Inspections – A Survive and Thrive Approach
Macadamian Articles: Software Design & Coding Practices
Slide 30

31. GCP/QA SIAC Jan. 28,2010 Thank You! Merci Tak, Tack, Takk Gracias
Obrigado
Spasibo
Nandri
Cobjai

32. Any Questions or Comments???