1. Building an External Quality Assurance & Improvement Program
Brian Kruk | CIA, CISA, CGAP, CCSA, CCA
Senior Director Quality Assessment Services

2. Agenda A brief history of QA Discuss the available QA&IP guidance
Examine common misconceptions in QA&IP development
Explore the differences between basic internal audit processes and effective components of a QA&IP
Utilization of the Old IIA PA to create an appropriate, right-sized QA&IP
Understand how a CMM can be used to facilitate the path to quality

3. Today’s Focus Has anyone recently completed a QA?
Has anyone performed as a validator?
Is anyone working on their Internal Assessment or Self Assessment?
What do you want out of today’s session?
Are there any questions before we begin?

4. “ Quality is not an act – it is a habit.”
- Aristotle
“ Quality means doing it right when no one is looking.”
- Henry Ford

5. Quality Assessment
The process of evaluating the efficiency and effectiveness of an internal auditing organization through a comprehensive, qualitative review of audit procedures, leading to recommendations for improving controls, reducing risk and the introductions of successful innovative best practices.
It should also ensure compliance with the International Standards for the Professional Practice of Internal Auditing and other relevant organizational and departmental policies and procedures.

6. Synopsis of QA History IIA first publication on QA in 1984
IIA recommended peer reviews in previous Standards
IIA began conducting QAs in 1986
Some QAs also conducted by other providers
GTF Brings Focus to Quality Initiative
QA Manual, 4th Edition, released in 2002
QA Manual, 5th Edition, released in 2006
QA Manual, 6th Edition, released in 2009
QA Manual, 7th Edition, released in 2013
QA Manual, 8th Edition, released in 2017

7. QAR 1984

8. Historical Situation Analysis on Standards
Consulting vs. Assurance Services
E-Commerce/Technology
Independence vs. Objectivity
Control Self-Assessment
Corporate Governance
Risk Management
Compliance Requirements
Not Covered by
Standards
Standards Coverage Inadequate
Standards in Conflict with Best Practices
Standards
Outdated

9. A Vision for the Future Professional Practices for Internal Auditing
Report of GTF to IIA Board of Directors
Adopt new framework
Revise definition of IA
Update Code of Ethics and Standards
Establish oversight committee
Develop guidance to support the Standards
The Guidance Task Force issued a report 2/99 called “A Vision for the Future”.
In it, they suggested that we needed to adopt a whole new framework to deliver professional guidance to internal auditors.
-They suggested that we needed to update the definition of internal auditing that was previously in use, and that this would lead to changes in the Code of Ethics and the Standards, too.
-They suggested we establish a new oversight committee to oversee the development of new professional guidance.
-The Guidance Task Force also said that we need to develop new and additional types of guidance to support the Standards – meaning additional information to help people interpret the Standards and put them to work.
CLICK

10. Continuous Improvement Highlights
Examples of Shortfalls
Addressing the applicability of the Standards for specialty groups
Further clarification of assurance and consulting services
Knowledge of key IT risk, controls and technology-based audit techniques
Periodic internal and external QA and ongoing monitoring as part of QA&IP
Inclusion of overall opinion and/or conclusion where appropriate, in final communications

11. Professional Practices Framework - 2002
The “Path to Quality” gets its formal start with the creation of:
7 New Quality Standards
5 Practice Advisories

12. Continuous Improvement Highlights
By Jan – 24 changes to the PFF
11 new Standards
13 additions to glossary
11 new practice advisories
5 revisions to PA’s

13. Continuous Improvement Highlights
July 2007 – Arrival of the New International Professional Practice Framework

14. Continuous Improvement Highlights
By the end of 2009 – changes to the IPPF
6 new Standards
19 new interpretations
13 additions to glossary
Practice advisories reduction to 58
3 new practice guides
New 13 GTAG’s
New 3 GAIT’s

15. Continuous Improvement Highlights
2010 to 2011 – changes to the IPPF
3 new, 1 deleted
15 revised Standards
9 new and revised interpretations
5 revisions to glossary
13 new practice advisories
8 new practice guides
3 new GTAG’s

16. QA Related Implementation Guides
IG Quality Assurance and Improvement Program
IG Requirement of the Quality Assurance and Improvement Program
IG Internal Assessments
IG External Assessments
IG1320 – Reporting on the Quality Assurance and Improvement Program
IG Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing”
IG1322 – Disclosure of Nonconformance
PG – Measuring Internal Audit Effectiveness and Efficiency
PG – Quality Assurance & Improvement Program
Old PA Managing the Risk of the Internal Audit Activity

17. Structure of Implementation Guides
Getting Started
Considerations for Implementation
Specific Related Topics
Example: IG1311 – Internal Assessments
On-going Monitoring
Periodic Self-Assessment
Considerations for Demonstrating Conformance

18. Continuous Improvement Highlights

19. Attribute Standards 1000: Purpose, Authority and Responsibility
1100: Independence and Objectivity
1200: Proficiency and Due Professional Care
1300: Quality Assurance and Improvement Program

20. Performance Standards
2000: Managing the Internal Audit Activity
2100: Nature of Work
2200: Engagement Planning
2300: Performing the Engagement
2400: Communicating Results
2500: Monitoring Progress
2600: Management’s Acceptance of Risks

21. QA Related Standards
1300 – Quality Assurance and Improvement Program (New)
The chief audit executive must develop and maintain a quality assurance and improvement program that covers all aspects of the IAA and continuously monitors its effectiveness. This program includes periodic internal and external quality assessments and on-going monitoring. Each part of the program should be designed to help the IAA add value and improve the organization’s operations and to provide assurance that the IAA is in conformity with the Standards and the Code of Ethics.
Interpretation:
A quality assurance and improvement program is designed to enable an evaluation of the internal audit activity’s conformance with the Definition of Internal Auditing and the Standards and an evaluation of whether internal auditors apply the Code of Ethics. The program also assesses the efficiency and effectiveness of the internal audit activity and identifies opportunities for improvement.

22. QA Related Standards Previous 1310: Quality Program Assessments
The internal audit activity should adopt a process to monitor and assess the overall effectiveness of the quality program. The process should include both internal and external assessments
Current 1310: Requirements of the Quality Assurance and Improvement Program
The QA&IP must include both internal and external assessments.

23. QA Related Standards Current 1311 – Internal Assessments
Internal assessment must include:
Ongoing monitoring of the performance of the IAA.
Periodic reviews performed through self-assessment or by other persons within the organization with sufficient knowledge of internal audit practices.

24. QA Related Standards (New IPPF)
Current 1311 – Internal Assessments
Interpretation:
Ongoing monitoring is an integral part of the day-to-day supervision, review and measurement of the IAA. Ongoing monitoring incorporated into the routine policies and practices used to manage the IAA and uses processes, tools and information considered necessary to evaluate conformance with the DIA, COE and Standards.
Periodic reviews are assessments conducted to evaluate conformance with the DIA, COE and Standards.
Sufficient knowledge of IA practices requires at least an understanding of all elements of the IPPF.

25. QA Related Standards Original 1312 – External Assessments
External assessments such as quality assurance reviews, should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization.

26. QA Related Standards
1st Subsequent Revision – External Assessments
External assessments should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization.
The potential need for more frequent external assessments as well as the qualifications and independence of the external reviewer or review team, including any potential conflict of interest, should be discussed by the CAE with the board.
Such discussions should also consider the size, complexity and industry of the organization in relation to the experience of the reviewer or review team.

27. QA Related Standards Current 1312 – External Assessments
External assessments must be conducted at least once every five years by a qualified independent reviewer or review team from outside the organization. The CAE must discuss with the board:
The need for more frequent external assessments.
The qualifications and independence of the external reviewer or review team, including any potential conflict of interest.

28. QA Related Standards (New IPPF)
1312 – External Assessments
Old Interpretation:
A qualified reviewer or review team demonstrates competence in two areas: the professional practice of internal auditing and the external assessment process. Competence can be demonstrated through a mixture of experience and theoretical learning. Experience gained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of a review team, not all members of the team need to have all the competencies; it is the team as a whole that is qualified. The chief audit executive uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified.
An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs.

29. QA Related Standards (New IPPF)
Current 1312 – External Assessments
Current Interpretation:
External assessment may be accomplished through a full external assessment, or a self-assessment with independent external validation. The external assessor must conclude as to the conformance with the COE and the Standards; the external assessment may also include operational or strategic comments.
A qualified assessor or assessment team demonstrates competence in two areas: the professional practice of internal auditing and the external assessment process. Competence can be demonstrated through a mixture of experience and theoretical learning. Experience gained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of an assessment team, not all members of the team need to have all the competencies; it is the team as a whole that is qualified. The chief audit executive uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified.
An independent assessor or assessment team means not having either an actual or perceived conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs. The CAE should encourage the board oversight in the external assessment to reduce perceived or potential conflicts of interest.

30. QA Related Standards
Old 1320 – Reporting on the Quality Assurance and Improvement Program
The CAE must communicate the results of the quality assurance and improvement program to senior management and the board.
Current 1320 – Reporting on the Quality Assurance and Improvement Program
The CAE must communicate the results of the quality assurance and improvement program to senior management and the board. Disclosure should include:
The scope and frequency of both the internal and externa assessments
The qualifications and independence of the assessor(s) or assessment team, including potential COI
Conclusion of assessors
Corrective action plans

31. QA Related Standards (New IPPF)
1320 – Reporting on the Quality Assurance and Improvement Program
Current Interpretation:
The form, content and frequency of communicating the results of the QA&IP is established through discussions with the senior management and the board and considers the responsibilities of the IAA and CAE as contained in the IA Charter. To demonstrate conformance with the COE, and the Standards, the results of external and periodic internal assessments are communicated upon completion of such assessments and the results of ongoing monitoring are communicated at least annually. The results include the assessor’s or assessment team’s evaluation with respect to the degree of conformance.

32. QA Related Standards
Old 1321 – Use of Conforms with the International Standards for the Professional Practice of Internal Auditing
The chief audit executive may state that the IIA conforms with the ISPPIA only if the results of the QA&IP supports this statement.
Current 1321 – Use of Conforms with the International Standards for the Professional Practice of Internal Auditing
Indicating that the IAA conforms with the ISPPIA is appropriate only if supported by the results of the QA&IP.

33. QA Related Standards Previous 1322 - Disclosure of Nonconformance
Although the internal audit activity should achieve full compliance with the Standards and internal auditors with the Code of Ethics, there may be instances in which full compliance is not achieved. When noncompliance impacts the overall scope or operation of the internal audit activity, disclosure should be made to senior management and the board.
Current Disclosure of Nonconformance
When nonconformance with the COE, or the Standards impacts the overall scope or operation of the IIA, the CAE must disclose the nonconformance and the impact to senior management and the board.

34. Full External Assessments

35. External QA Provider Resources
The IIA
Industry Groups
Consulting Firms
Local Peers

36. External Assessments Areas of focus
Review IA Activity’s charter, audit plans, policies and procedures
Review a sample of audit reports, special projects and supporting work papers
Review staff composition, supervision, professional development and response to client needs

37. External Assessments Areas of focus
Assess staff and client satisfaction through interviews and surveys
Specifically interview audit committee chairperson, a representative sample of officers, senior executives and management clients and the external auditing partner
Risk assessment methodology
Approach and adequacy of IT audit coverage

38. External Assessment Activities
Tools review
Self study/benchmarking
Customer/staff survey
On-site activities
Interviews (board, management, external auditor, staff)
QA Program
Work paper reviews
Issue report

39. QA – Assessment Objectives
Assess the efficiency and effectiveness of the internal audit activity in light of:
Its charter and mission
Expectations of the board, senior management, audit clients, and the CAE
Identify opportunities and offer ideas and counsel to the CAE and staff for:
Improving their performance
Increasing the value they add to the enterprise
Provide an opinion on the internal audit activity’s conformance to the spirit and intent of the Standards

40. QA – Assessment Approach
Advanced prep and CAE questionnaire
Survey of clients and staff
Interviews with senior managers and staff
Review tools (programs)
Background
Governance
Staff
Management
Process
Information technology
Rating of conformity with IIA Standards

41. QA – Conforming Evaluation Definitions
GC – “Generally Conforms” means the assessor has concluded that the Activity’s charter, structure, policies, and procedures, as well as the processes by which they are applied, are judged to be in conformity with a majority of the Standards with some opportunities for improvement being possible.
PC – “Partially Conforms” means the assessor has concluded that a good faith effort exists but deviations from conformity for a majority of the Standards exists and corrective action is needed. These deviations are not, however, significant enough to preclude the Activity from carrying out its responsibilities in an acceptable manner.
DNC – “Does Not Conform” means the evaluator has concluded that the Activity is not aware of, is not making good-faith efforts to comply with or is failing to achieve conformity with the majority of the Standards, thus impacting its ability to carry out its mission.

42. QA Overall Evaluation OVERALL EVALUATION Generally Conforms (GC)
Attribute Standards GC
1000 Purpose, Authority & Responsibility GC
1100 Independence & Objectivity GC
1200 Proficiency and Due Professional Care GC
1300 Quality Assurance and Improvement PC
Performance Standards GC
2000 Managing the IA Activity GC
2100 Nature of Work GC
2200 Engagement Planning GC
2300 Performing the Engagement GC
2400 Communicating Results GC
2500 Monitoring Progress GC
2600 Communicating the Acceptance of Risk GC
IIA Code of Ethics GC

43. QA – Potential Issues Reporting Categories
Opportunities to improve conformity with Standards
Opportunities for IA consideration
Suggestions for senior management
Verbal comments

44. Self-Assessment with Independent Validation

45. Self-assessment w/ Independent Validation
Benefits vs. Shortcomings
Perceived as less costly
Perceived as less intrusive
Can generates IA team buy-in
Can be a training & process improvement exercise
Documentation process more cumbersome
Perceived as less thorough
Less independent and objectivity
Lessen opportunity for best practice comparisons

46. Performing the Validation
Key points for consideration
General considerations
Planning and preparation
Interviews
Self-assessment fieldwork
Self-assessment results, recommendations and implementation plans

47. Performing the Validation
Key Points for Consideration
Perception of lower cost – more time invested by IA Activity
Project timeline controlled by IA Activity
No or limited best practice enhancements
Less independent as much of the work is done by the IA Activity
Key Point – validator should be qualified
Interview and survey limitations

48. Performing the Validation
Overview and Details
General considerations
Planning and preparation
Interviews
Self-assessment fieldwork
Self-assessment results, recommendations and implementation plans

49. Performing the Validation
General Considerations
Alternative means for complying with Standard external assessments
Benefits
Economics/practicality
Expand external assessments to more IA activities

50. Performing the Validation
General Considerations
Scope limitations
Scope more targeted/limited than full external assessment
Focused on basic IA expectations
Fulfillment of IA mission
Conformance to the Standards
Areas where in-depth analyses may be curtailed or excluded

51. Performing the Validation
Planning and Preparation
Designate project leader and team
Select external independent validator
Agree on scope and responsibilities
Prepare self study
Consider/conduct client surveys
Select audit/consulting engagements for review
Select interview candidates for team and validator

52. Performing the Validation
Interviews
Audit committee chair
Executive to whom the CAE reports
Senior and operating manager
CAE
IA staff
External auditor

53. Performing the Validation
Fieldwork
Departmental structure and organization
Risk assessment and engagement planning
Staffing skills and experience
IT review
Assessing productions and value added
Individual W/P file review

54. Performing the Validation
Results, Recommendations and Implementation Plans
Major results/findings with emphasis on
Opportunities for process improvement
Enhancing customer relations
Evaluation summary
Conclusion on conformity to the Standards

55. Performing the Validation – Recap
Validation Process
Independent validation of the self-assessment
Advance prep review
AMQ review
Report review
On-site review
Interviews
Documentation of self-assessment
Limited testing
Evaluation summary
Draft report/communication
Memorandum/closing conference/report

56. QA&IP Should Reveal the IAA is…
Efficient and effective
Structured and staffed appropriately
Has an approach that is adequate and meet stakeholder expectations
Fully complying with the Standards
Utilizes sound testing techniques, methods and technology
Considers innovative practices and adopted them, when appropriate

57. Guiding Concepts Design a program that fits your IAA
Utilize available internal resources
Treat as a project, start with a detailed plan
Promote total team involvement
Hold regularly scheduled update meetings
Educate all constituencies (IA staff, executive management, and the audit committee) on objectives and progress
Make the process as transparent, objective and participatory, as possible
Conceptualize on synergies with external QA

58. QA Related Practice Advisories
Old PA Managing the Risk of the Internal Audit Activity
Managing the risk of not achieving IA objectives
IA must manage its own risk
Three categories: audit failure, false assurance, and reputation risks
Where were the internal auditors?
IA can implement the practices to mitigate its risk:
QA&IP
Periodic reviews of audit plan
Effective planning
Effective audit design
Effective management review and escalation
Proper resource allocation
Six through 14 – additional topics of further guidance

59. Capabilities Maturity Model Example

60. Remember! “
You manage what you measure.
- Brian E. Kruk ”

61. Questions? Thank You! Brian Kruk | CIA, CISA, CGAP, CCSA, CCA
Senior Director Quality Assessment Services |