Measuring the effectiveness of government IT systems Current ANAO initiatives to enhance IT Audit integration and support in delivering Audit outcomes Wayne Jones Executive Director IT Audit Australian National Audit Office Kristen Foster Senior Director IT Audit Australian National Audit Office

Objective of Presentation Discuss the approach that the ANAO has taken to strategically expand and enhance its IT Audit coverage in delivering Audit outcomes in response to the changing IT environment

Session Overview Part 1: Introduction Part 2: Enhancing IT Audit capability –Background –Implementation approach

Part 1: Introduction The role and function of the ANAO Structure of the IT Audit program Structure of the IT Audit team

Part 2: Enhancing IT Audit Capability Background Overview of the Implementation program –Outcomes and capability –Key initiatives –Expected benefits –Overview of key program deliverables Where are we at now?

Project Background May 2009: ANAO received additional funding for enhancing IT Audit capability in support of Audit outcomes Implementation for enhanced IT Audit capability is for 5 years

Implementation Program Enhanced IT Audit Support Outcome A: Optimise use of IT Audit specialist and support tools Capability 1 : Skills development Capability 2: Infrastructure and Tools Capability 3: Methodology Outcome B: Increased IT Audit involvement with Performance Audit products Capability 4: IT Audit integration with Performance Audit Product Delivery

Overview of Initiatives Capability 1 : Skills development Technical training – Performance Audit Support Technical training IT Audit staff Capability 2: Infrastructure and Tools Software and IT asset support Management of contracted IT specialists Capability 3: Methodology Integration of IT Audit methodology and Performance Audit methodology. Review IT Audit Methodology and update with Performance Audit product requirements Capability 4: IT Audit integration with Performance Audit Product Delivery IT Audit Resource and program planning Develop and refine IT Audit products and services Advice/ consultation for Performance Audit Program Ongoing and increased performance Audit assistance Project management of IT technical audits

Expected Benefits More qualitative findings Better work papers and increased understanding of auditee business Investment in specialist analytical tools Inclusion of IT concepts and risks in overall Performance Audit program

Challenges Challenge 1: Methodology development Challenge 2: Development of Audit program Challenge 3: IT Audit Support – product development Challenge 4: Data analysis framework

Audit Approach– Before Government Compliance and Regulatory Framework (Protective Security Manual, Information Security Manual, Procurement Guidelines, Finance Minister’s Orders, Financial Management Act) Entity Governance and Accountability (Financial Management Policies – CEIs; Security Policies, Information Management Policies, IT Strategic Plan) Management Processes and Controls (IT and Corporate)(Accounting Registers, User Identity Management and Access Processes and Matrices) Network Processes and Configuration Controls Operating System Processes and Configuration Controls Application System Processes and Configuration Controls Data Management - Processes and Controls Nature, timing and extent of audit procedures

Audit Approach- After

Information Criteria Efficiency Effectiveness C.I.A. Compliance Reliability The Universe of IT Audit Domains IT Governance Continuity & User Support Operations &Network Support Systems Development Practices IT Security Management Information Systems and data

Program Development Challenge 2: –How to include IT concepts and Audit approaches to assist with performance auditing. –Impact of emerging technologies to program delivery –What are the benefits to Performance Audit in increasing –How do we measure the benefits and costs of IT?

IT Audit Products Recognition of importance of IT Better audit work-papers and findings Complex Data Analysis IT Technical Audits Specialist assistance

IT Audit Products TypeRationaleDescription IT specialist assistance Support performance Audit team with specialised IT staff. Diverse support requests Two types: ‘discrete’ assistance; and ‘integrated audits’. IT Audit support for discrete components only. May only require support to design Audit procedures, or to design and implement ‘simple’ audit workprograms. Generally audit procedures are designed to cover information criteria of C.I.A. IT technical audits Emerging industry/technological trends or Whole of government risks – ie. Disaster recovery, management of Human Resource Information Systems. Require significant IT technical expertise to assist with designing and implementing Audit procedures. 3 types: Implementation/upgrade of IT system, cross-agency audits (i.e. disaster recovery), project management of IT Security and Controls Better Practice Guides) Data analysisMethodology needed to support Tools Consistent across the audit service groups Two critical changes to analysis approach to emphasise the use of data as evidence and evaluation of evidence

Approach for Data Analysis 1. Understand the auditee’s business Identify sources of data and knowledge Determine data analysis goals Identify the best CAAT tool and approach 2. Data understanding Collect initial data/liaise with client Assist Audit team to identify sources of data and knowledge at the auditee Explore the data and data integrity 3. Data preparation Select/construct/format data (mainly for complex assessments) 4. Modelling and analysis Determine the appropriate analytical approach Generate test data and assess the ‘model’ Generate analytical tests 5. Evaluation What does the data telling us? Did we answer the right questions ? Do we have enough evidence? 6. Reporting Presentation of analysis/discussion of exceptions with Audit team and auditee

Status of Project –Methodology implemented –Planning for Audit program for 3 year period underway –Currently providing over 5000 hours support for Performance Audits – demand has doubled! –Year 2 – implementation of complex data analysis capability (tools and methodology)

Questions/Discussion