1. Performance Measures Indicating IT Project Effectiveness and Investment Success / 1 2010 Performance Measures Indicating IT Project Effectiveness and Investment Success INTOSAI Working Group on IT Audit The 6th Performance Auditing Seminar Beijing, China April 2010 Prepared By State Audit Bureau of Kuwait Information Technology Audit Technical Support Department

2. Performance Measures Indicating IT Project Effectiveness and Investment Success / 2 2010 Benefits of Performance Measurement 1.Benchmark the contribution and alignment of an IT investment with strategies and business goals. 2.Provides an accurate and a quantifiable assessment of improving and degrading measured areas. 3.Assist in assigning focus to new areas of improvement to help in directing funds in the appropriate direction and magnitude.

3. Performance Measures Indicating IT Project Effectiveness and Investment Success / 3 2010 IT Investment Measurement Approach The term IT investment is broad. It can be a product, service or program. Areas of measurement can include financial management, IT service/infrastructure availability, quality, delivery time, cost, customer satisfaction, resource utilization…etc

4. Performance Measures Indicating IT Project Effectiveness and Investment Success / 4 2010 IT Investment Measurement Approach Having a set of measurements to cover multiple IT areas helps in establishing a metric. Gradually building up with the investments goals in mind will yield more specific measurements. Linking of measurements to Organizational and IT goals gives a strong insight on areas with potential development.

5. Performance Measures Indicating IT Project Effectiveness and Investment Success / 5 2010 Measurements Focus based: Financial, Customer, Internal, Learning and Growth. Measurement unit: Percentage, number, Ratio …etc Measurement direction and Target

6. Performance Measures Indicating IT Project Effectiveness and Investment Success / 6 2010 IT Investment Case Initial audit conducted revealed findings that were reported. Measurements can be applied to in order to track performance areas of the findings.

7. Performance Measures Indicating IT Project Effectiveness and Investment Success / 7 2010 IT System Description 1.Our case required the audit of a Rationing system developed by the IT department of a Government agency to automate and optimize the delivery of essential subsidized commodities to eligible beneficiaries. The Ration Department is in charge of setting the rationing regulations and laws by Setting eligibility criteria and Determining quotas and prices. 2.The IT department of the ministry is in charge of the Rationing System to: Issue, renew, amend the ration card data for beneficiaries. Distribute the goods to consumers using the system in branches of Co-operative wholesale societies.

8. Performance Measures Indicating IT Project Effectiveness and Investment Success / 8 2010 Expected System Benefits As reported by the Ration Department, the system is expected to deliver the following benefits: 1.Provide the information on beneficiaries, ration cards and transactional data in an electronic format for easier accessibility and accuracy. 2.Non-issuance of duplicate ration cards per beneficiary. 3.Avoid the current manual issuance and indexing of ration cards.

9. Performance Measures Indicating IT Project Effectiveness and Investment Success / 9 2010 Expected System Benefits 4.Link the branches to the main system at the ministry to automate change. 5.Issue/print the new ration cards. 6.Better management of inventory and governing of transactions. 7.Provide accurate statistical reports in a timely fashion with ease.

10. Performance Measures Indicating IT Project Effectiveness and Investment Success / 10 2010 Audit Goal The goal of the audit is to: 1.Verify the efficiency and effectiveness of the system of issuing ration cards and the system of distribution/sales. 2.Verification of the systems availability. 3.Alignment to benefits and requirements. 4.Measure the security and safety of the systems.

11. Performance Measures Indicating IT Project Effectiveness and Investment Success / 11 2010 Initial Observation There was an absence of organizational and IT goals and objectives. Impact: Limits the possibility of relating the outcome of performance measures to the Organizational and IT goals as IT investment processes, resources and activities cannot be aligned/linked to a higher level business objectives. Thus, the measures will be focused on IT processes, resources and activities surrounding the Rationing system.

12. Performance Measures Indicating IT Project Effectiveness and Investment Success / 12 2010 About the Findings Next, we will discuss some of the findings of our audit. Since there were no pre-established measures, these audit findings will be used to suggest measures to the concerned agency.

13. Performance Measures Indicating IT Project Effectiveness and Investment Success / 13 2010 Audit Findings Where Performance Measures are Applicable Finding (1): Absence of documented and approved procedures and processes for project management. Potential performance measures: 1.Percent of meeting the predefined business benefit 2.Percent of budget deviation value compared to the assigned budget 3.Percent of project progress (on time and on budget)

14. Performance Measures Indicating IT Project Effectiveness and Investment Success / 14 2010 Audit Findings Where Performance Measures are Applicable 4.Percent of following project management standards and practices

15. Performance Measures Indicating IT Project Effectiveness and Investment Success / 15 2010 Audit Findings Where Performance Measures are Applicable Finding (2): Absence of standardized procedures and processes for IT resource procurement. Potential performance measures: 1.Amount of reduction of the purchasing cost 2.Percent of key stakeholders satisfied with suppliers 3.Percent of initial requirements addressed by the selected solution 4.Percent of compliance with standing procurement policies and procedures

16. Performance Measures Indicating IT Project Effectiveness and Investment Success / 16 2010 Audit Findings Where Performance Measures are Applicable 5.Percent of variance amongst budget, forecast and actual cost 6.Percent of overall cost that is allocated according to the agreed-upon cost models 7.Percent of disputed costs by business

17. Performance Measures Indicating IT Project Effectiveness and Investment Success / 17 2010 Audit Findings Where Performance Measures are Applicable Finding (3): Absence of technological direction. Potential performance measures: 1.Number and type of deviations from technology infrastructure plan 2.Percent of compliance with defined IT architecture and technology standards 3.Number of emergency changes related to the infrastructure components

18. Performance Measures Indicating IT Project Effectiveness and Investment Success / 18 2010 Audit Findings Where Performance Measures are Applicable Finding (4): Absence of IT investment feasibility study. Potential performance measures: 1.Percent of stated benefits that were not achieved due to incorrect feasibility assumptions 2.Percent of stakeholders satisfied with the accuracy of the feasibility study

19. Performance Measures Indicating IT Project Effectiveness and Investment Success / 19 2010 Audit Findings Where Performance Measures are Applicable Finding (5): Absence of disaster recovery and business continuity plan and incident logs. Potential performance measures: 1.Percentage of downtime due to unplanned outages 2.Percent of availability SLAs met 3.Frequency of service interruption of critical functions

20. Performance Measures Indicating IT Project Effectiveness and Investment Success / 20 2010 Audit Findings Where Performance Measures are Applicable Finding (6): Operation and user manuals are not updated. Potential performance measures: 1.Percent of business owners satisfied with application training and support materials 2.Number of incidents caused by deficient user and operational documentation and training 3.Satisfaction scores for training and documentation related to user and operational procedures

21. Performance Measures Indicating IT Project Effectiveness and Investment Success / 21 2010 Audit Findings Where Performance Measures are Applicable Finding (7): 1)The Ration card system does not restrict the issuance of cards to ineligible individuals. 2)The Ration card system does not implement any restrictions regarding the creation of non-existing beneficiaries. 3)The Ration card system allows the creation of beneficiaries with out names or contact information.

22. Performance Measures Indicating IT Project Effectiveness and Investment Success / 22 2010 Audit Findings Where Performance Measures are Applicable Potential performance measures: 1.Percentage of ineligible beneficiaries 2.Percentage non-existing beneficiaries 3.Percentage of beneficiaries with missing or incomplete information 4.Percent of data integrity 5.Frequency of internal control incidents 6.Cost of non-compliance, including settlements and fines

23. Performance Measures Indicating IT Project Effectiveness and Investment Success / 23 2010 Audit Findings Where Performance Measures are Applicable Finding (8): 1)The Ration distribution/sales system transactional database contains sales of commodities that are unidentifiable on the system. 2)The Ration distribution/sales system transactional database contains sales with miscalculated commodity prices.

24. Performance Measures Indicating IT Project Effectiveness and Investment Success / 24 2010 Audit Findings Where Performance Measures are Applicable Potential performance measures: 1.Percentage of transactions with unidentifiable commodity. 2.Percentage of transactions with miscalculations. 3.Percent of data integrity.

25. Performance Measures Indicating IT Project Effectiveness and Investment Success / 25 2010 Audit Findings Where Performance Measures are Applicable Finding (9): The distribution centers run a client software that saves the transactions on an unprotected local database that is reconciled with the main collective database of transactions at the end of the day due to design limitations. Some databases were reported to be found emptied at the time of reconciliation which suggests also the possibility of any other database being tampered with.

26. Performance Measures Indicating IT Project Effectiveness and Investment Success / 26 2010 Audit Findings Where Performance Measures are Applicable Potential performance measures: 1.Number of security incidents with business impact 2.Number and type of suspected and actual access violations 3.Number of occurrences of an inability to recover data critical to business process 4.Revenue lost due to loss of data

27. Performance Measures Indicating IT Project Effectiveness and Investment Success / 27 2010 Summary The audits findings we communicated to the agency in order to: 1.Resolves issues permanently where possible. 2.Apply appropriate controls and adopt suggested measures to follow up with actual performance measurement. The audit concluded for the Ration system provided the agency with the means to establish areas of improvement, measurement baselines and targets to achieve.

28. Performance Measures Indicating IT Project Effectiveness and Investment Success / 28 2010 Discussion Thank you…