Presentation is loading. Please wait.

Presentation is loading. Please wait.

ADRI A model for critical analysis A training resource to assist with CDU’s preparation for audit Prepared by A/Prof Martin Carroll PVC Learning, Teaching.

Similar presentations


Presentation on theme: "ADRI A model for critical analysis A training resource to assist with CDU’s preparation for audit Prepared by A/Prof Martin Carroll PVC Learning, Teaching."— Presentation transcript:

1 ADRI A model for critical analysis A training resource to assist with CDU’s preparation for audit Prepared by A/Prof Martin Carroll PVC Learning, Teaching & Community Engagement September 2010

2 Overarching Principle CDU will be proactive and transparent in its approach to audit. This is an opportunity to support our ongoing quality improvement efforts.

3 What is a Quality Audit? A public and systematic determination of whether: CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks; CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach); CDU’s actual practice conforms to the planned arrangements (i.e. the deployment); The arrangements achieve the desired results; CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

4 What is a Quality Audit? A public and systematic determination of whether: CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks; CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach); CDU’s actual practice conforms to the planned arrangements (i.e. the deployment); The arrangements achieve the desired results; CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

5 Reviewing Graduate Destinations ADRI Cycle of Quality Assurance Quality Audit ‘Double Loop’ Process APPROACH RESULTS DEPLOYMENT IMPROVEMENT Evidence examples: Uni graduate destination targets Industry needs analysis Careers Service Plans Curriculum demonstrating links to industry needs Evidence examples: Graduate destination statistics Evidence of employer satisfaction (e.g. survey results) Evidence examples: Internal program review reports Input obtained from industry & professional bodies Resulting action plans Reports on progress against action plans Evidence examples: Interviews with academic staff, careers service staff, recent graduates and employers Topic: Graduate Destinations External Review of the System (results in Audit Report section with Coms, Affs, Recs) Internal Review of the System (results in a Portfolio section with Strengths & OFI) ADRI is a model of quality assurance used by agencies in many countries around the world (e.g. AUQA) It can be applied to any topic – in this example, the analysis of a university’s graduate destinations

6 Key Point Each aspect of ADRI involves consideration of different types of evidence. Only the totality makes sense when analysing the big picture.

7 APPROACH RESULTS DEPLOYMENT IMPROVEMENT

8 Statements of intent (goals, objective etc.) will be found in multiple sources: – applicable Legislation – Strategic Plan and internal plans – Bylaws, policies, procedures, guidelines – Cycle 1 Audit Report – MCEETYA National Protocols – An unpacking of the Themes – External standards Approach What CDU proposes to achieve A DR I

9 Internal and external statements of intent must align. The processes for setting statements of intent require evidence of appropriate benchmarking and stakeholder involvement (this is AUQA’s proxy for “Fitness of Purpose” auditing). Approach What CDU proposes to achieve A DR I

10 Operational Plans – detailing what should be done by when, by whom, to what standard and with what resources. Manuals – detailing how processes should be implemented. Professional development and training aligned to the University’s operational needs. Alignment of resource allocation to plans. Approach How CDU proposes to achieve its purpose A DR I

11 What external requirements apply to CDU? How are they incorporated into the Strategic Plan? Does CDU have set of goals, objectives, strategies and targets that are clearly understood by the governing bodies and staff? Was there appropriate consultation, benchmarking and analysis in developing the Strategic Plan? What standards and external benchmarks have been applied? Approach Broad Questions A DR I

12 Does the planning process incorporate appropriate risk management? Does everyone know what they are supposed to be doing, how and why? Are goals well supported with plans, manuals and training? Are there clear means for monitoring progress against the goals? Approach Broad Questions (cont.) A DR I

13 An assessment of Approach, on its own, does not tell the whole story – only what is intended. Need to look at a wide range of sources to get the full Approach. Alignment of various goals and strategies ought to be by design, not accident. Equally, alignment of plans with resource allocation models ought to be by design. Approach Cautions A DR I

14 APPROACH RESULTS DEPLOYMENT IMPROVEMENT

15 Also known as ‘implementation’ or ‘process’. Looks at how CDU is implementing its approach. In other words, do the plans and bylaws happen in reality? This is best tested through interviews; tap into people’s ‘lived experiences’. Also includes consideration of input factors such as the quality of resources. Deployment Deployment Dimensions A DR I

16 What do people do? How do they know if they are doing a good job? Do all staff have the necessary authority and resources to deliver what is expected of them? Do they have the necessary skills and knowledge? Is the organisational structure a help or a hindrance to deployment? Deployment Broad Questions A DR I

17 Are there appropriate indicators for monitoring the effectiveness and efficiency of processes? How are these reported and used? Are there appropriate means for intervening if necessary? How well do they work? Where the approach is deliberately not being followed, why not? How are changes to the planned processes managed? Are people allowed to contribute ideas? Deployment Broad Questions A DR I

18 It is insufficient to only focus on deployment. It must relate to an approach and lead to results. It is essential to ‘triangulate’ anecdotal evidence about deployment (e.g. from interviews) with other sources of information, until the issue is ‘saturated’. Deployment Cautions A DR I

19 TRIANGULATE! Corroborate information by obtaining original evidence from more than one type of source SATURATE! Keep collecting evidence until nothing new of significance emerges Topic Information Source A Information Source B Information Source C

20 APPROACH RESULTS DEPLOYMENT IMPROVEMENT

21 Quality cannot be assured by only focusing on the goals, plans, inputs and processes. There must be an emphasis on what is actually achieved – the results! Every intent must have at least one reported result. Every result should link back to a statement of intent. Results Results Dimensions A DR I

22 7 + (-6) + 18 = 19 A B C It is essential that a causal relationship can be shown between the approach, the deployment and the eventual result. Otherwise the result may be just chance. If you know that A+B+C = 19 you still do not know what B is (i.e. whether each step in the process is adding value or not). Results Results Dimensions 7 + (-6) + 18 = 19 A DR I

23 Results will be both quantitative and qualitative. In order to be meaningfully interpreted, results ought to be expressed as trends over 3-5 years, with targets and benchmarks. Results for many goals will be aggregated from the results of its component objectives. Examples of good practices in reporting results follow... Results Results Dimensions A DR I

24 PERFORMANCE MEASURES Annual Result for Goal #1 Performance Measure 2007 With result Actual Result, targets Targeted Results & best practices & best practices Upper and Lower Quartiles, benchmark averages Peer Average, trend s (Ensure that the measurement scale is fully understood!) 2006

25 Staff input limited. Review processes limited. Improvements are random. Staff input allowed. Review framework. OFI and GP identified. Staff input encouraged. Reviews systematic. Benchmarking undertaken. Staff directly empowered. Self-reviews effective. Benchmark host. Ad hoc practices. Not linked to plans. Not monitored. Numerous good practices. Practice aligns with plans. Processes are analysed. Good practice systemic. Plans inform practice. Monitoring in place. Processes benchmarked. Practice informs plans. Good practices promoted. Results not linked to plans. Results not measured. Where measured, variable. Most plans have results. Most results reported. Most targets achieved. All plans have results. All results reported. Targets achieved. Stretch targets established. Results are analysed. Targets exceeded. Low awareness of issue. Ad hoc plans. Random training. Commitment attained. Planning framework. Training available. Systematic approach. Full set of plans. Training linked to plans. Leading-edge vision. Plans aligned & integrated. Training comprehensive. Annual Result for Goal #1 A R D I PERFORMANCE MEASURES One qualitative + quantitative approach

26 For each statement of intent, what are the results? Does CDU know exactly how and why those results were achieved? Have these results been appropriately contextualised (e.g. using targets, trends and benchmarks)? What meaning/interpretation does CDU derive from the results? Results Broad Questions A DR I

27 It is insufficient to only consider results. Results only make sense in the context of the approach and deployment. The manner in which a result is presented can influence how it is interpreted, so adopt an attitude of ‘healthy skepticism’. That is what the Auditors are trained to do. Results Cautions A DR I

28 APPROACH RESULTS DEPLOYMENT IMPROVEMENT

29 This dimension looks at what CDU knows about itself in order to get better and better. Intentions should be regularly recalibrated. Processes should get more efficient and more effective over time. Results should indicate increasing success. This requires a comprehensive system of review – not just consideration of results. Improvement Improvement Dimensions A DR I

30 What data about CDU performance is routinely collected and reported? How is the validity of the data ensured? What happens to the data? How is the Strategic Plan (and other plans) reviewed and revised? What review processes are in place for CDU’s major activities? How does CDU know that the review processes are effective? Improvement Broad Questions A DR I

31 Is the process of self review, learning and improvement endemic throughout the organisation? Are all staff empowered and encouraged to contribute to ongoing improvement efforts? What has changed/improved as a result of the review processes? What does CDU intend to change/ improve as a result of the review processes? Improvement Broad Questions (cont.) A DR I

32 If not driven by sound values, this aspect of ADRI can be very threatening and disempowering for staff (who may feel that their jobs are in jeopardy) and students (who may feel that their grades are in jeopardy). Findings from surveys and reviews are often not used effectively. ‘Wet Paint’ syndrome. This is fine – if it leads to improvements. Improvement Cautions A DR I

33 Key Point It is ok to use the audit to catalyse improvements. In fact, it is desirable as evidence of quality improvement in action. But new improvements must be reported as such, and set in an ongoing context.

34 Improvement Cautions A DR I

35 APPROACH RESULTS DEPLOYMENT IMPROVEMENT

36 ADRI Tips This is the overarching method of self review used for AUQA audits – by auditees and by auditors. For any given topic, at any point in time, CDU will be strong in some aspects of ADRI and less strong in others. This is normal. A quality audit looks for evidence of the quality improvement cycle in action. ADRI can be applied at macro and micro levels, but for audit purposes it is best applied at the subsection (specific topic) level.

37 Analysis vs. Problem Solving ADRI is a method for analysing a total QA system. It can be used internally or externally. It can identify strengths as well as opportunities for improvement (OFI). ADRI is not a planning framework, although it provides useful information for planning. ADRI does not identify the best solution to an OFI. To do so would require problem-solving methods (such as benchmarking). Therefore, AUQA recommendations arising from ADRI ought to focus on what needs to be improved, not how it needs to be improved.

38 APPROACH RESULTS DEPLOYMENT IMPROVEMENT Mission. Planning, designing processes and training staff Implementing plans and monitoring processes Analysing measures of outcomes against goals Reviewing & improving Approach, Deployment & Results

39 For further information: Martin Carroll, PVC Learning, Teaching & Community Engagement Tel: Fax: Sinead Vincent, Project Support Officer Tel: Fax:


Download ppt "ADRI A model for critical analysis A training resource to assist with CDU’s preparation for audit Prepared by A/Prof Martin Carroll PVC Learning, Teaching."

Similar presentations


Ads by Google