Oman Academic Accreditation Authority (OAAA)

Oman Academic Accreditation Authority (OAAA)
Institutional Standards Assessment Workshop A training module in preparation for Institutional Standards Assessment 4 June 2015

Aims of the Workshop To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual To provide staff with a clear idea of their role in the Institutional Standards Assessment process To provide guidance on how to carry out a self-assessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI To provide participating HEI staff with a chance to ask questions and clarify understanding

Program Session 1 8.30-8.45 Introduction
OAAA’s role in Institutional Accreditation Dr Salim Radhawi CEO, OAAA Session 2 Institutional Standards Assessment Manual and Process How to read the Standards Assessment Manual (SAM) Standards Assessment Process Timeline Dr Anna Scopaz Quality Assurance Consultant Session 3 How to Carry Out a Self-assessment/Prepare an Institutional Standards Assessment Application (ISAA) ISAA Template overview Dr Salim Radhawi Break Session 4 Activity 1 Types of evidence to review when preparing an ISAA OAAA Team Session 5 Methods of Analysis/Using ADRI for Introduction of methods of analysis for Institutional Standards Assessment including the use of ADRI as an analytical tool to support self-assessment Dr Tess Goodliffe Deputy CEO, Technical Affairs Lunch Session 6 Activity 2 Rating criteria and standards OAAA Team Session 7 Recap and Tips Features of a good ISA product and process Preparing for a smooth ISA Ms Susan Trevor-Roper Senior Quality Assurance Expert Session 8 Questions Opportunity for OAAA to provide further clarifications

OAAA Roles & Responsibilities
Developing a system for institutional and program accreditation Accrediting public and private HEIs and academic programs Conducting Quality Audits of HEIs Establishing a procedure for recognising foreign programs offered in Oman Signing mutual recognition MoUs with external QA agencies Updating and maintaining the OQF

HEI QA Processes HEI Licensure HEI Accreditation Stage 1 : Quality
Audit First cycle commenced 2008 4 years ≤4 years HEI Accreditation Certificate HEI Accreditation Stage 2 : Standards Assessment Appeal Met Met Not met 1-2 years on Probation Process Document Start/End KEY HEI Standards Reassessment HEI Accreditation Terminated 5

Difference between Quality Audit and Institutional Standards Assessment
Evaluates the HEI’s effectiveness against its own stated Mission and goals Evaluates the HEI’s effectiveness against a set of nine national standards which are applied to every HEI operating in Oman Formative in nature Summative in nature Results in Recommendations, Affirmations and Commendations which help to develop the emerging HE sector in Oman Results in Accreditation Outcome Quality Audit Report made public Standards Assessment Report not made public Accreditation Outcome and ratings against standards and criteria made public

Any questions?

Session 2: Institutional Standards Assessment Manual and Process
OAAA Session 2: Institutional Standards Assessment Manual and Process A training module in preparation for Institutional Standards Assessment

Session aims To gain an understanding of the structure of the Standards Assessment Manual (SAM) To learn how to make best use of the SAM To learn about the key milestones in the Institutional Standards Assessment process

Standards Assessment Manual (SAM)
Part A Institutional Accreditation Overview Part B Institutional Standards Part C Accreditation Outcomes and Ratings against Standards and Criteria Part D The Self-assessment Part E The Institutional Standards Assessment Part F Methods of Analysis Part G Appendices See SAM Introduction, Table 1

SAM Part B Institutional Standards
Only address those standards which are relevant to your HEI STANDARD 1: GOVERNANCE AND MANAGEMENT Governance and management of the HEI is ethical and ensures implementation of academic and non-academic systems and functions which support achievement of the HEI’s Mission and Vision and the protection of academic standards. Governance and Management structures, processes and mechanisms for accountability are appropriate... Criterion 1.1: Mission, Vision and Values The Mission, Vision and Values are appropriate, have been developed in consultation with stakeholders, formally approved and guide the HEI in all its activities. The HEI has Mission and Vision statements which clearly define the HEI’s purpose, whom it serves and what it intends to accomplish; and these align with the national priorities of Oman. The HEI has a defined set of institutional Values. Indicators The Mission, Vision and Values effectively guide the HEI, are consistent with the HEI’s purpose and its ability to meet the national priorities of Oman, and community expectations. The governing body has formally approved the Mission, Vision and Values. Key stakeholders have been consulted and support the Mission, Vision and Values. The Mission, Vision and Values are readily accessible and effectively communicated to stakeholders. The Mission, Vision and Values are regularly reviewed and reaffirmed or amended as appropriate in order to maintain relevance and effectiveness in guiding the HEI. 79 Criteria Address those criteria which are relevant to your HEI (state why others are Not Applicable) 4-6 Indicators per criterion. Provided as guidance NOT requirements

SAM Part C Accreditation Outcomes and Ratings
SAM Part C Accreditation Outcomes and Ratings against Standards and Criteria Criteria Ratings Met (ratings 2, 3, 4) Partially Met (rating 1) Not Met (rating 0) Excellent Good Met Not Met Accredited with Merit Accredited with Merit in one or more standards Accredited On Probation Not Accredited Standards Ratings Accreditation Outcomes See SAM section 5

SAM Part D The Self-assessment
HEIs are expected to undertake a self-assessment well before the ISAA submission date Self-rate HEI performance against criteria; use this to inform standard ratings Commentary must match ratings Respond to Quality Audit Report Commendations, Affirmations and Recommendations (CARs) See SAM section 10

We’ll check throughout the session
Workshop Activity The ISA indicative timeline highlights key tasks for: OAAA Executive Officer Panel HEIs Identify which activities are HEI responsibilities We’ll check throughout the session

SAM Part E The Institutional Standards Assessment
OAAA SAM Part E The Institutional Standards Assessment Prior to ISAA submission ISAA submission Preliminary Meeting Planning Visit SA Visit Report v5 OAAA Board approves outcome Outcome posted on OAAA website See SAM section 3.4

ISAA Submission Ensure all (relevant) criteria have commentary and ratings Ensure all Supporting Materials (SMs) are attached Ensure declaration is signed Nominate a Contact Person (this may happen earlier) See SAM section 11.4

OAAA Receives ISAA Undertakes a ISAA completion check Invoice issued
If not complete the ISAA will be returned to the HEI If complete, ISAA and SMs forwarded to Standards Assessment Panel Invoice issued See SAM section 14.3

Preliminary Meeting About 6 weeks after the ISAA is submitted
Standards Assessment Panel meets to discuss ISAA (international Panel Members via teleconference) Provides opportunity to focus attention on ISAA Panel discusses preliminary ratings and commentary Ensures Panel Members are very clear about tasks and expectations Helps with planning for rest of the Standards Assessment See SAM section 18.2

About 7 weeks before the Standards Assessment Visit
OAAA Planning Visit About 7 weeks before the Standards Assessment Visit Provides an opportunity: To facilitate the overall Standards Assessment Visit To request additional information To identify individuals Panel would like to meet To discuss logistics See SAM section 18.5

Standards Assessment Visit
OAAA Standards Assessment Visit About 7 weeks after Planning Visit Provides an opportunity for the Panel to verify whether the HEI’s ISAA is: Adequate in addressing all relevant standards and criteria Comprehensive in scope Accurate and complete Defensible in ratings against criteria and standards See SAM section 20

Draft Reports Draft Report v1
OAAA Draft Report v1 Prepared based on Panel’s provisional ratings and preliminary comments Draft report v2 Written after the Standards Assessment Preliminary Meeting and incorporates Panel discussion Draft Report v3 Written on last day of Standards Assessment Visit. Panel reaches consensus on standard and criteria ratings and provide precise, evidence-based commentary to support ratings Draft Report v4 Cross-checked against existing evidence. Undergoes internal and external moderation

Draft Reports Draft Report v5
OAAA Draft Report v5 Feedback from internal and external moderation is incorporated Report v5 sent to HEI for feedback on factual inaccuracies; and to OAAA Board Draft Report v6 OAAA Board approves accreditation outcome and ratings against standards and criteria Final Report Report sent to HEI under embargo for 10 days See SAM section 21.6

2 weeks after Final Report sent to HEI
Public Reporting OAAA 2 weeks after Final Report sent to HEI Accreditation Outcome and ratings against the standards and criteria posted on OAAA website Accreditation Certificate awarded to successful HEI See SAM section 21.6

After Accreditation Outcome Release
OAAA After Accreditation Outcome Release Stakeholder Feedback Sought Prior to ISAA submission ISAA submission Preliminary Meeting Planning Visit SA Visit Report v5 OAAA Board approves outcome Outcome posted on OAAA website OAAA seeks feedback from stakeholders on all aspects of the Institutional Standards Assessment process See SAM section 23

Any questions?

OAAA Session 3: Preparing the Institutional Standards Assessment Application (ISAA) A training module in preparation for the Institutional Standards Assessment

Session Aims To be aware of how to complete the ISAA
OAAA Session Aims To be aware of how to complete the ISAA To be aware of what self-rating involves To understand how to determine the standard rating To understand how to determine the Accreditation Outcome To understand how fees are calculated

Institutional Standards Assessment Application (ISAA)
OAAA The template will be available electronically The same template will be used by the Standards Assessment Panel The HEI provides commentary explaining how it has met each criterion and why its self-rating against each criterion is appropriate The Panel provides commentary against each criterion

Institutional Standards Assessment Application (ISAA)
OAAA Institutional Standards Assessment Application (ISAA) The ISAA includes: Introductory section in which the HEI provides an overview of its history and context Declaration Form Standards 1-9 & all criteria (HEIs provide commentary against each relevant standard and criteria) A list of Supporting Materials Summary Data Tables (see SAM Appendix P)

Required Supporting Materials must be submitted with the ISAA
OAAA Institutional establishment or licensing documentation (such as Decree establishing the HEI or similar) Official licenses for all programs offered Current catalogue (prospectus or similar) Most recent HEI Annual Report A campus map See SAM section

Declaration OAAA Declaration to be signed by HEI’s most senior representative stating that: the information contained in the application is complete and accurate the HEI meets all national laws and requirements not specified in OAAA Institutional Standards the HEI abides by health and safety regulations and the OAAA holds no responsibility in this regard (this is still under discussion) See SAM section 11.2

OAAA HEI Overview Include a brief history, campus location/s, a general description of the HEI and its context, and any special characteristics it may have. The description should include the academic and general structure of the organisation and a complete list of programs being offered, including details of their licensing and accreditation status (and if the HEI is not the body awarding the qualifications, which entity is the awarding body). Relevant data should be included in the Summary Data Tables. The overview should provide a context for the HEI. It should be brief and succinct See SAM section

Standards 1-9 OAAA In the template, each standard and criterion is written out in full: Commentary includes HEI responses to Affirmations and Recommendations in the Quality Audit Report and how Commendations have been maintained or further strengthened Standard text Rating HEI Commentary Criterion text Rating HEI Commentary

OAAA Completing the ISAA For each criterion provide a commentary which describes how the HEI has met its requirements. The commentary should refer to evidence. After commentary has been provided against each criterion, provide a brief overarching summary statement on overall performance against the standard. Provide an explanation against the standard statement (there is no need to reiterate commentary already provided against each criterion) See SAM section

Standard Rating & Commentary
OAAA STANDARD 2: STUDENT LEARNING BY COURSEWORK PROGRAMS Academic standards are maintained through the implementation of the HEI’s planned, well-managed approach to the design, delivery and assessment of all student learning by coursework programs. The HEI has clearly defined generic graduate attributes and program-specific learning outcomes which align with the Oman Qualifications Framework, the HEI’s Mission and Vision and stakeholder expectations. Program delivery is supported by effective use of appropriate teaching and learning methods and assessment is governed by soundly-based regulations, policies and procedures. The HEI’s processes support a culture of academic integrity. The HEI has a systematic approach to the monitoring of graduate destinations and employment that is used to inform the review of student learning by coursework programs. Rating Commentary [Insert a very brief summary of how the HEI meets the standard]

Criterion Rating & Commentary
OAAA Criterion 2.1: Graduate Attributes and Student Learning Objectives The HEI has defined generic graduate attributes which reflect stakeholder expectations and which graduates from all programs are required to attain. These are effectively communicated to all stakeholders and incorporated into program development, design and delivery. The HEI’s systems and processes for designing, developing and approving programs ensures that the academic standards of awards are set at an appropriate level and that program-specific learning outcomes are appropriate for the award and align with the Oman Qualifications Framework. The process for assessing student attainment of generic graduate attributes and program-specific learning outcomes is clearly defined and implemented. Rating Commentary [Describe how the requirements of the criterion have been met. Provide links to Supporting Materials that support claims made in the commentary]

Evidence used to support commentary statements
OAAA Evidence used to support commentary statements The HEI should provide evidence to support the application List all evidence referred to in the commentary in the table at the end of all nine standards. Consecutively number each item of supporting materials Supporting Material # Description of SM SM001 Institutional License SM002 Program License(s) SM0023 Current catalogue See SAM section

Summary Data Templates
OAAA Evidence to support claims made in the ISAA can be provided using the Summary Data Templates Number of students by program, year of study and gender Number of students by program, year of study and mode Attrition, progress and completion rates Number of staff by department, year, employment status and gender Number of staff by academic department, year, employment status and nationality Number of staff by academic department and highest qualification held Number of staff by administrative department, year, employment status and nationality See SAM Appendix P

Workshop Activity Refer to the worksheet which provides examples of panel commentary which support ratings ranging from 4 to 0 (see SAM p.70) Identify which panel commentary relates to which rating

Criterion Rating Definition
OAAA Criterion Rating Definition Rating Description 4 (Met) Definition: Provision or practice consistently meets the requirements of the criteria and exceeds requirements most of the time. Characteristics of provision or practice: Results are of high quality and significantly exceed the requirements of the criterion Results against the criterion are achieved through a deliberate and innovative approach Results against the criterion are sustainable and have significantly improved over time Results against the criterion are indicative of best practice See ISAM section 5.1

OAAA Example of Rating 4 A revised innovative academic advising system was successfully implemented and has been shown to have had a significant impact on retention and identifying ‘at risk’ students; the system has received significant positive feedback, been periodically reviewed for effectiveness, indicating continuous quality improvement. Rating against criterion 6.5, page 52

OAAA Criterion Rating Definition Rating Description 3 (Met) Definition: Provision or practice consistently meets the requirements of the criterion and exceeds the requirements some of the time. Characteristics of provision or practice: Results consistently meet the requirements of the criterion and exceed the requirements some of the time Results against the criteria are achieved through a deliberate approach which is consistently implemented Results against the criterion are sustainable and have improved over time See SAM section 5.1

OAAA Example of rating 3 As part of its continuous improvement system, the HEI has introduced a revised academic advising system in response to feedback; there is a comprehensive handbook and training for staff and students which has been consistently implemented; the system has had a positive impact; and the HEI has clear plans for how the system will be evaluated. Rating against criterion 6.5, page 52

OAAA Rating Description 2 (Met) Definition: Provision or practice meets the requirements of the criterion most of the time. Characteristics of provision or practice: Results meet the requirements of the criterion. Results against the criterion are achieved through a deliberate approach which is implemented most of the time. Results against the criterion are sustainable. See SAM section 5.1

OAAA Example of rating 2 The HEI has implemented an effective formal academic advisory system which overall, supports students in meeting their educational goals; the system has been evaluated and improvement plans have been implemented in most departments Rating against criterion 6.5, page 52

OAAA Rating Description 1 (Partially Met) Definition: Provision or practice does not fully meet the requirements of the criterion but the HEI has demonstrated an appropriate commitment to meeting the requirements of the criterion in the future; not fully meeting the requirements of the criterion does not have a significant impact on the overall achievement of the standard. Note: the combined effect of two Partially Met criteria should not have significant impact on the overall achievement of the standard (under discussion) See SAM section 5.1

Rating against criterion 6.5
OAAA Example of rating 1 The HEI has implemented a new institution-wide system for academic advising which replaced ad-hoc, faculty based approaches to academic advising. While the new system is designed to improve support to students, and students expressed a high degree of satisfaction with the new system to Panel members, it has not yet been evaluated for effectiveness. The HEI’s approach to evaluation is clearly specified and will be used for the evaluation of the HEI’s academic advising system at the end of the academic year. Rating against criterion 6.5 Rating against criterion 6.5, page 52

Criterion Rating Definitions
OAAA Criterion Rating Definitions Rating Description (Not Met) Definition: Provision or practice does not meet the requirements of the criterion. Not Applicable This criterion is not relevant to the HEI’s context. See SAM section 5.1

Example of Not Applicable
OAAA Example of ‘Not Met’ While the HEI has a policy for academic advising, the system has not been implemented throughout the HEI and has not been monitored or evaluated for effectiveness Rating against criterion 6.5 Example of Not Applicable Criteria 3.1 – 3.6 do not apply to XX HEI as it is classified as a College and does not deliver Student Learning by Research Programs (see SM011 HEI license approval).

Standard Rating Criteria rating informing standard rating OAAA
Description Standard rating Most of the criteria are rated 3, at least one criterion is rated 4, and no criteria are rated 1 or 0 Excellent Most of the criteria are rated 3, no more than two criteria are rated 1and no criteria are rated 0 Good Most of the criteria are rated 2, no more than two criteria are rated 1 and no criteria are rated 0 Satisfactory More than two criteria are rated 1 or one criterion is rated 0 Not met The standard is not applicable to the HEI’s context NA See SAM page 71 ‘Most’ in this context means more than 50%

Determining the overall rating for the standard (1)
OAAA Determining the overall rating for the standard (1) Criterion Rating 6.1 2 6.2 3 6.3 6.4 6.5 6.6 6.7 Standard 6 ????? Satisfactory

OAAA Determining the overall rating for the standard (2) Criterion Rating 5.1 3 5.2 5.3 5.4 5.5 5.6 4 Standard 5 ????? Not met

OAAA Determining the overall rating for the standard (3) Criterion Rating 8.1 4 8.2 8.3 8.4 8.5 8.6 8.7 3 8.8 1 8.9 8.10 Standard 8 ????? Good

OAAA Determining the overall rating for the standard (4) Criterion Rating 8.1 2 8.2 3 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 4 Standard 8 ????? Excellent

OAAA Determining the overall rating for the standard (5) Criterion Rating 4.1 2 4.2 4 4.3 4.4 4.5 4.6 3 4.7 4.8 4.9 NA Standard 4 ????? Satisfactory

Determination of Accreditation Outcome
OAAA Determination of Accreditation Outcome Description Outcome Most of the standards are rated Good and at least one standard is rated Excellent. No standards are rated Not Met Accredited with Merit with Merit Most of the standards are rated as a minimum Satisfactory and one or more standards are rated as Excellent Accredited with Merit in one or more standards Most of the standards are rated either Satisfactory or Good Accredited One or more standard is/are rated as Not Met On Probation One or more standard is/are Not Met (following Standards Reassessment) Not Accredited See SAM page 71 ‘Most’ in this context means more than 50%

Determination of Accreditation Outcome, Example (1)
OAAA Determination of Accreditation Outcome, Example (1) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 6 7 8 9 Accreditation Outcome Accredited with Merit in Student Learning by Coursework ?????

OAAA Determination of Accreditation Outcome, Example (2) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 6 7 8 9 Accreditation Outcome Accredited with Merit ?????

OAAA Determination of Accreditation Outcome, Example (3) Standard Rating 1 Satisfactory 2 Excellent 3 NA 4 Good 5 6 7 8 Not met 9 Accreditation Outcome On Probation ?????

OAAA Determination of Accreditation Outcome, Example (4) Standard Rating 1 Good 2 3 NA 4 5 6 7 8 9 Accreditation Outcome Accredited ?????

Submitting the ISAA ISAA written in the language of instruction
Professionally typeset, soft cover format 8 hard copies and e-version on data sticks Supporting Materials to be submitted electronically (on data stick or similar) Information referred to on a website must: be accessible by OAAA not be hindered by internal firewalls be printable and can be saved and downloaded See SAM section 11.4

Fees Royal Decree 54/2010, Article 18 permits OAAA to charge fees
Fees based on HEI institutional classification and number of full-time equivalent student enrolments at the end of the last academic year Number of students must be verified by an appropriate independent body Number of FTE = Number of ( FT PT)

Fees (cont.) University: 25,000 OMR + 2 OMR x total number FTE enrolments University College: 20,000 OMR + 2 OMR x total number FTE enrolments College: 15,000 OMR + 2 OMR x total number FTE enrolments Accreditation Outcome will not be released until full payment has been received See SAM section 11.5

OAAA Any questions?

OAAA Session 4: Activity 1 Types of evidence to review when preparing the ISAA A training module in preparation for the Institutional Standards Assessment

Workshop Activity Aims
To learn how to use the indicators as tools for self-assessment of the criteria To understand the types of questions to ask when undertaking a self-assessment of the criteria To understand the types of documents that should be reviewed as part of the self-assessment of the criteria

Workshop Activity Each group has been given a set of suggestions to guide the development of a self-assessment against the criteria. Answer the following: For which criterion is the guidance relevant? What other questions could be asked? What other documents could be reviewed? How can your HEI demonstrate that its processes and practices related to this criterion are effective?

Workshop Activity Reporting Back
Which criteria relates to which set of suggestions for self-assessment? How easy did you find the exercise? What was the most challenging part of the exercise? How will this help you prepare for the ISA?

OAAA Session 5: Methods of Analysis Using ADRI for Institutional Standards Self-Assessment A training module for preparation for the Institutional Standards Assessment

Aims and objectives of this session
Understand the different methods of analysis used in Standards Assessment Understand the importance of evidence in supporting an HEI’s claims Understand use of ADRI as a tool in self-assessment Appreciate the reasons why a robust method of analysis is important see SAM Part F, particularly section 25

Some Methods of Analysis Issues
Using Statistics and Types of Evidence Quantitative evidence Qualitative evidence Gaining a Comprehensive Picture Gaining Confidence in the Evidence Reaching Conclusions

Using statistics Standards Assessment outcomes are based on consideration of evidence. There are different types of evidence, each with its own methods of presentation and collection. A notable distinction is between Quantitative and Qualitative evidence. All types have something useful to offer in reaching a comprehensive conclusion. Therefore, Standards Assessment is a mixed method exercise.

Gaining a Comprehensive Picture
Saturation – a method used to explore an issue until no new information about it comes to light. Triangulation – a method for strengthening the analysis using combination of: Multiple original source of data (e.g. students, staff, other stakeholders) Multiple methods of data collection (e.g. surveys, interviews, literature) Different types of data Process Mapping – a method for depicting the steps in a process and their relationships.

Gaining Confidence in the Evidence
In reaching a rating decision, Panel Members must have confidence that the evidence is not only comprehensive, but also valid, reliable and honest! Examples of methods for gaining confidence: Non-attribution Rule Discourage ‘rehearsing’ for interviewees Random Interviews Non-attributable surveys (e.g. student evaluations of teaching) Independence/externality (of survey analysis, program reviews etc.)

A D R I Using ADRI for ISA

What is ADRI? ADRI (or a similar tool) helps to facilitate structured and systematic self-assessment. ADRI is used to show that achievements have not come about by chance. ADRI can show that activities are supported by systems and are sustainable. ADRI focuses on opportunities for improvement rather than weaknesses – the emphasis is on seeking opportunities to enhance the HEI’s provision. ADRI can be used at all levels of the HEI – institutional, department, program, class. ADRI is used by HEIs and by the Standards Assessment Panel.

What is ADRI? What strategies, structures and processes have been developed? Has the approach been benchmarked against best practice? What performance indicators have been developed to track progress? How are activities monitored and reviewed? What has been learned? How will improvements be made for next time? APPROACH RESULTS DEPLOYMENT IMPROVEMENT What has been achieved? How is this measured against goals, targets and performance indicators? Do these results show that the standard or criterion has been met? Have strategies, plans etc been put into practice? What is the extent of their implementation?

Using ADRI in Standards Assessments
The HEI implements systems and processes for academic advising which provide effective advice and support to students and to assist them in achieving their educational goals. Criterion 6.5 How do you know if your program meets this criterion? What results/evidence support your claim? How do you know if what you do is “good enough”?

ADRI for Internal and External Processes
ADRI Cycle of Quality Assurance Reviewing Academic Advising Evidence examples: Evidence of review of academic advising system e.g. external/internal reports Input obtained from staff and students Resulting action plans Reports on progress against action plans Evidence examples: Academic advising policy Staff handbook Student handbook Evidence of benchmarking APPROACH RESULTS DEPLOYMENT IMPROVEMENT Criterion 6.5: Academic advising Evidence examples: Academic advising data Evidence of student satisfaction Evidence of impact e.g. at risk student register, student progression rates Evidence examples: Ongoing staff and student feedback Staff and/or student training materials Records of academic advising It can be applied to any topic – in this case, the analysis of an HEI’s academic advising ADRI is a model of quality assurance used by agencies in many countries around the world Internal Review of the System External Review of the System

Conclusion Quality assurance requires a suitable method for analysing the effectiveness of the systems in place to show they meet external standards. The ADRI method is a useful tool. It can be used by small work teams, internal review committees and external review panels. Because ADRI combines an assessment of the quality system with a constructive analysis, it is not something extra to do, but rather a better way of doing what we should be doing anyway.

Session 6: Activity 2 Rating Criteria and Standards
OAAA Session 6: Activity 2 Rating Criteria and Standards A training module for preparation for the Institutional Standards Assessment

Aims of the activity To practice applying ratings to criteria
OAAA Aims of the activity To practice applying ratings to criteria To gain an understanding of the types of processes and practices and characteristics are associated with the ratings To develop an understanding of the types of evidence required to justify ratings

Workshop Activity Select any standard
Refer to the criterion rating definitions and characteristics provided in Table 3 SAM section 5 For each criterion related to the selected standard provide bullet points on the types of processes and practices that would normally be in place to support the criteria ratings Rating 4 3 2 1

Workshop Activity Reporting Back
Without discussing specific ‘real life examples’ what was the most challenging aspect to deciding on a rating? How difficult was it to determine the type of evidence required to support the rating? How did you achieve consensus within the group?

Any questions?

Session 7: Institutional Standards Assessment Recap and Tips
OAAA Session 7: Institutional Standards Assessment Recap and Tips A training module for preparation for the Institutional Standards Assessment

Session Aims Be reminded of the concept of self-assessment
Identify features of a ‘good’ institutional standards assessment application (ISAA) Reflect on features of a ‘good’ institutional standards assessment (ISA) self-assessment process Identify practices that will support a smooth ISA Visit

What is Self-Assessment?
In the context of quality in higher education, the term “self-assessment” is used interchangeably with “self-evaluation” and “self-study”. Core definition: “The process of critically reviewing the quality of one’s own performance and provision” Takes place in the context of external review; but is based on the principle that primary responsibility for quality and quality assurance rests with the HEI Involves: a process of self-reflection in the form of critical evaluation by the HEI being reviewed (based on evidence and analysis; identifies strengths and areas for improvement; evaluates performance against external standards ) and preparation of a document reflecting the self-assessment: the Institutional Standards Assessment Application (ISAA) based on Harvey, L., Analytic Quality Glossary

Think like a Panel member!
Workshop Activity Think like a Panel member! Scenario You are a member of an OAAA Standards Assessment Panel, for the Stage 2 Institutional Accreditation: Standards Assessment of an HEI. As a Panel Member, from your perspective, what are the features of a ‘good’ ISAA? To help you carry out your role, what do you need from the HEI’s ISAA? Please discuss in your group and list 10 features of a ‘good’ ISAA.

The Institutional Standards Assessment Application (ISAA)
The Product

General Features of a ‘Good’ ISAA
Complete, and the layout adheres to the template Demonstrates that the HEI meets the institutional standards Includes the HEI’s response to Quality Audit CARs Self-evaluative; analytical; based on evidence; demonstrates use of ADRI Clear; readable; understandable; ‘makes sense’ to an external reader A ‘single voice’: the ISAA is edited to a high standard A ‘clear voice’ : internal consistency in the content All key evidence (supporting materials - SMs) is submitted with the ISAA and clearly labeled (numbered, titled, dated) No SM submitted more than once; clear cross-references to other sections / supporting materials

Features of a ‘Good’ ISAA: Commentary and Use of Evidence
Commentary relates explicitly to each criterion Open and honest / not ‘promotional’: avoids marketing language Concise: as long as needed and no longer Appropriate balance between description and analysis; description should not greatly outweigh analysis Evidence used to: compare the HEI’s performance against the criteria and standards identify strengths and areas for improvement, in order to apply ratings justify evaluative statements made in the commentaries Draws on evidence that already exists in the HEI Selective in use of evidence Evidence used is: valid; reliable, accurate and relevant

Any additional features
of a ‘good’ ISAA?

The Institutional Standards Assessment (ISA)
Self-Assessment The Process

Workshop Activity Reflection on ISA Self-Assessment
Reflect on your ISA self-assessment process to date. List 5 features of a ‘good’ institutional self-assessment process

Features of a ‘Good’ ISA Self-Assessment Process (1)
Ownership and inclusivity: based on the principle that the main responsibility for quality and quality assurance rests with the HEI Based on an understanding of the cyclical, two stage institutional accreditation process (Quality Audit – Standards Assessment) and the concept of continuous quality improvement Builds on the Stage 1 Quality Audit Process (with special attention to CARs) Uses a planned approach (timeline; tasks; roles and responsibilities; procedures etc.) which is informed by the experience of previous self-review activities Appropriate balance of attention to evaluation (self-assessment) vs improvement Effective communication: all relevant stakeholders kept “in the loop” Support for all involved: training; acknowledgement of time and effort

Features of a ‘Good’ ISA Self-Assessment Process (2)
Effective Project Management E.g. Steering Committee and working groups established Time frame determined and resource requirements (time, budget etc.) identified and addressed Staff training conducted (e.g. in ADRI) Task broken down: clear responsibilities for evidence gathering and drafting the ISAA established Clear procedures for: approving ISAA drafts; circulating drafts; getting feedback on drafts Responsibilities for bringing all the sections together into a coherent overall ISAA and for final editing established. See SAM section 10.2

Any additional features of a ‘good’ ISA self-assessment process?

Preparation for the Institutional Standards Assessment (ISA) Visit

Panel Room Layout How can you prepare for the ISA visit?
See SAM section 20 How can you prepare for the ISA visit?

Preparation for the ISA Visit (1)
Use lessons learnt from the Stage 1 Quality Audit process (ADRI) Contact Person to establish and maintain a close and effective working relationship with the OAAA Executive Officer (EO) There are no ‘stupid questions’; seek timely clarification from the EO on any points that arise Be well prepared for the Planning Visit: allocate time in the days beforehand to: Prepare a written response to the Points for Clarification Consider the request for additional Supporting Materials and being to collect these ready for submission Consider the draft ISA Visit Program and be prepared to contribute constructively to finalizing this during the meeting Decide on the key venues and logistics for the visit in preparation for consideration by the OAAA See SAM section 20 + Appendix F

OAAA identifies the profile of interviewees, not named individuals. In populating the ISA Visit program, adhere to expected requirements to avoid last minute complications: E.g. Line managers are interviewed separately from their staff (also applies to “in situ” interviews) Interviewees are interviewed only once (unless special circumstances apply) Key roles undertaken by named individuals are indicated (e.g. Chair of Disciplinary Committee) Representatives of external stakeholder groups such as employers and student placement hosts are selected appropriately: relevant role vs seniority Interviewees who are Arabic rather than English Language users are identified The EO is informed of any changes in a timely manner See SAM section 29

Develop an approach to contacting and briefing all interviewees; it is in the interest of the HEI that all interviewees identified by the OAAA Standards Assessment Panel are available for interview in accordance with the Standards Assessment Visit Program Allow sufficient time for ensuring the availability of: Governance representatives: (e.g. senior government officials; chairs of members of governing bodies) External Stakeholders (e.g. employers, student-placement hosts; advisory group members, guest lecturers etc.) Affiliate representatives (may include external examiners) Alumni Students ( identify back ups) Ensure interviewees are well informed: make use of the briefing sheets provided (translate as required) SAM Appendices I,J, and K

Ensure everyone understands what Public Submissions and Random Interviews are; the Public Submissions notification in the local press is a standard part of the process for all HEIs See SAM Section 20 + Appendix G Ensure all interviewees understand the general format and protocols of the interviews See SAM section 29 E.g. Most will be interviewed as part of a group and will enter the interview room as a group Interviewees should bring their name cards with them to the interview No note taking or recording of the interview is permitted Interviewees may refer to documents but Panel Members are not permitted to receive documents from them Students are not being evaluated as individuals Post interview, interviewees should not report what they (or other people) said during the interview Trial Standard Assessment Visits: If conducted, avoid the pitfalls. Coaching of interviewees will be detected by the Panel and will hinder the process See SAM section 12

Standards Assessment Visit
During the Standards Assessment Visit

During the ISA Visit Support the smooth running of the ISA visit by:
Disciplined time keeping Effective management of interviewees by HEI Contact Person available at all times IT support readily available to Panel (through the Contact Person) Panel’s need to work confidentially respected Catering for the Panel sufficient but not excessive Additional points? Panel Departure: Differs from Stage 1 Quality Audit; no feedback on preliminary findings or conclusions (management of expectations) Photo opportunity

The OAAA is keen to “catch HEIs doing things well”!
The OAAA Institutional Accreditation process encourages and recognises excellence through its rating scale and accreditation outcomes with Merit: The OAAA is keen to “catch HEIs doing things well”! Any Questions?

OAAA Session 8: Q&A A training module for preparation for the Institutional Standards Assessment

Aims of the Workshop To provide participating HEI staff with an overview of the OAAA’s role in Institutional Standards Assessment To provide a guide for the process of the Institutional Standards Assessment and its link to sections of the Standards Assessment Manual To provide staff with a clear idea of their role in the Institutional Standards Assessment process To provide guidance on how to carry out a self-assessment and prepare an Institutional Standards Assessment Application including the methods of analysis and ADRI To provide participating HEI staff with a chance to ask questions and clarify understanding

Any questions?

Oman Academic Accreditation Authority (OAAA)

Similar presentations

Presentation on theme: "Oman Academic Accreditation Authority (OAAA)"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Oman Academic Accreditation Authority (OAAA)

Similar presentations

Presentation on theme: "Oman Academic Accreditation Authority (OAAA)"— Presentation transcript:

Similar presentations

About project

Feedback