Mia Alexander-Snow, PhD Director, Office for Planning and Institutional Effectiveness Program Review Orientation 1

Topics to Cover  Institutional Effectiveness & Assessment  program review overview  UK’s program review schedule & current process  administrative units participating in cycle  program review components: self study, external review, and implementation plan  role of Planning, Assessment, and Institutional Research  sources that inform review  program review calendar  contacts  questions Program Review Orientation 2

What is Institutional Effectiveness? “ “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission” (SACS Core Requirement 2.5) Program Review Orientation

What is Assessment? Assessment is the process by which … “the institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results” (SACS Comprehensive Standard 3.3.1) Program Review Orientation

What is Assessment at UK? “Assessment is used to monitor the University’s effectiveness in achieving its mission, vision, and goals. The (UK AR 1:4) “Assessment is used to monitor the University’s effectiveness in achieving its mission, vision, and goals. The University and its units shall demonstrate an explicit use of assessment results to facilitate resource allocation and budgeting decisions in support of their strategic plans and to ensure quality enhancement” (UK AR 1:4) Program Review Orientation

What assessment activities monitor UK’s Institutional Effectiveness efforts? Strategic Planning Identifies and prioritizes the actions the University and its units can take to help it best accomplish the University’s goals and fulfill its mission (AR 1:4) Program Review Orientation

What assessment activities monitor UK’s Institutional Effectiveness efforts? Annual Progress Reporting  Dynamic process for reviewing, updating and revising strategic planning efforts over a 3-5 year period  Answers the following questions in a systematic and thorough way :  “How are we doing? ---Actual Results  “What things are working? ---Reflection and Analysis  “What needs to happen next?” ---Improvement Action Program Review Orientation

What assessment activities monitor UK’s Institutional Effectiveness efforts? 6 yr Periodic Program Review : the primary vehicle for assessment of educational and administrative units and for documentation of institutional effectiveness (AR 1:4). 424 Units participate in Program Review 77 Administrative and Educational Support units 18 Colleges and Schools 307 Academic Departments and degree programs 22 Research Centers Program Review Orientation

The Role of the Office of Planning in Program Review Planning Learning Assessment Institutional Research Program Review Orientation

Program Review Overview  Background: Program Reviews in Kentucky  Governing Regulation IX-I  Administrative Regulations 1:4  required every 5-7 years for all academic and administrative units (exceptions may be negotiated to align with specialized accreditation cycle) Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 10

UK’s Program Review Schedule Schedule & Current Year Progress Updates  Purpose:  communicate to organizational entities the full 6-yr review cycle and when units can expect to undergo program review  Goals:  provide the transparent and accurate maintenance of the review schedule for the university’s educational (academic) and administrative units; and  monitoring unit progress  Administrative Units participating in Cycle  President: 2 units  Provost: 3 units  EVPFA: 3 units Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 11

Program Review Overview, continued  What is the purpose and goal of program review?  to improve the quality and effectiveness of teaching and learning, research, public service, and operations; and  to develop recommendations leading to organizational improvement based on internal evaluation with appropriate input from external experts Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 12

Program Review Overview, continued  Who is responsible for satisfying program review?  President, provost and executive vice presidents  deans, vice presidents, associate vice presidents, associate and vice provosts, department chairpersons, directors, and other administrators  Vice President for Institutional Research, Planning, and Effectiveness  unit/area faculty, staff, and/or appropriate personnel Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 13

What are the components of UK’s program review process? 14 Unit/Program Self – Study (internal Review) SWOT Analysis External Review within UK outside UK Recommendations Implementation Plan Quality Enhancement Agenda Annual Progress Report Documentation of enhancement results & Informs Strategic Planning & Budget Program Review Orientation

Program Review Components I.*Self-Study Report (include as appropriate):  program documents  resources  input from affected constituents  adherence to policies and procedures  evaluation of quality and productivity  analysis of strengths and recommendations for quality enhancement  Elements evaluated:  centrality  competitive /comparative advantage  cost effectiveness  demand  quality  distinctiveness *Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 15

What are the components of UK’s program review process? 16 Unit/Program Self – Study (internal Review) SWOT Analysis External Review within UK outside UK Recommendations Implementation Plan Quality Enhancement Agenda Annual Progress Report Documentation of enhancement results & Informs Strategic Planning & Budget Program Review Orientation

Program Review Components, continued II.i. External Review (completed by External Review Committee)  examine the self-study report;  use appropriate data collection techniques to assure objectivity;  assess validity of conclusions reached in self-study;  identify additional strengths and recommendations for quality enhancement; and  prepare a final report—report made available to faculty, staff employees, and students Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/06 Slide Reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 17

Program Review Components, continued II.ii. Administrative External Review Committee  appointed and charged by administrator to whom the unit head reports  consists of 4-5 members—stakeholders and constituencies affected by the unit program and services  4-5 faculty and staff employees, or students from outside the unit  1-2 ex-officio members, appointed to support external review committee  following external review, meets with unit and its leadership to discuss preliminary findings and writes report Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 18

Program Review Components, continued II.iii. External Review Committee Report considers the unit’s…  Program documents: strategic plan (i.e. mission statement, goals, and objectives, & criteria for measuring progress); primary contributions to UK’s mission and vision, organizational chart or structure, & annual progress reports  Resources: adequacy of budget, facilities, equipment, personnel, including faculty and staff numbers demographics, and support from other university units essential to effective operations (e.g., research, engagement, development, alumni affairs, human resources, facilities management, financial units, & information technology)  Input from Affected Constituents: evaluation data from faculty, staff, and students affected by the delivery of program and services to the unit. Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 19

Program Review Components, continued II.iii. External Review Committee Report considers the unit’s…  Adherence to Policies and Procedures: evidence of adherence to university policies and procedures (e.g., registration, student activity fees, hiring practices, etc.)  Evaluation of Quality and Productivity: evidence of quality of the collegial culture and climate  Faculty and staff employees, communications and interactions;  Orientation, advising, and other student services programs;  Learning outcomes;  Customer or client satisfaction;  Business and operating procedures; Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 20

Accreditation and the External Review Process  Fully Accredited Programs/Units: Accreditation Self- Study, Accreditation Review and Accreditation Report  May Substitute for:  UK’s self-study  UK’s External Review, and  UK’s External Review Committee Report  Will NOT Substitute for:  Program Review Implementation Plan  Partially Accredited Programs/Units: Accreditation Review and Accreditation Report  May ONLY Substitute for:  External Reviewer (s) for the UK External Review Program Review Orientation

What are the components of UK’s program review process? 22 Unit/Program Self – Study (internal Review) SWOT Analysis External Review within UK outside UK Recommendations Implementation Plan Quality Enhancement Agenda Annual Progress Report Documentation of enhancement results & Informs Strategic Planning & Budget Program Review Orientation

Program Review Components, continued III. Implementation Plan  Sets agenda for change and quality enhancement over the next 5-7 year cycle;  unit faculty, staff, and/or students under the leadership of unit head define unit agenda based on self-study and external review report/recommendations;  must be approved by unit head’s supervisor;  used by unit to document future plans and resource needs for consideration in budgetary decision-making; and  supports annual progress reporting Slide reference citation: University of Kentucky Administrative Regulations (AR)1:4:11/15/ Program Review Orientation 23

Sources that inform Review  unit website  peer benchmarking and “best practices”  last unit self-study reports ( , or )  annual progress reports (past 3 years)  most recent accreditation or certification results and recommendations  formative and summative assessments  qualitative– focus groups, interviews, etc.  quantitative—satisfaction surveys, employer surveys, etc.  Institutional data (provided by Office of Institutional Research) for examples see: Program Review Orientation 24

The Role of Institutional Research in Program Review Planning Student Learning Institutional Research Program Review Orientation

What IR sources inform Program Review? Student-Related Data  Freshman Profile (ACT Scores & High School GPAs)  Headcount Enrollments (10 years)  Degrees Conferred (10 years)  Retention and Graduation Rates  Surveys  Senior  Alumni  NSSE  Campus Climate Program Review Orientation

Institutional Data Sources Institutional data compiled by the UK Office of Institutional Research can be found at: Program Review Orientation

The Role of Assessment in Program Review Planning Student Learning Assessment Institutional Research Program Review Orientation

Student Learning Outcomes Assessment Process Student learning outcomes Assessment methods Data collection Program improvement Timeline/ schedule Unit staff achieve consensus on learning outcomes to be assessed, and map outcomes to curriculum Unit staff choose assessment methods appropriate to the outcomes, purpose of assessment, and student population Unit staff develop timeline for gathering assessment data, both formative and summative Unit staff work together to gather formative and summative assessment data and compile for review Unit staff collaborate to interpret assessment data and develop strategies to improve student learning outcomes Continuous improvement cycle Program Review Orientation

Program Review Calendar *Calendar  Purpose:  communicates steps and timeline for completing program review; and  ensures timely completion *Refer to administrative calendar for Program Review Orientation 30

Additional Program Review Questions General Program Review Process  Mia Alexander-Snow, PhD Director, Planning and Institutional Effectiveness Office phone: Year Schedule  Connie Vaughn Program Planning Coordinator Office phone: Program Review Orientation 31

Presentation Contact Information Mia Alexander-Snow Office for Planning & Institutional Effectiveness Website: Tara Rose Office of Assessment Website: Roger Sugarman Office of Institutional Research Website: Program Review Orientation