Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual Meeting Department of Institutional Research and Effectiveness St. Petersburg College P.O. Box St. Petersburg, FL (727) FAX (727) Revised version for the 2011 SACS Working Group

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting2 Issues with IE  SACS’s Peer Review Research Project revealed that in the area of Institutional Effectiveness (IE)  62% of institutions in off-site reviews were deemed non-compliant and  27% of institutions in on-site visits were deemed non-compliant  IE was the second most identified area behind faculty qualifications

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting3 SPC Experience with IE  As part of the recent SACS reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes  The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting4 SACS Requirements for IE  Core Requirement 2.5: Integrated Planning & Evaluation Processes Institutional Effectiveness  “…ongoing, integrated, and institution-wide research-based planning and evaluation processes…”  Comprehensive Standard 3.3.1: Institutional Effectiveness  “…identifies expected outcomes…,assesses whether the extent to which it achieves these outcomes, and provides evidence of improvement…”  Comprehensive Standard 3.5.1: General Education Core College Level Competency  “… identifies college-level general education competencies and the extent to which graduates have attained them. within the general education core and provides evidence that graduates have attained those competencies.”

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting5 SACS Requirements for IE  Additional IE related requirements include:  Comprehensive Standard 3.1.1: Institutional Mission  Federal Requirement 4.1: Evaluating Success Student Achievement  Federal Requirement 4.2: Program Curriculum

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting6 IE Evidence  SACS Suggested Documentation for CR 2.5  Evidence of linkage of IE to institutional mission  Institutional plans and budgets that demonstrate linkage of assessment findings to planning at all levels  Minutes of appropriate (IE related) unit, committee, task force meetings…  Documentation that relates to IE, such as budget preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports  Samples of specific actions taken to improve the IE process and/or results from that process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting7 Performance Improvement  From Compliance to Performance Improvement

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting8 SPC’s Definition of IE  “At SPC, Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.” “Closing the Loop”

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting9 IE Phases  IE Processes are yearly processes that consist of three distinct phases:  Evaluation/assessment phase  Strategic/budget planning phase  Implementation/operation phase

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting10 Evaluation/Assessment Phase  Reflective look back at the prior year’s assessment and evaluation results.  educational outcomes  institutional surveys  departmental-specific assessments  Individual assessments results are reviewed by the respective oversight group which in turn aggregates the information and presents recommendations to the President’s Cabinet.  These recommendations are evaluated and are routinely summarized in the form of the Institutional Objectives (IOs).

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting11 Strategic/Budget Planning Phase  Transition from assessment/evaluation results to planning for the upcoming operational year  Includes assessment-driven unit planning  Unit managers state their intent for the upcoming year in measurable objectives  Unit managers ensure alignment with the institution’s mission and goals, Strategic Directions and Institutional Objectives (SD-IOs)

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting12 Implementation/Operation Phase  Execution of the developed unit plans  Unit managers are required to provide periodic reports of their progress to ensure the fidelity and stability of these unit plans  Periodic reports include a “mid-year status report” and an “end-of-year final results report.” The end-of-year report includes assessment results in addition to the unit’s use of these results in improving their unit.

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting13 Individual IE Processes  Individual IE processes are designed to result in continuous improvement and comprise the following: a.Mission statement process b.Evaluation and assessment processes for academic programs, student services, and administrative services c.Strategic Directions and Institutional Objectives (SD- IOs) process d.Unit / budget planning process e.Collegewide operating budget development process f.Budget approval, implementation, and annual operating results

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting14 Mission Statement  IE process begins and ends with the College Mission statement  College’s Mission is formally reviewed every five years by a committee representing faculty, administrators, students, community leaders, and members of the Board of Trustees (BOT)  Mission is also reviewed annually by President’s Cabinet before the new strategic planning cycle begins a. Mission Statement Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting15 Assessing the Mission 5 SPC Mission – Assessments  Closing the Loop a. Mission Statement Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting16 Evaluation and Assessment  Evaluation and assessment processes at SPC are centered on mission-driven outcomes in the following program and service areas:  (i) 26 Organizational units comprising 40 Academic programs (lower division);  (ii) 8 Colleges and Schools comprising 20 Majors (upper division), and  (iii) key administrative and educational support services.  Analysis of outcome results is on-going and captured in various assessment reports.  SPC evaluates all student services to ensure students are provided the best support possible to ensure student success. b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting17 Assessment Areas  Academic Outcomes  General Education  BAS/BS  AAS/AS  Key Administrative and Educational Support Services  Viability  Importance  Satisfaction b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting18 Assessment Focus  Continually evaluate assessment instruments and current processes  Integrate assessment with educational process  Focus on fewer, yet meaningful improvements (action items)  Use multiple assessment methods  Involve faculty in process  Integrate timing of assessments (3-year cycle) Assessments ReviewsSurveys GenGen EdEd b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting19 Direct and Indirect Measures  Direct Measures  Internally Developed General Education  Externally Developed General Education (e.g., MAPP, iSkills)  AAS/AS Academic Program Assessment Report (APAR)  BAS/BS Program Assessment  Comprehensive Academic Program Review (CAPR)  Indirect measures  Entering Student Survey  Enrolled Student Survey  Graduating Student Survey  Recent Alumni Survey  Employer Survey  Community College Survey of Student Engagement (CCSSE) b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting20 Three-year Assessment Cycle Academic Program Viability Reviews (APVRs) conducted yearly APARs CAPRs b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) Key Assessments Academic Program Viability Review (APVR) Academic Program Assessment Review (3) (APAR) Comprehensive Academic Program Review (3) (CAPR) Oversight Group Report (Multiple Programs) 1)Program Description (1) 2)Number of Graduates – by year 3)Actual Enrollments – by last six semesters 4)SSH Productivity – by last six semesters 5)Program Profitability – RPI Index by year 1)Introduction w/ use of past results (1,2) 2)Major Learning Outcomes – program specific (1, 2, 3) 3)Assessment Methodology (3) 4)Criteria for success (3) 5)Summary of assessment findings 6)Discussion & analysis 7)Action plan & time table 8)Budgetary & planning implications 9)Follow-up report 1)Key APAR findings 2)Key APVR findings 3)Environmental scan of stakeholders – program specific (1) 4)Enrollment / Graduation trends 5)Employment / wages trends (1) 6)Discussion & analysis 7)Action plan & time table 8)Annual follow-up report 9)Cabinet Review & Approval 1)Outcomes Asmt. Sources (1) 2)Previous Years Plans & Results (1,2) 3)Current Year Asmt. Results (1,2) 4)Key Areas Requiring Improvements (not meeting criteria / expectations) 5)Recommended Action Plans 6)Cabinet Review & Acknowledgement Notes:1)SACS Compliance Requirement 2)Florida DOE Recommendations 3)BOT Rule 6Hx Requirement b. Evaluation and Assessment Processes 21

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting22 Academic Program Assessment  Academic Program Assessment Reports (APARs)  Prepared for the Gen Ed, A.S., and A.A.S. programs according to the three-year cycle  Include a summary of assessment findings and a discussion and analysis of those assessment findings (including use of results, action plan items, and budgetary implications)  Process was initiated college-wide in 1999  Program data is gathered and analyzed on a continuous basis b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting23 Academic Program Viability Review  Academic Program Viability Review (APVR)  Published yearly  Included measures:  Program Graduates  Course Enrollment  Student Semester Hour (SSH) productivity  Relative Profitability Index (RPI-T)  Economic Trend Data b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting24 Program Review Process  Comprehensive Academic Program Review (CAPR) is a summative evaluation that includes multiple measures such as:  program-specific performance  profitability measures  academic outcomes, and  stakeholder perceptions b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting25 Program Review Objectives  CAPR was designed to meet three objectives:  To provide a comprehensive report that summaries all elements of the program’s viability and productivity from a 360-degree perspective,  To provide comprehensive and relevant program specific information to key College stakeholders such as the President Cabinet members in order to make critical decisions regarding the continued sustainability of a program, and  To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement. b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting26 Ed Outcomes Online  To provide a medium for completing the draft educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (  College administration and instructional staff are provided access to “completed” assessment reports including the APAR (Program & Gen Ed) and the CAPR  Online access further encourages the use of assessment data as well as highlighting “best practices” across the college b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting27 College Access b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting28 Program Director Access b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting29 Student Survey Process  Data collection  Increased student awareness of survey process and accessibility  Minimized use of class time to complete surveys  Use multiple points of student contact  Online survey submission and scanning of paper forms  Quality of survey forms  Professional “look and feel” of survey format  Items designed to gather pertinent data  Included online branching models to reduce survey length b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting30 Strategic Directions  College’s SD-IOs are comprised of longer-term directional statements and shorter-term, annually measurable objectives.  Strategic Directions are:  Updated on a three-year cycle and are based on achieving the College’s mission.  Developed to make improvements in learning outcomes, quality, productivity, and other environmental-scan considerations, as well as inputs from IE Team, President’s Cabinet, faculty and student representatives, the BOT, and external/community stakeholders. c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting31 Institutional Objectives  Institutional Objectives are  Reviewed and updated annually, with key on-going initiatives updated based on progress made during the prior year and carried forward, where appropriate.  Designed for the purpose of achieving the corresponding longer-term strategic direction. c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting32 SD-IO Alignment Key Components SPC Mission Evaluation/ Assessments Strategic Plan SD & IOs (87 Total; A few selected below) Operating / Results (See full details: Year-End Report) Accessible Education  BS, BAS, AA, AS Degrees  Technical Certificates  Applied Tech. Diplomas  Distance Education  Developmental Education  Continuing Education 1. Education Oversight Group (EOG) Reports 2. Academic Program Assessment Reports (APARs) 3. Academic Program Viability Reviews (APVRs) 4. Comprehensive Academic Program Reviews (CAPRs) A.2: Use educational outcome data to improve General Education Outcomes A.11: Incorporate service learning into Honors Program courses I.4: Increase baccalaureate degree opportunities w/ targeted new programs A.2: Gen. Ed. outcomes for all A.S. programs implemented A.11: Service learning now required for Honors students I.4: Int’l Business, O&P, Paralegal Studies added Mission – Assessments - Closing the Loop Examples c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting33 Unit / Budget Planning  SPC’s unit/budget planning is centered on:  more than 150 Unit / Budget Managers,  development of departmental unit plans and budget requests, and  use of the national award-winning Strategic Planning Online (SPOL) system.  Unit / Budget Managers are urged and encouraged to access, review, and make use of all available assessment outcome data (i.e., College’s IE website, oversight group reports, etc.) d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting34 Unit / Budget Planning  Budget guidelines are prepared by the Collegewide Budget Committee (CBC), which provides  Information about the College’s expectations for State appropriations,  Collegewide budgetary outlook, and  Salary package and other key strategic expectations.  Budget guidelines also provide details on how capital expenditure requests are to be entered into the SPOL system  SPOL system requires the unit / budget managers to select one of the SD-IOs for each of the unit plan objectives entered d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting35 Unit / Budget Planning  Supporting tasks, expected outcomes, and related budget funding are also captured in the SPOL system  After all unit plans and budget requests are entered, they undergo management review in which refinements and adjustments may be made based on funding availability and overall College priorities  Approved Unit plans (by College President’s direct reports) are aggregated and submitted for further review and evaluation by the CBC d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting36 Collegewide Budget Committee  Each year, the President confirms the membership of a CBC (SVPs, VPs, provosts, and other key functional leaders)  Certain members of the committee are rotated on an annual basis for cross-training purposes.  The charge of the CBC includes the following:  Develop budget guidelines,  Review of all budget requests approved by department heads, vice presidents, and senior vice presidents,  Ensure that the SD-IOs are funded according to their priorities and funding availability, and  Develop the College’s Operating Budget. e. Collegewide Operating Budget Development Process

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting37 Budget Approval  Operating Budget is approved by the President’s Cabinet and recommended to the BOT  Once approved by BOT, the budget is communicated and uploaded into the College’s central financial system  College financials and operating results are monitored on a monthly basis and reported quarterly to the BOT f. Budget Approval, Implementation, and Annual Operating Results

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting38 Operating Performance and Results  Operating performance and results are centered on:  Executing SD-IOs  Supporting unit plans and quality improvement initiatives  Adhering to the operating budget  Operating performance and results are accomplished through the following key steps:  Implementing budgets and action plans  Monitoring achievements and related circumstances  Evaluating mid-year unit plan reviews  Review end-of-year unit plan reports f. Budget Approval, Implementation, and Annual Operating Results

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting39 IE Next Steps  Ultimate goal is to provide stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through:  Formatted (dashboard) style reports, and  On-demand customizable reporting, with  Valid, reliable, and standardized measures.

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting40 Links to Additional IE Materials

Using IE to Build a Culture of Performance Improvement Key Points  IE is more than assessment  A lot has changed since 2007  IE is less centralized  New planning system has been implemented  Assessment activities have expanded December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting41

Using IE to Build a Culture of Performance Improvement Next Steps  Need to capture/define the current processes  Need to ensure alignment among current processes  Need to incorporate the new Student Achievement Committee Structure  Finally, need to tell the story… December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting42

Using IE to Build a Culture of Performance Improvement December 10, 2007 (Revised ) 2007 SACS-COC Annual Meeting43 Questions

Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual Meeting Department of Institutional Research and Effectiveness St. Petersburg College P.O. Box St. Petersburg, FL (727) FAX (727) Revised version for the 2011 SACS Working Group