RACHEL BAKER & ELIZABETH DAYTON GRADUATE SCHOOL OF EDUCATION, STANFORD UNIVERSITY ANDREW LAMANQUE FOOTHILL-DE ANZA CCD KATHERINE MCLAIN COSUMNES RIVER COLLEGE Leading by Example: Assessing Institutional Research Office Outcomes

Background Institutional Research is becoming increasingly important for helping to measure institutional quality as witnessed by new reporting requirements from federal and state governments and accreditation. The senior institutions under WASC have a standard requiring an evaluation of the IR Office (Periodic Reviews of the IR Office and WASC Criterion for Review 4.2. Heather Brown, Sutee Sujitparapitaya, Rebecca Sorell, Michael Wrona. CAIR, 2013). While ACCJC does not have such an explicit requirement the new standards assume a highly functional IR Office. 2

Satisfaction, Outputs, and Outcomes Satisfaction How do people feel about our work? Outputs How many projects did we complete? Outcomes Campus leaders know how to interpret and apply research Improvements based on results Development of a Culture of Inquiry 3

Purposes of Assessing IR Outcomes Promotes Improvement Provides Confidence when Advising Others Assures Quality to Campus Community 4

FHDA IR AUOs 2012 Articulate 3 questions important to ask when starting a research project at FHDA. Interpret and draw correct conclusions from a cross tabulation of descriptive statistics such as course success rates by ethnicity. Using data provided by the IR office, identify and describe 3 key attributes of students served by the employees assigned work unit. Utilize data provided by the IR office to enhance programs and services. 5

IR AUO Assessment Cycle Developed AUOs Surveyed Clients Reflected on Results / Assessment of Outcomes Changes Made / Set new Benchmarks 6 Asked ‘test’ questions Dropped one AUO

Structured Interviews Spring We pursued two overarching questions: 1. Is the information received from IR useful? And specifically, is it presented in a way that is easily understood and applied? 2. How is information received from IR used to make decisions?

In discussing these questions, we often asked: 8 How often are you in touch with IR? In what capacity? (e.g., meetings, presentations, s) What kind of information do you tend to receive from IR? (i.e., tell me about the kinds of things you tend to learn from IR; do you have a recent example?) How do you tend to use the information you receive from IR? What decisions have you made based in information you have received from IR? Can you think of a specific example? How could the information you receive from IR be more useful? Are there ways in which the information IR provides you is mismatched with the kinds of questions you tend to have, or the kinds of decisions you need to make? Are there specific kinds of information you would especially appreciate from IR? Has your understanding of data, and how best to use it, developed through your interactions with IR? Do you feel you’ve learned to use data differently? What do you think IR does particularly well? Where do you think IR could best expand or improve its campus-interactions?

Interview Findings 9 IR informs decision making—heard in 10/10 interviews  Scheduling and curriculum decisions – 6 interviews  Hiring decisions – 1 interview  By complementing faculty, administrators’ “qualitative” knowledge – 8 interviews There are opportunities to grow—10/10 interviews  IR is perceived to benefit some departments more than others – 4 interviews  More longitudinal data would be valuable, both before and after enrollment – 4 int.  More readily accessed data could be valuable – 6 interviews IR provides additional value—9/10 interviews  Supporting grants and funding – 6 interviews  Providing individualized support – 8 interviews  Facilitating productive communication – 2 interviews IR identifies strengths and points of vulnerability—7/9 interviews  Curriculum and programs – 7 interviews  Faculty – 2 interviews  Students – 3 interviews

Assessment of Outcomes 10 Campus leaders know how to interpret and apply research  7/9 of those interviewed cited examples of IR skills that they had learned. {Goal = 90%} IR Data used to make improvements  9/9 of those interviewed cited examples of IR skills that they had learned. {Goal = 100%}

Establishing a Culture of Inquiry Definition of Culture - the behaviors and beliefs characteristic of a particular group Definition of Inquiry – the systematic investigation into a matter Working definition of a Culture of Inquiry - An organization with a culture of inquiry would be characterized by the regular and systematic investigation of practices and outcomes across the organization. Related Outcome - The college community has the information needed to adequately assess and improve the effectiveness of its instructional and non-instructional program and services. RP Conference Presentation 2014

Two Assessments RP Conference Presentation 2014 Matrix to analyze data dissemination (BricTap Inquiry Guide p. 13) apacity apacity Administrator survey (derived from the Strategic Data Project from Harvard University) resources/files/news-events/sdp-rubric-self- asssessment.pdf resources/files/news-events/sdp-rubric-self- asssessment.pdf

Data Integration Matrix RP Conference Presentation 2014 Website Postings Research Office Newsletter Action research teams College research committee Sharing of actions taken based on data Information Sessions Presentations Technical Assistance Meetings Briefings Data Integration Workshops Research and Assessment Methods workshops Research Agenda Development and Implementation Data Integration Strategy Matrix High Low High IMPACT SCOPE

Application of the Matrix RP Conference Presentation 2014 Research Briefs Research Web Site Assessment Reports Research Studies distribution of research Research Projects Innovation Grants Innovation Grant Reports Research Consultation Research Presentations Technical Assistance Resources (presentations and materials) Assessment Methods Workshops Research Briefings to inform planning To inform/follow up on management meeting dialog Data Integration Strategy Matrix High Low High IMPACT SCOPE

Reflection on the Assessment Tool RP Conference Presentation 2014 PROS  Easy to apply  Can be applied in a variety of contexts  Types of activities  Activity list/log  Research agenda  Provides a new way to track and evaluate activities  Quickly identifies possible areas of improvement CONS  Evaluates outputs not their impact  May not span all research office activities  Some activities span classification areas

Administrator Survey RP Conference Presentation 2014 Questions derived from the 7 principles about the strategic use of data from Strategic Use of Data Rubric Perception survey about  Application of these principles at the college  Evaluation of personal application of these principles  Relative important of Research Office Data dissemination Pilot-tested this semester in leadership team (n=16) Survey instrument available at _Office/Research_Office_Web-based_Surveys.htm

Application of the Survey RP Conference Presentation 2014 Institution  Strengths - Using data to set goals and monitor college performance  Challenge - Using data for program enhancement Individual  Challenge - Using data to adjust budgets Important research office activities  Internal research disseminated in response to request or related to a topic/issue being discussed  Website, data briefings, discussions about actions taken in response to research Relatively unimportant research office activities  External research that is related to a topic/issue being discussed

Reflection on the Survey Tool RP Conference Presentation 2014 PROS  Can be customized to different groups  First implementation can set a baseline  Can identify relative strengths and potential areas of change  Open-ended questions provide a wealth of data  Survey can lead to individual learning and meaningful dialog CONS  Diversity in the responses may reflect diversity of roles  Not appropriate for institution-wide assessment  Indirect assessment  Identifies potential growth areas not specific interventions  Many inputs contribute to the outcome

Questions? RP Conference Presentation 2014