1. Institutional Researcher’s Credo
I realize that I will not succeed in answering all of your questions. Indeed, I will not answer any of them completely. The answers I provide will only serve to raise a whole new set of questions that lead to more problems, some of which you weren’t aware of in the first place. When my work is complete, you will be as confused as ever, but hopefully, you will be confused on a higher level and about more important things.

2. Developing & Sustaining a Culture of Institutional Research
Victor M. H. Borden, Ph.D.
Professor of Educational Leadership and Policy Studies
Indiana University Bloomington

3. What’s Your Problem?
Select one current institutional issue that requires attention
What do you need to know to be better informed to pursue that issue?
What do you need to know to decide what course of action to take?
How will you be able to tell if that action is working?

4. What is IR?
…research conducted within an institution of higher education to provide information which supports institutional planning, policy formation and decision making.
Saupe, 1990 – The functions of IR

5. What Does an IRer Do?
The institutional researcher serves higher education and, in turn, his institution through critical appraisal and careful investigation of its processes and programs.
Suslow, 1972– A Declaration on Institutional Research

6. Who Does IR?
IR Staff
Anyone who gathers, evaluates, and interprets ‘data’ for use in decision making
Senior administrators and their senior staff
Academic department chairs and program directors
Administrative directors and managers
Faculty working on committees and in administrative support roles
Administrative assistants

7. Variations on U.S. IR Practice
From 1 to 25 person offices
Accountability
Enrollment management
Data administration and warehousing
Learning Assessment support
Survey research
Marketing and development
Planning and policy analysis
Other duties as assigned

8. An IR Office Case Study
IUPUI’s Office of Information Management and Institutional Research (IMIR)

9. IMIR mission
The mission of the Office of Information Management and Institutional Research (IMIR) is to provide and coordinate information support for planning, administering, and evaluating academic and administrative programs in ways that will continuously improve IUPUI.

10. IMIR Services and Functions
Management Reports and Analyses
developing for academic deans and other campus administrators a series of management reports and analyses that integrate information from a variety of institutional and external data sources;
Ad Hoc Requests
providing academic and administrative managers with information needed to address ad hoc problems and issues;
Information Management
creating organized, documented, and accessible data resources based on institutional, survey, and external databases;

11. IMIR Services and Functions
Survey Research
conducting survey research to assess the expectations, satisfaction, and outcomes of students, alumni, faculty, staff, employers, and other stakeholders;
Planning and Program Evaluation Support
providing direct support to specific campus, school, and program evaluation and planning activities;
Information Technology Development
developing computer network-based systems for collecting, accessing, and analyzing information in a more timely and cost- effective manner; and
Consulting
helping staff from other academic and administrative units to conduct institutional research, reporting, and analysis

12. Principles of Effective IR (old school)
Rational, data-driven decision making processes yield more informed and successful decisions
Improved access to well-managed data sources provides decision makers with the tools they need to inform decision processes
To be useful, data must be turned into information, and information into knowledge
Data, information, and knowledge can all be managed
The primary function of research and information support is to provide information needed by decision- makers and decision-making groups

13. but

14. Vic Borden FYE Assessment Keynote
Heresy
Being “data-driven” is not, in and of itself, a good thing
e.g., “selective use” of evidence to support a foregone conclusion
Torture numbers long enough and they’ll confess to anything
Effective use of data requires sharing diverse and often divergent perspectives
It’s not what the data say, it’s what you say about the data
Some disagreement and dissent is important to learning and innovation

15. Vic Borden FYE Assessment Keynote
Further Heresy
Building effective programs requires some level of irrationality and disorder
To learn from what we do requires that we unlearn some things that we often don’t want to unlearn
As if doing this by ourselves were not difficult enough, we must do this together
Organizational learning requires communities of practice

16. Vic Borden FYE Assessment Keynote
Beyond Data-Driven
Data-driven implies…
Rational, systematic testing of ideas through inspection of facts
sequential, often individual decision-making process
Learning-driven implies…
Going beyond what we already know and can do to gain new competencies
Deconstruction and reconstruction of ideas and beliefs
Becoming irrational to become re-rational

17. Vic Borden FYE Assessment Keynote
Beyond Data-Driven
We are engaged in the learning process as interdependent members of an organization (i.e., system)
Through collaborative, action-oriented, organizational learning, we learn (together) how to achieve organizational goals

18. Learning Organizations
Vic Borden FYE Assessment Keynote
Learning Organizations
…are organizations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together. (Senge, 1990)

19. Example: Learning Communities
Vic Borden FYE Assessment Keynote
Example: Learning Communities
Problem: Low retention rates at urban, commuter university associated with under-prepared new students, low grades, and non-existent campus life
Summer Retreat Involving Administrators, Faculty, Advisors, IR Staff
How do we increase engagement through the classroom?
Constraint: can’t “block-schedule” working commuters
Common first-year seminar for all beginners
100+ sections of 25 students, each with faculty member, advisor, student mentor and librarian/technologist
Common outcomes template

20. Vic Borden FYE Assessment Keynote
E.G., LCs (2)
Bottom line evaluation shows modest impact
Improved retention and grades among “treated students”
Overall retention rate goes south
Further (qualitative) analyses reveal problems with “one-size-fits-all” approach
Focus groups; interviews; faculty fellows
Adjustments
Agreement with local CC to defer some students
School-based models to appeal to different interests
Companion supports: summer bridge, critical inquiry, structured learning assistance, math assistance center
Block and thematic learning communities

21. Vic Borden FYE Assessment Keynote
E.G., LCs (3)
Improved Results
10 percentage point increase in retention over four years!
No improvement for two years!!
New problems—New approaches
Existing bridge programs adjusted to weed out students who aren’t likely to succeed
Further increases in retention!!

22. The Collaborative Action-Inquiry Approach
Vic Borden FYE Assessment Keynote
The Collaborative Action-Inquiry Approach
An organizational learning framework for evidence-based practice
Action research and beyond
Further examples

23. Herbert Simon’s Challenge
Vic Borden FYE Assessment Keynote
Herbert Simon’s Challenge
What information consumes is rather obvious: it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it. ~(Simon, 1971)

24. Kahneman & Tversky’s Challenge
Vic Borden FYE Assessment Keynote
Kahneman & Tversky’s Challenge
Cognitive biases
We seek information based on what we know and want to know
Biases and defense mechanisms prevent us from processing information objectively
Self-serving bias
Primacy and Recency

25. Vic Borden FYE Assessment Keynote
Challenges (cont.)
Decisions made within a social- organizational-political context
Rules and roles define/constrain process
Social and organizational relationships must be accommodated

26. Vic Borden FYE Assessment Keynote
A Learning Paradigm
Planning, evaluation, and improvement all fit within a rational model
Learning incorporates uncertainty, ambiguity, and multiple styles
Individual learning and organizational learning are compatible concepts
Learning is mission critical for all educational institutions

27. Single- and Double-Loop Learning
Vic Borden FYE Assessment Keynote
Single- and Double-Loop Learning
Learning is the detection and correction of error (unintended consequences)
“Governing Variables” are those things what we feel are important to keep within acceptable limits
“Action Strategy” is what we do or plan to do to keep the governing variables within limits
“Consequences” are the intended and unintended outputs and outcomes
Intended: confirm our theory in use
Unintended: suggests error in our theory in use

28. Vic Borden FYE Assessment Keynote
Single-Loop Learning
Governing variables not called into question
Adjustments made to action strategies at best
Defense mechanisms can readily arise to maintain single-loop learning
Governing
Variables
Action
Strategies
Conse-
quences

29. Vic Borden FYE Assessment Keynote
Double-Loop Learning
Questioning the role of the framing and learning systems which underlie actual goals and strategies
Reflection is fundamental
Basic assumptions are confronted
Hypotheses publicly tested
Falsification is sought
Ego is laid aside
Governing
Variables
Action
Strategies
Conse-
quences

30. Model I and II Org Learning
Vic Borden FYE Assessment Keynote
Model I and II Org Learning
Single- and double-loop learning at the organizational level
Model I: Organizational members prescribe to a common theory in use
Organizational policies and practices inhibit change
Model II: Governing values, policies, and practices promote double-loop learning

31. A Model I Learning Organization
Vic Borden FYE Assessment Keynote
A Model I Learning Organization
Governing Variables
Tow the line
Win at all costs
Suppress negative feelings
Emphasize rationality
Action Strategies
Control environment and task unilaterally
Protect self and others unilaterally
Discourage inquiry
Consequences
Defensive relationships
Low freedom of choice
Reduced production of valid information
Little public testing of ideas

32. A Model II Learning Organization
Vic Borden FYE Assessment Keynote
A Model II Learning Organization
Governing Variables
Valid information is most important
Free and informed choice
Shared internal commitment
Action Strategies
Shared control
Participation in design and implementation of action
Consequences
Minimally defensive relationships
High freedom of choice
Public testing of ideas

33. Jerome Bruner / Gilbert Ryle
Vic Borden FYE Assessment Keynote
Jerome Bruner / Gilbert Ryle
Learning about
vs.
Learning to be
Knowing that
vs.
Knowing how

34. Vic Borden FYE Assessment Keynote

35. Learning To Be / Know How
Vic Borden FYE Assessment Keynote
Learning To Be / Know How
Based on collaborative practice
Communities of practice
Knowledge as inseparable from the knower
Evidence as input
Sharing interpretations as process
Common priorities and strategies as output

36. Vic Borden FYE Assessment Keynote
What’s important is not what the data say,
It’s what we say and do about the data

37. Action Research
Continuous cycle of data collection  data analysis  data feedback  action plans  data collection
Stakeholder empowerment through active and on-going participation
Data feedback meetings promote collaboration, dialogue, and collective analysis
Active learning and discovery fostered by critical reflection process
Data-driven action plans developed = research linked to action

38. Action Research Research Question/Evaluation Focus Data Collection
Analysis and Interpretation
Presentation and Reflection
Action Plan
Evaluation of impact (Go to 2)
Focus on new issue (Go to 1)

39. Linking Research and Action
Vic Borden FYE Assessment Keynote
Linking Research and Action
Who does what?
Decides what actions are taken?
Is responsible for effective implementation?
Can devise appropriate evaluation protocols?
Has access to or can collect appropriate evidence?
Reviews the results and decides what to do?
What can be done to get these people to work together and in concert?

40. Determining the Research Question
Vic Borden FYE Assessment Keynote
Determining the Research Question
Traditional IR
Given to researcher
Top-down directive
Bottom-up request
Clarification of request
Discussion of context and use
Action Research
Developed together
Requester or researcher
Specific questions often deferred until vested parties brought together

41. Vic Borden FYE Assessment Keynote
Data Collection
Traditional IR
Researcher finds and collects data
Researcher accountable for integrity of information
Action Research
Stakeholders have role
Collecting data
Learning about nuances
Shared responsibility for integrity

42. Data Analysis and Interpretation
Vic Borden FYE Assessment Keynote
Data Analysis and Interpretation
Traditional IR
Researchers responsible through dissemination
May consult with stakeholders to gain insight into the results
Action Research
Stakeholders involved in stages of data analysis
Preliminary results presented and discussed
Further analyses shaped by those discussions

43. Report Presentation and Dissemination
Vic Borden FYE Assessment Keynote
Report Presentation and Dissemination
Traditional IR
Researcher prepares and often presents results to stakeholders
Action Research
Presentation and report writing responsibilities shared
Presentations involve
active discussion
facilitation of action plan development

44. Vic Borden FYE Assessment Keynote
Follow-up
Traditional IR
Some additional analyses may be requested or perhaps some clarification
Often the end of the process
Action Research
Stakeholders design action plan based on results
Data collection included in follow-up plan
Further lines of inquiry established for next cycle of research

45. Example: Evaluation of New Student Orientation
Vic Borden FYE Assessment Keynote
Example: Evaluation of New Student Orientation
Research Question and Evaluation Focus
reassessment of goals; incoming students’ needs; impacts on knowledge, attitudes, and behaviors
Data Collection
focus groups and questionnaires, sought perspectives of all major stakeholders
Data Reporting and Feedback
meetings with orientation leaders and faculty stakeholders

46. Example: Evaluation of New Student Orientation
Vic Borden FYE Assessment Keynote
Example: Evaluation of New Student Orientation
Development of Action Plans
facilitation of dialogue and data-driven proposals
Action
implementation of proposed changes
Assessment – on-going formative evaluation; re-administration of process and outcome instruments

47. Example: Indiana Project on Academic Success (IPAS)
Vic Borden FYE Assessment Keynote
Example: Indiana Project on Academic Success (IPAS)
Research-based inquiry for enhancing academic success
Four-stage method
Assessment
Organizing
Action Inquiry
Evaluation
Supported by use of state and institutional student tracking records

48. Vic Borden FYE Assessment Keynote
Stage 1: Assessment
Compare campus assessment information to statewide assessment results; identify possible challenges
Collect additional information from campus sources, such as prior reports and studies and focus group interviews
Organize teams of administrators, faculty, professional staff, and students to identify critical challenges on the campus
Prioritize the challenges, identifying two or three that merit special attention at a campus level

49. Vic Borden FYE Assessment Keynote
Stage 2: Organizing
Coordinate the assessment and inquiry process with campus-level planning and budgeting; integrate the challenges with strategic plans; coordinate budgeting to provide necessary support.
Appoint workgroups to address critical, campus- wide challenges; consider providing release time to team leaders to work on tasks for the campus.
Coordinate the inquiry process (activities of the workgroups) with campus planning and budgeting.

50. Vic Borden FYE Assessment Keynote
Stage 3: Action Inquiry
Build an Understanding of the Challenge
What solutions have been tried in the past, and how well did they work? What aspects of the challenge have not been adequately addressed? What aspects require more study? Develop hypotheses about the causes for the challenges using data to test the hypotheses. Do the explanations hold up to the evidence?
Look Internally and Externally for Solutions
Talk with people on campus about how they have addressed related challenges. Consider best practices for retention and how they might be adapted to meet local needs. Visit other campuses that have tried out different approaches to the problem. How well would these alternatives address the challenge at your campus?

51. Vic Borden FYE Assessment Keynote
Stage 3: Action Inquiry
Assess Possible Solutions
Consider alternatives in relation to the understanding of the problem developed in Stage 3, step 1. Will the solutions address the challenge at your campus? How can the solution be pilot tested? If you tried out the solution, how would you know if it worked? What information would you need to know how well it worked?

52. Stage 3: Action Inquiry (cont.)
Vic Borden FYE Assessment Keynote
Stage 3: Action Inquiry (cont.)
Develop Action Plans
Action plans should address the implementation of solutions that should be pilot tested. Consider solutions that can be implemented by current staff. If there are additional costs, develop budgets for consideration internally and externally. (Remember, seeking additional funds can slow down the change process.) Develop action plans with time frames for implementation and evaluation
Implement Pilot Test and Evaluate
Provide feedback to workgroups and campus coordinating team. Use evaluation results to refine the solution. Also, evaluation can be used as a basis for seeking additional funding from internal and external sources, if needed

53. Institutional Support for Organizational Learning
How do we organize/staff to support the “culture of institutional research?”
There are many kinds of supports possible
Institutional research and analysis
Measurement and instrumentation
Program review and evaluation
Quality enhancement
Performance management

54. Planning for Improvement
Assessable
Outcomes
Improvement 1.
Reporting to internal constituents 2.
Demonstrating accountability to
external stakeholders 3.
Applying findings in campus
improvement initiatives 4.
Proposing improvement initiatives 5.
Improving assessment methods
Web-based data
Electronic portfolios
Culture Of
Evidence
Application
of Findings
Instrumentation
Data Collection
Analysis

55. Planning for Improvement

56. Planning for Improvement
Evaluation
Academic and administrative program reviews
Evaluation of process effectiveness
Assessment of learning outcomes
In major
In general education
Course evaluations
Student assessment
Constituent surveys
Management information and analysis
Program cost analysis
Web-based evaluation tools
Annual campus performance report
General and specialized accreditation

57. Planning for Improvement
Reporting to internal constituents
Demonstrating accountability to external stakeholders
Applying findings in campus improvement initiatives
Proposing improvement initiatives
Improving assessment methods
Web-based data collection/reporting
Electronic portfolios

58. How Might You Do it?