1. When Good Plans Go Bad: The Flip Side of Assessment "Best Practice"
Laura Dean, University of Georgia Diane Cooper, University of Georgia
Andrew Wells, Macalester College
Jason Wallace, University of Georgia
Tuesday, March 6, Convention Center, Room 117
Laura

2. Learning Objectives
Familiarize the audience with a multi-part assessment design created to evaluate a complex campus-wide initiative
Describe examples of well-intentioned assessment strategies that did not work as intended
Discuss alternatives that can be employed to reach assessment goals
Consider assessment successes and failures and the development of alternative strategies
Laura

3. Intended Learning Outcomes
As a result of participating in this program, audience members will be able to do the following:
Describe key elements in the assessment plan design for a complex, campus-wide initiative
Identify characteristics of successful and unsuccessful assessment efforts
Create strategies for developing alternatives in the event that assessment strategies fail
Apply these lessons to examples from their home campuses
Recharacterize their assessment failures as opportunities for improved practice
Laura

4. Emory Integrity Project
Campus-wide initiative designed to create an inclusive campus community embracing integrity through encouraging “honor, humility, and helpfulness” in undergraduates - H3
Three-year, $2.6 million grant funded by the Templeton Foundation
Transform campus culture through the deliberate and strategic application of co-curricular change and campus- wide initiatives
External assessment team (University of Georgia, University of Iowa)
Diane

5. EIP Stakeholders
Center for Ethics in collaboration with Campus Life received grant funding for campus-wide initiative
Campus investment in culture of integrity
Campus Life staff
Academic administration
Faculty
Students
Grantors invested in EIP success, relevance for other institutions, and accountability for grant funds
Higher education/student affairs researchers investigating student development & campus environmental change processes
Diane

6. EIP Assessment Design
Investigating individual-level changes (inc. longitudinal cohort study) and campus climate changes (focus groups, surveys of students, staff, and faculty)
Quantitative elements
PSRI
Program attendance and service utilization statistics
Changes in such measures as academic honesty reports
Qualitative elements
Focus groups
Observation of campus environment
Document analysis
Mixed elements
Assessment of student feedback, evaluations of programs and services
Diane

7. EIP Project – Initial Challenges
Being assessors external to both the project and the institution
Understanding culture & norms
Ease of access to campus processes, units, constituents
Establishing credibility
Designing assessment when not designing initiatives & implementation
Partnerships across units
Student Affairs/Academic Affairs
Center for Ethics – dotted line relationship to Academic Affairs
Change in grant leadership
Involving students, faculty, staff
IRB challenges - who owns/manages the data?
Diane

8. Well, that didn’t work like we thought it would…
Assessment itself has a cycle & a need to close the loop
When the assessment strategy doesn’t work like you planned, how do you change it for the next round?
Laura

9. Example 1: A Vision for Myself Expected n=~1000 / Got n=~300
No way to know how representative the sample was
Used what we had to look for indicators of integrity constructs in the responses
Realized that the initial goal question yielded responses focused on academics (logical, given institution/student body)
Revised it to ask about goals “other than academics”
Laura

10. Example 2: Chalkboards
Both a program intervention & an assessment strategy
Data were transcribed, but difficult to read and to analyze
Some comments may have been jokes… but maybe not?
Discussion yielded hypotheses about what students were “saying”
Previous campus incident related to chalking may have influenced student responses – no way to tell for sure from the content
Might have followed up for additional/deeper insights
Andrew

11. Example 3: Focus Groups
“Really, I think there's some confusion as to what the intentions are, what are the, you know, the tangible initiatives that are taking place and then how to approach students about that and engage them. So -- … I cannot think of who would be behind it. Like, it seems very mysterious as to the students or the professors. …It -- It seems like a faceless group.”
Goal was to conduct focus groups to explore student understanding of integrity & perceptions of integrity & EIP on campus
Students expressed negative perceptions about EIP in particular  
We intended to talk about the effects of the intervention; what we heard was that the intervention wasn’t working 
Jason

12. Examples 4 & 5: PR Problems
PSRI instrument was sent to faculty Turned out to have a bad link Observation of orientation session Associate Dean confronted us because they hadn’t been brought into the loop
Diane

13. Now what?? Lessons learned…
Expect the unexpected; be ready to practice your flexibility
Communication & buy-in are crucial
Know & consider the context, institutional culture, current “hot topics”
Timing of chalkboards – issues on campus, national climate/election
Understand your relationship to the process/project
When results aren’t clear, may need to design another layer of follow-up
Attention to detail – both for credibility and accuracy
No one cares about your data as much as you care about your data
When you are an external assessor or consultant, you can convey information and make recommendations when something is not working, but you may not be in a position to intervene or change the initiatives
Laura & all

14. Lessons from the field When have your good plans gone bad?
What alternative strategies did you use?
What did you learn?
What successes have you had?
Laura & others

16. Thank you for joining us today!
Please remember to complete your online evaluation following the conference.
See you in Los Angeles in 2019!