Assessing Student Learning Outcomes in Student Development – Part II Student Development Division Retreat SUNY Oneonta May 27, 2008

Presenter Patty Francis, Associate Provost for Institutional Assessment and Effectiveness

Topics/Activities for Today Continued themes from May 9 division meeting Developing a unit assessment plan Writing good outcome statements Measuring student learning outcomes Hands-on opportunity to develop outcome statements/measures for department

Continued Themes from May 9 Meeting

Assessment Basics and Rationale Assessment as an ongoing, iterative process which must culminate with “closing the loop” Importance of congruence in the assessment process Assessment as an opportunity for ongoing dialogue, professional development and, most important, improving student services

Developing a Unit Assessment Plan

Overall, Unit Objectives Would Reflect: Institutional effectiveness performance indicators Documentation of all services and programs offered Tracking of use of services (and by whom) Student satisfaction with services/programs Direct impact of services/programs on students

Writing Good Outcome Statements

Issues to Address at the Outset Consistency between college, division, and unit mission statement (and, ultimately, outcome statements) Do all staff have the opportunity to provide input? Who are all your constituents? What results do you expect your programs and services to produce?

Components of a Good Outcome Statement Who is the target? What domain of student development is the target? What change is expected?

But Other Things to Keep in Mind Do you have a program/activity in place to bring about outcome? Can desired change in students be measured? How will you know you were successful? Do external standards apply (i.e., in case of external accreditation/certification)?

Outcome Writing Rule #1: Focus on Student, Not Program Good example: “Student workers will identify, provide, and implement technical equipment that is appropriate for specific union events.” Poor example: “College union work study program will teach student workers how to select and set up equipment for union events.”

Outcome Writing Rule #2: State Outcomes Using Concrete Language and Action Verbs Good example: “Students will negotiate necessary academic accommodations with faculty members.” Poor example: “Our objective is to enhance students’ independence and self-confidence.”

Outcome Writing Rule #3: Focus on Results, Not Process Good example: “Students will demonstrate increased job search skills (e.g., letter and resume writing, interviewing, employer research).” Poor example: “Students will participate in three 2-hour sessions on letter writing, interviewing, and employer research.”

Measuring Student Learning Outcomes

Basic Principles Measurement techniques must be rigorous, since unreliable data are of minimal value Best to use variety of quantitative and qualitative measures  Quantitative easier, but not often as rich  Qualitative often more informative, but require check on scoring (e.g., rubrics) Rely as much as possible on existing data sources

Types of Information to Include 1. Normative/benchmark information whenever possible (external sources, own performance over time) 2. Locally collected data Survey data  SUNY SOS  NSSE  Local surveys of student satisfaction/ perceptions National benchmarking data (e.g., ACUHO/EBI, ACUI/EBI) Performance-based data Focus groups

General Types of Measures (Maki, 2004) Direct measures – students actually demonstrate learning so that evaluators can match results to expectations  Standardized tests  Authentic, performance-based – embedded into students’ actual educational context Indirect – students’ perceptions of learning  Should never be used as sole indicator

From Measures to Criteria Assessment criteria reflect your expectations about student performance (i.e., how you know you were successful) Set criteria at reasonable but challenging levels Often take these forms:  “90% of students will …..”  “80% of students will score at least 70% on…”

Authentic, Performance-Based Assessment: The Value of Rubrics Rubrics provide reliable means of rating student performance in more qualitative way Steps in developing rubrics  Use entire staff to help develop and be as specific as possible in differentiating between performance levels  Use existing rubrics as guide  Pilot test to assure scoring is reliable

Rubric Example #1: Recreation & Sport Services (Univ. of W. Florida) VariableFails to MeetApproachesMeetsExceeds Respect Makes negative comments about others and recreation. Engages in disrespectful behavior during work and/or participation. Keeps self from making negative comments about others and recreation. Sometimes engages in disrespectful behavior during work and/or participation activities. Sometimes makes positive comments about others and recreation. Generally engages in positive behavior during work and/or participation opportunities. May engage in disrespectful behavior during work and/or participation activities but learns from the experience and corrects behavior. Frequently makes positive comments about others and recreation. Generally engages in positive behavior during work and/or participation opportunities. Rarely engages in disrespectful behavior during work and/or participation activities and always learns from the experience and corrects behavior.

Rubric Example #2: Career Services (Interviewing) (Univ. of W. Florida) VariableFails to MeetApproachesMeetsExceeds Appearance Did not dress in proper interview attire or was messy and wrinkled - wore shorts, flip flops, hat, t-shirt, or other apparel that was improper Wore casual khakis or jeans with a semi-casual shirt or blouse. Candidate may have not worn tie or professional heels Professional appearance, but had one or two items that stood out to be less professional - for example, too short of skirt Wore exceptional interview attire - conservative suit, proper length skirt, well- pressed garments and overall - well groomed First Impression May have been late to the interview, appeared too casual or uninterested initially, provided poor handshake or greeting, etc. Was on time to the interview, but offered poor greeting, handshake or appearance, etc. Was on time to the interview; looked interested and had a professional appearance - may need to practice greeting or handshake Was on time to the interview, offered a professional appearance, and gave a great handshake and greeting Non-Verbal Behavior May have used many distracting hand gestures or no eye contact - Candidate appeared completely uninterested or heavily slouched May have used distracting hand gestures or may have slouched in the interview for approximately half of their interview time Used some distracting hand gestures or may have slouched some of the time during the interview, but overall looked interested Used appropriate hand gestures and displayed good posture & appropriate eye contact - generally illustrated that they were interested Communi- cation May have organized thoughts in an unclear way, used slang, or may have been too concise or brief with answers given May need some work with organizing thoughts and using examples to strengthen issues discussed Organizes thoughts well, uses proper grammar, but may still need to use examples Gives solid introductions and conclusions to thoughts, expresses self in proper grammar, and uses examples

Congruence in the Assessment Process: A Detailed Example Handout #1

Your Turn! Handout #2