S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT ASSESSMENT ON THE GROUND What you need to know when conducting assessment or working with an assessment professional so that the results help you do your best work!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT SARA Mission Student Affairs Research and Assessment (SARA) provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Why is assessment important? It will help you do a better job It will help ensure that you have the resources to do that job It can help you prioritize your efforts It can contribute to the broader conversation and understanding

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT What do you need to know? Not everyone has to be an assessment expert But you do need to know how to make informed decisions ◦ To conduct an assessment ◦ To use the findings effectively Plan, Do, Check, Act (PDCA)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT You are the content expert Things to consider: ◦ The parameters of the activity you are assessing ◦ The conceptual model that underlies the activity ◦ The desired objectives/outcomes (learning or other) ◦ Past assessment efforts and findings ◦ How this activity/assessment relates to the strategic goals of your unit and student affairs ◦ What resources are available ◦ How you will use the results ◦ Where/how will you disseminate the results

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Basic Approaches (Schuh & Upcraft, 2001)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Key steps - PDCA Identify objectives Identify appropriate measures Choose an appropriate assessment method Choose an appropriate project design Collect the data Analyze the data Disseminate the findings Take action Start all over again!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Identify & articulate objectives You should always be able to tie what you are doing (programming, advising, etc.) to a specific objective Bloom’s taxonomy or similar learning constructs may be useful if you’re seeking to measure learning The focus of a learning objective is not on what you will teach or provide, but on what students will learn or gain

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT One helpful approach to writing learning outcomes is ABC(D): Audience: Who will accomplish the objective (e.g., students) Behavior: What is expected; most helpful when overt and observable Condition: Under what circumstances the behavior is expected (Degree: The acceptable performance level - often unstated)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Avoid the “weasel words” Learn Know Be aware of Be familiar with Have a firm grasp of Understand Appreciate

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Example using Bloom Remember – After attending the Women's’ Resource Center open house, students will be able to describe the resources available at the Center. Understand – After completing their first-year seminar, participants will be able to interpret university policies related to academic integrity. Apply – By the completion of the Study Skills series, participants will establish a realistic time management plan. Analyze – By the end of their first elected term, student government participants can find and provide credible evidence to support a planned initiative. Evaluate – After completing Leadership 101, class members will be able to debate the merits of various leadership styles and their application in different situations. Create – By the end of their first elected term, student leaders will be able to develop and present detailed plans for proposed projects. AudienceBehaviorCondition

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Determine appropriate measures Attitudes? Experiences? Knowledge? Abilities? Persistence? Graduation? Employment?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT More on measures Norm-referenced = report student performance relative to other students Criterion-referenced = compares to an absolute standard of achievement (e.g., you pass or you fail) Self-referenced = compares different scores/ratings from the same student

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT What kind of data? Quantitative ◦ Can be economical, generalizable, reliable, valid Qualitative ◦ Can provide rich data – can ask about relationships rather than infer Mixed methods

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Consider various types of data collection approaches Test of abilities or knowledge Survey Interview Focus group Other types

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Tests ◦ Can be used to measure a specific skill or knowledge base ◦ “Objective” ◦ Feasible? Surveys ◦ Attitudes ◦ Experiences ◦ Self-reported abilities & knowledge ◦ Can be confidential/anonymous ◦ Can reach large numbers of people

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Interviews ◦ Rich & detailed information ◦ Body language can be helpful ◦ Respondents may feel inhibited ◦ Repeated interviews can build trust/comfort ◦ Time-consuming Focus groups ◦ Many of the same benefits as interviews, but not as in-depth ◦ Participants “feed off of” each other ◦ Can reach more people than interviews

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Other ways to collect data Student self-assessments Portfolios Reflective journals Observation-based What others can you think of?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Project design Go back to your assessment question(s) ◦ What do you want to know? ◦ What are the limitations (e.g., time, resources, staff)? ◦ How will the assessment be used? The answer to these questions help you figure out: ◦ What your purpose/question is ◦ What approach to use ◦ What type of information to collect ◦ How to analyze your data

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Data collection

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Data analysis Make sure that your analysis is appropriate to your question. ◦ Does your data meet the basic assumptions of the statistical test you are using? ◦ Is your analyst free coding your transcripts when your goal was to analyze your data using a pre- assigned coding system based on the theoretical framework underlying your program? Make sure that you understand the assumptions and limitations of any analyses you rely on or report, whether you are the one who conducted them or not

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Reporting & Dissemination Think about your audience Administrators don’t have time to read a dissertation or even a journal-length article Executive summary of the executive summary! Put key points into easily digested tables, charts, graphs, and bullets Don’t get carried away!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Dissemination continued DO NOT FORGET to spell out the implications of your work ◦ How do the findings improve our understanding? ◦ What should we change? ◦ What else do we need to know? Make sure the information gets to key decision makers – understanding the structure and politics of your institution is very important (Terenzini’s “contextual intelligence”)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT You’re not done yet! Time for action. What are you going to do with what you learned? Plan, Do, Check, Act, Repeat

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Theory to practice In practice, you will not always have the resources to do assessment the way it is outlined in books and papers What’s important? ◦ Understand the impact of the compromises you will make ◦ When you act, you are doing so based on the best available information

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT QUESTIONS?

Your turn Get out a piece of paper Choose a student affairs area to represent Write at least one learning outcome for your unit How would you assess it?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Think about Type of approach (needs assessment, learning outcomes, satisfaction, etc.) Type of measure(s) (attitude, knowledge, etc.) Type of data (quant, qual, mixed) How will you collect it? (focus group, survey, etc.) What is your design? (pre-/post-test, longitudinal, etc.) How will you analyze it? (thematic analysis, mean comparisons, etc.)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Betty Harper