1. Foundations of Assessment II Methods, Data Gathering and Sharing Results

2. What do you still have questions about? How to effectively speak to students about problems How to include all stakeholders (develop partnerships) Developing effective questions What tools or mine-able data already exist that we can use in our assessment projects When are you asking too much info at once (i.e. mid-year reviews)? How to narrow down the focus of the assessment How to stay on top of assessment while working on other programs

3. What do you still have questions about? How to draw conclusions from assessment data (especially data from long-term assessments) Quick assessments and what they look like How long an assessment process should take How to choose the right assessment method How to conduct an assessment interview How to incentivize participation in data collection Further understanding how to “measure” qualitative data How much is too much data collection? How do you predict the info you will need or want?

4. Goals of Today’s Presentation 1.Understand the various methods available for assessment 2.Learn how and why to share results 3.Gain confidence to plan and conduct assessments in your department

5. The Assessment Cycle Website: Assessment Resources FormsAssessment Resources Forms

6. A few questions What methods have you used for data collection? How did you determine that those methods were the best way to gather evidence?

7.  Surveys  Rubrics  Interviews  Pre/Post evaluation  Reflection  Quick assessments  Document Analysis  Reviewing existing data  Focus Groups  Demonstrations  Observations  Written papers, projects  Posters and presentations  Portfolios  Mobile data collection  Tests, exams, quizzes Common Methods for Gathering Evidence Contributing Source: Campus Labs

8. Things to consider when selecting a method  How will you use the data?  Available resources (i.e., time, materials, budget, expertise)  Potential for collaboration  Timing: Religious holidays, large events, student schedules  Direct vs. indirect  Quantitative vs. qualitative Contributing Source: Campus Labs

9. What Type of Data Do I Need?  Direct Methods: Any process employed to gather data which requires students to display their knowledge, behavior, or thought processes.  Indirect Methods: Any process employed to gather data which asks students to reflect upon their knowledge, behaviors, or thought processes. Source: Campus Labs

10. What Type of Data Do I Need? Quantitative  Focus on numbers/numeric values  Easier to report and analyze  Can generalize to greater population with larger samples  Less influenced by social desirability  Sometimes less time, money Qualitative  Focus on text/narrative from respondents  More depth/robustness  Ability to capture “elusive” evidence of student learning and development  Specific sample

11. Questions for Choosing a Method  Is your method going to provide you with the evidence you need to make a statement about the learning that occurred?  If you are assessing satisfaction or service effectiveness, is the method going to give you the most detailed and accurate information?  Do you have the time and resources to use the specific method ? Adapted from Campus Labs

12. Surveys Advantages  Useful when you have a large population.  Easily administered  Easy to compare longitudinally  You can ask a variety of questions  Quick data turnaround  Good for statistical reporting Disadvantages  Indirect measure  Interpretation – lack of skill in understanding results  Low response rates  May need secondary direct measure to learn more information

13. Focus Groups: What is a focus group?  Qualitative research  Small group, open discussion  Gauges opinions, perceptions and attitudes

14. Focus Groups Advantages  Able to collect a lot of rich data in short time  Check perceptions, beliefs and options  Explore specific themes  Participants build off each other’s ideas  Can be a good follow-up to a survey Disadvantages  Facilitation requires skill  Not generalizable to population  Time needed for preparation and analysis  Lack of control over discussion  Can be difficult to attract participants  IRB may be required

15. Quick Assessment Techniques  Quick, easy, and systematic  Assessment does not HAVE to be time-consuming  Minimal resources  “Pulse” on how things are going  Specific techniques available on BC websiteBC website

16. Rubrics  Scoring tool for subjective assessment  Assess student performance on learning objectives  Clearly define acceptable and unacceptable performance  Training programs, interviews, projects

17. Rubric Example

18. Mobile Devices

19. How/When to Use Mobile Devices Campus Pulse Survey  Larger audience and more diverse sample  Dining Halls, residence halls, computer lounge  Topic suggestions: food, campus facilities, hot topics, world events Point of Service Survey  Distribute to students after they stop an office on campus  Focus on service experience  Locations: Career Center, SABSC, Residential Life After an Event Survey  Immediate response  After a concert, meeting, or dance

20. Activity: Choosing a Method Use your assessment project/idea What are the outcomes you are assessing? What is the best way to gather this information? Why? Report out to the group

21. Analyzing and interpreting your data requires as much diligence and strategy as the design process

22. Analysis  Read the data with an eye for themes, categories, patterns, and relationships  Have multiple people read the data and discuss the key themes  Identify contradictions surprises Interpretation  Make assumptions, add meaning, and come up with conclusions – keeping your own assumptions and beliefs visible.  Do not disregard outliers: Data that is surprising, contradictory, or puzzling can be insightful Source: InSites 2007 Qualitative Data

23. Quantitative Data  Campus Labs - sort your data, use crosstabs, and view it in graphs or pie charts.  Campus Labs does not help you analyze or interpret your data!  Your responsibility is to describe the data as clearly, completely, and concisely as possible. Interpretation  For each learning outcome, compare the results with the level of intended outcome. What does the data show? Analysis

24. Reporting Assessment Results In what ways have you seen assessment results reported? Source: Campus Labs

25. Why Focus on Reporting Results?  Role Modeling  Buy-in  Historical Documentation  Evidence trail  What about sensitive data and/or campus politics? Source: Campus Labs

26. Formats for Audiences Students Email invitations PR campaign (flyers, newspaper, TVs) Student government meetings Staff Roadshows Brief emails Newsletters Retreat Full reports Internal campus partners Cabinet meetings Elevator speech Exec. summary Annual reports External Constituents Presentations Website Press releases Source: StudentVoice

27. Policymakers are more likely to read information if: Information is in short bulleted paragraphs, not large blocks of type Charts or graphs are used to illustrate key points If provided in print rather than electronically Recommendations and implications are presented Source: Campus Labs

28. When do you need to report your results? Who is the audience of the report? Why is this information important to this audience? What are your options for reporting the results? What is the best method for reporting your results? What exactly should be or needs to be included? Questions to Consider when Planning a Report

29. Basic Structure of Assessment Report 1.Executive summary 2.Purpose of assessment 3.Methods 4.Description of participants 5.Findings 6.Discussion/implications and conclusion

30. Questions/Discussion