Making Your Assessments More Meaningful Flex Day 2015

Why do we assess?  To study the learning process in the discipline and in the larger learning arenas and discover new ways and methods of teaching.  To determine the extent to which the curriculum is working (design and implementation of the curriculum).  To inform decisions as to where time, energy, and/or money should be repurposed for continuous improvement to learning.  To help demonstrate our quality assurance to the community we serve.

Today we’ll talk about…  Creating and/or revising your means of assessment to ensure that your results are more meaningful.  New ways to analyze your data.

Making your Means of Assessment more Meaningful Ensure Consistency Across Course Sections  When conducting assessment, the same assessment instrument should be used across course sections to assess an SLO.  If you use exam questions to assess: The same SLO questions should be used on the exams across all sections of the course.  If you use a rubric to assess: Everyone needs to use the same rubric. The instructions and task for the essay or skill performance used for the assessment need to be the same across sections.  Why?  You can’t account for as much variance if you don’t standardize the assessment. In other words, it becomes more difficult to tell why students scores differed on the assessment.

Making your Means of Assessment more Meaningful Ensure a Consistent Time Frame for Conducting the Assessments  Timing is important!  Plan for all instructions teaching sections of a course to give the assessment on approximately the same date.  For pre-test/post-tests: The pre-test should be administered in all sections on the first day of class. The post-test should be administered in all sections on the final day of class.  For end-of-semester exams: The exam should be given on the last day of class.  For essays or skill performances: The assignment should be administered during the same week.

Making your Means of Assessment more Meaningful Sampling Techniques: Sample Size  What is sampling?  The process of selecting a sample of individuals (students) from a population (course population or program population). Through sampling, you attempt to estimate what is likely to be the situation for the total population.  NOTE: There will always be a difference between the scores of your sample of students and the scores of the population of students for a course or program.

Making your Means of Assessment more Meaningful Sampling Techniques: Sample Size What if I asked you to flip a quarter three times?

Making your Means of Assessment more Meaningful Sampling Techniques: Sample Size  Your sample needs to be large enough so that you can be confident that the sample best represents your population.  When is doubt, the bigger the sample, the better.

Making your Means of Assessment more Meaningful Sampling Techniques: Sample Design  Your sample of students will be biased if the sampling is done in a non-random method.  Selection can be consciously or unconsciously influenced by human choice.  Selection can be biased if it is not equally likely for each student to get selected into the sample.  What can you do?  Choose a random sample of all course sections and all students enrolled in the course.  How can you do this?  Use Excel’s random number generator

Making your Means of Assessment more Meaningful Revisit your Exam Questions  SLOs have changed over the years and sometimes the questions used to test those SLOs should be revisited.  When you meet with the instructors who teach your course (or all the faculty in your program), ask the group…  Are we really testing students on all aspects of this SLO?  Have we included enough questions in the assessment to adequately measure the SLO?

Making your Means of Assessment more Meaningful Revisit your Rubric  A well-designed rubric should allow evaluators to efficiently and effectively assess their students’ learning.  When choosing a rubric for assessment make sure that  The rubric’s categories are well-defined and that each scoring level is consistent in it’s increments.  For assessment, the rubric needs to be extremely clear and leave little room for interpretation.

Making your Means of Assessment more Meaningful Revisit your Rubric

Analyzing your Data Ensuring Inter-Rater Reliability with Rubric Scoring  When using an assessment tool with a rubric it is important to achieve inter- rater reliability when scoring the assignment.  Inter-rater reliability: The degree of agreement in scoring among the faculty raters of the assignment.  How can you achieve this?  Submit a random sample of the student assignment used to assess the SLO.  Make multiple copies of a selected number of student submissions.  Pass out the same student’s assignment to all the raters and have them score it based on the rubric.  Discuss how everyone rated that student’s work and come to agreement on how it should have been scored. Repeat until everyone is on the same page. Then pass out the rest to be scored individually by each rater.

Analyzing your Data Digging Deeper into the Data  An overall percentage of how many students successfully met the level of achievement for an SLO will not tell you much.  Consider looking at:  Differences in results between course section offerings at different times of the day (morning, afternoon, night).  Differences in results between full and part-time faculty.  Differences in student populations (by overall success rates at the college, overall GPA, socio-economic status)  Indirect assessments combined with your direct assessments of learning.

How Often Should you Assess?  I recommend collecting data for as many SLOs, for as many courses, and for as many programs as often as possible.  The ASLO subcommittee recommends that you close the loop for each course SLO at least twice during the course review cycle. The same goes for program SLOs.  By the time you go through course review, all your course SLOs should have been assessed. Same for program review. You want the results to inform your reviews!