Presentation on theme: "Assessing Student Learning Outcomes in Writing Courses Lisa Marchand Cosumnes River College Sacramento, California."— Presentation transcript:
Assessing Student Learning Outcomes in Writing Courses Lisa Marchand Cosumnes River College Sacramento, California
ACCJC Accreditation Standards California Community Colleges are required to establish well-documented, ongoing processes for evaluating effectiveness at all levels of the institution. The foundation of this evidence must be data collected and analyzed in a consistent way, over time.
By the Year 2012 Engage in an ongoing dialog about student learning Have a well documented process for evaluating Student Learning Outcomes Show tangible evidence that your program is continuously moving toward improved student learning.
Think about your own classes or program… How do you assess students writing skills? Are Student Learning Outcomes in place? Are they measurable? How are they measured? Take five minutes to talk it over!
Cosumnes River College Questions: How well can students demonstrate that they have met the Student Learning Outcomes for writing courses that they have passed? How well can students who have placed into a level through the assessment process demonstrate the skills specified in the prerequisite courses SLOs?
Existing Diagnostics Not specifically aligned to course SLOs (Either too general or too specific) Mining data from these diagnostics would be too difficult
Concerns We wanted to assess SLO skill sets but we didnt want to have these assessments affect the students grades. We didnt want to have the SLO assessments be used to evaluate the teachers in any way. Assessing SLOs at the end of a course wouldnt provide the best information on students who entered through the placement process.
Our Goal: Sustainable Continuous Quality Improvement Analyze Data and Discuss Observations Measure Student Learning Outcomes Make Adjustments
Ramping Up Develop student learning outcomes Define metrics to measure SLOs Experiment with various measures and discuss observations
Analyze SLOs to Identify Metrics Measurable elements of writing include: Syntactic structures and variety Accuracy of verb tense form and use Appropriate word forms Range of vocabulary Rhetorical patterns Development Use of cohesive devices Other
…include measures of meta skills Meta-linguistic skills are included in SLOs Examples: Identify X Label X Analyze X Recognize X Differentiate between X and Y An objective instrument can provide discrete measures of each of these
Build a Scoring Guide CRC chose an ETS style holistic scoring guide model with 5 levels: Insufficiently prepared = 1 Under-prepared = 2 Fairly prepared = 3 Well prepared for the level = 4 Demonstrates skill above level = 5
…write descriptors for each level Start with the SLOs – write a descriptive narrative to match expressive skills they describe This will be the score 4 – well prepared for the course To write descriptors for scores 1 – 3 and for score 5, examine the sample papers already marked by faculty
Select A Writing Prompt For the lowest levels, we chose a picture and guiding questions: Look at the picture. Who is the man? Where is he? What happened? How does he feel? Why does he feel that way? Where is he going? What will happen next? Write the story! For intermediate SLOs and beyond, we selected this prompt: Some people like to live in big cities, others do not. Tell where you prefer to live and explain why.
Validity Check According to Test Better, Teach Better by W. James Popham, content-related evidence of validity can be determined by relying on human judgement A team of expert colleagues agrees that each test item measures the skill it was designed to reflect. The team must also agree that the test items are accessible and free of bias.
Check for Clarity Conduct item-by-item study of the test Be sure that the language level matches the texts Be sure that vocabulary is appropriate and standard Be sure the content doesnt unfairly affect a students ability to respondit is not biased.
Check for Bias Content Bias Gender Culture Experiential Linguistic Ambiguity Vague wording Confusing grammar Low-frequency vocabulary Difficulty
Create a Score Tally in Excel student id SLO's 1 - 4 Holistic Score SLO #1 Evidence of Planning SLO #2 Logic and Develop ment SLO #4 Syntax SLO #3 Mechanic s Identifyin g Sentence Parts SLO #5 Items 1-7 (7Q's) Identifyin g Parts of Speech SLO #6 Items 8- 14 (7 Q's) Verb Tense Forms SLO #7 Items 15- 20 (6 Q's) Verb Tenses in Question s SLO #2 Items 21- 24 (4 Q's) 475549423225412 485646222124443 922766322224464 987991322215653 1020104323322332 1036656221114342
…include Columns for Student ID #s Holistic Writing Score Scores for component skills evaluation Scores from each section of the objective assessment
Analyze Data Collect descriptive statistics Make correlations Focus on evidence of student learning Provide data sets to all teachers for use in self-studies and program review Use the assessments over several semesters to enable longitudinal study of a significant sample of student work
Continue the Dialog Use collective observations to suggest and implement changes to Curriculum Methods Materials Select appropriate data to support resource requests like facilities and FTE Have faculty experienced in the process mentor and norm their colleagues
Other than ACCJC Standards, Why Assess SLOs? Find out what youre teaching that the students arent learning. Move faculty dialogue away from anecdotal discussion Confirm intuition (or not) Gather substantive support for resource requests
Example SLOs SLO 1: Identify and apply basic essay components SLO 2: Employ the writing process SLO 3: Interpret and respond to writing prompts SLO 4: Develop paragraphs appropriate to rhetorical mode SLO 5: Identify and correct grammatical errors through editing SLO 6: Write simple, compound, and complex sentences with correct punctuation SLO 7: Identify and correctly use grammar structures and word forms
Example of an SLO with Metrics SLO 7: IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS identify and use all basic verb tenses with very rare errors in form (simple and progressive aspect of present, past and future) identify and use some perfect verb tenses with occasional error in form and usage (present perfect, present perfect progressive, past perfect) identify and use singular, plural and non-count forms and appropriate quantifiers and articles without errors in form identify parts of speech and use correct word forms in writing
W 50 SLO#7 IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS
Convert to a Comparable Form Do some simple arithmetic: Divide the total possible into 100 100÷5 = 20 Multiply the answer by the average score 20 x 2.18 = 43.6%
W 50 SLO#7 IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS
Observations from the Data Neither the students ability to correct grammar errors on the objective assessment nor the teachers evaluation of students grammar in the writing sample were significantly correlated with their success on the holistically graded writing sample. There was no significant correlation between students writing sample and the objective assessment.
Significant Correlations The students holistic writing scores were directly correlated with the score the rater gave them on their essay organization. The students holistic writing scores were directly correlated with the score the rater gave them on their essay development.