1. Where have we been? Where do we go from here? Spring 2012 http://www.mus.edu/writingproficiency/index.asp

2. Why MUSWA?

3. College Readiness: A National Concern The laments about America’s higher education system are long and loud………….. And remedial education -- the ‘catch-up’ work now required for the nearly 40 percent of students who come to college lacking basic skills needed to succeed -- is a prime candidate for elimination on almost everybody’s list. Because colleges have not clearly articulated the skills that students must possess to be college-ready, students are blindsided when they are placed into remedial courses, and high schools don’t have a clear benchmark for preparing students for success. Jane Wellman and Bruce Vandal, Inside Higher Education, 2011

4. Writing Proficiency Policy A. Any student seeking full admission to a four-year degree program… must earn a minimum score of: 7 on the Writing Subscore or 18 on the Combined English/Writing section of the Optional Writing Test of the ACT; or 7 on the Essay or 440 on the Writing Section of the SAT; or 3.5 on the Montana University System Writing Assessment; or 3 on the AP English Language or English Literature Examination. D. A student who has not yet demonstrated the ability to meet these standards may be admitted (without condition) to a two-year degree program or admitted provisionally to a four-year degree program on any campus of the Montana University System. E. Before gaining full admission status to a four-year program, the student may prove that he/she has the appropriate proficiency in the following ways: 1) retake one or more of the listed writing assessments to earn the required score; or 2) within 3 semesters, earn a grade of C- or better in the composition course that is the prerequisite to the composition course that satisfied the general education program requirements described in Board Policy 301.10.

5. MUSWA Turned Policy into Practice Defined College Readiness in Writing Provided high school students with information about placement into college-level or developmental courses Provided feedback to HS staff on curriculum and instruction for the transition to college Established a forum for K-12/Higher Ed collaboration Provided ongoing professional development in writing Gave students & high schools an avenue for celebrating student success in writing

6. Evolving

7. The Result? Improved Achievement This graph shows percent at or above Proficient level and includes data on ALL students, not just those with college aspirations. N = 3,365 in 2001; 7,685 in 2011

8. The Outcome? Reduced Remediation in College Composition Actual Placements

9. MUSWA: Sustained Over Time 2001 First Grader 2011 Junior Holden Pepprock, Shelby High School, Earned a “6” on 2011 MUSWA

10. The MUSWA Evolution 2001 2011 7,685 students tested 138 high schools 320 scorers 8 regional sites 3.9 average score 1.7% earned “6” 75% scored college-ready 77% of tests word- processed and submitted online (2% handwritten) 3,365 students tested 73 high schools 96 scorers 3 regional sites 3.0 average score 0.6% earned “6” 37.8% scored college-ready 100% of tests handwritten

11. MUSWA in Three Modes

12. MUSWA Adapts to Change From ACT to MUSWA Expansion & Refinement 2001-2003 Training by ACT Qualifying Set 2004 Training of Trainers Calibration Set Strengths and Weaknesses College Credits 2005-2006 Online testing Mixed prompt packets 2007 Eight scoring sites Over 300 scorers 2008 Consensus Set Learning > Scoring 2009 The AHA! Essay 2011 Common Core

13. Hundreds Participate 300 to 370 workshop participants give two days to MUSWA each year. (Renewal Units, College Credit) 40-50 Trainers give two days to preparation and two days to workshops each year. Benefits: Improved teaching skills and confidence Useful tools—prompts, rubric, training process Important role of MUSWA for students, teachers, state Collaboration among teachers Personal regeneration from stimulating conversations

14. How do we score?

15. Data in Useful Formats WEBSITE for score retrieval: www.muswa.com with password:www.muswa.com Distribution tables for state, school, classroom Student scores (with S & W) by teacher and class Spreadsheet, by student, with all data Individual memos, by student, to parents explaining score MAILINGS with: AwardsScores honored in Transcript labels UT, WA, ID, ND & SD Newsletters

16. MUSWA Recognizes Excellence Awards of Merit for Schools in Top Quartile Letters of Recognition for students with scores of 6 and 5.5 Poplar High School Won Awards of Merit in 2009 and 2010 In 2011, nine American Indian students received Letters of Recognition for earning scores of 5.5 or 6.

17. Ensure accuracy, reliability, validity Ensure accurate scoring Does EVERY score match the rubric? Is EVERY writer scored fairly? Ensure reliable scoring Would you give the same score later? Is everyone giving about the same score? Ensure that the samples are valid Were they written in class by the student? Did the student use a suitable mode?

18. Question Features of Writing Does this essay address the prompt? What is the organizational pattern and is it logical, coherent, appropriately sequenced? Is this “a little” elaboration or “some” elaboration? Is this “precise” word choice, or is it even appropriate? How does an essay generated in a testing environment differ from one produced as an assignment for a particular classroom teacher?

19. Scoring Protocol Refer to the rubric often Compare to anchor papers Discuss questionable scores Weigh strengths and weaknesses Print and bubble scores carefully and accurately For online tests, print and bubble test number Write score at bottom of essay itself

20. Strength and Weakness Data Purpose: Provide feedback to schools and students Procedure: Mark a strength or weakness that impacts the score (that feature keeps the score at “x” or drops/raises the score to “x” Mark a strength or weakness to “set aside” a feature that prevents you from scoring other features fairly Both scorers may bubble in, but they must not be contradictory

21. Take care with score sheets If student didn’t do it, bubble in prompt number If printed prompt is wrong, bubble in right number Bubble in Solution: 1, 2, or 3 (other) All score sheets must have Reader 1 & 2 Reader 3 (resolver) bubbles in 2 scores All readers share Comments: Strength and Weakness Online score sheets MUST show two matching test #’s After scoring, scorers may check for agreement Table leader should check for accuracy and discrepancies, then hold discussions outside— particularly early in the scoring process

22. Keep Materials Neat Keep score sheets with tests until information and scores are final (use paper clips) Ensure score sheets are scanner-ready Completed neatly and correctly Stacked with same orientation Stack tests by score for easy research Collect complete, ordered training materials: ready for next scoring site

23. Remember: Each essay represents a student What strengths does this writer demonstrate? How does this essay reflect the rubric at score X? At score Y? How would you help this writer improve? Does this writer demonstrate the capacity to succeed in a college-level composition course?

25. MUSWA Faces the Future In 2013, all of Montana’s juniors will have the opportunity to take the ACT Plus Writing, paid for through new GEAR UP funding. In 2014, all of Montana’s juniors will be required to take Smarter Balanced Assessment Consortium tests. Can a professional development program modeled after MUSWA be designed and funded?

26. Comparing Writing Tests MUSWA Choice of 2 prompts Choice of handwritten, word-processed, online 40 minutes One class period, chosen by teachers Scored collaboratively by Montana teachers ACT One prompt Handwritten only 30 minutes After 4-hour MC test on a state test date Scored individually by paid scorers on computers

27. Put the Writing Summit on Your School Calendar! September 23-25, 2012 Red Lion Colonial Inn, Helena Co-sponsored by MATELA & Title II Featuring social event, luncheons, banquet, speakers such as Carol Jago (CA), Kathleen Blake Yancey, Donna Miller, Beverly Ann Chin, and others Presentations from MUSWA trainers and teachers on writing instruction, writing research, writing assessment, college readiness, and Common Core.

28. Participate in a New Writing Program The MUS Writing Alternative A limited number of students take the MUSWA, providing student samples for Writing Assessment Workshops across the state. Integrating Literacy in Science and Technical Subjects Schools form teams of English, science, and library media teachers to learn, through an online class, readings, and/or workshops to teach students to read and write arguments developed with relevant data and sources. Teams would convene to score these papers, much like a science fair, but based entirely on student writing. Arguing with Statistics Teachers would take a course to help students select appropriate statistics to analyze and solve an economic, policy, or social problem. Their arguments would be scored by groups of teachers.

29. Can you help us plan? Will you participate in the Writing Summit? Would your school participate in another kind of Writing Assessment/Professional Development effort? Would you school take the MUSWA, in addition to or as an alternative to Plus Writing?

30. Contact Jan Clinard, Ed.D. The University of Montana Helena 1115 North Roberts, Helena, MT 59601 406-444-0652 Jan.clinard@umhelena.edu http://www.mus.edu/writingproficiency/index.asp On Facebook: MUS Writing Assessment.com