Presentation on theme: "Implementing a Standardized Performance Test of Writing."— Presentation transcript:
Implementing a Standardized Performance Test of Writing
The Swimmers Raymond Moy – University Director of Assessment. 2 ½ years at CUNY, 25 years at the United Nations in language proficiency testing and organizational development. Melissa Uber – University Director of Testing. 23 years at CUNY – 12 at Hunter College and 5 at New York City College of Technology.
CUNY: Who are we? The University is comprised of 23 institutions: 11 Senior Colleges 6 Community Colleges (a 7 th opens Fall 2013) An Honors College, the Graduate School and University Center, the Graduate School of Journalism, the School of Law, the School of Professional Studies, and the School of Public Health. Fall 2010 undergraduate enrollment 228,484: 58.6 % are female 28.4 % are 25 years or older 43.6 % have a native language other than English 57.3 % are Pell Grant Recipients 44.2 % are First Generation in College
College Readiness: Points of View Open admissions High schools want more liberal standards College faculty want tighter standards Tying budgets to enrollments Administration wants more liberal standards Governmental funding bodies want tighter standards Raising the quality of academic programs Requires clear understandable standards
CUNY Assessment Testing: A History CUNY’s mission to make quality higher education accessible to all New Yorkers Admission standards liberalized to admit all students with a NY state high school diploma in 1970’s CUNY Board of Trustees mandates evidence of college proficiency by 60 th credit beginning in 1976 Political concern about declining academic standards and ineffective use of public funds.
A Framework Emerges: Annual audits to ensure college faculty were grading to the rubric Certification training to qualify faculty as scorers Establishment of borough grading sites to ensure standardization of grading University Testing Database and Scheduling System deployed
Placement vs. Exit Single score did not provide adequate feedback to developmental students or faculty Test task was artificial and did not resemble the writing expected in college Students wanted instructors to teach to the test and test-taking techniques
Test Development Phases From the Standards for Educational and Psychological Testing, AERA, APA, & NCME (1999): 1. Delineation of the purposes(s) of the test and the scope of the construct or the extent of the domain to be measured; 2. Development and evaluation of the test specifications; 3. Development, field testing, evaluation, and selection of the items and scoring guides and procedures; 4. Assembly and evaluation of the test for operational use.
Requisite Skills for Freshman Composition Reading Read with understanding at minimum level of the class Summarizing Summarize text Summarize and use a text Summarizing: recognizing viewpoint, thesis Critical thinking Responding to content Ability to compose a thesis statement that answers the question State ideas, describe experience and reflect on it. Critical thinking: reading, understanding, responding Distinguishing fact from opinion Supporting ideas Support with evidence from a text Organization Overall organization (beginning, middle, end) Organizing: beginning, middle, end Move from general to specific and reverse Verbal accuracy Edited English Reflect on one’s own writing: revise and edit
Measurement Construct Understand and critically evaluate the ideas in a text Use ideas in the text to support and develop student’s own ideas Organize a focused response that demonstrates a clear and coherent progression of ideas Choose sentence structures and vocabulary for clarity and precision Adhere to the conventions of English grammar, usage, and mechanics
CATW Score Scale
Test task design Assignment Begin by reading the passage below. Hype Advertisements are the most prevalent and toxic of the mental pollutants. From the moment your alarm sounds in the morning to the wee hours of late-night TV, commercial pollution floods your brain at the rate of about three thousand marketing messages per day. Every day an estimated 12 billion display ads, 3 million radio commercials, and more than 200,000 TV commercials are dumped into North America’s collective unconscious. The increase in commercial advertising has happened so steadily and relentlessly that we haven’t quite woken up to the absurdity of it all. No longer are ads confined to the usual places: buses, billboards, stadiums. Anywhere your eyes can possibly come to rest is now a place that, in corporate America’s view, can and ought to be filled with a logo or product message. You fill your car with gas, and there’s an ad on the nozzle. You wait for your bank machine to spit out money and an ad scrolls by in the little window. You drive through the countryside and the view of the wheat fields is broken at intervals by enormous billboards. Your kids watch Pepsi and Snickers ads in the classroom. A company called VideoCarte installs interactive screens on supermarket carts so that you can see ads while you shop. (A company executive calls the little monitors “the most powerful micromarketing medium available today.”) There is nowhere to run. No one is exempt and no one will be spared. In the silent moments of my life, I often used to hear Beethoven’s Ninth Symphony play in my head. Now I hear that kid singing the Oscar Meyer wiener song. Excerpted from Kalle Lasn. “Hype,” Signs of Life in the USA: Readings on Popular Culture for Writers, 4th ed. (Sonia Maasik & Jack Solomon, eds.) Boston: Bedford/St. Martin’s,
Writing prompt Writing Directions Read the passage above and write an essay responding to the ideas it presents. In your essay, be sure to summarize the passage in your own words, stating the author’s most important ideas. Develop your essay by identifying one idea in the passage that you feel is especially significant, and explain its significance. Support your claims with evidence or examples drawn from what you have read, learned in school, and/or personally experienced. Remember to review your essay and make any changes or corrections that are needed to help your reader follow your thinking. You will have 90 minutes to complete your essay.
Pilot Test Questions Do forms make a difference? Does text length make a difference? Are ESL students disadvantaged? Can readers score an analytic rubric as fast as a holistic rubric? Can readers score an analytic rubric reliably?
Results of the Pilot Test We had a 90% perfect or adjacent scoring agreement between raters on all test forms. For passage selection, length and readability are not as important as well-written passages with clearly stated main ideas. Students were comfortable with topic selections and the amount of time allotted Most important dimensions in determining level of writing are ability to organize a focused response and to choose vocabulary and sentence structures effectively Analytic scoring only takes slightly longer than holistic and can be just as accurate
Field Testing Development of parallel forms Text selection specifications Editor to manage the process Faculty contributions/review Testing with CUNY developmental students, ESL, and Freshman comp Standardizing grading Reader training and certification Anchor papers Practice papers Text summary and anchor paper articulations
Plot of Test Form Means
Distribution of Rater Difference Scores
Setting the Proficiency Standard Rubric performance level – 56 (which represents a 7 on each dimention) Score that maximizes the distinction between developmental and Freshman Comp students – 55 Panel of expert judges classifying borderline papers - 57
Implementation: the Logistics Data systems modifications Copyright clearance Certification training for faculty Informational sessions for all stake- holders Test prep materials Test administration manual Student score reports/rosters
Student Score Report
Class Roster Scores
Impact of the CATW Faculty buy in for the test Established a definition of college level writing – what is to be learned and what is to be taught High schools can be more focused on preparing their students Faculty have a unified goal for their writing courses Transferability of the proficiency requirement Clearer placement of students in developmental courses Identification of effective teaching practice CATW conference Better preparation for academic performance
CUNY Testing Website: More information regarding the CUNY Testing Program and the CATW can be found at: