2.3.3 Learning From Our Assessment Through Chalk and Wire Jessica Dawes and Cindy Harvel Chalk & Wire Chicks Did you hear about that new ePortfolio 2? Yeah, it sounds GREAT!

Or…

Learning From Our Assessment Through Chalk and Wire Jessica Dawes (Dutchess of Rubrics) and Cindy Harvel (Queen of Standards)

Or…

Palmetto, Practice & Performance Princesses

Understanding the Task Had 3 years between implementation and NCATE visit – important to NOT change the TOC! Document when rubrics are revised & keep old rubrics ePortfolio Rubricmarker Desktop Reporter CWAssess Training Ideas 1) Give a 90 minute training session to students 2) Leave training with a portfolio and a consent form (legal to simply send) Some scanned printed form and sent.

Implementation of Chalk & Wire by Oral Roberts University  Faculty were part of the implementation process.  They Defined exactly how faculty were supposed to assess student work.  Assessed progress of students through peer reviews, self reviews, tests, and observations.  Campus-wide implementation happened by…  Organizing retreats with faculty  Selecting a C&W liaison for each department  Training more students  Training all computer personnel and library workers.  Only required artifacts that were needed. Up to THREE artifacts maximum on each standard.  Used dispositions on Chalk & Wire with a rubric and had advisors grade.  Set specific deadlines for student work to be turned in and for assessors to grade.

Standards and Artifacts Our Standards need to be linked to each criterion in each rubric I’ve done this with our pilot information to show you what is possible 3 artifacts per standard makes a pattern Our Table of Contents needs to be the stable force that holds our data together The Table of Contents needs to be consistent for several years in order for data to be valid

Rubrics:  Adoption of a scale  We have a 1- 4 scale.  Adding NA (non-applicable) would be helpful for many assessors who feel that a certain criteria doesn’t exactly fit.  NCATE wants to see credibility of scoring  We need to be able to show what a score of a 1,2,3 & 4 looks like.  Dispositions and a C&W Consent Form were placed on C&W and Rubrics were used for assessment by an advisor.

Tables of Contents:  Variety of organizational structure for TOC’s  Used standards, courses, and program structure for organization.  Some Tables of Contents were set up for…  Administrative Elements  Dispositional Assessements  Data-based Decision Making  Accountability

How well are we doing? Percentage of students who are meeting their learning goals to each specific level of our scoring.

How good is our assessment system? We have used the INTASC standards, coupled with our 3 C’s to evaluate. Our conceptual framework seemed to be on par with the other universities present.

Are the rubrics set up correctly? This report shows how consistently criteria have been assigned across all the rubrics in the system.

Are the rubrics linked to standards? For every standard, this report shows every class, and every criteria within that class that has been linked to this standard.

Do our rubrics really measure performance? Using this report, you could for example compare assessments of coursework with subsequent assessments of practicum performance to see if the one predicts the other. Regression analysis and scatter diagrams will soon be added to this report.

Is each portfolio clear and easy to navigate? Deep portfolios are hard to navigate

Are artifacts being submitted correctly? This report provides an overview of which artifacts have been uploaded and what proportion have been submitted, based on the rubric structure.

Are we collecting good demographics? In light of what we can compare on CWREPORTER, we may want to reassess our questions.

Is feedback prompt and comprehensive? Blank scores after a submission indicate assessment is lacking. These tend to be consistent as you browse through rubrics. Certain faculty consistently have missing grades which may indicate further training is needed.

Is assessment accurate and fair? Is there any course in which the artifact is reassessed by someone else? Some universities present sent certain key selected artifacts to a different university for reassessment by a non-biased party.

Where should we focus our efforts to improve school performance? I opened the standard report Mean and Standard Deviation by Standard. We should look for areas where performance is lower than average. If the variation is high, these may indicate inaccurate assessments.

Once we see a standard area has issues, “drill down” into it to see more details. In this case, the low percentage under planning comes from several rubrics which didn’t allow a 4 in this criteria.

I can see a problem area, how do I identify the cause? Choose rubrics we want to reassess. Below (you cannot see it on this screen) you can allocate another assessor. Existing rubrics should not be altered within the C & W system once assessments have been made.

Is it the students? For example, for PEL, do years in education affect how they meet the Standards? To identify which groups are likely to perform less well, we look at the standard report section and select the topic and question we have asked in the demographic survey or other survey.

Is it the teaching? Choose an assessor’s name in the explore section of CW Reporter. That will show all the assessments made by that individual.

 Setting Up a Support System  Help Desk Staff & Lab  More student workers/ tech support. Tuition waivers were used for students.  More training for students and faculty on eP 2 and RM2.  On-line comment cards to assess help received by students.  Adding to Our Support System  Camtasia: on-line tool to walk students through the C&W program.

Implementation Possibilities and Hurdles to Consider Consider putting all data with Kathy’s ACCESS data Put all standards in TOC – can link each rubric to an INTASC standard and use CWReporter to display Phase 1 – Pilot began with gates Phase 2 – First year used Conceptual Framework Phase 3 – Keep the Conceptual Framework and link it all to INTASC standards for reporting. Consider 4 year optional password payment plan

Implementation Continued Faculty Don’t grade twice! 2 week deadline to assess work Have on hand or use for workshops: How To Develop A Personal Portfolio (book about how to write reflections) and Portfolio and Performance Assessment In Teacher Education by Dorothy M Campbell Training Team – our most valuable asset Need incentives Class field credit Account for free C & W t-shirt Food!