Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessment Rubric for Critical Thinking Rubric Validation Process Second Workshop Quality Enhancement Plan QEP Team and Faculty Champions.

Similar presentations


Presentation on theme: "Assessment Rubric for Critical Thinking Rubric Validation Process Second Workshop Quality Enhancement Plan QEP Team and Faculty Champions."— Presentation transcript:

1 Assessment Rubric for Critical Thinking Rubric Validation Process Second Workshop Quality Enhancement Plan QEP Team and Faculty Champions

2 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan2 Authentic Assessments  Authentic assessments serve dual purposes of  encouraging students to think critically and  providing assessment data for measuring improved student learning.  These assessment techniques fall into three general categories:  criterion-referenced rubrics,  student reports (reflection or self-assessments), and  student portfolios.

3 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan3 Rubrics What is a rubric?  Scoring guidelines, consisting of specific pre-established performance criteria, used in evaluating student work on performance assessments

4 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan4 Criterion-referenced Rubrics  Complex, higher-order objectives can be measured only by having students create a unique product, whether written or oral, which may take the form of in-class essays, speeches, term papers, videos, computer programs, blueprints, or artwork (Carey, 2000).

5 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan5 Rubrics SPC currently uses rubrics in such programs as…  College of Education  College of Nursing  Paralegal

6 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan6 Assessment Rubric for Critical Thinking  A global rubric template developed to provide a snapshot view of how student learning is being affected by the critical thinking initiative.  Designed to be flexible enough to address a number of student project modalities including written and oral communications.  Will evaluate the student’s use of critical thinking skills in the development of the paper as opposed to specifically evaluating the quality of student’s writing skills.

7 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan7 Assessment Rubric for Critical Thinking  Development of a rubric is an iterative process and will be improved and strengthened as it is used more widely

8 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan8 Assessment Rubric for Critical Thinking ARC was designed by the QEP staff and the Faculty Champions to…  Enhance the QEP  Align with the College’s definition of critical thinking  Be flexible for use in multi-disciplines

9 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan9 Rubric Development Process 1.Re-examine the learning objectives to be addressed by the task  2.Identify specific observable attributes students should demonstrate  3.Describe characteristics of the identified attribute  4.Write narrative descriptions for each level of continuum  5.Collect samples of student work  6.Score student work and identify samples that exemplify various levels  7.Revise the rubric as needed  Repeat as Needed

10 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan10 Assessment Rubric for Critical Thinking

11 December 12, 2008 Quality Enhancement Plan11 Assessment Rubric for Critical Thinking

12 December 12, 2008 Quality Enhancement Plan12 Assessment Rubric for Critical Thinking

13 December 12, 2008 Quality Enhancement Plan13 ARC Assignment Profile  ARC Assignment Profile is designed to provide consistency and accuracy in the evaluation of the ARC at the institutional level as well as provide guidelines for the use of the assessment at the course level.  For a tool to be effective it must be used in the correct situation or ‘job.’ The ARC is essentially a ‘tool’ to evaluate critical thinking, but for a tool to be effective it must be in the correct situation or ‘job.’  The purpose of the ARC Assignment Profile is to outline the most appropriate course assignment.

14 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan14 ARC Assignment Profile 1. Participating faculty should have one assignment during the course that can be evaluated using the ARC scoring rubric. The course assignment could be a graded homework assignment or a major assessment for the course.

15 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan15 ARC Assignment Profile 2. The course assignment for the ARC should include all of the elements of the rubric and should be aligned with the task outlined for each element. Assignments that only evaluate some of the elements or are not aligned with the specific ARC tasks will be considered incomplete and not used in the institutional analysis.

16 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan16 ARC Assignment Profile 3. Faculty may add additional discipline specific rubric elements (such as grammar and punctuation in a composition class), but must maintain the ARC elements as listed.

17 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan17 ARC Assignment Profile 4.Students should be provided a copy of the assignment rubric (ARC and any additional discipline specific elements). The specific elements and tasks include:  Communication: Define the problem in your own words.  Analysis: Compare & contrast the available solutions within the scenario.  Problem Solving: Select one of the available solutions and defend it as your final solution.  Evaluation: Identify the weaknesses of your final solution.  Synthesis: Suggest ways to improve/strengthen your final solution (may use information not contained within the scenario).  Reflection: Reflect on your own thought process after completing the assignment.  “What did you learn from this process?”  “What would you do differently next time to improve?”

18 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan18 ARC Assignment Profile 5. The evaluating scenario (selected or created) should be stated in such a manner to allow the student to address each of the tasks. The QEP team is willing to assist with the creation of the scenario or identify possible sources of existing scenario that could be used.

19 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan19 ARC Assignment Profile 6. At the end of the semester, please send the completed student assignments to the Janice Thiel, QEP Director, TE (X3110). Completed student assignments should include a copy of the scenario, the assignment provided to the student (with the rubric), the students work and the final graded rubric.

20 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan20 ARC Assignment Profile Competency (KSA)Problem with Multiple Solutions Premise with Multiple Perspectives Communication Define Problem Define Premise Analysis Compare & Contrast Solutions Compare & Contrast Alternative Perspectives Problem Solving Select & Defend Final Solution Select & Defend Final Perspective Evaluation Identify Weaknesses Final Solution Identify Weaknesses Final Perspective Synthesis Suggest Improvements Final Solution Suggest Improvements Final Perspective Reflection Reflect on Thought Process Reflect on Thought Process

21 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan21 Sample Scenario (Deer) Three teenagers were seriously injured in a car accident when swerving to avoid a deer on a two-lane road near a small, rural town in Florida. The residents of the town have seen more and more deer enter the town’s populated areas over recent years. Local law enforcement has been called numerous times this year to remove the animals from backyards and neighborhood streets, and one deer even caused considerable damage as it entered a restaurant in town. The mayor has been charged by the city leaders to keep the town residents safe.

22 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan22 Sample Scenario (Deer) Local crops have even been damaged by the animals. Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population. One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas. Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.

23 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan23 Sample Scenario (Deer) Possible Solutions: 1.Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population. 2.One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas. 3.Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.

24 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan24 Rubric Development Process 1.Re-examine the learning objectives to be addressed by the task  2.Identify specific observable attributes students should demonstrate  3.Describe characteristics of the identified attribute  4.Write narrative descriptions for each level of continuum  5.Collect samples of student work  6.Score student work and identify samples that exemplify various levels  7.Revise the rubric as needed  Repeat as Needed

25 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan25 ARC Scoring Workshop Process 1.After the completion of this PowerPoint Presentation, the workshop will begin with introductions from the participants 2.Workshop participants will be provided the ARC as well as scoring worksheets. Additional instruction will be provided on the scoring process. 3.A sample test item will then be presented on the screen, and various responses will be discussed and scored based on the scoring rubric given for that specific item. 4.Each scorer will then review the response provided for the first item on his/her first assessment, and scored it based on the scoring rubric. This process will be repeated for each of the five items on the assessment.

26 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan26 ARC Scoring Workshop Process 5.Scorers who encountered a response which did not clearly follow the rubric will discuss the response with the group for clarification. 6.Each scorer will then passed the scored assessment to their scoring partner, and the same assessments will be scored by the second scorer. 7.In the event that two scores differed significantly, the facilitator will provide the assessment to a third scorer, and a third score will be recorded. 8.When all scoring for an assessments is completed, the assessment will be provided to the facilitator.

27 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan27 ARC Scoring Workshop Process 9.Finally, steps 1 through 8 will be repeated for each assessment as time allows. 10.Workshop participants will complete the ARC Validity and Reliability Form at the end of the workshop. 11.Interrater reliability will also be calculated from ARC ratings after the completion of the workshop. 12.Rubric results will be reevaluated after each administration, and additional refinements and modifications may be made to the instrument as the assessment development and validation is intended to be an on-going dynamic process designed to provide the very best indicator of a student’s skills.

28 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan28 Validity and Reliability

29 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan29 Validity  Does the Rubric measure what it is suppose to measure?  “Validation is the process of accumulating evidence that supports the appropriateness of inferences that are made of student responses…” (AERA, APA, & NCME, 1999)

30 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan30 Validity 1.Consequences: The effects of the assessment 2.Content Coverage: Comprehensiveness of assessment content 3.Content Quality: Consistency with current content conceptualization 4.Transfer and Generalizability: Whether assessment is representative of a larger domain 5.Cognitive Complexity: Whether level of knowledge assessed is appropriate 6.Meaningfulness: The relevance of the assessment in the minds of students 7.Fairness: Fairness to members of all groups 8.Cost and Efficiency: The practicality or feasibility of an assessment

31 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan31 Validity Consequences The effects of the assessment 1.Is the assessment likely to produce results that will be used to improve instructional programs or otherwise improve student learning? Content Coverage Comprehensiveness of assessment content 2.Does the assessment comprehensively cover the content and processes assessed? 3.Is the content covered in sufficient breadth and depth? 4.Does the assessment represent important (not trivial) components of the content? 5.Together, will the assessments provide sufficient evidence about the content?

32 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan32 Validity Content Quality Consistency with current content conceptualization 6.Is the assessment consistent with the best available conceptualization of the knowledge or skill assessed? 7.Does the assessment represent current, rather than outdated, perspectives? Transfer and Generalizability Whether assessment is representative of a larger domain 8.Can the assessment results be generalized to the broader domain (knowledge, skill, or learning outcome) they are intended to represent?

33 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan33 Validity Cognitive Complexity Whether level of knowledge assessed is appropriate 9.Do the assessment tasks or questions represent the cognitive complexity of the knowledge or skill that it is intended to assess? (For example, if an outcome includes higher order or critical thinking skills--such as problem solving or synthesis--does the assessment measure them?) 10.Does the assessment actually require students to use higher-level knowledge or skills, or can students simply respond from memory without having to think? Meaningfulness The relevance of the assessment in the minds of students 11.Are assessment items or tasks meaningful to students? 12.Is the assessment relevant to problems students will encounter again in school, work, or daily living? 13.Does the assessment provide students with worthwhile or meaningful experiences?

34 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan34 Validity Fairness Fairness to members of all groups 14.Is the assessment biased against students who are members of various racial, ethnic, and gender groups or students with disabilities? Does it contain stereotypes of any groups? 15.Do students of similar ability, regardless of group membership, score the same? Cost and Efficiency The practicality or feasibility of an assessment 16.Is the assessment a reasonable burden on teachers, instructional time, and finances? 17.Is resulting information worth the required costs in money, time, and effort?

35 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan35 Reliability  Consistency of the assessment scores  Types of reliability…  Interrater Reliability – scores vary from instructor to instructor.  Intrarater Reliability – scores vary from a single instructor from paper to paper  A test can be reliable and not valid, but never valid and not reliable

36 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan36 Reliability Concerns Reliability 1.Are the score categories well defined? 2.Are the differences between the score categories clear? 3.Would two independent raters arrive at the same score for a given student response based on the scoring rubric?

37 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan37 Improving Scoring Consistency  Provide rubric to students prior to assessment  Anonymous papers  Anchor papers defining levels of proficiency for reference  Use of multiple scorers  Interrater reliability statistics during training and grading

38 Assessment Rubric for Critical Thinking December 12, 2008 Quality Enhancement Plan38 Next Steps  The new faculty champions will administer coursework using the ARC rubric within their programs during the Spring semester  Faculty Champions will use the ARC assignment profile to ensure consistency  Process will be repeated (Steps 5 - 7)

39 Assessment Rubric for Critical Thinking Rubric Validation Process Second Workshop Quality Enhancement Plan QEP Team and Faculty Champions


Download ppt "Assessment Rubric for Critical Thinking Rubric Validation Process Second Workshop Quality Enhancement Plan QEP Team and Faculty Champions."

Similar presentations


Ads by Google