Presentation on theme: "Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951."— Presentation transcript:
Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951 and use PIN/code 240228 Parts of this presentation were adapted from materials created by: Noelle C. Griffin PhD, Assistant Director, UCLA Center for the Study of Evaluation Original material can be found at: http://www.lmu.edu/Page18044.aspxhttp://www.lmu.edu/Page18044.aspx
Housekeeping Maximize your CCC Confer window. Mute your phone (*6) if you have visitors or noise in your office. Please note phone audio may be in presenter-only mode. Ask questions over the phone when the presenter prompts. Ask questions throughout presentation via the chat window. Turn on or off Closed Captioning by clicking on the icon. Save the presentation or chat by clicking on the icon.
Agenda Background and Terms Planning your Evaluation Strategy - Defining your Goals, Objectives & Outcomes - Choosing your Measurements - Online Quizzes and Exams - Performance based assessments Analyzing data – rubrics
Background and Terms Enormous body of literature on this subject Download list of resources at http://nlight.com/ONE/evalsSurveys_resources.doc
Background and Terms Some Basics Assessments versus evaluations Reliability - repeated measurements are same or very similar Validity – you are measuring what you say you are
Background and Terms Kinds of Measures Quantitative: generates numeric data Qualitative: generates descriptive data
Background and Terms Collection Methods Survey, Quizzes, and Exams Interviews Observations Work Products
Background and Terms Authentic – Performance & Behavioral Assignments Projects – individual and group Behavioral observations –General Participation –Discussion contributions Portfolio
Planning Your Evaluation Strategy Define your course goals, learning objectives and outcomes Determine what kind data and measurement tools will best indicate how well your Learning Objectives & Outcomes are being met?
Planning your Evaluation Strategy Goals, Objectives, Outcomes Goals - Represent the overall mission Objectives - Broader perspective on what generally students should know, value or do Outcomes - What students should demonstrate/represent relative to objective
Why bother with Goals, Objectives, and Outcomes? They provide the basis for: –Course content –Evaluating students –Communications Students Other faculty and administration Outside world –Curriculum planning
Example Goal: The course will prepare students to become a rocket scientist Objective: Students will be able to understand the working components of a rocket and methods for a successful launch Outcome: Students will be able to build and launch a rocket.
Measurement Tool – Surveys, Quizzes & Exams Types of Items Quantitative (numbers) often closed ended (limited responses) True/False Multiple choice Ratings Single answer Qualitative (narrative & content) often open ended items Short answer Essay
Measurement Tool - Quizzes & Exams Examples of Closed Ended Items What is your gender? Male – Female What is your age ? (10-25) (26-40) (41-55) (55+) Which college do you attend? EVC Butte Foothill De Anza Who is your favorite artist of the 20 th Century? Picasso Joan Miro Marc Chagall Salvador Dali
Measurement Tool - Quizzes & Exams Examples of Open Ended Items Describe the stylistic elements of Art Deco. Why did Van Gogh cut off his ear? What is art? What is your life story until this point?
Measurement Tool - Quizzes & Exams Watch for Bias in Constructing Items How long ago did you stop using street drugs? Just this week A month ago A year ago Over a year ago
Measurement Tool Performance Assessments Portfolios Capstone assignments Performances/projects Internships/field activities Students evaluating students
Measurement Tool – Performance Assessments Potential Strengths Direct evidence of student learning Students motivated Realistic context Can be integrated into the grading process Data collection can be relatively unobtrusive to students
Measurement Tool – Performance Assessments Potential Weaknesses Additional development/coordination time Faculty trust/buy in critical Faculty time for scoring Reliability/validity uncertain Lack of norms/comparative data
Measurement Tool – Performance Assessments Portfolios What is it? A collection of multiple targeted pieces of student work that provide evidence of a learners achievements along a continuum of learning. Can be formative or summative (i.e., developmental vs. showcase) Can be online or old school
Measurement Tool – Performance Assessments Portfolio Pitfalls Kitchen sink approach Lack of pre-planning Retrospective re-creation Request vs. requirement (i.e., built into program) Under-estimating faculty time involved
Measurement Tool – Performance Assessments Capstone Assignment/Project What is it? A culminating project or assignment that provides evidence of achievement of learning goals; generally integrative in nature. Does not only have to be last year; can be spaced in terms of developmental milestones in the program providing longitudinal/comparative data
Measurement Tool – Performance Assessments Capstone Considerations Does assignment capture the breadth of program learning expectations? Does assignment link back to what is actually being taught in the courses?
Measurement Tool – Performance Assessments Data Collection Process How will assignments be collected/saved for analysis? Integrate into regular classroom process, otherwise compliance difficult Will collection be online or hard copies? Where will data be stored? How will anonymity be maintained?
Measurement Tool – Performance Assessments Scoring Criteria Rubrics: A set of criteria that identify expected dimensions of achievement regarding specific assignments, as well as levels of achievement along those dimensions Checklists Other approaches
Measurement Tool – Performance Assessments Strategies for Developing Rubrics Professional org/ discipline standards Review representative student work (failing though exemplary) and analyze what criteria led to those grades Student feedback Analysis of grading criteria already used by faculty to grade similar assignments Ask faculty to grade sample work and provide criteria they used, looking for commonalities Pilot/practice use
Measurement Tool – Performance Assessments Students Students evaluating themselves Students evaluating other students –Need to train students in critiquing guidelines http://nlight.com/ONE/critiquingGuidelines.doc http://nlight.com/ONE/critiquingGuidelines.doc –Use of rating scales useful Evaluating Team Projects
References Assessing for Learning (2004). Peggy L. Maki, Stylus Press Assessment Clear & Simple: A Practical Guide for Institutions, Departments, and General Education (2004). Barbara E. Walvoord, Jossey-Bass. A Practical Guide to Alternative Assessment (1992). Joan L. Herman, Pamela Aschbacher & Lynn Winters, ASCD. Scoring Rubrics in the Classroom (2001). Judith Arter & Jay McTighe, Corwin Press.
Q & A Ask now, or via email if you prefer: Marti Atkinson, firstname.lastname@example.org@cccone.org Bill Doherty, email@example.com firstname.lastname@example.org
Evaluation Survey Link http://tinyurl.com/24n7lq
Upcoming/Archived Seminars For upcoming desktop seminars and links to recently archived seminars, check the @ONE Web site at: http://www.cccone.org/seminars/index.php
Thanks for Attending For information on @ONEs trainings go to: http://www.cccone.org/index.php http://www.cccone.org/index.php