Presentation is loading. Please wait.

Presentation is loading. Please wait.

Module Three Assess the Outcomes

Similar presentations


Presentation on theme: "Module Three Assess the Outcomes"— Presentation transcript:

1 Module Three Assess the Outcomes
Davidson County Community College May

2 Module Three: Assess the Outcomes Topics
The Assessment Toolkit Using Authentic Assessment to Assess SLOs Selection of Direct and Indirect Methods of Assessment Selection Of Test Question Types To Measure To Specific SLOs. Test Item Analysis Outcomes: At the completion of Module Three faculty should be able to: Explain why a variety of assessments are necessary. Distinguish between direct and indirect methods of assessment. Discuss the concept of authentic assessment and design an authentic assessment for your content area. Select appropriate methods of assessment to match SLO to be measured. Explore the use of CATs to assess student knowledge of content, attitudes and self assessment and reactions toward teaching and teaching materials. Identify questions you want answered about your teaching and learning and select an appropriate CAT to answer the question. Use a rubric to assess Student Learning Outcomes (SLO). Utilize Bloom’s Taxonomy to design test questions to align with student learning outcomes being measured. Evaluate tests items to validate that they indeed measure the SLO. Identify the strengths and weakness of different types of test questions. Select appropriate type of test questions to measure specific SLOs. Create an assessment plan that outlines the specific methods that will be used to assess the specific expected student learning outcomes for a course. Create a matrix that shows the appropriate activity and assessment for specific SLOs. Write effective test items by applying helpful hints. Use a Test Blueprint to evaluate tests items to validate that they indeed measure the SLO and provide challenges for students at higher levels of learning. Updating Curriculum to Support Learning DCCC Perkins Grant

3 Table of Contents – Module Three
Stage Three: Assess the Outcomes What is Assessment and Why Do IT? 4 Direct and Indirect Assessment Authentic Assessment 5 The Assessment Toolbox 6 Toolbox of Authentic Assessment 7 Designing Authentic Assessment Worksheet 8 CATs – (Classroom Assessment Techniques) 9-10 Using CATS to Answer Questions About Teaching and Learning 11 Using Rubrics for Direct Assessment of Student Work 12 Rubric Worksheet 13 Outcomes/ Assessment Alignment Worksheet 14 Using Bloom’s Taxonomy to Write Test Questions 15 Direct Verbs Provide Clues 16 Pros and Cons of Types of Tests 17 How to Write Effective Questions in Various Formats 18-19 Test Blue Print 20 Teaching Approach Assessment Matrix 21 My Plan 22 Updating Curriculum to Support Learning DCCC Perkins Grant

4 Stage Three: Assess the Outcomes
How do I assess student learning outcomes? What is Assessment and Why Do It? Assessment is an ongoing process that investigates student learning. The continuous improvement aspect of the assessment cycle models the commitment to the quality of learning within a program and a college. The Assessment Process Source: Fulks, Janet. (2004) “ The SLO Toolkit.” Bakersfield College. Updating Curriculum to Support Learning DCCC Perkins Grant

5 Using Authentic Assessment Traditional Assessment
Types of Assessment Direct assessment methods are measures of student learning that require students to display their actual knowledge and skills rather than report what they think their knowledge and skills are. (Summative) Indirect assessment methods ask students to reflect on their learning rather than to actually demonstrate it. (e.g., student opinions about what they learned and or student ratings of instruction. Also, other self report data like employer surveys, advisory group.) (Formative) Capstone project Classroom Assessment Techniques (C.A.T.s) Case studies Focus groups Demonstrations Graduation and retention rates Exams, quizzes, and tests Interviews Internship and/or clinical evaluations Job/graduate school placement statistics Licensure exams Reflective journals Performance assessments Surveys (employer satisfaction) Portfolio evaluation Advisory Group suggestions Pre and post-tests Simulations What is direct evidence of student learning? 08/10/2009 Direct evidence of student learning is tangible, visible, self-explanatory, and compelling evidence of exactly what students have and have not learned. Imagine a critic of your college or your program--someone who thinks it's a complete joke, a total waste. Direct evidence is the kind that even that critic couldn't argue with. If one of your goals is effective writing, for example, a critic would say that self-ratings, logs of time spent, and even grades aren't truly convincing evidence that your students can write effectively. They can be very helpful, but they're indirect evidence. But it would be hard for that critic to argue papers, tests, and portfolios--accompanied by rubrics or other evaluation criteria that expect appropriate rigor. Using Authentic Assessment What is it? “A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills., says Jon Mueller. Traditional Assessment Authentic Assessment Students select an answer or recall information to complete the assessment such as in test like multiple-choice, true/false, matching or fill-in-the blanks Student perform a task which is simulated to a real word situation Focuses on recall/recognition Requires higher order thinking Norm-referenced that can compare or rank students Criterion-referenced that can identify strengths and weaknesses Updating Curriculum to Support Learning DCCC Perkins Grant

6 The Assessment Toolkit
Tools Capstone Projects: Assessment of the integrated the skills, knowledge, and abilities of students at the completion of a program of study. CATs: Classroom assessment activites used to gather information about student levels of knowledge about the content, about how they perceive they are learning and what they perceived about the instructor’s ability and the instructional materials. Expected Learning Outcomes Statements Survey: Assesses directly how well students feel that they have mastered the expected learning outcomes for the course. Students complete a brief survey that asks them to rate how well they have learned each outcome. The simplest way to do this is to list each stated expected learning outcome and have students indicate on a scale how well they feel they have mastered each outcome. Leave space for students to provide evidence that they successfully mastered the outcome. Performance Assessment: Students demonstrate skills and abilities in a real-time situation. Performance assessments, like portfolios, require well-designed instruments, criteria, rubrics, and norming between reviewers. Portfolios: A display of students’ abilities through a collection of artifacts. Portfolios are good ways to assess change over time. Instructions to the students must be explicit, based upon the purpose and uses of the portfolio. Rubrics should be available to show students what type of evidence needs to be collected and what criteria they must achieve. Non-Graded Quizzes: Activities that provide great feedback on student learning. Non-graded quizzes can be given in a variety of formats (paper & pencil, questions in a PowerPoint presentation, etc.). Be sure to provide students with the correct answers to the quiz items, so they can learn from their mistakes! This choice of assessment is a great way to provide immediate feedback to the student. Reaction Papers: Require students to write down their reactions (thoughts, feelings, attitudes, etc.) to course material or topics. Although reaction papers can be formal graded assignments, they can also be used solely for the purposes of assessing student learning. In this case, faculty members ask their students to take a few minutes of class time and write down their reaction to a fact, opinion, skill, procedure, attitude, critique, etc. that was addressed in the course. The CAT called the “Minute Paper” is a sample of a reaction paper. Rubric: A scoring tool that lays out the specific expectations for an assignment.  Rubrics divide an assignment into its component parts and provide a detailed description of what constitutes acceptable or unacceptable levels of performance for each of those parts. Student-Generated Exam Questions A process where instructors ask students (either alone, in pairs, or in groups) to generate potential exam questions. Having students create potential exam questions helps the instructor assess (1) If students can identify key concepts; (2) What students see as fair or reasonable exam questions; and (3) How well the students can answer these potential exam questions. Using this approach will motivate students to read the assignments and prepare better for tests. Updating Curriculum to Support Learning DCCC Perkins Grant

7 Tool Box of Authentic Assessment
Assignments / Assessment Descriptions Application Cards On an index card have student choose from a list of principles or concepts you just discussed and indicate how they will apply it. Concept Map These are visual images of important concepts. Uses a variety of ways of learning, visual, spatial, linguistic, logical, mathematical and kinesthetic. Engages both sides of the brain. Students remain involved throughout the session. There are cluster Maps… flow chart maps, Burger Map, Free Flow map, and star-burst. Graphic Organizers Use such things as a categorization grid allows you to organizes concepts and characteristics of those concepts; a Fishbone diagram to discover “cause and effect”; ; Ranking Ladder to sort a list and then rank items by placing them in a particular order; T-Graph is useful during brainstorming. You see issues and concerns related to both sides of a topic a time line to show historical events; 5 W’s chart is an introductory experience to help you gather information and summarize; Venn-diagram to “compare and contrast” concepts, In the News Have students bring to the class articles, new items, editorials or cartoons related to the topic. Divide the class into small groups and ask members of each subgroup to share their item with one another and to choose the 2 or 3 most interesting items. Reconvene and ask members from sub-groups to share. Media Diaries Each student tracks a current issue on TV, radio, and in print sources and compares and contrast the like and different ways each focuses on the issue. Discipline Specific Examples of Authentic Learning Geometry You want to install new carpet and a baseboard in you room, but you need to know how much carpet and wood to purchase. Write a set of directions to follow, measure and calculate the square footage of each item needed, and determine the cost of carpet and pad (use the Internet to research). Economics While studying the stock market and its effects on the economy, students will “purchase” a stock or stocks with $5000 and plot their earnings over a three month period. English Research a career that interests you and report on the educational and job requirements, career outlook, job description, etc. Health Record and analyze your food consumption over a given amount of time. Health Record and analyze your food consumption over a given amount of time. Updating Curriculum to Support Learning DCCC Perkins Grant

8 What Do You Already Do To Assess Students?
Appropriate Assessment Strategies “…best identified through careful consideration of the nature of the task, the cognitive level of the objective, and the most feasible context for the assessment.” Source: Garavalia, L, Marken, P and Sommi, R. (2003) “Selecting Appropriate Assessment Methods: Asking the Right Questions” American Journal of Pharmaceutical Education Vol. 66, Summer 2003 p. 111 What Do You Already Do To Assess Students? _____________________________________________ Designing An Authentic Assessment Worksheet Student Learning Outcomes Authentic Assessment Idea Updating Curriculum to Support Learning DCCC Perkins Grant

9 Examples of CATs CAT Description
Assessing Course Related Knowledge And Skills Application Cards After teaching about an important theory, principle, or procedure, ask students to write down at least one real-world application for what they have just learned to determine how well they can transfer their learning. Background knowledge probe Used to determine what students already know to determine a starting point for new instruction Categorizing Grid Give students a grid containing two or three main categories plus a scrambled list of terms and have them sort the terms into the categories. Concept Maps Students draw or diagram the mental connections between a major lecture concept and other concepts they know. This helps students to see connections. Direct Paraphrasing Ask students to write a layman’s “translation” of something they have just learned to assess their ability to comprehend and transfer concepts. Documented Problem Solutions Students are asked to document the steps they take in solving a problem. Students will gain more awareness of and control of their problem solving routines. Focused Listing Give students a main term or concept and have them generate all the related terms they can think of. Practicing this technique can help students learn to focus attention ,improve recall and become familiar with terminology. Muddiest Point Have students write what they find least clear or most confusing about a particular lesson or topic. “What point is confusing to you?” One Minute Paper During the last few minutes of the class period, ask students to answer on a half-sheet of paper: “What is the most important point you learned today?” One Sentence Summary Students summarize knowledge of a topic by constructing a single sentence that answers the questions ,”Who does what to whom, when, where, how, and why?” The purpose is to require students to select only the defining features of an idea. Pro and Con Grid Have students create a grid of the pros and cons for a given decision. This forces students to search for two sides of an issue or question. Updating Curriculum to Support Learning DCCC Perkins Grant

10 Examples of CATs CAT Description
Assessing Learner Attitudes, Values, and Self-Awareness Classroom Opinion Polls Students respond to a survey or by hand-raising and quick counts to items presented by the instructor. Diagnostic Learning Logs Students keep records about assignments they have done and the kind of errors or successes they experienced. At regular intervals they go back over these logs and reflect on their learning strategies. Double-entry Journals In taking notes on class or readings, students keep a simultaneous running commentary of questions, ideas and connections in a second column. Interest/Skill/ Knowledge Checklist The instructor prepares a list of course-specific skills, interests and knowledge and students assess themselves with regard to their interest or level of skill. Punctuated Lectures Periodically during the lecture, the lecturer pauses to allow students to reflect and write on their own attention and behavior during the class in order to identify how actively they are listening or participating. Assessing Learner Reactions to Instruction Do it Yourself Quiz Students are asked to write out the five (5) questions about the key points in the reading and then, answer each. This assignment allows the teacher to see who has done the reading and what comprehension they have of the reading. Film Feedback After viewing a film, the instructor asks students to answer the following two questions: 1. Explain (in as much detail as you can) the main idea illustrated by the film. 2. What can you suggest that I do to make using films a more useful learning experience for a student like you? Handouts Feedback I need your help to decide which handouts are most useful to you. 1. The handout that was most useful to me was . . . 2. The reason this handout was so useful was . . . 3. The handout that was least useful to me was . . . 4. The reason this handout was least useful was . . . Teacher Feedback Have students answer the following questions anonymously. 1. One thing that you (the instructor) do that helps my learning is . . . 2. One thing that you do that interferes with my learning is . . . 3. A practical suggestion I have that will improve the class for me is . . . Updating Curriculum to Support Learning DCCC Perkins Grant

11 Questions You Want Answered About Ways to Improve Instruction
Teaching and Learning ________________________________________ CAT Results Form Course: ___________________________________________________________________________ Name of CAT Purpose CAT seeks to get information or to answer the following: Results Ways to Improve Instruction Updating Curriculum to Support Learning DCCC Perkins Grant

12 Using Rubrics for Direct Assessment of Student Work
What are the parts of a rubric? Task Description: ______________________ Scale Level I Level 2 Level 3 Dimension 1 Dimension 2 Dimension 3 Descriptions of Dimensions A task description (the assignment) A scale of some sort (levels of achievement, possibly in the form of grades).  Scales typically range from 3 to 5 levels. The dimensions of the assignment (a breakdown of the skills/knowledge involved in the assignment) Descriptions of what constitutes each level of performance (specific feedback) Checklist Rating Scale Holistic Rating Scale Descriptive Rubric Four Formats for Rubrics Updating Curriculum to Support Learning DCCC Perkins Grant

13 __________________________________________ Rubric
Rubric Worksheet __________________________________________ Rubric Task Description: _________________________________________ Dimensions Scales Build your rubrics online at: Updating Curriculum to Support Learning DCCC Perkins Grant

14 Outcome AssessmentAlignment Worksheet
Select an SLO from your course syllabus and describe how you would teach to that outcome and how you would assess it. Student Learning Outcome: Ways I can teach the outcome Ways I can assess or measure the outcome. Updating Curriculum to Support Learning DCCC Perkins Grant

15 Using Bloom’s Taxonomy to Assess Student Learning Outcomes
Student will experience more success in your classes if they know up front what is expected of them. Levels of Learning Action Verbs Create Putting together ideas to develop an original idea. Action Verbs: assemble, compose, create, construct, design, develop, formulate, imagine, invent, plan, write Example: Plan appropriate science activities for young children. Construct a basic web page using HTML. Evaluate Judging the value of ideas, materials and methods by developing and applying standards and criteria Action Verbs: appraise, argue, assess, critique, defend, evaluate judge, measure, predict, prioritize, rate, select, support, validate, value Examples: Select the project that reflects use of quality animation and sound. Defend the use of use of the Lock-out/Tag-Out system. Analyze Breaking information down into its parts Action Verbs: analyze, compare, contrast, criticize, debate, detect, differentiate, discriminate, distinguish, examine, experiment, inquire, inspect, investigate, question, separate, test, troubleshoot Troubleshoot a piece of equipment by using logical deduction. Inspect equipment’s safety features before start up. Apply Using strategies, concepts, principles and theories in new situations Action Verbs: apply, calculate, choose, compute, demonstrate, determine, develops, dramatize, draw, employ, illustrate, interpret, manipulate, practice, operate, schedule, show, sketch, solve, use. Choose the appropriate color for a clients hair texture. Operate tools associated with the HVAC trade. Understand Explaining the meaning of information Action Verbs: classify, describe, discuss, explain, identify, locate, recognize, report, restate, translate, paraphrase Describe the parts of a computer. Translate passages from your Spanish text into English. Remember Recalling or recognizing specific information Action Verbs: define, duplicate, label, list, match, memorize, outline, quote, recall, recite, repeat, reproduce, show, state, Outline the steps in giving CPR. Label the diagram of the respiratory system. Bloom’s Revised Taxonomy Source: Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. New York: David McKay Co., Inc. Updating Curriculum to Support Learning DCCC Perkins Grant

16 Questions about Writing Test Items
_____________________________________________ Directive Verbs provide “Clues” Objective Essay O.& E Updating Curriculum to Support Learning DCCC Perkins Grant

17 Pros and Cons of Type of Tests
Test Type Pros Cons Multiple Choice More answer options reduce the change of guessing Measures knowledge, application and analysis goals Can cover a wide sampling of content or outcomes Student have to spend more time reading more answers so reduces the number of questions to be presented Difficult to write 4 or 5 reasonable choices Takes more time to write them. True False Can present many items Easy to score Used to assess misconceptions, cause-effect reactions Most difficult question to write objectively Ambiguous terms can confuse Increases chance of guessing Matching Efficient Used to assess student understanding of associations, relationships and definitions Difficult to assess higher-order outcomes. Essay Measures higher order outcomes Encourages more appropriate study habits Easier to construct Takes more grading time Not efficient to test large amounts of content If you give the choice of 3 or more essay options, you can find out what they know, but not what they don’t know. Performance Assessments Focus on ability of students to apply skills or knowledge in real life situations. Labor and time-intensive Tests student individually and are not for testing groups. Low reliability Problem Based Scenarios Allows for student creativity Focuses on student application of knowledge Time consuming to write Answers may vary Updating Curriculum to Support Learning DCCC Perkins Grant

18 How to Write Effective Questions in Various Formats
Check the hint(s) you need to adopt. Check Multiple Choice - Helpful Hints Eliminate excessive wording or irrelevant information from the stem. Include in the stem any word(s) that might have otherwise been repeated in each alternative. State the stem as a direct question rather than as an incomplete statement Make all alternatives plausible and attractive to the less knowledgeable or skillful student. Make sure there was only one correct or best response per item. Make alternatives approximately equal in length. Present alternatives in some logical order (e.g., chronologically, most to least). Randomly distribute the correct response among the alternative positions throughout the test having approximately the same proportion of alternatives a, b, c, d, and e as the correct response. Use at least four alternatives for each item. Make the alternatives grammatically parallel with each other, and consistent with the stem Use negatively stated stems sparingly. When used, underlined and/or capitalized the negative word(s). Use the alternatives "none of the above" and "all of the above" sparingly. When used, such alternatives were occasionally the correct response. Source: How to Write Better Tests Updating Curriculum to Support Learning DCCC Perkins Grant

19 How to Write Effective Questions in Various Formats
Check Matching Helpful Hints Avoid grammatical or other clues to the correct response Explain whether or not a response could be used more than once and indicate where to write the answer. Include directions which clearly states the basis for matching the stimuli with the response. Include more responses than stimuli. Keep the lists of items short and put the brief responses on the right When possible, arrange the list of responses in some systematic order (e.g., chronologically, alphabetically). When possible, reduced the amount of reading time by including only short phrases or single words in the response list. Writing Essay Questions - Helpful Hints Ask questions that will elicit responses on which experts could agree that one answer is better than another. Avoid giving the student a choice among optional items as this greatly reduces the reliability of the test. Indicate for each item a point value or weight and an estimated time limit for answering. It is generally recommended for classroom examinations to administer several short-answer items rather than only one or two extended-response items. Prepare essay items that elicit the type of behavior you want to measure. Phrase each item so that the student's task is clearly indicated. Use the verbs from Bloom’s Taxonomy to write essay questions to test different levels of thinking. Source: How to Write Better Tests Updating Curriculum to Support Learning DCCC Perkins Grant

20 Test on: _________________________
Test Blueprint Worksheet Date: ____________ # Items: __________ Test on: _________________________ Levels of Thinking Outcomes Remember Understand Apply Analyze Evaluate Create Total/% LO Instructions: 1st: Analyze the test you have written by examining each question (item) and determine at which level of critical thinking this is written. 2nd Determine which Student Learning outcome each question measures. Write the number of the question in the box that coincides with the correct SLO tested and the level of Critical Thinking. Once all items have been checked add the total items for each outcome on the vertical total line, as well as for each level of critical thinking on the horizontal total line. The percentages under the vertical column will indicate how much of the test is written with higher level thinking in mind. You can also determine which SLOs need more questions to adequately measure them. Updating Curriculum to Support Learning DCCC Perkins Grant

21 Teaching Approach / Assessment Matrix
Write the SLOs in the first column on the left. In the center column indicate what you do to teach to that SLO and in the last column on the right indicate how you assess the outcome. SLO Teaching Approach Assessment Updating Curriculum to Support Learning DCCC Perkins Grant

22 My Plan for Assessing Outcomes
Course in which I wish to make changes: ______________________________________________ Effective assessment principles I will adopt: Strategies I plan to use: ____________________________________________________________________________________________ “I never teach my pupils; I only attempt to provide the conditions in which they can learn.” Albert Einstein Updating Curriculum to Support Learning DCCC Perkins Grant

23 Session Evaluation One thing I will take from the workshop that I can apply is … _________________________________________________________________________________________________________________________________________________________________________________ I would have learned better if you would have… The handout materials were… Using the scale of 1 to 5, please place a check by the number where would you rate how you felt your expectations were met for this session. 1 2 3 4 5 Did not meet my expectations It was ok I learned some things It met my expectations It met and exceeded my expectations Updating Curriculum to Support Learning DCCC Perkins Grant


Download ppt "Module Three Assess the Outcomes"

Similar presentations


Ads by Google