Presentation on theme: "Assessing Higher Order Thinking Skills Vickie Mohnacky April 19, 2011."— Presentation transcript:
Assessing Higher Order Thinking Skills Vickie Mohnacky April 19, 2011
Assessing Higher Order Thinking Skills To identify personal, social and academic student outcomes in the PreK-Grade 12 Programming Standards To align student outcomes with different types of assessments Today’s Objectives
What do you want your students to learn as a result of gifted education? What do I value the most? (As a teacher – as a human being)
Following directions? Timeliness? What do we value most? Speed? Accuracy? Knowledge of content?
What do we value most? Complex thinking Originality Research Understanding systems Seeing connections Self Knowledge Perspective Abstract Concepts Leadership Civic Responsibility Empathy
How will we use the information? Teachers’ responses:
How will we use the information? Grade? Class rank? Determine entrance to program? Embarrass the student? Punishment? Predict success? Determine coursework? Measure growth/progress in learning? Plan instruction?
Assessment Uses Determines what the student already knows; Determines readiness Pre-assessment A process during learning As the learning is forming Formative Assessments An event after learning Sums up learning Benchmark – Summative Assessments
Not high stakes/Not for accountability. Used to adjust instruction/ improve student learning. Not for report card grades. Examples: Teacher informal questioning Warm-ups Homework K-W-L Chart Formative Assessment An assessment FOR learning. Occurs while learning is forming.
Not high stakes. Examples End of chapter tests Acuity DIBELS Benchmark testing An assessment OF learning – Interim points Benchmark/Interim Assessment
Not high stakes. Used to adjust overall curriculum, programming More Examples: Attitude scales Interest Inventories Established rubrics for class product and performances Critical thinking tests/checklists (Cornell, Watson- Glaser, Test of Critical Thinking) AP/IB course taking and performance An assessment OF learning – Interim points Benchmark/Interim Assessment
High stakes. Reporting and accountability Used to evaluate the overall effects of programming Examples State Standardized Test (WESTEST 2) NAEP (National Assessment of Educational Perf.) PISA (Program of International Student Assessment Act Explore/ACT/SAT An assessment OF learning; Sums up; Occurs at set points Summative Assessment
Difference in Formative and Summative Assessments? How they are used Level of student involvement
Critical Consumers Responsible Producers What are the ultimate student outcomes? Brian Housand
What is the ultimate outcome? Solve
Houston, we have a problem bBS8sHrDgA&NR=1 bBS8sHrDgA&NR=1
Rick Wormeli 2006 Fair Isn’t Always Equal The point is that we have to be clear in our objectives before we can differentiate instruction and properly assess our students’ attainment of those objectives.”
Look at Your Data What does it measure? Will it give you the information you need?
2010 Gifted Education Programming Standards Students identified with gifts and talents demonstrate important learning progress as a result of programming and services (Assessment Criterion 2.5) Standards Focus on Student Outcomes
Learning and Development Student Outcomes Demonstrate self-knowledge of interests, strengths, identities and needs. Possess developmentally appropriate understanding of how they learn and grow and recognize the influences of their beliefs, traditions and values on their learning and behavior Demonstrate an understanding and respect for similarities and differences between themselves and their peer group and others in the general population. 1.1 – 1.3 Self-Understanding
Learning and Development Student Outcomes Access resources from community to support cognitive and affective needs. Receive support from family and communities who understand similarities and differences with respect to the development and characteristics of advanced and typical learners. 1.4 – 1.5 Awareness of Needs
Learning and Development Student Outcomes Benefit from meaningful and challenging learning activities. Recognize preferred approaches to learning and expand their repertoire. Identify future career goals that match their talents and abilities and resources needed to meet goals. 1.6 – 1.8 Cognitive and Affective Growth
Curriculum Planning and Instruction Student Outcomes Demonstrate advanced and complex learning as a result of using multiple, appropriate and ongoing assessments Demonstrate important learning progress as a result of programming and services Assessment
Curriculum Planning and Instruction Student Outcomes Demonstrate growth commensurate with aptitude. 3.1 Curriculum Planning
Curriculum Planning and Instruction Student Outcomes Become more competent in multiple talent areas and across dimensions of learning. Develop their abilities in their domain of talent and/or area of interest. 3.2 – 3.3 Talent Development
Curriculum Planning and Instruction Student Outcomes Become independent investigators. Develop knowledge and skills for living and being productive in a multicultural, diverse and global society Instructional Strategies
Curriculum Planning and Instruction Student Outcomes Benefit from programming that provides a variety of high quality resources and materials. 3.6 Resources
Learning Environments Student Outcomes Demonstrate growth in personal competence and dispositions for academic and creative productivity (self- awareness, self-advocacy, self-efficacy, confidence, motivation, resilience, independence, curiosity, risk taking). 4.1 Personal Competence
Learning Environments Student Outcomes Develop social competence manifested in positive peer relationships and social interactions. Demonstrate personal and social responsibility and leadership skills. 4.2 – 4.3 Social Competence - Leadership
Learning Environments Student Outcomes Value their own and other’s language, heritage, and circumstance. Possess skills in communicating, teaming and collaborating with diverse individuals and across diverse groups. Use positive strategies to address social issues, including discrimination and stereotyping. 4.4 Cultural Competence
Learning Environments Student Outcomes Develop competence in interpersonal and technical communication skills. Demonstrate advanced oral and written skills, balanced biliteracy or multiliteracy, and creative expression. Display fluency and technologies that support effective communication. 4.5 Communication Competence
Specify clearly and exactly what it is you want to assess Design questions and tasks that require students to demonstrate this skill or knowledge
Understanding By Design (UBD) 1. Identify desired results (what the student will know) 2. Determine acceptable evidence (how will the student demonstrate knowledge) 3. Plan instruction; Implement the design
“I began by seeing assessment as judging performance, then as informing teaching, and finally as informing learning.” Carol Ann Tomlinson
Types of Assessment Standard - Selected Response (Multiple Choice) Constructed Response (Short Answer) Fill in the Blank True/False Performance-based or Product- Rubrics Rating scales Checklists
High School Entrance Exam – 1928
Standardized – Large Scale Mass produced Mass graded
Standard/Objective: Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge TargetsReasoning TargetsPerformance Skill Targets Product Targets What knowledge would students need to master this skill? What reasoning proficiencies (if any) would students need to master this skill? What performance skills (if any) would students need to practice to master this skill? What products (if any) would students need to practice creating to master this skill?
Example from real life: Drive with skill. Type: Knowledge Reasoning Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge TargetsReasoning TargetsPerformance Skill Targets Product Targets Know the law Understand informal rules of the road Understand what different parts of the car do Read signs and understand what they mean Understand what “creating a danger” means Understand what “creating a hazard” means Other? Analyze road conditions, vehicle performance, and other driver’s actions Compare/contrast this information with knowledge and past experience Synthesize information and evaluate options to make decisions on what to do next Evaluate “Am I safe?” and synthesize information to take action if needed. Other? Driving actions such as: steering, shifting, parallel parking, looking, signaling, backing up, braking, accelerating, etc. Fluidity/automaticity in performance driving actions. Other? None Since the ultimate type of target is a performance skill, there are no embedded product targets
Standard/Objective: SS.O : Interpret quotes of famous Americans from various periods of history Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard/objective? Knowledge TargetsReasoning TargetsPerformance Skill Targets Product Targets What knowledge would students need to master this standard? What reasoning proficiencies would students need to master this standard? None?
Standard/Objective: SC.O : Construct... charts, graphs and tables for various purposes... Type: Knowledge Reasoning Performance Skill Product Learning Targets What are the knowledge, reasoning, skill or product targets underpinning the standard or benchmark? Knowledge TargetsReasoning TargetsPerformance Skill Targets Product Targets What knowledge would students need to master this standard? What reasoning proficiencies would students need to master this standard? What performance skills would students need to practice to master this standard? What products would students need to practice creating to master this standard?
Informal Questioning Concept Questions - Why do we care about this? Why are we studying this? Open ended Broad, over-arching, outcome oriented questions Will motivate students and target higher-order thinking. What is the big concept you are trying to uncover? Do the standards have any “big concept” words? Content Questions – What are the facts? What are the procedures? Closed ended. Will help strengthen and develop students’ understanding of the larger questions.
Informal Questioning Concepts: Conflict – How can conflict be resolved? (How could this particular conflict have been resolved?) Change - How can we cope with change? Friendship – What does it mean to be a friend? Freedom (of speech) – Is pure freedom of speech desirable in today’s world? Why or why not? Questions Resource
Teachers who do not specifically plan classroom discussion questions ahead of time to tap particular higher-order thinking skills, but rather ask extemporaneous questions “on their feet,” are likely to ask recall questions. Susan M. Brookhart, 2010
Checklists Checklist of items - the least complex form of assessment. Yes or No – Is it present or not? There is no value attached to the performance. All elements weighted the same. Skill On-task throughout time period Participates in class discussions Collaborates with other students Score To determine a score, the total number of checked items or the percentage of total possible. No quality is attached unless specified in the item. Example, “three paragraphs required” or “neatly” or “500 words.” Meaning is then attached to the score. For example, what is the minimum score that would be considered proficient?
Checklists Example of basic speech skills: The student: Maintains eye contact with the audience Speaks loudly enough to be heard in all parts of the room Enunciates clearly Stands up straight (does not shift from foot to foot) Does not go over the allotted time Has notes Uses notes sparingly Score
Rating Scales Graphic rating scale on a continuum – example : NeverSeldomSometimesUsuallyAlways Turns in lessons on time Uses correct capitalization Completes projects
Rating Scales Numerical rating scale – example : 1 to 10 with 10 being the highest PerformanceRating Time on task Participation in class discussion Developing a plan/Setting goals Perseverance during project Total:
Rating Scales Numerical rating scale with descriptions example: 1 = typical for grade/age 3. Quite advanced for grade/age 2 = above average for 4 = Remarkable for grade/age grade/age (1 in 50) Rating Learns quickly Shows power of concentration Enjoys “adult” conversations Has many and/or intense interests Asks many questions Invents, creates Total:
Rubrics A quality is attached to each skill. Start with the skill you want to assess. Example: Does the student reason inductively from the examples to arrive at a clear, accurate description of physical and chemical changes (example from book)? Completely and clearly – Response give clear evidence of reasoning from the examples. Partially – Response is accurate, but reasoning from examples isn’t clear or is only partial. No – Response does not demonstrate reasonable conclusions from the examples.
Rubrics A quality is attached to each skill. Very generalized example of SCALE: Scale refers to numerical or word ratings Exceptional or Distinguished Above Mastery Proficient or Mastery Below Mastery or Proficient Limited or Novice Scoring Performance Assessments: Scoring is not always simple and straightforward. Students generate their own responses.
Rubrics DevelopingCompetentProficient Preparation Organization of the portfolio Another example. Very generalized example of Portfolio (Paulson & Paulson, 1996) Rubrics may have holistic areas or specific dimensions. may be used for self-monitoring and self-assessment may be used in authentic contexts may be used formatively, summatively or as interim assessments
Rubrics DevelopingCompetentProficient Reflection Does not accurately self assess strengths and weaknesses Accurately self assesses strengths and weaknesses Focuses on what and how the knowledge or skill was learned Goal setting Goals vague; takes no responsibility for planning work Accepts responsibility/ describes progress toward goals Identifies future goals that link to strengths and weaknesses A quality is attached to each skill. More descriptive example of Portfolio (Paulson & Paulson, 1996)
Rubrics SkillCriteria Accuracy of Information 100% accurate information Inaccurate information CraftsmanshipWell organized, logical/ clear (strong word choices, good sentence variety, powerful images) Reasoning Skills Reasons are relevant and to the point the writer is making. Justifications Define the highest performance level first. A three-level rubric is common, but five levels allow for some “gray areas.”
4321 Score FluencyI can think of many ideas. I can think of some ideas If I get some help, I can think of ideas I have a hard time thinking of ideas FlexibilityI notice what is surprising and unusual I notice unusual things around me When someone reminds me, notice I hardly ever notice unusual things EvaluationI know several ways of deciding I can tell which ideas are worth working on With help, I can tell which ideas worthwhile I cannot tell which ideas are worthwhile Risk-takingI like to try new ideas I try new ideas Sometimes I try new ideas I do not try new ideas Seeking Challenges Goal setting (etc.) Goal settingGoal setting (etc.) I do not set goals ElaborationWhen I have good idea, I add details to make great I can usually add details to make better Sometimes, I can think of way to make better I do not know how to make better
CriteriaExemplary (4-5)Good (2-3) Needs Improvement (0-1) Initial QuestionsQuestions are probing and help clarify facts All questions may not be relevant Few or no questions formulated Understanding the problem Clearly defines the problem Statement has some vagueness or missing information Problem defined incorrectly Seeking information Identifies several sources of information Relies on few sourcesNot clear as to what is needed Risk-takingI try new ideasSometimes I try new ideas I do not try new ideas Integration of knowledge Effectively applies previous knowledge Applies limited amount of prior knowledge Unable to connect previous knowledge
Criteria210 Thesis (judgment of credibility, identification of assumption or persuasive tactic, etc.) Thesis is clear, is complete, and answers the question posed by the problem or task. Thesis is clear and at least partially answers the question posed by the problem or task. Thesis is not clear or does not answer the question posed by the problem or task. EvidenceEvidence is accurate, relevant and complete. Evidence is mostly clear, relevant and complete. Evidence is not clear, relevant or complete. Reasoning and clarity The way in which the evidence supports the thesis is clear, logical and well explained. The way in which the evidence supports the thesis is mostly clear and logical. Some explanation is given. The way in which the evidence supports the thesis is not clear, is illogical or is not explained. General Rubric for Critical Thinking Involving Judgment
The Crow and the Pitcher A crow, dying of thirst, came upon a pitcher which had once been full of water. When the crow put his beak into the mouth of the pitcher, he found that only very little water was left in it, and he could not reach far enough down to get at it. He tried and tried, but at last had to give up in despair. Then a thought came to him. He took a pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the pitcher. At last he saw the water rising toward him, and after casting a few more pebbles into the pitcher, he was able to drink and save his life.
The Crow and the Pitcher A3A3 A2A2 A1A1 Consequences and Implications What would have happened if the crow had done the following: a) Kept putting his beak in the bottle? b) Flown away? c) Broken the bottle? d) Waited for rain? Cause and Effect What caused the water to reach the crow? What overall effect did it have on the crow? Sequencing What steps did the crow use to get water? Outline them below in order: 1. _________________ 2. _________________ 3. _________________ 4. _________________
The Crow and the Pitcher C1C1 C3C3 C2C2 Main Idea/Theme Inference What made the crow successful in getting a drink of water? Why did his plan work? Characterization What are the crow’s most important qualities? What other characters have you read about that show similar life qualities? How were their situations similar or different from the crow’s situation? What main idea(s) did you get from this story?
Assessment/Response Form Teacher Comments: There’s not a book I read about that was similar to this one. The crow’s most important qualities are water and food. The crow was successful because when you put the pebbles in the water, the water came up. That helped him to get water. He had to get pebbles and keep doing that until water came up. Crows get thirsty. Crows are smart. The crow kept putting pebbles in the water until it could drink. I didn’t even know that. Main Idea Inference Characterization
Learning – Thinking Skills Rubric
Susan M. Brookhart (2010) Many people may be surprised that higher- order thinking can be assessed with “well- written” multiple choice test items.
NAEP (reasoning with data) The table below shows information about the weather in four cities on the same day. In which city did snow most likely fall at some time during the day? A. City 1 B. City 2 C. City 3 D. City 4 City 1City 2City 3City 4 High Temperature65 ◦80 ◦48 ◦25 ◦ Low Temperature56 ◦66 ◦38 ◦10 ◦ Precipitation – Rain or Snow (inches) 2 inches0 inches1 inch
WESTEST 2 Do you use your phone to access your ? Do you make a lot of calls on your cell phone? Do you spend a lot of time on your cell phone? Do you IM with your friends? Have you ever thought about reading a book on your phone? It’s all the rage in Japan! Cell phone novels, called keitai shosetsu, are becoming increasingly trendy in Japan and are starting to gain popularity in China and South Korea. The spread of cell phone novels is mainly due to their appeal to young adults, the same audience with whom blogging, IMing, text messaging, and cell phone use are all the rage. Since cell phone screens are so small, only about a … What is the effect of the series of rhetorical questions at the beginning of this article? A. They draw readers into the article by making them think about their answers to the questions. B. They illustrate the point that so many people now have cell phones at their disposal. C. They point out that the article’s main purpose is to talk about the many uses of cell phones. D. They show how American readers can become successful novel writers using their cell phones.
“How to Assess Higher Order Thinking Skills” (example from book) Which of the following scenarios describes behavior that is legal because of the First Amendment? A.Mr. Jones threw a rock through the front window of Mr. Smith’s house. Around the rock was tied a paper that called Mr. Smith nasty names. B.Mr. Jones waited until Mr. Smith left for work one morning, then got in his car and followed him, honking and yelling. C.Mr. Jones doesn’t trust his neighbor, Mr. Smith. Jones believes Smith is a dangerous person and a threat to the peace of the neighborhood. Therefore, Mr. Jones buys a gun. D.Mr. Jones wrote a letter to the editor of the local paper. Mr. Smith heads a local environmental committee, and Mr. Jones called his position “disastrous.”
“How to Assess Higher Order Thinking Skills” (example from book) Essay questions assessing the ability to make deductive conclusions: 1. Select one of the amendments in the Bill of Rights of the U.S. Constitution. Describe a specific example of one of the rights in the Bill of Rights. The example can be from a real event or something you make up yourself, but it must be a clear illustration of one of the rights in the Bill of Rights. Tell the story of your example. Then explain which right your story exemplifies, from which amendment, and tell why. Criteria for feedback or rubric: Appropriate identification of a particular right and amendment. Appropriateness of example. Appropriateness of evidence. Soundness of reasoning and clarity of explanation.
“How to Assess Higher Order Thinking Skills” (example from book) Questions assessing critical thinking involving judgment: 12.The poster shown above was made during the First World War. What was the poster designed to do? A.Make people feel that it would be easy to win the war. B.Make people feel guilty for thinking that war is harmful. C.Get people to join the army by making them feel responsible for starting the war. D.Get people to join the army by appealing to patriotic feelings.
“How to Assess Higher Order Thinking Skills” (example from book) Questions assessing critical thinking involving judgment: 12.The poster shown above was made during the First World War. What was the poster designed to do? Explain how you came to this conclusion. Criteria for feedback or rubric: Clear, appropriate statement of the main point. Appropriateness of evidence. Soundness of reasoning and clarity of explanation.
Multiple Choice Examples from PISA – Program for International Student Assessment
Test Items from ACT
What is the primary aim of assessment? Who will use the information? How will the information be used? In Review:
What other measures are there? How do we measure the football program? How do we measure the music/dance program?
What other measures are there? Competitions/Games Exhibits Bulletin Boards Websites
What other measures are there? Portfolio – of successes/best work. Knowing that the “failures” along the way were lessons learned on the way to success.
Resources Julia Roberts NAGC’s WOW series Susan K. Johnson NAGC’s WOW series Brookhart, Susan M. (2010) Assess Higher-Order Thinking Skills Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership, September 2010, Vol. 68 No. 1 Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association Wormeli, Rick (2006) Fair Isn’t Always Equal; Assessing & Grading in the Differentiated Classroom, p. 14, Stenhouse Publishers, Portland, Maine.
Resources New Online IEP Form: Policy 2419 at Resources for teachers at Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership September 2010, Vol. 68 No. 1 Pink, Daniel (2006) A Whole New Mind, Riverhead Books Published by the Penguin Group. New York, NY. Reis, Sally & Renzullli, Joseph Curriculum Compacting: A Systematic Procedure for Modifying the Curriculum for Above Average Ability Students The National Research Center on the Gifted and Talented - University of Connecticut. Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association Van-Tassel-Baska, Joyce (2003) Content-Based Curriculum for High-Ability Learners, Waco, TX, Prufrock Press, Inc.