Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessment Literacy for the Middle Level Educator Jennifer Borgioli Learner-Centered Initiatives, Ltd.

Similar presentations


Presentation on theme: "Assessment Literacy for the Middle Level Educator Jennifer Borgioli Learner-Centered Initiatives, Ltd."— Presentation transcript:

1 Assessment Literacy for the Middle Level Educator Jennifer Borgioli Learner-Centered Initiatives, Ltd.

2 Organizational Focus Assessment to produce learning… and not just measure learning.

3

4 Do you honestly want to know what X exactly is? Is your life going to be improved by momentarily knowing what x is? No. Absolutely not. This whole problem is a conspiracy against hardworking American students. Let me tell you, solving for X right now is not going to stop the recession. It fact, it’s not going to do anything. And another thing. When have you ever had to know what is X is in your long esteemed professional career? Exactly. This is a futile attempt for “educators” in this district to boast of their student’s success rate. I am going to go the rest of my life not knowing what X is. Because what is X when you really think about it? A letter, the spot, two lines crossing each other. I don’t think anyone will ever really know what X truly is because the essence of X is beyond our brain potential. In conclusion, Harry S. Truman’s middle name was just the letter S, not an actual name. Now that is a letter that’s actually being utilized. See, you learned something, and it was not because of this logarithm. The End.

5 Talking about the science of our profession does not discredit the art.

6

7 “Less than 20% of teacher preparation programs contain higher level or advanced courses in psychometrics (assessment design) or instructional data analysis.” Inside Higher Education, April 2009

8 Implications Minimize interruptions. Make them worthy.

9 To be assessment savvy….

10 Essential Element #2 2f. The degree to which the middle-level educational program includes ongoing Standards-based assessments

11 Assessment Definition: The strategic collection of evidence of student learning. (Martin-Kniep, 2009) Assessment: test as dogs: pitbull A thing and a process

12 “And while the exams may be a thoroughly vetted, sophisticated means of measurement, they are an inadequate, constricted form of expression.” March 12, 2013 SEATTLE’S LOW-STAKES TESTING TRAP Posted by Michael GuerrieroMichael Guerriero Read more: trap.html#ixzz2NWx8bv00http://www.newyorker.com/online/blogs/newsdesk/2013/03/seattles-low-stakes-testing- trap.html#ixzz2NWx8bv00

13 What are the implications of chasing the pineapple?

14 “Standards-Based Assessment” RL.05.06a: Recognize and describe how an author’s background and culture affect his or her perspective. (NYS)

15 Assessment considerations Why? Purpose Assessment of Learning Assessment for Learning For Whom? Audience Student Teacher Parent Administration (NYSED) What? Learning Targets Knowledge Skills and Abilities ReasoningDispositions When? Timing PeriodicDiagnostic FormativeSummative How? Types RecallProduct Demonstration Process

16 1999 APA Testing Standards

17

18 “The higher the stakes of an assessment’s results, the higher the expectation for the documentation supporting the assessment design and the decisions made based on the assessment results.” (Section 13)

19

20 Traditional Assessment Performance- Based Assessment

21 Performance-Based Assessments (PBAs) A performance task is an assessment that requires students to demonstrate achievement by producing an extended written or spoken answer, by engaging in group or individual activities, or by creating a specific product. (Nitko, 2001)

22 PBA’s versus Traditional Liskin-Gasparro (1997) and Mueller (2008) AttributeTraditionalPerformance Assessment activitySelecting a responsePerforming a task Nature of activityContrivedEmulates real life Cognitive level Knowledge/ comprehension Application/ analysis/synthesis Development of solutionTeacher-structuredStudent-structured Objectivity of scoringEasily achievedDifficult to achieve Evidence of masteryIndirectDirect

23 PBA’s versus Traditional Liskin-Gasparro (1997) and Mueller (2008) AttributeTraditionalPerformance Assessment activitySelecting a responsePerforming a task Nature of activityContrivedEmulates real life Cognitive level Knowledge/ comprehension Application/ analysis/synthesis Development of solutionTeacher-structuredStudent-structured Objectivity of scoringEasily achievedDifficult to achieve Evidence of masteryIndirectDirect

24 Validity = Accuracy

25 How do we ensure alignment and validity in assessment? Degrees of Alignment Strong 1.The assessment clearly aligns to the target; the assessment and the target are almost the same. 2.The language of the standard is explicit. 3.You can confidently conclude the level of student learning/ understanding of the target. Moderate 1.The assessment addresses the target; the target is included in the assessment but is not the primary focus. 2.The language of the standard is only partially used. 3.You need more data points to confidently infer the level of student learning/understanding of the target. Weak 1.The assessment misses the target; it might prepare kids for the target, but doesn’t address it. 2.The language of the standards is missing or barely referenced. 3.You cannot assess level of student learning/understanding of the target.

26 How? Types RecallProductDemoProcess

27 Goal is Best Fit

28 If you want to assess your students’ ability to perform, design, apply, interpret then assess them with a performance or product task that requires them to perform, design, apply, or interpret.

29 I cannot claim my assessment is valid if I do not have some type of blueprint

30 Minimum

31 Basic

32 Articulated

33

34

35 How many? – 5 standards in a PBA (reflected in rows in the rubric) 3 – 5 items per standard on a traditional test

36 Reliability = Consistency

37 I cannot claim my assessment is reliable if I do not have statistics to support my claim

38 Reliability Indication of how consistently an assessment measures its intended target and the extent to which scores are relatively free of error. Low reliability means that scores cannot be trusted for decision making. Necessary but not sufficient condition to ensure validity.

39 Three Types of Measurement Error Subject effect Test effect Environmental effects

40 Subject Effects

41 Others… Fatigue Sleep deprivation Illness Disability

42

43 Testing Fatigue Test Familiarity Bias Score

44 In what ways do we knowingly (or unknowingly) contribute to measurement error?

45 Test Effects

46

47

48 Examples Not enough space for a response Confusing items Typos Misleading (or lacking) directions Scorer inconsistencies

49 10. Format the item vertically instead of horizontally. From A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment by Haladyna, Downing, and Rodriguez

50 21. Place choices in logical or numerical order. Students should not have to hunt to find an answer. Answers should be provided in a logical, predictable pattern.

51 Compare with...

52

53

54 Final Eyes isn’t about editing rather “is this what you want the students to see/read?”

55

56 From Haladyna: 26. Avoid All-of-the-above. 28. Avoid giving clues to the right answer, such as specific determiners including always, never, completely, and absolutely

57

58 Test from Period 1 Test from Period 2

59

60

61 Engage in peer review “Final Eyes” – Is each item aligned to a standard?* – Is each item rigorous? – Is each item fair? – Does each item have one, unambiguous correct key?* – Are all plausible/text based? – Are all tasks meaningful and build upon student comprehension? *Very hard to answer without a test map

62 3. Develop Context-Dependent Item Sets for Content Areas

63 Develop Test Maps and Item Analysis Procedures The higher the stakes of an assessment, the more we need to play by the rules If it’s a mid-term or final exam, there should be a test map. Consider also: – Item analysis – Using choice E (primarily for pre-assessments)

64 Environmental Effects

65 In what ways might we knowingly (or unknowingly) contribute to measurement error?

66 Reliability = Consistency

67 3 general ways to collect evidence of reliability Stability: How consistent are the results of an assessment when given at two time-separated occasions? Alternate Form: How consistent are the results of an assessment when given in two different forms? Internal Consistency: How consistently do the test’s items function? Inter-rater reliability: How consistently do scorer’s use the scoring rubric?

68 Cronbach’s Alpha “In statistics, Cronbach's (alpha) is a coefficient of reliability. It is commonly used as a measure of the internal consistency or reliability of a psychometric test score for a sample of examinees. Alpha is not robust against missing data.” (Nitko, 2012)

69

70 Item Analysis “This is unfamiliar to me”

71 Percent of Students Selecting Choice “E”

72 One assessment does not an assessment system make.

73

74 WHEN DESIGNING A PRE/POST PERFORMANCE TASK the standards and thinking demands must stay the same. the modality that students express their thinking through must also stay the same. the content of the baseline and post must be different. the rubrics for the pre/post will be the same in terms of thinking and modality, but the content dimension will be different.

75 MS General Music Example Pre AssessmentPost Assessment Students listen to 3 songs from Western African Your mother has just returned from a business trip and is playing a song from the country she visited. (Students listen to song) Explain in a blog post how you know what country or region she went to based on the song you heard. Students listen to 3 songs from Spain Your task as a member of your school newspaper is to write an introduction to the new exchange student. Write a brief paragraph that introduces people to the music of her country. Music 4a. Identify the cultural contexts of a performance or recording and perform (with movement, where culturally appropriate) a varied repertoire of folk, art, and contemporary selections from the basic cultures that represent the peoples of the world W.6-8.4: Produce clear and coherent writing in which the development, organization, and style are appropriate to task, purpose, and audience. W.6-8.2d: Use precise language and domain-specific vocabulary to inform about or explain the topic.

76 HS PE Example Pre AssessmentPost Assessment The superintendent of a neighboring school is proposing cutting PE to 40 minutes once a month. Explain to the superintendent why her idea is a good or bad one. Our State Congresswoman is advocating for increasing PE to 40 minutes a day for all students who are not playing a formal sport in school. Explain to the Congresswoman why you agree or disagree with her idea. PE 2. iii. Accept physical activity as an important part of life. Self-renewal, productivity as a worker, energy for family activities, fitness, weight control, stress management, and reduction in health-care costs are understood as benefits of physical activity W.6-8.4: Produce clear and coherent writing in which the development, organization, and style are appropriate to task, purpose, and audience. W.6-8.2d: Use precise language and domain-specific vocabulary to inform about or explain the topic.

77 HS Art Example Students will create their own artwork that communicates the theme of personal identity as an artist through the purposeful use of the elements and principles of art. (NYSVALS Commencement 1.b & 1.c) Students will respond critically to their artwork through dictation or writing by using the language of art criticism to describe, analyze, interpret, and make a judgment regarding that work of art. (NYSVALS Commencement 3a & WHST d) Students will create their own artwork that communicates the theme of personal identity as a learner through the purposeful use of the elements and principles of art. (NYSVALS Commencement 1.b & 1.c) Students will respond critically to their artwork through dictation or writing by using the language of art criticism to describe, analyze, interpret, and make a judgment regarding that work of art. (NYSVALS Commencement 3a & WHST d)

78 Grade 8 ELA (Local Assessment) Pre AssessmentPost Assessment Does setting influence character? Select a character from a book you read last year. Write a brief scene (with dialogue, an established setting, and plot) that places your character in NYC on September 10, Write a second scene that places your character in NYC on September 12, Select a character from a book you read this year. Write a brief scene (with dialogue, an established setting, and plot) that places your character in Philly on July 3, Write a second scene that places your character in Philly on July 5, 1776.

79 It’s Write a first draft of a Wikipedia entry for these two photos. Use what you learned in SS 8 to compare, contrast, and make connections to US history, culture, laws, and society.

80


Download ppt "Assessment Literacy for the Middle Level Educator Jennifer Borgioli Learner-Centered Initiatives, Ltd."

Similar presentations


Ads by Google