1. Education 3504 Week 3 reliability & validity observation techniques checklists and rubrics

2. Reliability and Validity validity –extent to which you are measuring what you think you are measuring reliability –extent to which information you are gathering is consistent and unambiguous

3. Reliability chance factors –characteristics of student –characteristics of test / evaluation –conditions of administration –variations in scoring

4. Validity danger of thinking you are measuring one thing, actually measuring another –trivial knowledge –lowest-common denominator –regional knowledge/one culture/gender

5. Validity face validity –appears important, relevant, interesting construct validity –appears to capture intended quality or behaviour - e.g. what does “good citizenship” consist of?

6. validity and reliability are related but can vary independently

7. Checklists and Rubrics

8. checklist construction Checklist must be based on objectives of the course, unit, or lesson each item should be clear and precise each item should avoid generalities and focus on specifics

9. checklist construction checklist should be designed for feedback to students and parents –so need to be understandable –might want to distribute to students before hand must do so if intended for grading

10. checklist construction if students can use them, can be used for peer and self- evaluation focus on a few specifics, so keep checklist short –if list needs to be longer, focus on only small section at a time anecdotal summary comments can be tied to checklists

11. rating scales 5 -Extremely active 4 -somewhat active 3 -average active 2 -somewhat active 1 -extremely active

12. rating scales Extremely Active (5 points) always out of seat rarely sits without squirming makes noise, yells a good deal of the time

13. Rubrics scoring schemes

14. clear expectations Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

15. …but flexible Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

16. levels of mastery Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

17. multi-dimensional Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

18. parallelism Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

19. high but attainable standards: cumulative criteria Generic Essay OriginalitySubject content awareness of issues critical thinking EXCELLENT A (85 - 100) A Markedly Exceptional Performance originality, insight, and creativity are demonstrated; the paper goes beyond repeating what others have said and contributes something new to our understanding of the topic a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed

20. allowance for part marks Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

21. allowance for part marks

22. correspondence to real world Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A (85 - 100) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B (70 - 84) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C (55 - 69) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

23. evolving changes if you discover papers that do not fit criteria –add descriptors that accurately describe what you are seeing

24. evolving changes as your teaching changes changes as students change changes as environment changes

25. ease of use

27. rubrics workshop Objective: Produce the best cookie possible

28. rubrics workshop Objective: Produce the best cookie possible Was it difficult to come to a consensus on criteria for “best”?

30. why we evaluate motivation administration accountability certification

31. motivation problems: –test anxiety

32. Norm referenced describes student performance relative to other students –e.g.: ranks 11th in group of 30

33. Criterion referenced describes student performance compared to a clearly defined criteria or task –e.g.: “adds single-digit whole numbers”

34. motivation problems: –overemphasis distorts learning process emphasis on external motivation

35. motivation problems: –overemphasis distorts learning process not everything we teach can be assessed (e.g., non-cognitive skills, attitudes, etc.)

36. administration general trend towards greater emphasis on assessment general shift from selection to accountability

37. accountability government to taxpayers –2003, K-12 = 3.8 billion dollars…

38. accountability teachers –accountable for mandated curriculum –accountable for student learning –to oneself over time

39. accountability students to their parents

40. certification right of passage selection and allocation (gatekeeper)