1. Evaluating Open Educational Resource (OER) Objects An Introduction to the Achieve OER Rubrics CC BYCC BY Achieve 2013

2. 2 Open : Containing an open license - no restrictions on remixing or reusing [Having no enclosing or confining barrier; not restricted to a particular group or category of participants] Open Educational Resources: [With Webster’s Definitions] O E R Educational: Used for teaching and learning [Pertaining to the action or process of educating or being educated] Resource: Object used to support an effort or task [ A source of support or aid, especially one that can be readily drawn upon when needed]

3. 3 What are OER? OER are teaching, learning, and research resources that contain an open license. They provide extraordinary opportunities for educators to freely share knowledge and resources. They offer great potential for instructional innovation and networks for sharing best practices. There are millions of OER objects available online. Open Educational Resources (OER)

4. 4 OER objects may include (but are not limited to): Images Applets Games Worksheets Lesson plans Original-source texts Assessments Units Textbooks Teacher/learner support materials o NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson). Open Educational Resources (OER)

5. 5 The Achieve OER Evaluation Rubrics How and why were the Achieve OER Rubrics created? Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students. The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way. There are two ways of approaching the evaluation of a resource: o Holistically evaluating the object with a single rating, (similar to the star-rating systems used on Netflix, Yelp and Amazon) o Separately evaluating each of the components of quality, as with the Achieve OER Rubrics. The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies. The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify, evaluate and sort objects based on the specified essential elements of quality.

6. 6 How do the Achieve OER Rubrics work? The rubrics represent an evaluation system for objects found within Open Educational Resources. They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently. OERCommons.org They are applied to any content area. o NOTE: At this stage only Common Core Standards for English Language Arts/Literacy and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available. Each rubric is applied independently to an object. They rate the potential, not the actual, effectiveness of an object. They use a five-point scoring system to describe levels of quality. Applying the Achieve OER Rubrics

7. 7 There are Seven Rubrics on the Achieve OER Evaluation Tool Rubric I. Degree of Alignment to Standards Rubric II. Quality of Explanation of the Subject Matter Rubric III. Utility of Materials Designed to Support Teaching Rubric IV. Quality of Assessment Rubric V. Quality of Technological Interactivity Rubric VI. Quality of Instructional Tasks and Practice Exercises Rubric VII. Opportunities for Deeper Learning o NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate. The Rubrics

8. 8 The Detailed Scoring Descriptors are Specific to Each Rubric 3: Superior 2: Strong 1: Limited 0: Very weak / None N/A: Rubric not applicable to the object o NOTE: N/A should be used when a particular rubric does not apply to the object being rated. This is not a pejorative score; it simply means it would be inappropriate to apply this rubric to the particular object. The Rubrics’ Common Scoring Scale

9. 9 Before the degree of alignment can be rated, an object must be aligned to standards. (In the online tool only CCSS Math and ELA/Literacy are available at this time.) Some objects will not have an alignment identified, either in the object or in the online tool. Some objects may provide alignment suggestions or have been aligned by earlier users in the online tool. However a rater may need to verify those alignments. When using the online tool an alignment should be entered whenever possible so that Rubric I can be used in the evaluation and so that accurate data might be collected. Aligning objects accurately allows future users to sort by subject, grade level, and even by a specific standard. Rubric I: Degree of Alignment to Standards

10. 10 The following steps are not part of the OER evaluation process but are required for reviewers who may need to align an object before rating or want to verify the standards selected by previous raters. Six Steps in Alignment to the CCSS: 1.Review the object thoroughly, working through tasks and/or exercises. 2.Compare your work with keys/solutions and other instructional support materials. 3.Identify the content and performance(s) required in the object. 4.Identify the grade level for the object. 5.Match the grade level, content, and performance requirements to the appropriate CCSS, including the Mathematical Practices and texts for ELA Literacy. 6.Make a note of any requirements in the object that are not addressed in the standard(s) and any requirements in the standard that are not addressed in the object. Rubric I: Degree of Alignment to Standards

11. 11 This rubric is applied to objects that have suggested alignments to the standards. It is designed specifically for the Common Core State Standards (may possibly be used with other standards in the future). Rubric I can be completed only after alignment has been proposed. Alignment review involves two major aspects: o Content o Performance expectations Rubric I is not applicable for an object for which no alignment has been (or can be) suggested. Rubric I: Degree of Alignment to Standards

12. Rubric I – Superior Degree of Alignment to Standards 3: An object has superior alignment only if both of the following are true: All of the content and performance expectations in the identified standard are completely addressed by the object. The content and performance expectations of the identified standard are the focus of the object. While some objects may cover a range of standards that could potentially be aligned, for a superior alignment the content and performance expectations must not be a peripheral part of the object. [Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this section.] The Rubrics’ Common Scoring Scale: Rubric I 12

13. Rubric I – Strong Degree of Alignment to Standards 2: An object has strong alignment for either one of two reasons: Minor elements of the standard are not addressed in the object. The content and performance expectations of the standard align to a minor part of the object. [Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this section.] The Rubrics’ Common Scoring Scale: Rubric I 13

14. 14 Rubric I – Limited Degree of Alignment to Standards 1: An object has limited alignment if a significant part of the content or performance expectations of the identified standard is not addressed in the object, as long as there is fidelity to the part it does cover. [Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this section.] The Rubrics’ Common Scoring Scale: Rubric I

15. Rubric I – Very Weak Degree of Alignment to Standards 0: An object has very weak alignment for either one of two reasons: The object does not match the intended standards. The object matches only to minimally important aspects of a standard, cluster or domain. These objects will not typically be useful for instruction of core concepts and performances covered by the standard. This rating should be used for any standard that has been misaligned to the object [Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this section.] The Rubrics’ Common Scoring Scale: Rubric I 15

16. Rubric I – Degree of Alignment to Standards – Not Applicable N/A: This rubric does not apply for an object that has no suggested standards for alignment. NOTE: For Rubric I a 0- or 1-rating is preferable to N/A for those misaligned standards selected by previous raters since only numerical ratings (0, 1, 2, or 3) are averaged into the overall results for an object. All standards that are aligned should be rated and an N/A would only be appropriate if it is not possible to align standards to the object or if there is no intention for the object to align to standard, as in a set of raw data or a map. [Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this section.] The Rubrics’ Common Scoring Scale: Rubric I 16

17. MATH ALIGNMENT RATING EXAMPLES: An object has been aligned to CCSS 2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.” Ratings would be… 3 – If the primary focus of the object is counting to 1000 and skip counting by 5’s, 10’s, and 100’s, with any other content/performance requirements secondary to this. For example, in an object that focuses on all these requirements, there may be an interactive game that requires counting by 2’s, or relating counting to addition and subtraction, or an activity that requires measurement by counting whole units. These would be peripheral to the overall focus of the object and would not detract from the 3-rating. 2 – If the object requires counting to 1000 and skip counting by 5’s and 10’s, it aligns with the majority of the standard’s requirements, with a minor part not addressed (counting by 100’s). OR If it addresses all of the standard’s content and performance but only a peripheral way. For example, counting by 100’s might be addressed only in an extension activity, which users may opt not to use or may not even see. 1 – If the object requires counting with numbers to 500, it would be considered to have limited alignment: the object aligns very closely with a limited part of the standard. 0 – If the object only addresses skip counting by 10’s, with no clear indication stated as to the limitation (count to 1000). Or if the object requires only counting within 100. Rubric I: Degree of Alignment to Standards 17

18. Rubric I: Degree of Alignment to Standards ELA ALIGNMENT RATING EXAMPLES: An object is aligned to CCSS RI.9-10.8, “Delineate and evaluate the argument and specific claims in a text, assessing whether the reasoning is valid and the evidence is relevant and sufficient; identify false statements and fallacious reasoning.” Ratings would be… 3 – If the object requires that students evaluate the argument of a text, assessing the validity of its evidence and reasoning including identifying false statements and reasoning. The text and instructional material must be constructed to include some false statements and reasoning. 2 – If the object requires that students evaluate the argument of a text, assessing the validity of its evidence and reasoning, but did not include identifying false statements and reasoning. OR if the object included all the range of evaluative performances, but the evaluation was a minor part of the activities. 1 – If the object requires only delineating the argument but does not require students to evaluate its reasoning or evidence. 0 – If the text of the object includes an argument, but none of the activities require that students delineate or evaluate it. 18

19. 19 Rubric II: Quality of Explanation of Subject Matter Rubric II is used to rate how well the subject matter is explained or otherwise revealed in the resource. Subject explanation could be aimed at either the teacher or a student. This rubric could apply to objects used for a whole class, small groups, or an individual student (e.g. self-tutoring object). Sometimes the subject matter explanation is included in teacher support material and is intended to help a teacher learn or be refreshed on a specific topic (the teacher becomes the learner). Do not use Rubric II to rate the quality of the explanation of how to present the lesson (planning instructions) for an object. Use an N/A-rating only if there is no explanation of subject matter included in the object or for objects not designed to explain subject matter (e.g. a formula sheet or a map), even if it might be possible to use the object to explain a subject.

20. Rubric II – Superior Quality of Explanation of the Subject Matter 3: An object is rated superior only if all of the following are true: The object effectively provides comprehensive information so that the target audience should be able to understand the subject matter. The object connects the subject matter with important associated concepts. The object does not need to be augmented with additional explanation or materials. The main ideas of the subject matter addressed in the object are clearly identified for the learner. For example a lesson on multi-digit addition provides a clear explanation of multiple strategies and makes connections with place value, rather than simply showing procedure. Or a lesson clearly explains why and how to analyze how an author develops ideas across extended text would make connections among the various developmental steps and the various purposes the author has for the text. These examples would be considered superior for explanation of subject matter. The Rubrics’ Common Scoring Scale: Rubric II 20

21. Rubric II – Strong Quality of Explanation of the Subject Matter 2: An object is rated strong if it explains the subject matter in a way that makes skills, procedures, concepts, and/or information understandable. It falls short of superior in that it does not make the connections among important associated concepts within the content. For example if an object’s focus is to compare decimals to hundredths, it would be considered strong if it thoroughly explains how to compare 0.25 and 0.6 by comparing digits in the same place value position. However it may not make the connection to number line placement or conversion of the two decimals to fractions with a common denominator and then comparing the numerators. The Rubrics’ Common Scoring Scale: Rubric II 21

22. Rubric II –Limited Quality of Explanation of the Subject Matter 1: An object is rated limited if it explains the subject matter correctly but in a limited way. This is a cursory treatment of the content that is not developed enough to serve someone attempting to learn the content for the first time. The explanations are not thorough enough to serve as more than a review for most learners. For example an object addressing comparing the mean to the median and mode for a set of data provides procedure but does not explain how the measures of center are used or why one is preferable to another in a given situation. This is of limited quality for explanation of subject matter. The Rubrics’ Common Scoring Scale: Rubric II 22

23. Rubric II – Very Weak Quality of Explanation of the Subject Matter 0: An object is rated very weak or no value if its subject matter explanations are confusing or contain errors. There is little likelihood that this object will contribute to understanding. For example an object designed to teach synthesis of information gathered from narrative texts, data collections, and simulations found in various science publications, is missing a clear explanation of the process and elements of synthesis. It is very weak for quality of explanation of subject matter. The Rubrics’ Common Scoring Scale: Rubric II 23

24. Rubric II – Quality of Explanation of the Subject Matter - Not Applicable N/A: This rubric is not applicable (N/A) for an object that is not designed to explain subject matter. It may be possible to apply the object in some way that aids a learner’s understanding, but that is beyond any obvious or described purpose of the object. For example Rubric II would be not applicable for a sheet of mathematical formulae, a set of raw data, or a map. The Rubrics’ Common Scoring Scale: Rubric II 24

25. 25 Applied to objects designed to support teachers in planning or presenting subject matter. Primary user would be the teacher. Evaluates the potential utility of an object for the majority of instructors at the intended grade level. Not applicable if the object is not designed to support teachers in planning and/or presenting subject matter, even though it might be possible for an educator to find such an application. Rubric III: Utility of Materials Designed to Support Teaching

26. Rubric III – Superior Utility of Materials Designed to Support Teaching 3: An object is rated superior only if all of the following are true: The object provides materials that are comprehensive and easy to understand and use. The object includes suggestions for ways to use the materials with a variety of learners. These suggestions include materials such as “common error analysis tips” and “precursor skills and knowledge” that go beyond the basic lesson or unit elements. All objects and all components are provided and function as intended and described. For larger objects like units, materials facilitate the use of a mix of instructional approaches (direct instruction, group work, investigations, etc.). For example a superior object for this rubric includes clear descriptions of the lesson’s activities with suggestions for interventions, extensions, and for use with English language learners. Also the time needed for lesson planning and presentation appears to be accurately estimated, materials list is complete, and all explanations make sense. The Rubrics’ Common Scoring Scale: Rubric III 26

27. Rubric III – Strong Utility of Materials Designed to Support Teaching 2: An object is rated strong if it offers materials that are comprehensive and easy to understand and use but falls short of “superior” for either one of two reasons: The object does not include suggestions for ways to use the materials with a variety of learners (e.g., error analysis tips). Some core components (e.g., directions) are underdeveloped in the object. For example an object would be considered strong if it includes clear lesson directions, with suggestions for students with special needs, but does not include a list of materials needed or time estimates. Or the object has clear directions for the teacher, including student materials, but does not provide suggestions for intervention, extension, or for English language learners students. The Rubrics’ Common Scoring Scale: Rubric III 27

28. Rubric III – Limited Utility of Materials Designed to Support Teaching 1: An object is rated limited if it includes a useful approach or idea to teach an important topic but falls short of “strong” for either one of two reasons: The object is missing important elements (e.g. directions for some parts of a lesson are not included). Important elements do not function as they are intended to (e.g. directions are unclear). Teachers would need to supplement this object to use it effectively. For example an object, a lesson plan for teaching division of fractions has clear directions and general suggestions for the types of activities for students but has no specific student materials, assessments, or suggestions for how to adjust the lesson for students with special needs. It is considered limited since, in order to use the object, the teacher will need to create questions or problems, student worksheets and/or assessments and will need to make adjustments for intervention and extension. The Rubrics’ Common Scoring Scale: Rubric III 28

29. Rubric III – Very Weak Utility of Materials Designed to Support Teaching 0: An object is rated very weak or no value for instructional purposes if it is confusing, contains errors, is missing many important elements, or is for some other reason simply not useful. For example an object, a lesson plan for teaching division of fractions, would be considered very weak if there are some materials included designed to support teaching but several student support documents are missing and they include only a few of the needed solution keys, with errors throughout. The Rubrics’ Common Scoring Scale: Rubric III 29

30. Rubric III –Utility of Materials Designed to Support Teaching – Not Applicable N/A: This rubric is not applicable (N/A) for an object that is not designed to support teachers in planning and/or presenting subject matter. It may be possible that an educator could find an application for such an object during a lesson, but that would not be the intended use. For example a game that is designed to be used by students as an intervention or pre- assessment of skill knowledge would not necessarily need to have teacher support materials included. In this case Rubric III would be not applicable to the object. The Rubrics’ Common Scoring Scale: Rubric III 30

31. 31 Applied to objects which seek to determine what a student knows. Assessments may be used to assess knowledge before, during, or after a topic is taught. Applied to entire sets of assessment items when multiple items are included in an object. Not applicable for an object not designed to have an assessment component, even when one might imagine aspects of the object that may be used for assessment purposes. Rubric IV: Quality of Assessments

32. Rubric IV –Superior Quality of Assessments 3: An object is rated superior only if all of the following are true: All of the skills and knowledge assessed align clearly to the content and performance expectations intended, as stated or implicit in the object. Nothing is assessed that is not included in the scope of intended material unless it is differentiated as extension material. The most important aspects of the expectations are targeted and are given appropriate weight/attention in the assessment. The assessment modes used in the object, such as selected response, long and short constructed response, or group work require the student to demonstrate proficiency in the intended concept/skill. The level of difficulty is a result of the complexity of the subject-area content and performance and of the degree of cognitive demand, rather than a result of unrelated issues (e.g. overly complex vocabulary used in math word problems). For example a superior object for Rubric IV has appropriately focused and weighted assessments included, of varying types and modes, with accurate solution keys provided. The Rubrics’ Common Scoring Scale: Rubric IV 32

33. Rubric IV –Strong Quality of Assessments 2: An object is rated strong if it assesses all of the content and performance expectations intended, but the assessment modes used do not consistently offer the student opportunities to demonstrate proficiency in the intended concept/skill. For example an object with an assessment component intended to determine a student’s ability to write a proof or present an argument uses only multiple choice or matching assessment items. This object is considered strong for Rubric IV because it has clear requirements that students know the structure and parts of the presentation of a proof or argument, but lacks an assessment mode that offers students an opportunity to fully demonstrate the intended skill/knowledge: writing a proof or argument. The Rubrics’ Common Scoring Scale: Rubric IV 33

34. Rubric IV – Limited Quality of Assessments 1: An object is rated limited if it assesses some of the content or performance expectations intended (as stated or implicit in the object), but omits some important content or performance expectations and/or fails to offer the student opportunities to demonstrate proficiency in the intended content/skills. For example an object’s stated intention is to teach students to write fractions, with denominators of 2, 4, 5, 8, 10, and 100, as decimals, but assessment items address only measurements with the denominators 2, 4 and 8. The object is considered limited in the information and performance expectations it assesses. The Rubrics’ Common Scoring Scale: Rubric IV 34

35. Rubric IV –Very weak Quality of Assessments 0: An object is rated very weak or no value if assessments contain significant errors, do not assess important content/skills, are written in a way that is confusing to students, or are unsound for other reasons. For example the following are considered very weak for Rubric IV: 1) an object that includes general suggestions for the type of assessments to be used (formative, summative, self-, pre-, etc.) but is missing specific questions or even question types; 2) one that has attached student assessments that address content not included in the lesson; or 3) one that includes some assessment items with a lesson, but with incorrect solutions. NOTE: This list is not meant to be exhaustive. There are other possible examples of 0- rated objects for Rubric IV. The Rubrics’ Common Scoring Scale: Rubric IV 35

36. Rubric IV – Quality of Assessments – Not Applicable N/A: This rubric is not applicable (N/A) for an object that is not designed to have an assessment component. Even if one might imagine ways an object could be used for assessment purposes, if it is not its intended use, not applicable is the appropriate score. For example an object is an interactive student-directed game that introduces a new concept. There is no intention by the designer to include assessments so Rubric IV is not applicable to the object. The Rubrics’ Common Scoring Scale: Rubric IV 36

37. 37 The rubric does not apply to students interact with each other, but rather how the technology responds to an individual student user. For interactive objects designed for student use, the rubric is applied to the entire object. This is not a rating for technology in general, but only for technological interactivity between the user and the object. For objects designed for teacher use, for example lesson plans, which may include interactive components for use by students, the rubric is applied only to those interactive parts. Does not apply to objects where interaction with the object is limited to, for example, opening a PDF attachment. Rubric V: Quality of Technological Interactivity

38. Rubric V – Superior Quality of Technological Interactivity 3: An object is rated superior only if all of the following are true: The object, or interactive component of an object, is responsive to student input in a way that creates an individualized learning experience. This means the object adapts to the user based on what s/he does, or the object allows the user some flexibility or individual control during the learning experience. The interactive component is purposeful and directly related to learning. The interactive component of the object is well designed and easy to use, encouraging learner use. The interactive component of the object appears to function flawlessly on the intended platform. For example an interactive student-directed game is considered superior for Rubric V because it responds to individual students by increasing the level of challenge when students successfully complete sections of questions and offers hints when incorrect answers are given and moves the student to intervention questions when they are unable to give the correct answer. The Rubrics’ Common Scoring Scale: Rubric V 38

39. Rubric V – Strong Quality of Technological Interactivity 2: An object, or interactive component of an object, is rated strong if it has an interactive feature that is purposeful and directly related to learning, even if it does not provide an individualized learning experience. Similarly to the superior objects, strong interactive objects must be well designed, easy to use, and function flawlessly on the intended platform. For example an object has a component that reads aloud sections of text, making it easier for some students to stay focused or to hear the voice in a passage of literature. Even though this interactive feature is well designed, purposeful, and directly related to the learning, there is no personal feedback or individualized response. This object is considered strong for Rubric V. The Rubrics’ Common Scoring Scale: Rubric V 39

40. Rubric V – Limited Quality of Technological Interactivity 1: An object, or interactive component of an object, is rated limited if its interactive element neither relates to nor detracts from learning. This kind of interactive element might be designed to increase student motivation and to build content understanding by rewarding or entertaining the learner and extending the time the user engages with the content. For example even though the interactive element of a student-directed game may motivate the student to maintain the connection with the object, it is still considered limited for Rubric V. The game’s interactive feature includes earning points, leading to achieving levels, as a reward for correct answers. The interactivity component does not connect directly to the subject matter, provides no opportunity to learn from incorrect answers, or, in some cases, even to recognize when an incorrect answer was given. The Rubrics’ Common Scoring Scale: Rubric V 40

41. Rubric V – Very Weak Quality of Technological Interactivity 0: An object, or interactive component of an object, is rated very weak or no value if it has interactive features that are poorly conceived and/or executed. The interactive features might fail to operate as intended, distract the user, or unnecessarily take up user time. For example in a student-directed game a group of clowns tumble across the screen with each correct answer, distracting the student and slowing their progress. This interactive feature is considered very weak or of no value. The Rubrics’ Common Scoring Scale: Rubric V 41

42. Rubric V – Quality of Technological Interactivity – Not Applicable N/A: This rubric is not applicable (N/A) for an object that does not have an interactive technological element. Rubric V is not applicable if there is no interactive feature in the object or if technological interaction with the object is limited to, for example, opening a user-selected pdf. The Rubrics’ Common Scoring Scale: Rubric V 42

43. 43 Applied to objects that contain instructional tasks and/or sets of practice exercises designed to strengthen skills and knowledge. The purpose of these exercises is to deepen understanding and to routinize foundational skills and procedures. (When concepts/skills are introduced providing sufficient practice is critical. When integrating skills into complex tasks, the number of exercises is less important than their richness.) Applied to a group of exercises as a single object. Not applicable if the object does not include practice exercises. Even if one can imagine ways to use the object as practice, if that is not the intention of the object, N/A is the correct rating. Rubric VI: Quality of Instructional Tasks and Practice Exercises

44. Rubric VI – Superior Quality of Instructional Tasks and Practice Exercises 3: An object is rated superior only if all of the following are true: The object offers an more than enough exercises needed for the average student to facilitate mastery of the targeted skills, as stated or implied in the object. For complex tasks, one or two rich exercises may be considered more than enough. The exercises are clearly written and supported by accurate answer keys as applicable. There are a variety of exercise types and/or the exercises are available in a variety of formats, as appropriate to the targeted concepts and skills. For example a superior object for practice is a lesson addressing operations with polynomials includes an adequate number of practice problems using all four operations with polynomials. In addition there are extra practice sets designed for warm up, some for students needing remedial practice, and more complex problems for those needing a greater challenge. All practice sets provide varying formats for the problems and all include accurate solutions. The Rubrics’ Common Scoring Scale: Rubric VI 44

45. Rubric VI – Strong Quality of Instructional Tasks and Practice Exercises 2: An object is rated strong if it offers only a sufficient number (for the average student) of well-written exercises to facilitate mastery of the targeted skills, supported by accurate answer keys or scoring guidelines, with little variety of exercise types or formats. For example a sixth grade object addressing conventions of standard English, capitalization and punctuation rules, includes a practice worksheet with an adequate number of exercises to allow for practice for students presently reading at grade level. Answer keys and scoring suggestions are included. This object is considered strong rather than superior because even though it offers adequate practice, it does not provide varying formats or provide for a variety of student learning needs. The Rubrics’ Common Scoring Scale: Rubric VI 45

46. Rubric VI – Limited Quality of Instructional Tasks and Practice Exercises 1: An object is rated limited if it has some, but too few exercises to facilitate mastery of the targeted skills, is without answer keys, and provides no variation in format or type. For example an object is limited for practice in a lesson addressing two-digit multiplication which has a set of five practice exercises. That is too few to facilitate mastery of two-digit multiplication but enough to be considered practice. The Rubrics’ Common Scoring Scale: Rubric VI 46

47. Rubric VI – Very Weak Quality of Instructional Tasks and Practice Exercises 0: An object is rated very weak or no value if exercises provided do not facilitate mastery of the targeted skills, contain errors, or are unsound for other reasons. For example an object addressing division of fractions and mixed numbers has included a set of practice problems. However of the ten problems, none require that the student convert a mixed number to an improper fraction. In addition the answer keys for the practice set includes three with incorrect answers and none show steps in the process. Since the teacher will likely need to create new practice exercises, this object is very weak for Rubric VI. The Rubrics’ Common Scoring Scale: Rubric VI 47

48. Rubric VI – Quality of Instructional Tasks and Practice Exercises - Not applicable N/A: This rubric is not applicable (N/A) to an object that does not include practice exercises. Even though one might imagine ways it could be used in this way, not applicable is the appropriate score if practice is not the intention of the object. For example many objects are not intended to be used for practice of a skill but rather to introduce a concept, teach a subject, assess learning, etc. When practice is not the intention of the object, Rubric VI is not applicable. The Rubrics’ Common Scoring Scale: Rubric VI 48

49. 49 Rubric VII is applied to objects designed to engage learners in at least one of the following deeper learning skills: Think critically and solve complex problems Work collaboratively Communicate effectively Learn how to learn Reason abstractly Construct viable, logical arguments and critique the reasoning of others Apply discrete knowledge and skills to real-world situations Construct, use, or analyze models o Not applicable when an object does not appear to be designed to provide the opportunity for deeper learning, even though one might imagine how it could be used to do this. Rubric VII: Opportunities for Deeper Learning

50. Rubric VII – Superior Opportunities for Deeper Learning 3: An object is rated superior only if all of the following are true: At least three of the deeper learning skills from the list are required in the object. The object offers a range of cognitive demand that is appropriate and supportive of the material. Appropriate scaffolding and direction are provided. For example a superior object for deeper learning addresses peer conferencing in the process of revision of written drafts. It provides adequate scaffolding and direction for both the teacher and the student in its requirements that students work collaboratively, communicate their thinking effectively in sharing their work with peer partners, critique the reasoning of peer partners, and construct a logical argument supporting their final draft. The Rubrics’ Common Scoring Scale: Rubric VII 50

51. Rubric VII – Strong Opportunities for Deeper Learning 2: An object is rated strong if it includes one or two deeper learning skills identified in this rubric. For example an eighth grade object is considered strong in its opportunities for deeper learning since it involves solving a complex mathematical problem that requires critical thinking and abstract reasoning skills to reach a solution The Rubrics’ Common Scoring Scale: Rubric VII 51

52. Rubric VII – Limited Opportunities for Deeper Learning 1: An object is rated limited if it includes one deeper learning skill identified in the rubric but is missing clear guidance on how to tap into the various aspects of deeper learning. For example as part of a fifth grade writing lesson, an object includes a provision for learners to collaborate, but the process and product are unclear. Since there is a suggestion to use collaboration, Rubric VII would apply but without clear explanation of how the collaboration is to take place, the object is limited in its opportunities for deeper learning. The Rubrics’ Common Scoring Scale: Rubric VII 52

53. Rubric VII – Very Weak Opportunities for Deeper Learning 0: An object is rated very weak if it appears to be designed to provide some of the deeper learning opportunities identified in this rubric, but it is not useful as presented. For example an object describes an activity based on poorly formulated problems and has unclear directions. It is unlikely that this activity will lead to skills like critical thinking, abstract reasoning, constructing arguments, or modeling. It would be considered very weak for Rubric VII. The Rubrics’ Common Scoring Scale: Rubric VII 53

54. Rubric VII – Opportunities for Deeper Learning - Not Applicable N/A: This rubric is not applicable (N/A) to an object that does not appear to be designed to provide the opportunity for deeper learning, even if one might imagine how it could be used to do this. For example an object focuses on practicing long division with two-digit divisors through the use of multiple exercises. Because it does not clearly describe the process to be used in presenting the exercises, the object does not provide clear opportunities for deeper learning and Rubric VII would be not applicable. The Rubrics’ Common Scoring Scale: Rubric VII 54

55. Rubric VIII – Assurance of Accessibility Accessibility is critically important for all learners and should be considered in the design of all online materials. Only those who are qualified to make judgments about an object’s accessibility should use this rubric. Rubric VIII is used to assure materials are accessible to all students, including students identified as visually impaired or print disabled. It was developed to assess compliance with U.S. standards and requirements, but could be adapted to accommodate differences in other sets of requirements internationally. Rubric VIII: Assurance of Accessibility 55

56. Rubric VIII – Assurance of Accessibility Identification of certain characteristics will assist in determining if materials will be fully accessible for all students. Assurance that materials are compliant with the standards, recommendations, and guidelines specified assists educators in the selection and use of accessible versions of materials that can be used with all students, including those with different kinds of challenges and assistive devices. Rubric VIII does not ask reviewers to make a judgment on the degree of object quality. Instead, it requests that a determination (yes/no) of characteristics be made that, together with assurance of specific Standards, may determine the degree to which the materials are accessible. 56 Rubric VIII: Assurance of Accessibility

57. Rubric VIII – Assurance of Accessibility – Yes/No Rating Yes: The object displays the characteristic or complies with the standards, recommendations or guidelines. No: The object does NOT display the characteristic or comply with the standards, recommendations or guidelines. The Rubrics’ Scoring Guide: Rubric VIII 57

58. Slides developed by Achieve, Inc.: www.achieve.org 1400 16th Street, NW / Suite 510 Washington, DC 20036