1. GCAT-SEEK Workshop, 2013 Dr. Tammy Tobin, Susquehanna University Adapted from Mary J. Allen, SACS-COC Summer Institute Developing and Using Rubrics to Assess Vision and Change Outcomes

2. Designing and Assessing Student Activities for Vision and Change Learning Goals - Overview Use “Backwards” Course Design  Identify student learning goal(s) first....  THEN decide on activities/topics that best allow students to gain the relevant skills, knowledge and/or abilities  THEN decide on a rubric (if needed) and make sure that it actually addresses the learning goal in ways that are appropriate for the activity

3. Identifying Student Learning Goals: V&C Core Concepts Evolution Structure and Function Information Flow, Exchange and Storage Pathways and Transformations of Energy and Matter Systems

4. Identifying Student Learning Goals: V&C Core Competencies Ability to Apply the Process of Science Ability to Use Modeling and Simulation Ability to Use Quantitative Reasoning Ability to Tap Into the Interdisciplinary Nature of Science Ability to Communicate and Collaborate with Other Disciplines Ability to Understand the Relationship Between Science and Society

5. Decide How to Best Assess Student Learning Direct vs. Indirect Assessment  Authentic Assessment Formative vs. Summative Assessment Imbedded vs. External Assessment

6. The GCAT SEEK Assessment Instrument The Challenge: Develop an assessment instrument that is broad enough to assess the outcomes of ALL GCAT- SEEK projects for ALL V&C concepts and competencies… Stage One: Develop broad learning goals for GCAT SEEK – Using V&C categories as guide Stage Two: Develop and implement a survey/quiz in Google Forms to assess participant progress on (some of) these learning goals Google Forms Stage Three: Refine the instrument/identify gaps in V&C coverage Stage Four: Use data to improve GCAT-SEEK activities

7. Rubrics Classify continuous behaviors (V&C Competencies!) into categories Can be developed to classify any student product Can be used to provide formative feedback to students, to grade students, and/or to assess programs Are most effective when they are generated at the same time as the assignment

8. Creating Rubrics Adapt a pre-existing rubric to your needs Loyola Marymount University has a good database of samples, including those from the AAC&U VALUE project. http://www.lmu.edu/about/services/academicplanning/assessment/ Assessment_Resources/Rubrics/Example_Rubrics.htm

9. Steps for Creating Your Own Rubrics 1. Write down the learning outcome(s) to be assessed: Vision and Change Core Competencies: (STUDENTS WILL BE ABLE TO) TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE. (STUDENTS WILL BE ABLE TO) COMMUNICATE AND COLLABORATE WITH OTHER DISCIPLINES. (STUDENTS WILL) UNDERSTAND THE RELATIONSHIP BETWEEN SCIENCE AND SOCIETY.

10. Steps for Creating Your Own Rubrics 2. Identify measurable central components or skills for the learning goals, using the Vision and Change descriptors as as your guide. Ability to tap into the interdisciplinary nature of science Interdisciplinary science practice may be achieved in several ways. For future biologists, one way is through developing expertise not just in an area of biology, but also in a related discipline. That way, students will develop the vocabulary of both disciplines and an ability to think independently and creatively in each as well. A second, less intensive approach is to develop deep expertise in one area and fluency in related disciplines. A third option is to serve as a biologist on a multidisciplinary team. All of these routes develop a student’s facility to apply concepts and knowledge across traditional boundaries. For those not majoring in biology, the inherent interdisciplinary nature of biology practice lends itself to forming connections between biology and other sciences and, in so doing, can help all students understand the way science disciplines inform and reinforce each other.

11. Steps for Creating Your Own Rubrics 3. Develop an assessment instrument: GCAT-SEEK Essay: Science is an interdisciplinary, collaborative process that can be of great use to society, but only if it is conveyed in a manner that makes its importance clear to both scientists and non-scientists. We are interested to see if you have this ability. Write a short (2-3 paragraph) essay in which you try to convince a charitable foundation to fund an interdisciplinary next generation sequencing project. In your essay you should briefly explain the project that you will be performing, why it is important to society, and how next generation sequence data will help you complete the project. Also convince the foundation to fund the work of at least one individual who is not a molecular biologist, but without whom your research project would be less meaningful or effective.”

12. Steps for Creating Your Own Rubrics 4.Decide which rubric type to use (holistic or analytic). Holistic Scoring - One global, holistic score for a product or behavior. See rubric examples 1-2 at end of packet Analytic Rubrics - Separate, holistic scoring of specified characteristics of a product or behavior See rubrics 3-4 at end of packet

13. ABILITY TO TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE. 5 Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective. Steps for Creating Your Own Rubrics 5. Create a descriptor for the highest student performance level. Keep it short, simple and measurable. Make sure the rubric asks the student to perform the task! Science is an interdisciplinary, collaborative process that can be of great use to society, but only if it is conveyed in a manner that makes its importance clear to both scientists and non-scientists. We are interested to see if you have this ability. Write a short (2-3 paragraph) essay in which you try to convince a charitable foundation to fund an interdisciplinary next generation sequencing project. In your essay you should briefly explain the project that you will be performing, why it is important to society, and how next generation sequence data will help you complete the project. Also convince the foundation to fund the work of at least one individual who is not a molecular biologist, but without whom your research project would be less meaningful or effective.”

14. Creating Your Own Rubrics 6. Identify the word(s) in the descriptor that can vary to convey different levels of performance Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective. Vary these words when writing your lower performance level descriptions.

15. Creating Your Own Rubrics 7. Create a descriptor for the lowest student performance level. Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective Does not accurately describe how either of the chosen techniques are necessary to make the research project chosen more meaningful or effective

16. ABILITY TO TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE 5 Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective 4 Accurately and completely describes how either next generation sequencing or at least one other discipline is necessary to make the research project chosen more meaningful or effective. However, the description of the role of the other technique, while accurate, is superficial 3 Accurately but superficially describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective 2 Accurately describes how either next generation sequencing or at least one other discipline is necessary to make the research project chosen more meaningful or effective. However, the description of the role of the other technique is inaccurate 1 Does not accurately describe how either of the chosen techniques are necessary to make the research project chosen more meaningful or effective 8. Decide how many levels to use, and write descriptors for the intermediate levels.

17. Practice, Practice, Practice … In teams, identify the central components or skills for the learning goals below, using the Vision and Change descriptors as as your guide (V&C page 15). (STUDENTS WILL BE ABLE TO) COMMUNICATE AND COLLABORATE WITH OTHER DISCIPLINES. (STUDENTS WILL) UNDERSTAND THE RELATIONSHIP BETWEEN SCIENCE AND SOCIETY:

18. Practice, Practice, Practice… Next, create a descriptor for the highest and the lowest student performance levels. Make sure they are measurable, and that they measure something the student was actually asked to do.

19. 5 4 3 2 1 Varying the appropriate terms, write descriptors for all of the levels of achievement in the chart below

20. Using Rubrics Pilot Testing the Rubric Managing Group Readings One reader per document Requires careful reader calibration! Two independent readers per document Paired readers

21. Analyzing the Data Which level of performance is "good enough“? Consider using a preformed computer spreadsheet to quickly enter and process your data as you go. Modify the rubric/prompt for the next cycle, as needed. Reward yourselves!