1. A Way to Approach the Research Design Criteria
For Empirical/Quantitative Skills Assessment
Gray Scott, Ph.D.,

2. The Situation
Suppose you are teaching research design skills and want to assess them. You could have each student design a research project, but in large science classes the resulting grading load could be severe. You could have them design in teams, cutting down on the number of projects, but then it’s no longer possible to evaluate the skills of an individual on that team. You could have students evaluate an existing research design, and then assess their skills based on what they say in the evaluation. Unfortunately, if the research methods were professionally developed, students may not be skilled enough to critique them.
But you could have them evaluate research designs created by other students...

3. Overview
The basic idea behind this Assignment is that students work in teams to develop (rapidly, and in- class) a research plan or methods statement for how they might address a research question that you have posed.
Then, writing singly, each student evaluates another team’s drafted design, focusing on criteria that you have selected. We recommend assigning the evaluations so that each student within a group evaluates a different team’s project. Those evaluations are submitted as Artifacts for assessment.
When the students return to class, you can then have students assemble into new groups based on which designs they reviewed. (Example: Everyone who evaluated Team 1’s design gathers to compare notes. This is henceforth referred to as the “review group.”) The review group reaches a consensus about what its target design team should do differently. The best review team should also have a chance at extra credit.
The design teams then reassemble to review their feedback. They may revise based on what they read.

4. Grading the Assignment
The Assignment described on the previous page is designed to spare the faculty from a lot of grading. Though it does consume class-time as a trade-off, the time spent is likely to be pedagogically worthwhile, as long as your goal is to teach research design skills.
How the sequence is graded:
The design group stage: As credit/no-credit participation in an in-class activity.
The individual evaluation stage: Completed individual evaluations are required to participate in the review groups and recorded (credit/no-credit) as homework.
The review team stage: As participation in an in-class activity. The team that writes the best review earns a small amount of extra credit.
The (optional) revision stage: The team with the best research design earns a small amount of extra credit. By discussing your reasons for picking one project over the others, you “close the loop” on the sequence.

5. Building the Assignment
The slides that follow describe, in steps, how to build such an assignment.

6. Step 1: Find Your Criteria

7. Where to Find Assessment Criteria
Address assessment/7008.asp
Navigation TWU / Academic Assessment / Core Assessment / (Google Drive)
What to Do Open the Empirical/Quantitative Skills rubric Keep that rubric open as you move to Step 2.

8. Step 2: Review the Research Design Criteria
The next page lists criteria often associated with research design work. You can find those criteria on the Empirical/Quantitative Skills rubric.

9. Which Criteria to Review
Assignment EVALUATING RESEARCH DESIGNS
Relevant Objective EMPIRICAL/QUANTITATIVE SKILLS
Appropriate Criteria 1. Define Problem
2. Assumptions 3. Propose Solutions/Hypotheses
4. Research Design

10. Step 3: Read the Performance Descriptors
Those are the cell-by-cell descriptions of high-, middle-, and low-quality performances. After reading them, you should choose 3 to 5 of the criteria that best apply to what the students are doing. It’s okay to pick only three.

11. Step 4: Create Your Prompt
Come up with a research question for your research design teams. In the prompt that you create, include the criteria that the teams should be trying to meet--and that the individuals will be assessing later. They should know what the standards are.

12. Step 5: Create Teams and Run the Activity
The larger the class, the more time scrambling among teams can take. If you already have methods for creating teams within a large hall, feel free to skip the next slide.

13. A Team-Creation Strategy
To set up the initial teams, take a student roster and re-order the list of students by first name, student ID, or a similar variable. Take the first 4, 5, or 6 students and put them on Design Team 1. Do the same for the next 4-6 students, but aim for roughly equal-sized groups each time. (Try not to go higher than 6 students per team.) Continue in this manner until all students are assigned to a Design Team.
Print out sheets of paper, each with the number of one of the teams on it and the names of the team members. Have students tape up the team sheets around the room, evenly distributed. Then have students find their design teams.
As soon as they’ve found their design teams, have each member of the team choose the number of another team, with no two students from the same design team overlapping in their choices. Have them record those choices. The teams they choose will be the ones that they review. When they assemble into review groups, they will meet with other students who picked and reviewed the same teams.

14. Step 6: Fill Out an Assignment Profile on CASSIE
An Assignment profile tells us what you have asked your students to do. By completing the form, you tell us what you did and which criteria our raters should be scoring. CASSIE is our online core-assessment system, a link to which appears on the following page, along with instructions.

15. CASSIE Creating Assignments
Address
Navigation TWU/Pioneer Portal/My Tools/Core Assessment
Tasks 1. Click Assignments on the top menu
2. Pick term, course
3. Enter title and instructions. Mention you’re using this approach.
4. Identify objective(s)
5. Identify relevant criteria (3-5 per objective)
6. Attach any supplemental information that raters might need.

16. Step 7: Submit Artifacts Through CASSIE
Submit individual evaluations that students wrote as Artifacts through CASSIE by following the instructions on the next page. Do this for each student in your sample.

17. CASSIE Uploading Artifacts
Address
Navigation TWU/Pioneer Portal/My Tools/Core Assessment
Tasks 1. Choose Artifacts from the top menu
2. Pick term, course, student(s) (up to 3 at a time)
3. Choose Assignment that artifact fulfills
4. Attach artifacts
5. If student was not peer-rated, check the No Artifact Available option.

18. Step 8: That’s It!