1. Blackboard Rubrics: The good, the bad, and the ugly
Adel Gordon & Pepijn Kalis
University of Northampton | Blackboard
10th April 2013

2. Introductions Adel Gordon @adelgordon
University of Northampton
Learning Technologist
Been at Northampton for 13 years, an LT for 6
Pepijn Kalis
Blackboard EMEA
Sr. Specialist
Working for Blackboard for 5 years

3. Rubrics What are rubrics? Scoring guide Authentic assessment tool
Working guide for students

4. Let’s do some tasting assessing

5. Chocolate Chip Cookies4
Delicious 3
Good 2
O.K. 1
Poor
Number of Chips
Chocolate chip in every bite
Chips in about 75% of bites
Chocolate in 50% of bites
Too few or too many chips
Texture
Chewy
Chewy in middle, crisp on edges
Texture either crispy/crunchy or 50% uncooked
Texture resembles a dog biscuit
Color
Golden brown
Either light from overcooking or light from being 25% raw
Either dark brown from overcooking or light from undercooking
Burned
Taste
Home-baked taste
Quality store-bought taste
Tasteless
Store-bought flavor, preservative aftertaste – stale, hard, chalky
Richness
Rich, creamy, high-fat flavor
Medium fat contents
Low-fat contents
Nonfat contents

6. Let’s see the cookie rubric in Blackboard
Demo

7. The good,…
Rubrics for teaching
Rubrics for learning
In practice

8. Rubrics for teaching Clarify learning goals from the offset
Design materials and activities that address those goals
Communicate those goals to students
Guide feedback on students’ progress
Assess products to degree to which the goals are met
Andrade, 2005
Enable timely grade allocation whilst justifying them at the same time
Feedback > Feed Forward …
Where as a lot of student feel that rubrics help with their learning by setting out goals, tutors perspectives of using rubrics tend to lean towards the grading advantages... they serve a good purpose of marking, enabling them to mark consistently and with more objective focus.
Andrade reported that in areas where resistance/hesitation to rubrics existed resulted from a lack of understanding of the learning and teaching benefits of a rubric. It was felt that tutors would be more receptive if they understood how rubrics can enhance learning and teaching.

9. … teaching Rubric design based on learning outcomes
PLAN
TEACH
ASSESS
REFLECT
Make adjustments to teaching based on reflections
Emphasise the use of rubrics
Identify common areas of strengths and weaknesses
Teaching model adapted slightly from Introduction to Rubrics by Stevens and Levi.
Northampton CAIEROS
- module development process
- teaching teams start from scratch
- learning and teaching specialist
- learning technologist
- librarian
- student
- support and align assessment with learning outcomes, activities
- use a rubric to layout criteria, aligned with LOs
* appropriate/understandable language
A few studies that I've read promote the design of rubrics for assessment with the students. Positing that students are more likely to use one if they played a part in designing it than when it see it only as a way for the tutor generate grades. > takes us back to "emphasising the use of rubrics to students"...
Jessica Poettecker student union - Develop Partnerships students. Involved them in the development of their learning... Including them in the developments of rubrics can allow for this
I would like to look into the use of a rubric for the module development process for programme/module design and evaluating programmes
At Northampton we use a module development process where teaching teams start from scratch to develop a module. Throughout this process the teaching team, together with a learning and teaching specialist, learning technologist, student and librarian set out learning outcomes, activities etc to support and align assessment with outcomes. Using a rubric to lay out the criteria for the assessments - aligned with learning outcomes - makes everything really simple as long appropriate/understandable language is used in the marking criteria.
Look for patterns
Score student work using rubric
Adapted from Stevens & Levi, 2013

10. Rubrics for learning Learning Learner Face to face Online Experiential
Surface
Strategic
Deep thinker
Learning:
Theme throughout all types of learning is similar – the teaching model is used where refinements are made throughout the overall end to end process learning is improved.
Tailored
Transparent goals
Reducing uncertainty by identifying critical issues in an assignment
Evaluating their own performance in order to get immediate feedback
Focus efforts for subsequent assignments
"Instructional illuminators" Popham, 1997
Learner:
Surface learner – bare minimum
Strategic – do what is expected to get best grade they can
Deep thinker – everything plus more

11. In practice
Rubrics set out expectations to aid the student to understand what they’re being graded against
Rubrics let students know how their grade was calculated and where they could improve their work
Develops/drives a level of professionalism and enables me to provide more timely feedback
Moderation can take place immediately and feedback is ready to be released speedily
Rubrics give the marker confidence that you can been more objective then subjective
Makes calculating the overall grade easier when using a multifaceted approach to assessment criteria
The definitions defined in the marking criteria provide detailed explanations of what a student must do to demonstrate a skill, proficiency, or criterion in order to attain a particular level of achievement. Excellent, very good, good satisfactory etc...
Scoring, or marking, work based on these definitions makes the students job of interpreting the tutors judgment more easy to understand the targets for their learning
When marking in teams consistency needs to be monitored. In the Tii case study on marker was marking much more harshly than the others which, because of problems with moderation (later), wasn't picked up until after grades were released.

12. Overview of workflow for assessment process
Student submission
First grading
Internal moderation
Provisional grade & feedback to students
Record grade in SIS
External moderation
Overview of workflow for assessment process
External moderation
Record grade in SIS
Provisional grade & feedback to students
Internal moderation
First marking
Student Submission

13. … the bad, Reporting An actual report
However, this report works when working to identify patterns of strengths and weaknesses in achievement of assessment criteria across a set of students.
Practicalities of using the report within the workflow…
Outcomes system… ?

14. .. and the ugly Intuitiveness Usability Importing/exporting Saving
Integration
It works, but mostly it feels more like a workaround or something that is an after thought that hasn’t quite been integrated properly
Next slide... Demo...

15. In practice Let’s have a look: http://nile.northampton.ac.uk
Go into Adel-apr browsers?
Demo Kaltura, Turnitin, journal, presentation.

16. Blackboard vs Turnitin
Feature
Blackboard rubric
Turnitin rubric
Group submissions
Tutors can add personalised feedback to all/any of the marking criteria
Multiple markers
Export marked work with rubric summary
Marking by more than one tutor –
Bb - use smart views
Tii – Turnitin groups
Exporting
Bb – No, reporting issue
Tii – GradeMark paper download includes rubric
Use either Blackboard or Turnitin rubrics for:
Individual assignments
Setting an indefinite number of criteria and score ranges
Setting marks by points or percentage or no marks (feedback only)
Specifying feedback for each possible score/criterion in advance
Automatically adding up marks
Manually overriding marks if required
Re-use with future assessments
Exporting and importing rubrics (to allow sharing with others)
*Limitations
Blackboard rubrics cannot be printed by academics once completed – for example – for moderation purposes.   (A change request has been submitted to Blackboard for this but this will not be available until at least 2013.)
If adding comments to a Blackboard rubric, remember to save your work often – it does not auto-save so will otherwise be lost in the event of a time-out or computer crash.
University of Manchester

17. References
Andrade (2005), Teaching with Rubrics: The Good, the Bad, and the Ugly. College Teaching, [online] Available at: Stevens, D. D. and Levi, A. J., Introduction to Rubrics. Virginia: Stylus Publishing University of Manchester, Rubrics – What are they? Why and how should I use them? [online] Available from: Utah Education Network – Original Chocolate Chip Cookies. [online] Available at:

18. Contact Adel Gordon Pepijn Kalis University of Northampton
Pepijn Kalis
Blackboard