1. Jean Hughes Office of the Vice President for Learning Innovation, DCU

2. Overview DCU Approach Framework Implementation Network
Learning Outcomes
General – Assessment of Learning Outcomes
Common Questions

3. DCU Already Modularised and Semesterised (1997)
Module Descriptors use ‘Learning Outcomes’
European Credit Transfer System (ECTS) well established
But
Mainly administrative exercise
Academic Framework for Innovation (AFI) – for Deep Curriculum Reform (3 year project)
Bologna Compliance
Demonstrable Alignment with NFQ
Use of Learning Outcomes – for real change in assessment practice
Flexibility – Designed in and through new Marks & Standards

4. Framework Implementation Network
Created by the IUA and the NQAI
Membership
7 Universities
NUI Colleges
Purpose – to collectively address outstanding issues relating to Bologna and the National Framework of Qualifications
Three separate Working Groups
Award Titling
Disciplinary Learning Outcomes
Assessment of Learning Outcomes

5. Key Issues Ensuring that Assessment, Teaching and Learning are aligned
Moving from assessing content to assessing outcomes
Ensuring all outcomes are assessed and all assessment instruments actually assess outcomes
Avoiding over-assessment

6. Learning Outcomes
Clearly identify what a learner can demonstrate as a result of successfully completing a part of a learning programme
They describe student attainment which is demonstrable and assessable.

7. Benefits of Learning Outcomes
Guide students in their learning - what is expected of them - in turn helping them to succeed in their studies.
Help lecturers to focus on exactly what they want students to achieve in terms of both knowledge and skills.
Provide a useful guide to inform employers about the general knowledge and understanding that a graduate will possess.

8. Challenges Traditional view
inputs = quality of learning outcomes
Moving from content (Teaching) focus to Outcome (Learning) focus
Aligning teaching, learning and assessment

9. Award and Module Outcomes
NFQ describes generic, high level outcomes for awards at each level under 3 strands and 8 substrands
How do we demonstrate achievement of award outcomes?
We don’t assess at award level we assess at module level
Need to ensure that module learning outcomes collectively deliver the award outcomes

10. PO1 PO2 PO3 PO4 PO5 PO6 Alignment of Programme and Module Outcomes
PO – Programme Outcome
Mod - Module

11. Assessing Learning Outcomes

12. Constructive Alignment
is the underpinning concept behind the current requirements for programme specification, declarations of intended learning outcomes and assessment criteria, and the use of criterion based assessment.
(Biggs, 1999)

13. Example – Project Management
Learning Outcome
(Mismatched)
(Matched)
Tasks
Students will understand how to plan a multi-disciplinary project
Students will demonstrate ability to plan a multi-disciplinary project
Assignment 1
Students will complete an in-class test on Project Management
Students will devise an appropriate project plan from a scenario/business case study
Assignment 2
Students will create a Project Plan using MS Project during labs
Students will create a Project Plan in MS Project from the plan in Assignment 1. They will be demonstrate appropriate use of Base Lines, Serial and Parallel Tasks, Variances, Resource Allocation and Critical Path, through an in-lab demonstration.
Final exam
1 Question on final exam
Case Study exam question requiring analysis of a Business Problem, design of appropriate project plan and discussion of variety of project planning issues.

14. Student Perspective
From our students’ point of view, assessment always defines the actual curriculum (Ramsden, 1992)
Biggs (2003)
Teacher Perspectives:
Objectives Learning Outcomes Teaching Activities Assessment
Assessment Learning Activities Outcomes
Student Perspectives
Alignment
If curriculum is reflected in the assessment, the teaching activities and the learner activities are directed towards the same goal – in preparing for the assessment students will be learning the curriculum

15. Assessment – Impact (Boud, 1995)
“There is probably more bad practice and ignorance of significant issues in the area of assessment than in any other aspect of higher education”.
“The effects of bad practice are far more potent than they are for any aspect of teaching. Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment”.

16. Assessment - Terminology
Learning Outcomes
What a student should be able to demonstrate
(May imply the assessment criteria)
Assessment Criteria
The basis on which a judgement of the adequacy of the student work is made
Assessment Methods
The task(s) undertaken by the student – writing an essay, answering an exam question etc- that is subject to assessment.
Many assessment methods may be appropriate to the same outcomes – e.g. Written, Oral, Visual, Project etc
Must be valid and reliable – Fit for purpose

17. Over-Assessment Typical with semesterisation and modularisation
High risk with learning outcomes
Tend to be quite granular and have several independent outcomes
To avoid
Try combining the assessment of more than one LO
LOs only need to be assessed once
Use assessments which inform/follow on from each other
Check what you are assessing

18. Checking Assessment         Issues:
What does assignment 3 assess?
How is Learning Outcome 2 assessed?
Are Learning Outcomes 1 and 4 over-assessed?

19. Refining Assessment Perhaps design a new assessment 3 to re-balance? 

20. Common Questions

21. Must all outcomes be assessed?
Yes, but
Not independently
Look at how you write the learning outcomes
Students are expected to be able to
Write a business report
Present orally
Design a visual presentation
versus
Communicate effectively using a combination of written reports, oral presentations and visual media
This reduces the number of independent outcomes and also recognises the combination of communication mechanisms
BUT
For certain disciplines, for example languages, you may need these to be independently demonstrated

22. Example In a Business Module (1 outcome)
Students are expected to be able to communicate effectively in a business context using appropriate communication tools including Written, Oral and Visual
The outcome is communication, the tools are some means of demonstrating that outcome
In a languages Module (4 outcomes)
Students are expected to be able to demonstrate written proficiency in French
Students are expected to be able to translate a piece of French text
Students are expected to be able to comprehend a piece of French text
Students are expected to be able to deliver an oral presentation in French

23. Must all outcomes be passed?
In theory, yes, but…
Threshold Achievement
Implies that ALL learning outcomes must be passed in order to pass a module
Very strict interpretation of LO approach
Leads to reductionist or ‘Tick Box’ approaches
Typical Achievement
Learning outcomes are written (and assessed) with the ‘typical’ or ‘’average’ student in mind
Grading criteria will allocate marks to degree of achievement

24. But, we don’t currently assess everything….
Exams often require only 3 out of 5 questions
Therefore we don’t assess everything
But
Generally this refers to content
Outcomes should be about demonstrating knowledge/understanding aided by knowledge of content, not content in its own right
Assessment criteria and methods should satisfy you that students can demonstrate achievement of the outcomes
Worst-case scenario – no worse than current situation!

25. What if some outcomes are failed?
Depends on the criticality of the outcome
If particular outcomes must be passed then passing other outcomes should not be able to compensate
Need to look at ‘Threshold’ and ‘Typical’ Outcomes
Need to write outcomes carefully
Need to ensure that supplemental/repeat assessment is also valid

26. What about repeats? Did they actually fail?
“Students will present orally on 20th century politics”
If they don’t attend they fail….. Is this what you meant? or
“Students will effectively communicate on 21th century politics”
You may prioritise Oral assessment and allocate more marks, but this allows more choice of methods
If a student has to repeat the supplemental assessment must be valid if different from the original
Look at how you write the outcomes

27. Is it an Outcome, An Assessment or an Efficiency?
Often mix up pedagogical (or efficiency) approaches with learning outcomes
Example Group Working
Learning Outcome
‘Students will demonstrate effective group working skills’
Group working needs to be ‘taught’ and assessed
Can’t assume that putting people in groups will automatically lead to effective group working!
How does an individual student repeat?
‘Students will be able to evaluate group dynamics and effective group working’
Would this suffice but give an alternative assessment approach for repeat students?
Pedagogical Approach
Group working is used to facilitate learning of a particular outcome
Not assessing the group working – assessing the actual outcome
Efficiency
Often used to reduce the volume of marking
Need to be careful that it is pedagogically sound
Not assessing group working

28. Making the Tacit Explicit
Some disciplines rely on expertise of the individual lecturer
‘I know a good piece of work when I see it’
‘You can’t possibly express Creativity, Performance, Medieval English….. Using learning outcomes
NOT about subjectivity – expertise, tacit knowledge etc
How do students know what to aim for?
How do novice lecturers know about standards?
How can we demonstrate consistency, transparency etc?

29. Example First, rewrite the outcome: Redesigned:
“By the end of this module students will understand the importance of post-renaissance European art in the context of art history”
Redesigned:
By the end of this module students will be able to:
Evaluate and criticise post-renaissance European art within its historical context
Date and identify key works of this specific period
Demonstrate an understanding of the particular art-historical debates that have taken place about this period
Show ability to argue why this particular place and time raise interesting issues about the discipline as a whole
University of Sussex

30. Then, use a rubric Also called grading criteria or scoring guide
A set of guidelines for marking or scoring
Can be a list, chart, or guidelines
They state all dimensions being assessed
They contain a scale
They assist the marker on rating assessments on the scale

31. Rubrics - 3 Types Checklists Rating Scales Holistic Scoring Guide
Indicating list of criteria
Rating Scales
Checklist with rating scale added
Show degrees by which students meet criteria
Holistic Scoring Guide
Used where assignments can vary i.e. projects
Narrative descriptions of characteristics of different grades of work
Not list of criteria

32. Rubric Design Criteria Achievement Levels Objective 1 Objective 2
Excellent
Good
Needs Work
Not acceptable
Objective 1
Accepted Minor Major Rejected
revision revision
Objective 2
Expert Advanced Intermediate Novice
Objective 3

33. Simple Checklist Rubric

34. Rubric is for a written Business Assignment

35. This assignment asked students to take a position on a debateable issue regarding interpretation of literature they had studied. The lecturer has identified 3 major criteria – Position, Support and Acknowledgement of Alternative Points of View and has weighted them different levels of attainment of each criterion. Opposite is the detailed rubric for the Position criterion.

36. In Summary
Need to write learning outcomes so that they can be demonstrated and assessed
Need to identify assessment criteria and appropriate assessment methods
Need to be sure exactly what it is students need to demonstrate
Making the tacit explicit – Clearly written outcomes and the use of rubrics to articulate achievement

37. Questions?