1. Using Learning Outcomes and Assessment Criteria
Peter Noakes
Department of Electronic Systems Engineering
University of Essex

2. LTSN Workshop at University of Essex
Purpose of this Talk
To provide an introduction to topics and related background information that will help us to provide more concise module specifications
To improve your efficiency in generating appropriate Learning Outcomes, Assessment Methods and Assessment Criteria for the modules you teach
At the same time provide better transparency for students by making module specifications clearer and unambiguous
and improve students commitment to self learning by clarifying what is expected of a student
Consequently this should improve student progression and encourage better student retention
LTSN Workshop at University of Essex
26th June 2003

3. LTSN Workshop at University of Essex
Structure of Talk
Background
Learning Outcomes
Level Descriptors (QAA, SEEC, EPC)
Assessment Methods
Assessment Criteria
An Example for You
Report Back and Discussion
LTSN Workshop at University of Essex
26th June 2003

4. LTSN Workshop at University of Essex
Reference Sources
This presentation is based on the content of the following publications:
“How to use Level Descriptors” by Jenny Moon, SEEC*, 2002
“How to Use Learning Outcomes and Assessment Criteria” by David Gosling and Jenny Moon, SEEC*, 3rd Edition 2002
“Assessment: A Guide for Lecturers” by George Brown, LTSN Generic Centre, 2001
“Guide for Busy Academics”, LTSN Generic Centre
*Southern England Consortium for Credit Accumulation and Transfer
( )
LTSN Workshop at University of Essex
26th June 2003

5. LTSN Workshop at University of Essex
Background
A Programme defines study or learning required to achieve an award or qualification
A Programme Specification is required by the QAA for each award or qualification and defines the threshold learning outcomes for the programme
A Programme comprises a number of Modules each of which is separately assessed and earns credit when successfully completed
Using the outcomes model each Module Description defines the intended (threshold?) learning outcomes, the syllabus coverage and the assessment methods and criteria for the module.
Achievement of Module Learning Outcome should contribute to a student’s satisfaction of the programme learning outcomes
LTSN Workshop at University of Essex
26th June 2003

6. Learning Outcome-based Model
Traditionally an academic would first define the syllabus coverage, then develop how its taught and finally determine the method of assessing the student’s absorption of the material.
The outcome-based model has three interconnected components:
An explicit statement of learning intent (intended learning outcome) which focuses on what the student is expected to know and be able to do by the end of the module, expressed in a form that permits their achievement to be demonstrated and measured
The processes and resources to enable the outcomes to be achieved and demonstrated (curriculum, teaching, learning methods and materials, assessment and support and guidance methods)
The criteria for assessing whether the intended learning outcomes have been achieved and for differentiating the performance of students.
They are dependent on the “level” at which the module is targeted
LTSN Workshop at University of Essex
26th June 2003

7. Level / Qualification Descriptors?
A Level is an indicator of the relative demand, complexity, depth of study and learner autonomy
A Level (Qualification) Descriptor is a generic statement describing the characteristics and context of learning expected at each specific level.
Module Learning Outcomes and Assessment Criteria are reviewed with respect to a level descriptor when developing a module and assigning credit at the appropriate level.
QAA defines Qualification Descriptors in the Qualification Framework for Higher Education as Level C, Level I, Level H and Level M. These broadly correspond to Years 1, 2, 3 and first year postgraduate level respectively
SEEC defines Level Descriptors as Level Zero, Level 1, Level 2, Level 3 and Master’s Level. These broadly correspond to preliminary Year, Years 1, 2, 3 and first year postgraduate level respectively
LTSN Workshop at University of Essex
26th June 2003

8. LTSN Workshop at University of Essex
Level Descriptors?
Think of Level Descriptors as a means of communication about expectations of student’s study
They are not rigid but developmental
Student Learning is commonly described in terms of :
complexity of knowledge and understanding
standard of cognitive skills
key or transferable skills achieved
the expected responsibility of the learner
the autonomy or independence of the learner
amount of guidance required by the learner
Be careful of differences in the implied standard of learning!
Is it defined for a threshold student, an average (or typical!) student or the best student?
LTSN Workshop at University of Essex
26th June 2003

9. Hierarchy of the Cognitive Domain
Evaluation Ability to make a judgement of the worth of something
Synthesis Ability to combine separate elements into a whole
Analysis Ability to break a problem into its constituent parts and establish the relationships between each one
Application Ability to apply rephrased knowledge in a novel situation
Manipulation Ability to rephrase knowledge
1 Knowledge That which can be recalled
Based on Bloom’s Taxonomy of Educational Objectives
Bloom first published his Taxonomy of Educational Objectives in 1956 and it is helpful to review his Hierarchy of the Cognitive Domain shown here in relationship to the ideas of level descriptors. The hierarchy starts at the lowest level with knowledge that which can be recalled and moves through what are increasingly more difficult levels to the highest Evaluation where the ability to make judgment is expected.
LTSN Workshop at University of Essex
26th June 2003

10. Programme Learning Outcomes
Guidance is provided by:
QAA Level H Descriptors and Benchmark Statements for
Engineering
Computing
Biosciences
SEEC’s generic HE Level 3 Definition
Engineering Professors Conference’s 26 “Ability to” Statements for Engineering Programmes
Also see the accreditation requirements used by Professional Bodies (e.g. Engineering Council’s SARTOR 97 or latest UK SPEC)
Programme Learning Outcomes are now required for all degree programmes. QAA provides guidance in preparation of these through the Qualification Framework’s Level H Descriptors and the relevant Benchmark statements which relate to the output from the final year of the degree programme.
The Southern England Consortium for Credit Accumulation and Transfer (SEEC) has developed Level Descriptors which target each year of a 3 year degree. They allow a broader awareness of progression of learning through the 3 years.
With respect to Engineering the EPC has produced 26 ability to statements which it believes a new graduate should be able to demonstrate on completion of an Engineering degree. Our experience is that these are useful for overall degree outcomes but are difficult to link to module outcomes. An investigation is underway to see if it is possible to merge the EPC “ability to” statements with the Engineering Benchmark guidelines.
In some disciplines Professional accreditation requirements also have an impact on defining programme outcomes.
LTSN Workshop at University of Essex
26th June 2003

11. QAA General Honours Descriptors Level H - A
Students successfully completing programme requirements at this level will have demonstrated:
a systematic understanding of key aspects of their field of study, including acquisition of coherent and detailed knowledge, at least some of which is at or informed by, the forefront of defined aspects of a discipline;
an ability to deploy accurately established techniques of analysis and enquiry within a discipline;
conceptual understanding that enables the student:
to devise and sustain arguments, and/or to solve problems, using ideas and techniques, some of which are at the forefront of a discipline; and
to describe and comment upon particular aspects of current research, or equivalent advanced scholarship, in the discipline;
an appreciation of the uncertainty, ambiguity and limits of knowledge;
the ability to manage their own learning, and to make use of scholarly reviews and primary sources (e.g. refereed research articles and/or original materials appropriate to the discipline).
The next two slides provide a copy of the QAA Level H descriptor and I guess these are the most influential in our generation of programme descriptions for the output of degree programmes. This slide defines what activities should be included within a programme. I am not going to talk about these in any detail but I suggest you review them later.
LTSN Workshop at University of Essex
26th June 2003

12. QAA General Honours Descriptors Level H - B
Typically, successful students at this level will be able to:
apply the methods and techniques that they have learned to review, consolidate, extend and apply their knowledge and understanding, and to initiate and carry out projects;
critically evaluate arguments, assumptions, abstract concepts and data (that may be incomplete), to make judgements, and to frame appropriate questions to achieve a solution - or identify a range of solutions - to a problem;
communicate information, ideas, problems, and solutions to both specialist and non-specialist audiences;
and will have:
qualities and transferable skills necessary for employment requiring:
the exercise of initiative and personal responsibility;
decision-making in complex and unpredictable contexts; and
the learning ability needed to undertake appropriate further training of a professional or equivalent nature.
The ability statements here provide guidance on expected programme learning outcomes.
LTSN Workshop at University of Essex
26th June 2003

13. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors: Development of Knowledge and Understanding
HE Level 1
HE Level 2
HE Level 3
The Learner has a given factual and /or conceptual knowledge base with emphasis on the field of study and appropriate technology
The Learner has a detailed knowledge of major theories of the discipline and an awareness of a variety of ideas, contexts and frameworks
The Learner has a comprehensive / detailed knowledge of a major discipline with areas of specialisation in depth and awareness of the provisional nature of knowledge
The Learner can demonstrate awareness of ethical issues in current areas of study and is able to discuss these in relation to personal beliefs and values
The Learner is aware of the wider social and environmental implications of area of study and is able to debate issues in relation to more general perspectives
The Learner is aware of personal responsibility and professional codes of conduct and can incorporate a critical ethical dimension into a major piece of work
Knowledge Base
The next 8 slides detail the SEEC Generic Level Descriptors. Loosely Level 1 approximates to Year 1, Level 2 approximates to Year 2 and Level 3 approximates to Year 3.
However these are really only generic guidelines are not definitive in each way. It is very helpful to look across the development of the various sections across the three levels as it illustrates the growth of expected learning with time.
This slide considers knowledge and understanding
Ethical Issues
LTSN Workshop at University of Essex
26th June 2003

14. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Cognitive / Intellectual Skills - A
HE Level 1
HE Level 2
HE Level 3
The Learner can analyse with guidance using given classifications / principles
The Learner can analyse a range of information with minimum guidance using given classifications / principles and can compare alternative methods and techniques for obtaining data
The Learner can analyse new and / or abstract data and situations without guidance, using a range of techniques appropriate to the subject
The Learner can collect and categorise ideas and information in a predictable and standard format
The Learner can reformat a range of ideas and information towards a given purpose
The Learner with minimum guidance can transform abstract data and concepts towards a given purpose and can design novel solutions
Analysis
The next two slides consider and intellectual skills. Here we look at the use of words to define a clear development of analysis and synthesis skills over the three levels.
Synthesis
LTSN Workshop at University of Essex
26th June 2003

15. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Cognitive / Intellectual Skills - B
HE Level 1
HE Level 2
HE Level 3
The Learner can evaluate the reliability of data using defined techniques and / or tutor guidance
The Learner can select appropriate techniques of evaluation and can evaluate the relevance and significance of the data collected
The Learner can critically evaluate evidence to support conclusions / recommendations, reviewing its reliability validity and significance. Can investigate contradictory information / identify reasons for contradictions
The Learner can apply given tools / methods accurately and carefully to a well defined problem and begin to appreciate the complexity of the issues
The Learner can identify key elements of problems and choose appropriate methods for their resolution in a considered manner
The Learner is confident and flexible in identifying and defining complex problems and can apply appropriate knowledge and skills to their solution
Evaluation
As indicated in Bloom’s Hierarchy evaluative skills are important and are shown here developing over the three levels. It is not restricted to later levels.
Application
LTSN Workshop at University of Essex
26th June 2003

16. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Key / Transferable Skills - A
HE Level 1
HE Level 2
HE Level 3
The Learner can work effectively with others as a member of a group and meet obligations to others (e.g. tutors, peers and colleagues).
The Learner can interact effectively with a team / learning group, giving and receiving information and ideas and modifying response where appropriate.
The Learner can interact effectively with a team / learning group / professional group, recognise, support or be proactive in leadership, negotiate in a professional context and manage conflict.
The Learner can work within an appropriate ethos and can use and access a range of learning resources.
The Learner can manage learning using resources for the discipline. Can develop working relationships of a professional nature within the discipline.
The Learner with minimum guidance can manage own learning using a full range of resources for the discipline. Can work professionally within the discipline.
Group Working
The next 4 slides refer to the development of Key and Transferable skills and probably need little comment.
Learning Resources
LTSN Workshop at University of Essex
26th June 2003

17. Information Mangement
SEEC Generic Level Descriptors : Key / Transferable Skills - B
HE Level 1
HE Level 2
HE Level 3
The Learner can evaluate own strengths and weakness within criteria largely set by others.
The Learner can evaluate own strengths and weakness, challenge received opinion and develop own criteria and judgement.
The Learner is confident in application of own criteria of judgement and can challenge received opinion and reflect on action. Can seek and make use of feedback.
The Learner can manage information, collect appropriate data from a range of sources and undertake simple research tasks with external guidance.
The Learner can manage information. Can select appropriate data from a range of sources and develop appropriate research strategies.
The Learner can select and manage information, competently undertake reasonably straight-forward research tasks with minimum guidance
Self Evaluation
Information Mangement
LTSN Workshop at University of Essex
26th June 2003

18. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Key / Transferable Skills - C
HE Level 1
HE Level 2
HE Level 3
The Learner can take responsibility for own learning with appropriate support.
The Learner can take responsibility for own learning with minimum direction.
The Learner can take responsibility for own work and can criticise it.
The Learner can communicate effectively in a format appropriate to the discipline and report practical procedures in a clear and concise manner.
The Learner can communicate effectively in a manner appropriate to the discipline and report practical procedures in a clear and concise manner in a variety of formats.
The Learner can engage effectively in debate in a professional manner and produce detailed and coherent project reports
Autonomy
Communications
LTSN Workshop at University of Essex
26th June 2003

19. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Key / Transferable Skills - D
HE Level 1
HE Level 2
HE Level 3
The Learner can apply given tools / methods accurately and carefully to a well defined problem and begin to appreciate the complexity of the issues in the discipline.
The Learner can identify key areas of problems and choose appropriate tools / methods for their resolution in a considered manner.
The Learner is confident and flexible in identifying and defining complex problems and the application of appropriate knowledge, tools / methods to their solution.
Problem Solving
LTSN Workshop at University of Essex
26th June 2003

20. LTSN Workshop at University of Essex
SEEC Generic Level Descriptors : Practical Skills
HE Level 1
HE Level 2
HE Level 3
The Learner can operate in predictable, defined contexts that require use of a specified range of standard techniques.
The Learner can operate in situations of varying complexity and predictability requiring application of a wide range of techniques.
The Learner can operate in complex and unpredictable contexts, requiring selection and application from a wide range of innovative or standard techniques.
The Learner is able to act with limited autonomy, under direction or supervision, within defined guidelines.
The Learner is able to act with increasing autonomy, with reduced need for supervision and direction, within defined guidelines.
The Learner is able to act autonomously, with minimal supervision or direction, within agreed guidelines.
Application of Skills
The final slide defining the SEEC Level descriptors looks at the development of practical skills.
Autonomy in Skill use
LTSN Workshop at University of Essex
26th June 2003

21. LTSN Workshop at University of Essex
Engineering Professors Conference EPC Generic “Ability to” statements A
Ability to exercise Key Skills in the completion of engineering-related tasks at a level implied by the benchmarks associated with the following statements.
Key Skills for engineering are Communication, IT, Application of Number, Working with Others, Problem Solving, Improving Own Learning and Performance.
Ability to transform existing systems into conceptual models.
This means the ability to:
Elicit and clarify client's true needs.
Identify, classify and describe engineering systems.
Define real target systems in terms of objective functions, performance specifications and other constraints (i.e. define the problem).
Take account of risk assessment, and social and environmental impacts, in the setting of constraints (including legal, and health and safety issues).
Select, review and experiment with existing engineering systems in order to obtain a database of knowledge and understanding that will contribute to the creation of specific real target systems.
Resolve difficulties created by imperfect and incomplete information.
Derive conceptual models of real target systems, identifying the key parameters.
Finally the next three slide tabulate the EPC Ability 2 statements. Again these are provided for reference rather than detailed discussion.
LTSN Workshop at University of Essex
26th June 2003

22. LTSN Workshop at University of Essex
Engineering Professors Conference EPC Generic “Ability to” statements B
Ability to transform conceptual models into determinable models.
This means the ability to:
Construct determinable models over a range of complexity to suit a range of conceptual models.
Use mathematics and computing skills to create determinable models by deriving appropriate constitutive equations and specifying appropriate boundary conditions.
Use industry standard software tools and platforms to set up determinable models.
Recognise the value of Determinable Models of different complexity and the limitations of their application.
Ability to use determinable models to obtain system specifications in terms of parametric values.
Use mathematics and computing skills to manipulate and solve determinable models; and use data sheets in an appropriate way to supplement solutions.
Use industry standard software platforms and tools to solve determinable models.
Carry out a parametric sensitive analysis.
Critically assess results and, if inadequate or invalid, improve knowledge database by further reference to existing systems, and/or improve performance of determinable models.
LTSN Workshop at University of Essex
26th June 2003

23. LTSN Workshop at University of Essex
Engineering Professors Conference EPC Generic “Ability to” statements C
Ability to select optimum specifications and create physical models.
This means the ability to:
Use objective functions and constraints to identify optimum specifications.
Plan physical modelling studies, based on determinable modelling, in order to produce critical information.
Test and collate results, feeding these back into determinable models.
Ability to apply the results from physical models to create real target systems.
Write sufficiently detailed specifications of real target systems, including risk assessments and impact statements.
Select production methods and write method statements.
Implement production and deliver products fit for purpose, in a timely and efficient manner.
Operate within relevant legislative frameworks.
Ability to critically review real target systems and personal performance.
Test and evaluate real systems in service against specification and client needs.
Recognise and make critical judgements about related environmental, social, ethical and professional issues.
Identify professional, technical and personal development needs and undertake appropriate training and independent research.
LTSN Workshop at University of Essex
26th June 2003

24. Working with Level Descriptors
They should be seen as helpful guides rather than dictates
They are generic and may contain sections not appropriate to a particular programme
They may not cover all possible learning that is relevant to the programme.
The words become more meaningful if you look at descriptors at the previous and next level
Look at the relationship between descriptors at the same level as they do not function independently of each other
Use them to provide an appropriate vocabulary to describe learning
LTSN Workshop at University of Essex
26th June 2003

25. A Module Specification from a Clean Sheet!
Generic Level Descriptors
Identify Aim of Module
Translate Level Descriptors into Subject Descriptors
Develop the module and rethink it including the initial learning outcomes
Write Learning Outcomes
Let us first consider the writing of a module description for a new module – in other words assuming a clean sheet of paper and no previous information.
Need to first develop from the generic level descriptors the appropriate subject descriptors at the target level.
Next establish the Aim of the Module which allows the first draft of the Intended Learning Outcomes to be developed. These will certainly be at the threshold level but may also include desirable learning outcomes which will allow grading of student achievement.
From these the Method of Assessment for the learning outcomes can be evolved. We will be considering later alternative methods of assessment.
Having established methods of assessment for each learning outcome assessment criteria can be established.
Once having formulated the learning outcomes the assessment methods and the assessment criteria the teaching strategy can be developed in order to deliver the learning opportunities required.
In practice this process will need to be iterated in order to refine the outcomes, assessment methods and assessment criteria and delivery.
Write Threshold and Grading Assessment Criteria
Develop Assessment Method to test achievement of assessment criteria
Develop a teaching strategy to enable learners to reach the learning outcomes / assessment criteria
LTSN Workshop at University of Essex
26th June 2003

26. Writing Module Specifications
Clearly identify the intended level of the module
Formulate clear and unambiguous intended threshold and possibly desirable learning outcomes for the module
The threshold learning outcomes identify the essential learning to merit the award of the credits for this module
Desirable learning outcomes can be included to provide guidance of learning above threshold which will be assessed to provide grading
Identify assessment criteria that encourage learning at the appropriate level
Threshold assessment criteria should specify how satisfactory performance of the threshold module learning outcomes can be demonstrated
Grading-related assessment criteria are used to provide incentive for higher achievement above threshold performance
The outcomes approach requires the teacher to pose and answer the questions:
What do I intend students to learn? Or
What learning outcomes do I want them to achieve?
What teaching methods and curriculum design will I use to encourage students to behave in ways that are likely to achieve these outcomes?
What assessment tasks and criteria will tell me that students have achieved the outcomes I intend?
LTSN Workshop at University of Essex
26th June 2003

27. Writing a Module Description in Reality!
Existing Aims, Objectives and Syllabus Content of Module
Generic Level Descriptors
Translate Level Descriptors into Subject Descriptors
Review Aim of Module
Existing Assignments and Laboratories
Rewrite Objectives as Learning Outcomes
Develop the module and rethink it including the initial learning outcomes
The real world! Usually we do not have a clean sheet of paper! We have an existing course syllabus which we wish to rewrite in a form that ties in with QAA or institutional requirements or maybe we wish to make it more transparent for students! In this case we often start with AIMS, OBJECTIVES and SYLLABUS content plus some for of assessment policy for the module..
In this case we need to review aims and convert objectives into achievable and measurable intended threshold and desirable (for grading) learning outcomes. The assessment methods need to be reviewed in order to check that they are measuring the achievement of learning outcomes. Alternative methods of assessment should be considered and a check made to ensure that over assessment of particular learning outcomes is not introduced. Once the learning outcomes and assessment methods have been agreed the detailed assessment criteria can be written.
This process may lead to changes in the way that teaching of the module is undertaken!
Finally the both learning outcomes, assessment methods, assessment criteria and the teaching methodology should be reviewed!
Write Threshold and Grading Assessment Criteria
Modify Existing Assessment Methods to test achievement of assessment criteria
Modify current teaching strategy to enable learners to reach the learning outcomes / assessment criteria
LTSN Workshop at University of Essex
26th June 2003

28. Writing Learning Outcomes
A well written learning outcome is likely to contain :
A verb that indicates what the learner is expected to be able to do at the end of the period of learning
Word(s) that indicate on what or with what the learner is acting. If the outcome is about a skill then the word may describe the way the skill is performed
Word(s) that indicate the nature (in context or terms of standard) of the performance required as evidence that the learning was achieved
LTSN Workshop at University of Essex
26th June 2003

29. Examples of Learning Outcomes
The learner is expected to be able to:
demonstrate understanding of the purpose and operation of the hardware and software components present in personal computers, microprocessors and embedded processor applications (Level 1 Computer Systems)
explain the physical basis of the operation of Metal Oxide Semiconductor Transistors (MOSFETs) and obtain the small signal model of a MOSFET (Level 2 Electronic Devices)
LTSN Workshop at University of Essex
26th June 2003

30. Features of Good Learning Outcomes
Must be achievable by students within the time available and at the level of learning at which the students are. Be realistic!
Written in terms of the “learner being expected to be able” or as “intended learning outcomes”
Should specify areas of learning rather than specific curriculum
There should be in the range of 4 to 10 learning outcomes per module – too many makes statements of the assessment criteria unmanageable
Should be written in a language that is understood by all and is unambiguous
Each intended learning outcome should represent a major achievement expected by students at the end of the module
Learning outcomes must be assessable by a reasonable and manageable form of assessment within the time allocated to the module
Achievement of each threshold learning outcome is essential in order to pass the module
LTSN Workshop at University of Essex
26th June 2003

31. Vocabulary for Writing Learning Outcomes and Assessment Criteria - A
Verbs which require evidence of knowing:
Be aware of, define, describe, extract, identify, know, label, list, match, measure, name, organise, outline, present, recall, recognise, recount, relate, repeat, select, state, underline, write.
Verbs which require evidence of comprehension:
Clarify, classify, compare, comprehend, contrast, convert, defend, describe, discuss, distinguish, estimate, exemplify, explain, express, extend, find, formulate, generalise, give examples of, identify, illustrate, indicate, infer, interpret, judge, justify, name, paraphrase, perform, predict, present, report, represent, restate, rewrite, select, summarise, translate, understand.
Verbs which require evidence of knowledge / understanding:
Apply, arrange, assess, change, choose, compute, construct, demonstrate, discover, draw (up), exemplify, explain how, find, give examples, illustrate, manipulate, modify, operate, order, practice, predict, prepare, produce, relate, select, show, solve, use, verify
The next two slides suggest some words which might be used in learning outcomes and assessment criteria. I am sure there are others! Avoid words that are vague and can not lead to an assessable outcome.
LTSN Workshop at University of Essex
26th June 2003

32. Vocabulary for Writing Learning Outcomes and Assessment Criteria - B
Verbs which require evidence of analysis:
analyse, break down, calculate, categorise, compare, conclude, contrast, criticise, devote, diagnose, differentiate, distinguish between, divide, elucidate, evaluate, examine, identify, illustrate how, infer, justify, outline, point out, precis, question, recognise, relate, resolve, select, separate, subdivide.
Verbs which require evidence of synthesis:
account for, alter, argue, build up, combine, compile, compose, conclude, create, derive, design, develop, devise, engender, enlarge, explain, formulate, generalise, generate, integrate, manage, modify, order, organise, plan, prepare, present, produce, propose, put together, rearrange, reconstruct, relate, reorganise, report, restate, revise, select, structure, suggest, summarise, synthesise, teach, tell, write.
Verbs which require evidence of evaluation
appraise, assess, choose, compare, conclude, contrast, criticise, defend, describe how, determine, discriminate, estimate, evaluate, judge, justify, measure, question, rate, value.
LTSN Workshop at University of Essex
26th June 2003

33. Examples of Learning Outcomes
Discuss and comment on the following threshold learning outcomes:
Describe the structure of telecommunications networks (Level 1)
Understand the ideas of differentiation and integration (Level 1)
Understand the principles of human cognitive systems and motor performance when operating interactive computer systems (Level 2)
Demonstrate oral and written communication skills (Level 2)
Appreciate the use of the z-transform in digital signal processing (Level 3)
Improve at working in software engineering teams (Level 3)
Describe the principal characteristics of human vision, hearing and speech relevant to audiovisual communication and their exploitation in image, video, and audio compression (Level 3)
Imprecise – suggests assessment by essay questions, no context or depth of knowledge required
Imprecise understand not measurable – explain clearly the meaning of differentiation and integration and apply to functions of one variable
OK but do not use Understand - replace with Explain the ……
OK but context needs defining – add by preparing a technical paper on a topic and then preparing and delivering an oral presentation on this topic
Appreciate – what level of knowledge expected? How is is assessed? Define the z-transform and discuss its use as a technique for describing the transfer function of a second order digital signal processing system
Improve – difficult to measure as need to know starting point. Demonstrate the ability to work and make a significant contribution as a member of software engineering team developing a complex real time application.
This one looks OK? Any comments?
LTSN Workshop at University of Essex
26th June 2003

34. Purposes of Assessment
To provide a licence to proceed to the next stage of the programme or to graduation
To classify the performance of the student in rank order
To improve student learning by providing feedback on his or her strengths and weaknesses
To motivate the student to learn
To provide information for future selection or as a licence to practice
To evaluate the course and improve teaching
These may overlap or conflict!
A common error is to assume that the results of an assessment task used for one purpose are appropriate for another purpose
For example
Assessment tasks to provide a licence to proceed to the next stage should be based on threshold knowledge and skills and the pass/fail threshold should be absolutely secure.
If the primary purpose of the assessment is to place students in rank order, the assessment task should be designed to differentiate the capabilities of a wide range of students.
From the student point of view assessment should provide feedback as to the student’s progress in understanding the material but marks awarded should provide motivation to the student.
Reflective diaries or records of achievement can be useful to support learning but if they are used to make judgments or recommendations for employment students may be reluctant to record honestly their thoughts and feelings.
Finally assessment can provide feedback on the course contents and the teaching.
LTSN Workshop at University of Essex
26th June 2003

35. Some Principles of Assessment
Assessment shapes learning
therefore change assessment to change learning
Match the assessment tasks to the learning outcomes
Match the assessment criteria to the task and the learning outcomes
Keep the assessment criteria simple
Be fair, reliable and valid in your marking
Provide meaningful and timely feedback
Do we apply these principles?
Assessment shapes learning therefore in order to improve student learning we may need to change the assessment method
We certainly need to match the assessment tasks to the learning outcomes and then match the assessment criteria to the assessment task and the learning outcomes whilst keeping the assessment criteria simple
Once having received the students’ submissions be fair, reliable and valid in your marking and provide meaningful and timely feedback.
DO WE APPLY THESE PRINCIPLES?
LTSN Workshop at University of Essex
26th June 2003

36. Some Assessment Methods
Formal Examinations
Knowledge and Understanding
Analytical Ability
Problem Solving
Communication Skills
Progress tests
Simple Problem Solving
Formal Practical Experiments
Following Instructions and Recording Results
Practical Ability
Written Reports or Software Documentation
Projects
Requirement Analysis, Research, Problem Solving / Synthesis
Practical Development
Oral and Written Communication
Possibly Group Working
Other Coursework Assignments
Formal Examinations
allow a student to demonstrate recall of knowledge but can encourage rote learning, understanding of the course content, analytical ability, application of acquired knowledge to solving problems, some written communication skills in the form of clarity of explanation
Progress tests,
usually multi-choice and optically marked or computer based, can be used to provide formative and summative feedback by demonstrating recall of knowledge and understanding and can assess simple problem solving
Formal Practical Experiments
can be used to demonstrate ability to follow instructions and record results using for example Log Books. They can assess practical skills such as using equipment / tools as well as and understanding and achievement. Written reports or software documentation can be used to assess written communications skills.
Projects
A requirement analysis demonstrates communication with customer and develops understanding. Undertaking the background research demonstrates the process, use of tools to find information, and the ability to sort and filter relevant information. They can develop problem solving and synthesis by the application of knowledge and skills. Practical Development develops implementation, practical skills, and use of equipment / tools. Reports and presentations develop written and oral communication skills. Group projects develop team working, tolerance and management skills.
LTSN Workshop at University of Essex
26th June 2003

37. Common Weaknesses in Assessment
Tasks do not match the stated outcomes
Criteria do not match the tasks or outcomes
Criteria not known to and/or not understood by the students
Overuse of one method of assessment
Overload of students and/or staff
Insufficient time for students to do the assignments
To many assignments with the same deadline
Insufficient time for staff to mark examination or assignment
Absence of well defined criteria so consistency is difficult to achieve
Unduly specific criteria which create a straightjacket for students
Inadequate or superficial feedback provided to students
Wide variation in marking between modules and within assessors
Variation in assessment demands of different modules
LTSN Workshop at University of Essex
26th June 2003

38. Designing Assessments
Some questions to be considered
What are the learning outcomes to be assessed?
What are the capabilities / skills either implicit of explicit, within the learning outcomes?
Is the method of assessment chosen appropriate to the outcomes and skills?
Is the method relatively efficient in terms of student time and staff time?
What alternatives are there and what are their advantages and disadvantages?
Does the specific assessment task match the outcomes and skills?
Are the marking schemes or criteria appropriate?
LTSN Workshop at University of Essex
26th June 2003

39. Assessment Criteria – a definition
Not to be confused with assessment methods or tasks
Assessment Criteria provide a clear indication of how achievement may be demonstrated
Often specified with respect to each learning outcome, they describe what a learner is expected to do in order to demonstrate that the learning outcome has been achieved.
Assessment Criteria may be used in three ways:
To confirm achievement of threshold standards
To define what is expected in order to achieve each of the grades being awarded
To specify a template of characteristics or qualities against which the students’ performance of the assessment task will be judged
LTSN Workshop at University of Essex
26th June 2003

40. Writing Assessment Criteria
Consider the learning outcome being tested
Consider the assessment task set
Brainstorm requirements for, or attributes of, successful performance of the assessment task
If necessary specify the range to clarify contextual factors and the level
Focus on what is essential and categorise the requirements or attributes into clearly worded criteria
Check that the criteria are measurable or assessable in valid and reliable ways and that the criteria are clear and unambiguous
Repeat steps 3, 4 and 5 until you are fully satisfied
Publish the assessment criteria with the assessment task and identify the intended learning outcome or outcomes that the task is assessing
LTSN Workshop at University of Essex
26th June 2003

41. LTSN Workshop at University of Essex
An Example
The learner is expected to be able to:
demonstrate understanding of the purpose and operation of the hardware and software components present in personal computers, microprocessors and embedded processor applications (Level 1 Computer Systems)
Assessment Task?
Hardware Laboratory: Test and evaluate the operation of computer hardware components by constructing a serial adder on a logic patch-board using available TTL devices.
Assessment Criteria:
Connect correctly the following functional elements on the patch-board provided: EX-OR, Half Adder, Full Adder, 4 to1 Multiplexer, D-type bistable carry store, 4-bit register, 4-bit counter, and links modules to form a 4-bit serial adder.
Test each module and record results
Complete the multi-choice test and score better than 70%
LTSN Workshop at University of Essex
26th June 2003

42. Assessment Criteria – Exercise 4 U
Final Year Individual Project 30 credits:
Aim: The project provides the student with the opportunity to apply knowledge and practical skills gained during the degree programme to the solution of a problem agreed with the supervisor.
Form Groups of 3 or 4
Write a Threshold Learning Outcome for this module
Develop an Assessment Task to address this learning outcome
Write Assessment Criteria for this task
Repeat 1, 2 and 3 for a desirable graded learning outcome!
LTSN Workshop at University of Essex
26th June 2003

43. Report Back and Discussion
Each group to report their
learning outcomes
assessment tasks
assessment criteria
Comments
Discussion
LTSN Workshop at University of Essex
26th June 2003

45. Achievement of Module Threshold Learning Outcomes
Peter Noakes
Department of Electronic Systems Engineering
University of Essex

46. Current Developments at Essex
Internally Funded Project
The objective is to develop module specifications and structures where the threshold and extended learning outcomes, and associated assessment methods and criteria are clearly defined for students.
As a result a student’s satisfaction of threshold learning outcomes should be readily determined
Initially concentrating of Year 1 Modules offered by ESE
Working with colleagues in Departments of Computer Science and Biological Sciences
LTSN Workshop at University of Essex
26th June 2003

47. The Module Description
For each module Staff and Students will have a clear view of :
the threshold learning outcomes
their method of assessment and
the assessment criteria used.
the extended (desirable) learning outcomes
the assessment activities used for grading
the assessment criteria used for grading above threshold
the expected time commitment for the activities
This should ensure that a graduating student attains his or her maximum potential
LTSN Workshop at University of Essex
26th June 2003

48. LTSN Workshop at University of Essex
Project Proposals
In future a module description will identify separately the threshold and extended (desirable) learning outcomes and syllabus
The threshold syllabus defines the topics and associated skills, their method of assessment and associated assessment criteria, that will be used to demonstrate threshold level achievement for the module
The extended syllabus defines the topics, their method of assessment and associated assessment criteria, that will be used to demonstrate understanding and application of their knowledge above the threshold level for the module allowing grading above Third class.
The threshold aspects of the module will be taught conventionally with clear guidance to required reading, and supported by regular formative on-line MCQ testing
The extended aspects of the module will be taught by a combination of special topic lectures and directed self study with associated supporting problem classes
LTSN Workshop at University of Essex
26th June 2003

49. Module Specifications
Module specifications to be captured by filling form to provide entry to a central database
Data to be extracted for various audiences by running different reports
Linked to marks database to provide clear abstraction of satisfactory threshold achievement
LTSN Workshop at University of Essex
26th June 2003

50. Module Assessment Current Approach
Largely based on End of Year Examinations
All have an in-term Multi-choice Progress Test
Many modules have a Practical Laboratory (Hardware / Software / CAD / Database / Networks / Microprocessor / Matlab / Web) – Assessment in various ways including log book, demonstration, presentation, report, oral, OMR test
Some have Assignments – electronic submission, paper submission, demonstration, oral
Some modules include Projects – demonstration, presentation, report, oral
Module mark generated by forming a weighted aggregate of the marks awarded to individual elements
A 40% overall aggregate does not ensure that all learning threshold outcomes have been satisfied!
LTSN Workshop at University of Essex
26th June 2003

51. Proposed Module Assessment
Each module has 100 marks available
Progress Test – threshold material 10 marks
Examination – 2 hour paper – 2 parts
Part A – say 8 to 20 questions on threshold material to produce say 20 marks
Part B – 3 to 5 on extended material answer 2 or 3 for full marks – applying knowledge, problem solving – producing say 40 marks
Laboratory or Assignment Coursework – contribute say 20 marks on threshold material and say 10 marks on extended material.
LTSN Workshop at University of Essex
26th June 2003

52. Graphical Representation
Module Marks
100
Part B examination 90 80
Degree Graded Assessments 70 60
Practical Activity 50
Part A examination 40 30
Threshold Assessments
Practical Activity 20 10
Progress Test
LTSN Workshop at University of Essex
26th June 2003