Assessment – Preparing, Delivering, Assessing Damian Gordon

Last Week: Instructional Design

The Joy of Teaching Then Jesus took his disciples up to the mountain and gathered them around him, he taught them saying, "Blessed are the poor in spirit, for theirs is the kingdom of heaven. Blessed are the meek. Blessed are they that mourn. Blessed are you when you suffer. Be glad and rejoice, for your reward is great in heaven." Then Simon Peter said, "Are we supposed to know this?" And Andrew said, "Will we have a test on this?" And Philip said, "I don't have any paper." And Bartholomew said, "Do we have to turn this in?" And John said, "The other disciples didn't have to learn this." And Matthew said, "May I go to the bathroom?" Then one of the Pharisees who was present asked to see Jesus' lesson plan and inquired of Jesus, "Where are your objectives and your required competencies in the cognitive domain?" And Jesus wept.

Overview Part 1 (Why?) Part 2 (What?) Part 3 (How?) Part 4 (Who?) Purpose of assessment Formative -v- summative assessment Part 2 (What?) Learning outcomes Constructive alignment Part 3 (How?) Types of assessment Grading of assessment Part 4 (Who?) Self / peer assessment

Assessment How do you get a new belt in Karate, how accurate does your throws, blocks and punches have to be ? 40% 50% 75% 90% 100%

Assessment How do you get a new belt in Karate, how accurate does your throws, blocks and punches have to be ? 40% 50% 75% 90% 100% >>>>>>> How should it be assessed? <<<<<<<<

Assessment To qualify as a doctor, what average result do you need to get to pass ? 40% 50% 75% 90% 100%

Assessment To qualify as a doctor, what average result do you need to get to pass ? 40% 50% 75% 90% 100% >>>>>>> How should it be assessed? <<<<<<<<

What is the purpose of assessment ?

John Biggs Born in Tasmania (5th Generation) an educational psychologist and novelist developed the SOLO Taxonomy and the model of constructive alignment Discusses “Backwash”

Purpose of assessment I Reasons for assessment Select students Control / motivate students Direct teaching / learning Identify strengths / weaknesses of teaching / learning

Purpose of assessment II Backwash Students learn what they think they will be assessed on Assessment determines what and how students learn more than the curriculum does This must be capitalised on and employed

Michael Scriven Born 1928 in Britain Degree in mathematics and a doctorate in philosophy made significant contributions in the fields of philosophy and evaluation Coined the terms “formative evaluation” and “summative evaluation” in 1967

Purpose of assessment III Types of assessment Formative assessment Used for feedback throughout the course Should not (all) contribute to marking Useful for monitoring learning progress Summative assessment End of course Used to grade students

Peter Knight Born in Britain Died in 2007 Educational research and assessment expert Author of “Children’s Understanding of People in the Past” Author of “Being A Teacher In Higher Education”

Purpose of assessment IV Reason for assessment Authorities agree that assessment has two main roles. It warrants or certifies achievement…….It also influences learning, partly by signaling what is really valued… Peter Knight, SEFI Annual Conference, (2001)

…and one other expert(s) we should listen to

Purpose of assessment V Student’s views “Helps me find out what I know and don’t know” (87.3%) “Guides my learning” (75.5%) “Helps me link theory to practice” (84.3%) “Helps lecturers evaluate student knowledge” (81.4%) “Enables lecturers to evaluate their teaching” (75.5%) “Used by lecturers to rank students” (44.1%)

Formative -v- Summative Assessment “When the chef tastes the sauce it is formative assessment, when the customer tastes it, it is summative” Robert Stake (possibly)

Formative Assessment Students’ Top 5 reasons for assessment all formative Formative assessment cannot be separated from teaching The more effective the teaching method, the better its ability to provide formative feedback “Helps me find out what I know and don’t know” (87.3%) “Guides my learning” (75.5%) “Helps me link theory to practice” (84.3%) “Helps lecturers evaluate student knowledge” (81.4%) “Enables lecturers to evaluate their teaching” (75.5%) “Used by lecturers to rank students” (44.1%) Admitting / recognising error is crucial for learning Error is used to instruct, not punish Student should be able to take the formative role on himself / herself in the future

Summative assessment To report a grade!

One line answer Why assess? We assess to help students learn (formative) and to judge how much they learned (summative)

Overview Part 1 (Why?) Part 2 (What?) Part 3 (How?) Part 4 (Who?) Purpose of assessment Formative -v- summative assessment Part 2 (What?) Learning outcomes Constructive alignment Part 3 (How?) Types of assessment Grading of assessment Part 4 (Who?) Self / peer assessment

Learning Outcomes Without a clear understanding of the programme goals students are likely to become disorientated and waste time trying to discover what it is we want them to learn Learning outcomes consist of three parts: the student action the content the standard required to meet the objective

Industry-Oriented Learning Outcomes Graduates to be equipped with core competencies Who will be able to become productive in a short time with training Graduates to be equipped with the necessary skills to adapt to changing needs as appropriate and acquire new skills quickly

Layers of Learning Outcomes Programme Module Session

Constructive Alignment LEARNING OUTCOMES ASSESSMENT METHODS TEACHING & LEARNING

Constructive Alignment I Learning outcomes, teaching methods and assessment are all inter-dependant Only by truly integrating these components together, do we get efficient student learning Meaning is not imposed or transmitted by direct instruction - it is created by the student's own learning activities

Constructive Alignment II Knowledge and understanding Software Engineering; Problem Solving and Critical Analysis; A range of programming languages and environments; Basic principles of computer hardware, data transmission and local area networking; Operating systems, Business context; Engineering practice; Teaching/learning methods The knowledge required for the basic topics is obtained via lectures, exercises, practicals, assignments and project work. Appropriate IT and other software packages are taught. Practical demonstrators and project supervisors advise students, and feedback is provided on all continually assessed work. As the course progresses students are expected to show greater initiative. Assessment Most knowledge is tested through a combination of practicals, assignments and formal examinations. Students write reports on many assignments, and also make oral presentations of their work.

Constructive Alignment III Intellectual Skills Select and apply appropriate computer based methods; Analyse and solve problems; Transfer appropriate knowledge and methods from one topic to another; Plan, conduct and write a report on a project or assessment; Prepare an oral presentation. Teaching/learning methods Appropriate software, mathematical, scientific and IT skills and tools are taught in lectures, and problems to be solved are given as projects or assignments. Project planning is part of the Part 4 project, and written and oral presentations are required for various assignments and projects. Assessment Skills 1-3 are assessed partly by examination, though sometimes also by project or assignment work. Skills 4 and 5 are assessed as part of project work.

Constructive Alignment IV Practical Skills To design, implement and test well structured and documented computer programs using a wide variety of programming languages, with a focus on object-oriented development, To develop applications using the main web programming technologies; Database manipulation and administration; Research into a computer science problem; Present work in both written and oral form. Teaching/learning methods Software tools are introduced in lectures and their use is assessed by examinations and assignments. Programming assignments are set, and students may write programs to solve other projects. Practicals and projects are used to teach about skill 3, and projects are used for skills 4, 5, 6 and 7. Assessment Skills 1 to 4 are tested by assignments and projects, is addition to coursework. Skills 1, 2 and 3 are assessed in practicals. Skills 4 and 5 are assessed through project work.

Constructive Alignment V Transferrable Skills use software tools. acquire, manipulate and process data. use creativity and innovation. solve problems. communicate scientific ideas. give oral presentations. work as part of a team. use information resources. manage time. Teaching/learning methods and strategies Software tools are taught partly in lectures, mainly through practical sessions and assignments. Data skills are acquired in laboratory and projects. Creativity and innovation and problems solving are experienced through projects, as are team working, time management and presentations. Use of information resources, such as the library and IT methods is experienced through projects and assignments. Assessment Some skills, like the use of software tools and ability to communicate orally and in written form are directly assessed, in assignments or projects

3D Alignment Horizontal Vertical Constructive O'Leary, C., Lawless, D., Gordon, D., Carroll, D., Mtenzi, F., and Collins, M., “3D Alignment in the Adaptive Software Engineering Curriculum” 36th Annual Frontiers in Education Conference, San Diego, USA, October 28th - 30th 2006

we recall Bloom’s Taxonomy

Blooms Taxonomy I 6 Levels (Cognitive Domains) Knowledge Comprehension Application Analysis Synthesis Evaluation

Blooms Taxonomy II Knowledge Year 1 list name identify show define recognize recall state Year 1

Blooms Taxonomy III Years 1, 2 Comprehension summarize explain put into your own words interpret describe compare paraphrase differentiate demonstrate visualize find more information about restate Years 1, 2

Blooms Taxonomy IV Years 2, 3 Application solve illustrate calculate use interpret relate manipulate apply classify modify put into practice Years 2, 3

Blooms Taxonomy V Year 3 Analysis analyze organize deduce choose contrast compare distinguish Year 3

Blooms Taxonomy VI Years 3, 4 Synthesis design hypothesize support schematize write report discuss plan devise compare create construct Years 3, 4

Blooms Taxonomy VII Year 4 Evaluation Evaluate Choose Estimate Judge Defend Criticize Justify Year 4

Types of Knowledge

Types of knowledge I Functioning Conditional Declarative Procedural

Types of knowledge II Declarative Knowing something Facts – recorded in books / delivered in lectures Accrues from research, not from personal experience Verifiable, usable

Types of knowledge III Procedural Skill based knowledge Sequences of actions Having the right competencies

Types of knowledge IV Conditional Subsumes declarative and procedural Knowing when to do something, what to do, why to do it… Turns procedural knowledge into functioning knowledge

Types of knowledge V Functional Performance of understanding Knowledge within the experience of the learner who can put declarative knowledge to work by solving problems

Types of knowledge VI Curricula at third level generally focus on declarative, something reflected in forms of assessment What is required for professional competence is functioning knowledge Assessment should reflect this.

SOLO Taxonomy

Levels of understanding I SOLO Taxonomy Structure of Observable Learning Outcome Extended abstract Prestructural Unistructural Multistructural Relational Quantitative Qualitative

Levels of understanding II Prestructural understanding Learner misses the point No observable understanding

Levels of understanding III Unistructural understanding Simple naming Terminology Focus on one conceptual issue in a complex task Low level understanding

Levels of understanding IV Multistructural understanding Disorganised collection of items Knowledge telling

Levels of understanding V Relational understanding Using a concept that integrates a collection of data Understanding how to apply the concept to a familiar data set or to a problem

Levels of understanding VI Extended abstract Relating to existing principle so that unseen problems can be handled Questioning and going beyond existing principles

Assessment Criteria

Assessment criteria I State the Type of knowledge Level of understanding required to meet a particular learning outcome

Assessment criteria II “the standard required to meet the outcome” Criteria are developed by analyzing the learning outcomes and identifying the specific characteristics that contribute to the overall assessment How many criteria are needed that contain neither irrelevant or miss important areas?

Assessment criteria III Criteria serve two functions For formative feedback, learners can See how they performed on the various components of the assessment Identify strengths and weaknesses For summative feedback, A contract is in place between teacher and learner

Assessment criteria IV Questions to answer when setting an assessment What learning outcomes do you intend to assess? How does your approach to assessment align with these outcomes? What are the assessment criteria? What other skills are assumed e.g. skills from previous years? How do you ensure that your standards are consistent?

Formative Feedback I Feedback helps students to improve and prevent them from making the same mistakes again. It is useless if feedback comes back too late in the semester Students sometimes wait on the feedback before they feel capable of going onto the next bit Grades do not equal feedback. Examinations are usually a summative form of assessment and therefore not used for feedback.

Formative Feedback II For students, high quality feedback consists of following three things: A clear criteria against which to judge the comments. Comments that are detailed and related to specific aspects of their work. Comments that are improvement focussed.

Formative Feedback III Students’ views Agree Disagree Don’t know Check understanding of course material 81.4% 12.7% 5.9% Help students improve their work 99.0% 1.0% 0.0% To explain the grade 83.3% 14.7% 2.0%

Formative Feedback IV 75.5% agree that they compare the grade received to the feedback Does the feedback reflect the grade given? “I feel lecturers should consult students about their grades not just tell them the grade. Recently I got some assignments back and the comments were ‘excellent, good conclusion, good reflections etc.’ Nothing bad was written on it, but I only received 50%. I can’t understand why it was so bad if all the comments were so positive.”

Formative Feedback V Nearly a quarter of the students surveyed indicated that they would not approach the lecturer for clarification of feedback: “I think it would be helpful if lecturers took specific time to speak with all their students to discuss exam results etc. At present lecturers leave it up to the student to come forward. This doesn’t work because students who haven’t done well are less likely to come forward to speak to their lecturer.”

Formative Feedback VI Feedback should be provided by the assessor to the assessed student High level feedback to the class can be useful Feedback should be correlated to the assessment criteria Asking the student to self-assess against criteria can prove useful

One line answer What to assess? We assess all learning outcomes so that we can guarantee that a student who passes a module will know (declarative) or be able to do (functioning) everything listed in the outcomes to a certain level.

Overview Part 1 (Why?) Part 2 (What?) Part 3 (How?) Part 4 (Who?) Purpose of assessment Formative -v- summative assessment Part 2 (What?) Learning outcomes Constructive alignment Part 3 (How?) Types of assessment Grading of assessment Part 4 (Who?) Self / peer assessment

General approaches to assessment I Norm referenced assessment (NRA) Judgements about people Measurement model Criterion referenced assessment (CRA) Judgements about performance Standards model

General approaches to assessment II Measurement model Grading on a curve Marking by counting bottom up and averaging Assessment separated from teaching Multistructural, not relational or extended abstract thinking Verbatim responses gain marks Easily fit into most structures

General approaches to assessment III Standards model Standards or criteria set as objectives Most students should be able to reach standards at an acceptable level Different performances can reflect the same standards Performances can be judged against the criteria Holistic assessment – functional and declarative knowledge working together

Types of Assessment Examinations Normally unseen by students prior to the day they are attempted Sample a range of student learning and test the student’s own work Encourage students to memorise information rather than attempting to understand it as a component of their overall course

Types of Assessment Take home assignment Students given a period of time ranging from weeks to months Gives time to reinforce theory delivered in class Allows students develop own model, through trial and error Requires students to work without assistance for the majority of the time

Types of Assessment Laboratory Work Immediate feedback Can observe process Identify strengths and weaknesses

Types of Assessment Class Presentations Opportunity to assess on both structure and content The criteria should be made clear to the students Communication skills are key transferrable skills

Types of Assessment Essays / Research Assess, Analyse, Evaluate ......... Two approaches to grading Analytic Holistic

Types of Assessment Online Tests Multiple choice questions Complete objectivity in marking the test Largely used to test factual material and the understanding of concepts Used for testing large groups Time consuming to set No credit for partial information May encourage guessing

Types of Assessment Problem Based Learning An instructional method that challenges students to learn to learn Work cooperatively in groups to seek solutions to real world problems Used to engage students' curiosity and initiate learning the subject matter Prepares students to think critically and analytically, and to find and use appropriate learning resources

Types of Assessment Reflective Journal Develop an awareness of processes Write in your reflective journal as soon as possible after an event

Assessment for Learning

Assessment for Learning Students know at the outset of a unit of study what they are expected to learn. At the beginning of the unit, the teacher will work with the student to understand what she or he already knows about the topic as well as to identify any gaps or misconceptions (initial/diagnostic assessment). As the unit progresses, the teacher and student work together to assess the student’s knowledge, what she or he needs to learn to improve and extend this knowledge, and how the student can best get to that point (formative assessment). Assessment for learning occurs at all stages of the learning process.

Assessment for Learning According to The UK Assessment Reform Group (1999) improving learning through assessment depends of five factors: The provision of effective feedback to the students. The active involvement of students in their own learning. Adjusting teaching to take account of the results of assessment. A recognition of the profound influence assessment has on the motivation and self esteem of the students, both of which are crucial influences in learning. The need for students to be able to assess themselves and understand how to improve.

Assessment for Learning How do we do this ? Sharing learning goals with students. Involving students in self assessment. Providing feedback which leads to students recognizing and taking the next steps. Being confident that every student can improve.

Assessment for Learning What barriers are there ? A tendency for teachers to assess quantity and presentation of work rather than quality of learning. Greater attention given to marking and grading, much of it tending to lower self esteem of students, rather than providing advice for improvement. A strong emphasis on comparing students with each other, which demoralizes the less successful learners.

Primary Users Assessment of Learning Policy makers Program planners Supervisors Assessment for Learning Students Teachers Parents Knowing who is going to be using the results of the assessment are important in determining the type of assessment that is required. There is no one assessment that can be effectively used by all of these people.

Typical Uses Assessment of Assessment for Learning Learning Certify competence Sort students according to achievement Assessment for Learning Help students see and hit the target Help teachers identify and respond to student needs Knowing how the assessment results are going to be used will also influence the type of assessment needed. There is no one assessment that that can effectively provide for all of our needs.

Teacher’s Role Assessment of Learning Follow test administration procedures Use results to help students reach standards Assessment for Learning Inform students of targets Build assessments Adjust instruction Involve students in assessment The major difference for teachers between the two assessments is timing. In an Assessment of Learning, the teacher helps in the administration of the test and is to use the results to assist students. In an Assessment for Learning, the teacher is gathering evidence of where students are, where they need to go, and how they are going to get there. The communication with students allows them to become more active participants in their own learning.

Student’s Role Assessment of Learning Assessment for Learning Study to meet standards Take the test Strive for the highest score possible Avoid failure Assessment for Learning Strive to understand the target Act on classroom assessment to improve Encourage success The major difference for students between the two assessments is the degree of involvement and understanding of their progress. In an Assessment of Learning, students take the assessment. In an Assessment for Learning, students respond to information provided in class that helps them see where they are, where they are heading, and how they are going to get there.

The Assessment Goal To create and maintain a balanced assessment system that includes high quality assessments of and for student learning.

Assessment of learning = Summative Learning ?? Question Assessment of learning = Summative Learning ?? Assessment for learning = Formative Learning ??

Teacher-centred = Summative Learning ?? Student-centered = Question Teacher-centred = Summative Learning ?? Student-centered = Formative Learning ??

Student Centred Assessment Multi-choice Questions As we discussed previously from a teacher-centred perspective multi-choice questions have little to recommend them, but from a student-centred point-of-view multi-choice questions are telling the student “Here are N questions and I expect you to know the answers to all of them, if you find yourself guessing a lot of answers, you really don’t know the material as well as I want you to.”

Overview Part 1 (Why?) Part 2 (What?) Part 3 (How?) Part 4 (Who?) Purpose of assessment Formative -v- summative assessment Part 2 (What?) Learning outcomes Constructive alignment Part 3 (How?) Types of assessment Grading of assessment Part 4 (Who?) Self / peer assessment

Involving Students in Assessment I Ultimately the goal of any university course is for students to be able to make judgments on their own and others work The more time students spend on higher-level abilities like analysing and evaluating, the better they will get at assessment One approach to getting students involved in their assessment by having them develop their own assignment questions If students are concerned with the subject’s assessment they will be encouraged to engage in their subjects in deeper and more meaningful ways

Involving Students in Assessment II Peer assessment Students comment on and judge their colleagues work Vital role to play in formative assessment Can also be used as a component in a summative assessment package studies have found student ratings of their colleagues to be both reliable and valid

Involving Students in Assessment III Peer assessment – advantages Helps students to become more autonomous, responsible and involved Encourages students to critically analyse work done by others, rather than simply seeing a mark Helps clarify assessment criteria Gives students a wider range of feedback More closely parallels possible career situations where judgment is made by a group Reduces the marking load on the lecturer

Involving Students in Assessment IV Peer assessment – disadvantages Students may lack the ability to evaluate each other. Students may not take it seriously, allowing friendships, entertainment value, etc. to influence their marking. Students may not like peer marking because of the possibility of being discriminated against, being misunderstood, etc. Without lecturer intervention, students may misinform each other.

Involving Students in Assessment V

Involving Students in Assessment VI Self assessment the involvement of the students in identifying standards and/or criteria to apply to their work making judgements about the extent to which they have met these criteria and standards this form of assessment is commonly a supplement to teacher assessment of students, but in some cases it may replace it.

FORMATIVE SUMMATIVE BSc Computer Science I Categories of Assessment “diagnostic use of assessment to provide feedback to teachers and students over the course of instruction” “generally takes place after a period of instruction and requires making a judgment about the learning that has occurred e.g. by grading a test”

BSc Computer Science II Categories of Assessment FORMATIVE SUMMATIVE Laboratory Work Assignments Assignments Examination

Learning Outcomes Programme Aims and Content Programme Learning Outcomes Develop and Evaluate Learning & Teaching Strategies Assessment Criteria Assessment Methods Outline of Module Content Module Learning Outcomes Module Aims Programme Aims and Content

One line summary Assessment is the senior partner in the three way relationship also involving learning outcomes and teaching and learning strategy, because it heavily influences how learners approach a course and what they will ultimately know and do.

Have a look at… John Biggs, Teaching for Quality Learning at University, Open University Press, UK, 2003