The CAP Self-evaluation Survey Goal: to improve the assessment (the CAP survey is designed for a formative purpose)

Self-evaluation: procedure Step 1- organize an introduction session: to explain the criteria and concept of CAP and specify what kind of evidence is needed in the survey. Step 2-fill out the online questionnaire: fill out the CAP online self-evaluation questionnaire individually. Step 3- organize group interview(s): depending on your situation, you can organize a group interview (selected students and educators interviewed together) or organize two group interviews in which selected students and educators are interviewed separately.

By organizing an introduction session, you, as CAP organizer, want to ensure that survey participants :  understand the concept of the CAP and clarify the purposes and procedure of the CAP self-evaluation  understand the quality criteria and indicators stated in the survey  list all the assessment methods they will evaluate  understand what type of evidence is needed  have a clear idea on how to fill out the survey Why do I need to organize the introduction session?

Who should I invite? Who should I invite? Invite all stakeholders of the to-be-evaluated programme Participants’ profiles: Have various positions (students, teachers, coordinators, external evaluators, etc.) Or A group of people from the teaching team who share different visions of the to-be-evaluated programme Number of people: (can be more) per programme

Understanding CAP You can use the ppt entitled ‘supportive information for the introduction session’ together with the Word document entitled ‘a guide for conducting self-evaluation survey’ to introduce the concept of CAP and explain the 12 quality criteria.

Make an assessment list (CAP) The online questionnaire will ask each participant to make a list of assessment methods on which they will self-evaluate. Therefore, in the introduction session, please go through the table (see the table on the next page) to ensure that everyone knows what needs to be filled out in the table.

A description of your CAP Assessment methodWhat is assessed?Who assesses?When/ how often is an assessment carried out? For summative or/and formative purpose (s)? E.g. knowledge test (MCQ, T/F, etc.), project, observation during internship,… What competencies / core task / workflows E.g. teacher / experts in the field / fellow students E.g. at the end of acourse / few times per course ☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score ☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score ☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score ☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score ☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

What kind of evidence is needed? A collection of evidence & examples: Participants can refer to the following resources to indicate the quality of the assessment: - Student / course evaluation (written & oral; per thematic block and overall evaluation) - Report of grades / results of external examiners - Internship evaluation - Field experts feedback

Give a few examples to indicate what is expected and what is not: – Invalid evidence is, for instance, “that is obvious” – Solid evidence: such as “at the beginning of the course a document was provided illustrating the sections (specify how students will be evaluated against what criteria)” (see more examples given in another ppt) their opinion/ experience can also be indicated as evidence : do colleagues or others share the same opinion or experience? What kind of evidence is needed?

Step 2: filling out the online questionnaire  B e aware that filling out the online questionnaire takes time: about 1 hour!  You need to give at least 1 week to participants to fill out the online survey.  Ask participants to create login names and passwords.  Send reminder.

Step 2: filling out the online survey Ask all participants to:  Go to: 296/index.html 296/index.html  Log in with username and password.  Try to respond to all questions and provide evidence!  Check ‘Don’t known' if you really do not know the answer.  Fill out the questionnaire before the given deadline!

Step 3: Group interview The goals of the group interview are :  to understand different opinions  to add evidence  to correct the "errors"  for spontaneous improvement The general goal: consensus on the quality -“what are the strengths and weaknesses of our assessment”

Group interview preparation A print out of the results for all participants (only if not too many) Otherwise just give a short PowerPoint presentation For each criterion show: - Average scores of the indicators. - Highest and lowest score / standard deviation. - Summary of evidence. Note: can transfer the Likert scale into 1-6 point score!

The following slide is an example that shows what kind of issues the online survey may detect.

TRANSPARENCY assessment criteria and standards are communicated and understandable Problems: When the assessment, for instance, is report writing, the assessment criteria used to score students’ submitted report are only in the mind of the teacher and not explicitly communicated between teacher and students. The key criteria are written down. But how they are used to give scores are vague and not easy to interpret.

The following slide is an example that shows how you can demonstrate the issues the online survey may detect in the group interview.

Competency requirements are well described on paper, but are not reflected in the class. No clear link between competencies and assessment. Students do not understand what is expected of them in assignments. Criteria for presentations unclear. External assessors do not always fully understand the assessment context. Transparancy average score Lowest score Highest score Percentage of the qualified evidence 1. Students know and understand which competence is assessed. 2. Students know and understand assessment criteria (a property or characteristic by which the quality of something may be judged) and standards (a defined level of achievement/quality/performance aspired to or to be attained.) 3. Teachers know and understand the content of assessment tasks, and the criteria and procedures used in the CAP.

What to discuss? On the basis of the printout of the survey results, the following two situations may occur : Situation 1- There is consensus: High average score + no low scores (variations) (Almost) everyone finds sufficient OR Low average score + no high scores (variations) (Almost) everyone is insufficient Suggested discussion questions: 1. Is it true that this is the strength/ weakness of the CAP ? 2. Is all the evidence valid? Are the indicators that have a high score substantiated by strong evidence? 3. When there is little evidence: is there any evidence to substantiate the score? 4. For the weak points: what are the opportunities for improvement?

What to discuss? Situation 2- There is no consensus: Very different scores Big difference between highest and lowest scores Some abnormal scores Suggested discussion questions: Questioning people results in different scores. Ask them to provide explanations. Based on what do you make your judgment? For which part of the CAP? Are his/her scores supported by valid evidence or is it more “a gut feeling"? Look for evidence : is the provided evidence different for the high scores and lower scores ? Are they different because of different perspectives? Confront each other's opinion: are these new insights? Summarize the evidence: what are the positive and negative points? Seek consensus.

Tips for the group interview(s)  Make it clear that different opinions are welcome.  Interviewees are welcome to correct/restate their statements of particular indicators.  Explicitly question the people who express their dissent in a contrary opinion.  Avoid going astray (stick to the key topics).  The quality criteria are interrelated. Hence, sometimes certain topics come back again in the discussion. Try to make sure that you are the one who leads the conversation to go through the criteria one by one. Please avoid that the discussions jump back and forth between the criteria.  Discussion often arise naturally, because people disagree.

Some practical tips for data collection Determine in advance the time to discuss each quality criterion. For research purpose: check the recording equipment ask permission before recording the group discussion send the preparation materials (e.g., the survey results and discussion points /PowerPoint) to Lai send the interview report / record to Lai