1. Briefing session on Online Student Evaluation of Teaching and Learning
Prepared by
Social Sciences Research Centre of
The University of Hong Kong 1

2. SETL process
Online process covers all courses with undergraduate and TPG students (including Common Core courses) on 2 semesters system or summer course with teacher data in SIS.
Non-semester courses can be evaluated on-line if the course information and end date have been input into SIS, otherwise use paper. 2

3. 3 SETL process For RPG only courses, the departments/centres/schools
/institutions can choose to use paper or online
For the summer courses, the results will be merged into the first Semester of the following academic year
The online SETL evaluation will normally be conducted for 4 weeks until the start of the assessment period (i.e. for first semester courses, from Nov 9 to Dec 7, 2017) . 3

4. SETL process
Separate SETL forms for courses, teachers, tutors, demonstrators and practicum
Uses SIS database record of course/teacher/tutor/demonstrator (so these must be input accurately into the SIS)
SIS allows joining different courses of the same level in the evaluation, only one overall report will be generated for the parent course to the core department
Courses of different level will always be assessed separately, even if the same course code
MBBS and Dentistry online SETL evaluation are handled separately 4

5. SETL process
Multiple teachers refer to a number of teachers teach the same student in the same course code and in the same classroom
Sub-group identifiers refer to a number of teachers teach different students in the same course code in different classrooms
Each course will be evaluated only once before the end of the course e.g. a two-semester course will be evaluated before the end of the second semester 5

6. SETL process
Must use standard forms unless Chairman of TLQC approves
Online yields lower response rates, but better open- ended responses, which is good for improvement
To ensure good response rates, need to ensure that students are given time at the end of a class to complete the form (although they can complete any time before the deadline) 6

7. SETL process
Cannot currently change teacher names in system, which come from Human Resources
All teachers/tutors/demonstrators must have staff ID/University No. (HKU requirement and needed for staff review)
For the Department Items, a maximum 5 extra questions for both on-line and paper evaluation, please contact SSRC through (on or before mid October, for this semester) 7

8. SETL process
For the online SETL, all courses with enrollment of at least 2 students of the same level must undertake SETL, although the teacher/tutor/demonstrator parts are only done if there are at least 2 students per teacher/tutor/demonstrator
For the paper SETL, all courses with enrollment of at least 5 students must undertake the evaluation, although the teacher/tutor/demonstrator parts are only done if there are such persons for the class 8

9. SETL process
Up to 25 teachers can be evaluated in each course by a student, but only 1 tutor/demonstrator selected by a student from a list of up to 25 names (or evaluated as a group) of each course
Tutor means person running tutorials, demonstrator means person responsible for lab or field sessions (role descriptions, not job titles) 9

10. SETL Workflow 10 Prepared by Social Sciences Research Centre of
The University of Hong Kong 10

11. SETL Workflow of the Online SETL
Departments/Centres/Schools/Institutions must enter/update the course enrollment information in the Student Information System (SIS) especially the staff number linked to the name of teacher/tutor/demonstrator in each course and sub-class information 10 weeks before the assessment date (i.e. before end of September for this semester and finish date for non- semester courses)
All information must be updated for the appropriate semester
Student enrollment should be updated automatically for courses using SIS for enrollment, but the rest of the information relies on departments/centres/schools to update 11

12. SETL Workflow of the Online SETL
SIS system exports the course enrollment information of all undergraduate, TPG and RPG courses of which complete SIS data is available including:
Course code;
Course title;
Term (Semester 1/2/full Year)
Assessment period start date if not semester based (starting next semester)
Student list in each course (student name and university no.) by course level;
Teacher name list in each course (teacher name and staff number);
Tutor and Demonstrator name list in each course (teacher name and staff number); and
Jointly evaluated courses (course list) 12

13. SETL Workflow of the Online SETL
The course/teacher/tutor/demonstrator lists and indication of whether online or paper process will be sent out to departments/centres/schools/institutions on or before 16 October 2017 (for this semester), so departments/centres/schools/institutions can check for errors and missing information.
Departments/Centres/Schools/Institutions should update any amendments in the SIS system between 16 and 30 October 2017 (for this semester). 13

14. SETL Workflow of the Online SETL
Online process will cover all courses with undergraduate and TPG students (including Common Core courses) on 2 semesters system or summer course with teacher data in SIS.
Non-semester courses can be evaluated on-line if the course information with an end date later than the end of first semester and the end date have been input into SIS, otherwise use paper.
For RPG only courses, the departments/centres/schools
/institutions can choose to use paper or online
For the summer courses, the results will be merged into the first Semester of the following academic year 14

15. SETL Workflow of the Online SETL
The final course/teacher/tutor/demonstrator lists and indication of whether online or paper process will be sent out to departments/centres/schools/institutions (i.e. on or before 3 November 2017 for this semester). If any errors occurs, please contact SSRC through immediately (i.e. on or before 6 November for this semester).
For information, SSRC will send out the invitation s to all HKU students on 9 November and 2 reminder s to all HKU students on 30 November and 4 December, 2017 for this semester 15

16. SETL Workflow of the Online SETL
Non-semester
Non-semester courses must input dates through SIS system so they can be evaluated on-line and then the course/teacher/tutor/demonstrator lists will be sent out to departments/centres/schools/institutions 2 weeks before the evaluation date (ONLY ONE circulation will be sent). If any errors are found, departments/centres/schools/institutions must respond within one week after receiving the lists from SSRC and please contact SSRC through immediately. 16

17. Instructions for Online SETL
Before administration
The teacher concerned should inform the students in advance the scheduled time set aside for the online evaluation (i.e. soon after 9 November for this semester), and that the students are encouraged to bring a mobile internet device (such as laptop, tablet or smartphone) to class at the scheduled time. 17

18. Instructions for Online SETL
On the day of administration
Similar to the paper exercise, the SETL questionnaire should NOT be administered by the course teacher, but faculty/department staff.
The teacher should briefly explain the objectives of the student evaluation and then leave the room leaving it to faculty/department staff to oversee the process. 18

19. Instructions for Online SETL
On the day of administration
Faculty/Centre/Institution/Department staff should read out:
Please go to the weblink where you should see a list of all forms that you need to complete. Your evaluation will be saved anonymously, without any identification. There are separate forms for the course, teacher (and if appropriate) tutor and demonstrator. There are instructions, an FAQ link and a link for you to report any missing courses at the weblink. 19

20. Instructions for Online SETL
During the evaluation period
Students should
Access the SETL system via the weblink where there are instructions, FAQ and a list of forms for completion (separate links for the course, teacher, tutor, demonstrator forms)
Complete the forms for the class during four weeks until the start of the assessment period (this semester from Nov 9 to Dec 7, 2017)
If they cannot find the course that they have enrolled in, send an to SSRC using the link on the page so SSRC staff can rectify this promptly 20

21. Workflow for Paper SETL
Departments/Centres/Schools/Institutions are responsible for ensuring accuracy of SIS data before SSRC generates the SETL reports and inform SSRC about the course enrollment information for printing purpose.
Departments/Centres/Schools/Institutions should inform SSRC for printing at least 2 weeks before the evaluation date. Departments collects the blank forms one week before the evaluation date and carries out the evaluation.
For the paper evaluation forms, departments should deliver to SSRC all completed forms, completed cover sheet on the top of each envelope. 21

22. SETL Workflow for Reporting
The SETL results will be generated upon notice from SSRC after the examination result release date (20 January 2018 for this semester) and the following report will be provided to each department/centre/school/faculty and the reports will be saved in MS Excel format (worksheets) :
an individual report of every teacher for each course;
an overall teacher report for every course;
a report for the tutor/demonstrator for every course
(if applicable);
a course evaluation report for every course;
a soft copy of all open-ended questions’ answer for each course using online SETL (in MS excel format by question number); 22

23. SETL Workflow for Reporting
The overall SETL report will be provided to each department/centre/school/faculty and the reports will be saved in MS Excel format (worksheets) :
an overall report on the undergraduate courses;
an overall report on the TPG courses; and
an overall report on the RPG courses 23

24. SETL Workflow for Reporting
Additional requests may be entertained at cost
SSRC will upload SETL results to the APA for teachers and courses on or after 21 January for this semester (after the examination result release date 20 January 2018)
SSRC will upload ALL SETL results to the SSRC online SETL report system on or after 21 January for this semester (after the examination result release date 20 January 2018)
SSRC will upload SETL yearly course averages to SIS at the end of the year 24

25. Instructions for Online SETL
I will now demonstrate Online SETL, so you can see how the process works in practice:
I will now demonstrate Online SETL report system, so you can see how the process works in practice: 25

26. Instructions for Online SETL
The SSRC Guide (Third Edition, April 2015)
Briefing session on Online Student Evaluation of Teaching and Learning (24 October 2017)and SETL Online DEMO are available: 26

27. Using SETL data effectively
SETL data is used to evaluate all courses and can facilitate comparisons across courses and time. It is only one component of the QA process, but it is our focus for today.
SETL results can assist:
ensuring courses are well designed and being delivered well
improving course design
improving teaching
reassure students that concerns they have about courses and teaching are being addressed 27

28. 28 SETL context (I) Q: How many students in the class?
A: If very small, then the results may be less reliable, but the teacher should have been able to communicate better with the class)
Q: How many responded (what is the response rate)?
A: A low response rate may mean less valid results due to bias. This may reflect any of: a problem with administration of the forms in class (e.g. no time set aside) students did not turn up to class (i.e. serious dropout rate), students do not see this course as important enough to respond, students do not believe the programme team are responsive to their concerns. All of these are problems in different ways.
Q: Is this an award bearing course?
A: if yes, students will normally have higher expectations 28

29. 29 SETL context (II) Q: Part-time or full-time course?
A: Part-time students are often more motivated and may be more tolerant of teacher weakness.
Q: New course?
A: If yes, essential to check that the course design is appropriate, else the evaluation may be poor even if the teacher is relatively effective.
Q: New teacher?
C: If yes, the teacher may not understand the needs of the students well yet and if inexperienced, may need additional support initially to teach more effectively 
Q: Service course or required course?
C: Students may be more demanding about a required course, because they recognize its importance, but may be negative about a service course that they do not believe addresses a real need. 29

30. Interpret results
The overall questions are the primary focus for administrators:
Overall, the course (or teacher) was effective in helping me achieve the course learning outcomes
The report shows the % who strongly disagreed, disagreed, neutral, agreed or strongly agreed and converts them into mean, standard deviation (SD), standard error (SE)
How can we interpret the mean and SE?
Responses are scored from 100 (strongly agree) down to 0 (strongly disagree), so a mean of 50 would mean students were equally positive and negative about the course or teacher and a mean of 75 would mean that the average response was agree that the course/teacher was effective. SE gives an indication of how much variation is likely due to chance (taking into account variability across students and the number of responses). With a mean of 50 from 64 students, SE can be anywhere from 6 (half the students rated 0 and half rated 100) down to 0 (all rated 50). SE can be interpreted as that by chance, the mean will vary by about +/- 2 x SE, so we can safely ignore differences of less than 2 x SE. 30

31. Reference
To decide whether the mean is good or poor, we need a reference, which is normally the average score for that department in that semester, so a course with mean score lower than the average – 2 x SE should perhaps be checked more carefully.
Alternatively, we could use a criterion reference. The difficulty is deciding what that criterion reference should be – it could be 50 for tenure track and 70 for others.
In any event, it is important to identify remedies for low scores. 31

32. Improvement
My personal experience is that the open-ended comments on aspects that can be improved is the most useful element for identifying things I can improve, even if some of the comments cannot be addressed by me (e.g. scheduling) or reflect unreasonable expectations (teach advanced materials in a short course). 
It is important to realise that students may not know how your course or teaching can be improved. In my case, my scores (for teaching graduate students) improved most when I started providing screen videos after each class (so they can rewatch if they did not understand what I said) and when I started offering personal feedback on assignments during class breaks (in case they did not understand the written feedback), neither of which had ever been requested by students. 32

33. Transparency
Transparency with students is a good thing for course effectiveness, reassuring students that action is taken when problems are identified.
HKU now shows the average yearly course effectiveness for each of the last 3 years for all courses, together with feedback about any actions taken to address problems. I believe this will be much more effective than the previous process which, for most departments in HKU, involved sharing only through staff-student consultative committees.
However, for teacher effectiveness feedback, I see it best not being made public, especially as some students may write abusive comments about individual teachers, but it is essential for improvement and mentoring.
We have an implicit contract with our students in that their role is to provide honest feedback to assist teachers in continuous quality improvement. 33

34. Thank you! I am happy to answer questions.
Prepared by
Social Sciences Research Centre of
The University of Hong Kong 34